Federated Dataset Simulation: Inducing Label-Free Heterogeneity Across Tasks
Organizations: Department of Computer Science, City St George’s, University of London · Ultromics · ELLIS Institute Finland · Department of Electrical Engineering and Automation, Aalto University ·
Abstract
Federated Learning enables collaborative model training across decentralized clients without sharing raw data. While the field has seen rapid growth, the datasets underpinning FL evaluation have not kept pace: real federated datasets remain scarce, and most studies rely on small-scale centralized classification datasets partitioned via Dirichlet label skew, a one-dimensional proxy for the multi-dimensional heterogeneity present in real federations. Moreover, label-based partitioning is by construction inapplicable to non-classification tasks such as segmentation and detection, for which no principled simulation method exists. We cast the conversion of centralized datasets into federated benchmarks as the task of Federated Dataset Simulation (FDS), and decompose it into a representation function and an assignment mechanism . We provide two label-free instantiations -- geometric partitioning via pretrained model embeddings and semantic partitioning via vision-language model-derived criteria -- applicable to classification and dense prediction tasks alike. We evaluate across four datasets spanning classification, segmentation, and detection, and show that both methods induce heterogeneity that matches or exceeds that of label-skew partitioning, with measurable impact on downstream FL performance. We release fds, a modular open-source package for end-to-end federated dataset simulation, heterogeneity measurement, and FL training, at https://github.com/siomvas/fds.
Figures & tables
| JSD | MMD ( ) | FID | Centroid | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Method | 5 | 10 | 20 | 5 | 10 | 20 | 5 | 10 | 20 | 5 | 10 | 20 |
| Tiny | Random | .003 | .005 | .010 | 0.00 | 0.00 | 0.00 | 3.3 | 6.6 | 13.2 | 0.9 | 0.9 | 0.9 |
| Label Dir. =0.1 | .559 | .572 | .569 | 0.40 | 0.55 | 0.75 | 29.2 | 39.0 | 49.4 | 4.9 | 6.0 | 7.0 | |
| Semantic =1.0 | .128 | .168 | .207 | 0.70 | 0.93 | 1.19 | 22.0 | 30.9 | 43.0 | 4.8 | 5.5 | 6.2 | |
| Semantic =0.1 | .152 | .240 | .315 | 0.58 | 1.33 | 2.15 | 23.2 | 42.1 | 62.2 | 4.8 | 6.7 | 8.7 | |
| Geometric | .630 | .655 | .668 | 5.31 | 9.70 | 18.0 | 111 | 146 | 201 | 13.8 | 21.6 | 31.1 | |
| MMD ( ) | FID | Centroid | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | Method | 5 | 10 | 20 | 5 | 10 | 20 | 5 | 10 | 20 |
| ADE20K | Random | 0.00 | 0.00 | 0.01 | 11.1 | 21.8 | 41.3 | 0.9 | 1.3 | 1.8 |
| Semantic =1.0 | 2.13 | 2.27 | 2.16 | 47.1 | 60.8 | 80.5 | 6.4 | 6.8 | 7.0 | |
| Semantic =0.1 | 2.80 | 4.56 | 5.10 | 59.4 | 92.6 | 124.6 | 8.1 | 10.1 | 11.0 | |
| Geometric | 13.1 | 16.5 | 19.0 | 167 | 191 | 247 | 20.7 | 25.6 | 31.9 | |
| Pascal VOC | Random | 0.01 | 0.02 | 0.02 | 52.3 | 87.4 | 129.6 | 2.0 | 2.8 | 3.9 |
| Dataset | Metric | Partition | |||
|---|---|---|---|---|---|
| Tiny ImageNet | Acc. | Random | 74.53 (0.00) | 74.63 (0.00) | 74.29 (0.00) |
| Label Dir. | 71.74 (2.79) | 71.56 (3.07) | 71.74 (2.55) | ||
| Label Dir. | 74.30 (0.23) | 74.19 (0.44) | 73.57 (0.72) | ||
| Semantic | 73.69 (0.84) | 70.43 (4.20) | 72.99 (1.30) | ||
| Semantic | 73.56 (0.97) | 71.32 (3.31) | 72.31 (1.98) | ||
| Geometric | 67.41 (7.12) | 64.84 (9.79) | 69.24 (5.05) |
| Random initialization | Pretrained weights | |||||
| Method | Label Dir. | Geometric | Semantic | Label Dir. | Geometric | Semantic |
| FedAvg | 32.97 0.09 (3) | 24.89 0.22 (4) | 35.14 0.35 (2) | 54.73 0.23 (3) | 42.11 0.16 (4) | 55.97 0.34 (5) |
| FedDyn | 30.92 0.13 (6) | 25.11 0.04 (2) | 33.91 0.08 (5) | 53.06 0.10 (7) | 36.41 0.33 (6) | 56.73 0.02 (3) |
| FedNova | 32.91 0.36 (4) | 20.03 7.69 (7) | 33.76 0.06 (6) | 54.22 0.05 (4) | 26.32 1.86 (7) | 55.72 0.39 (7) |
| FedProx | 33.07 0.09 (2) | 24.92 0.24 (3) | 34.98 0.09 (4) | 54.17 0.12 (5) | 42.95 0.08 (2) | 55.98 0.08 (4) |
| FedSAM | 30.16 0.18 (7) | 25.97 0.10 (1) | 32.66 0.26 (7) | 55.45 0.03 (2) | 45.66 0.08 (1) | 57.27 0.01 (2) |
| Dataset | Partition | Label JSD | Feature MMD | -NN consistency |
|---|---|---|---|---|
| Fed-ISIC2019 | Natural | 0.115 | 0.027 | 0.260 |
| Geometric | 0.028 | 0.192 | 0.530 | |
| Office-Caltech10 | Natural | 0.010 | 0.016 | 0.446 |
| Geometric | 0.684 | 0.334 | 0.982 | |
| FEMNIST | Natural | 0.130 | 0.043 | 0.127 |
| Geometric | 0.643 | 0.465 | 0.654 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Criterion | Clients | FID | MMD | NN sep. | Centroid |
|---|---|---|---|---|---|---|
| Tiny ImageNet | main object | 7 | 115.3 | 0.0368 | 0.840 | 13.7 |
| location | 5 | 73.2 | 0.0237 | 0.781 | 10.9 | |
| size | 4 | 60.2 | 0.0160 | 0.621 | 8.6 | |
| color | 7 | 41.3 | 0.0092 | 0.482 | 7.0 | |
| time | 4 | 29.8 | 0.0073 | 0.583 | 6.2 | |
| ADE20K | scene | 4 | 122.1 | 0.0396 | 0.853 | 14.2 |
| (a) Criterion (rows) against taxonomy (columns), NMI | ||||
|---|---|---|---|---|
| Texture | Shape | Color | Count | |
| Texture | 0.524 | 0.063 | 0.012 | 0.003 |
| Shape | 0.004 | 0.860 | 0.002 | 0.005 |
| Color | 0.025 | 0.008 | 0.793 | 0.002 |
| Count | 0.002 | 0.016 | 0.003 | 0.297 |
| Dataset | Criterion | |||
|---|---|---|---|---|
| Tiny ImageNet | main object c | 0.483 | 0.515 | 0.486 |
| size s | 0.073 | 0.084 | 0.085 | |
| time | 0.080 | 0.081 | 0.075 | |
| color s | 0.053 | 0.059 | 0.072 | |
| location | 0.241 | 0.237 | 0.233 | |
| Food101 | cuisine c | 0.117 | 0.148 | 0.255 |
| Dataset | Client | Signature |
|---|---|---|
| Tiny ImageNet | 0 | location=Aquatic (3.66 ), main object=Animal (2.65 ), size=Small (1.79 ) |
| 1 | main object=Building (6.44 ), size=Expansive (5.70 ), main object=Landscape (5.33 ) | |
| 2 | main object=Inanimate Object (2.73 ), time=Unknown (1.98 ), time=Night (1.87 ) | |
| 3 | main object=Animal (3.53 ), location=Nature Outdoors (2.60 ), color=Brown/Yellow (1.97 ) | |
| 4 | main object=Vehicle (10.55 ), location=Urban Outdoors (4.34 ), color=White (2.01 ) | |
| 5 | main object=Human (4.68 ), size=Medium (1.86 ), time=Unknown (1.74 ) |
Federated Learning
Federated learning trains shared models across data holders without requiring them to pool their raw data.
See all 1,035 papers