Federated Dataset Simulation: Inducing Label-Free Heterogeneity Across Tasks
Authors: Vasilis Siomos, Lam Ngo, Jonathan Passerat-Palmbach, Giacomo Tarroni
Organizations: Department of Computer Science, City St George’s, University of London · Ultromics · ELLIS Institute Finland · Department of Electrical Engineering and Automation, Aalto University · Flashbots · Department of Computing, Imperial College London
Federated Learning enables collaborative model training across decentralized clients without sharing raw data. While the field has seen rapid growth, the datasets underpinning FL evaluation have not kept pace: real federated datasets remain scarce, and most studies rely on small-scale centralized classification datasets partitioned via Dirichlet label skew, a one-dimensional proxy for the multi-dimensional heterogeneity present in real federations. Moreover, label-based partitioning is by construction inapplicable to non-classification tasks such as segmentation and detection, for which no principled simulation method exists. We cast the conversion of centralized datasets into federated benchmarks as the task of Federated Dataset Simulation (FDS), and decompose it into a representation function Φ and an assignment mechanism π. We provide two label-free instantiations -- geometric partitioning via pretrained model embeddings and semantic partitioning via vision-language model-derived criteria -- applicable to classification and dense prediction tasks alike. We evaluate across four datasets spanning classification, segmentation, and detection, and show that both methods induce heterogeneity that matches or exceeds that of label-skew partitioning, with measurable impact on downstream FL performance. We release fds, a modular open-source package for end-to-end federated dataset simulation, heterogeneity measurement, and FL training, at https://github.com/siomvas/fds.
Figures & tables
JSD ↑
MMD ( ×10−2 ) ↑
FID ↑
Centroid ↑
Dataset
Method
5
10
20
5
10
20
5
10
20
5
10
20
Tiny
Random
.003
.005
.010
0.00
0.00
0.00
3.3
6.6
13.2
0.9
0.9
0.9
Label Dir. α =0.1
.559
.572
.569
0.40
0.55
0.75
29.2
39.0
49.4
4.9
6.0
7.0
Semantic α =1.0
.128
.168
.207
0.70
0.93
1.19
22.0
30.9
43.0
4.8
5.5
6.2
Semantic α =0.1
.152
.240
.315
0.58
1.33
2.15
23.2
42.1
62.2
4.8
6.7
8.7
Geometric
.630
.655
.668
5.31
9.70
18.0
111
146
201
13.8
21.6
31.1
Table 1: Heterogeneity metrics on classification datasets across client counts K∈{5,10,20} . JSD measures label-distribution divergence; MMD, FID, and centroid distance measure feature-distribution divergence (computed in InceptionV3 activation space). Higher values indicate greater heterogeneity. Geometric partitioning uses no labels yet induces the highest heterogeneity across all metrics, including JSD.
MMD ( ×10−2 ) ↑
FID ↑
Centroid ↑
Dataset
Method
5
10
20
5
10
20
5
10
20
ADE20K
Random
0.00
0.00
0.01
11.1
21.8
41.3
0.9
1.3
1.8
Semantic α =1.0
2.13
2.27
2.16
47.1
60.8
80.5
6.4
6.8
7.0
Semantic α =0.1
2.80
4.56
5.10
59.4
92.6
124.6
8.1
10.1
11.0
Geometric
13.1
16.5
19.0
167
191
247
20.7
25.6
31.9
Pascal VOC
Random
0.01
0.02
0.02
52.3
87.4
129.6
2.0
2.8
3.9
Table 2: Heterogeneity metrics on dense prediction datasets. Dirichlet label-skew partitioning is inapplicable to segmentation and detection, so Random is the only existing baseline. Geometric and semantic partitioning are the first methods to produce non-trivial heterogeneity for these tasks.
Dataset
Metric
Partition
K=5
K=10
K=20
Tiny ImageNet
Acc.
Random
74.53 (0.00)
74.63 (0.00)
74.29 (0.00)
Label Dir. α=0.1
71.74 (2.79)
71.56 (3.07)
71.74 (2.55)
Label Dir. α=1.0
74.30 (0.23)
74.19 (0.44)
73.57 (0.72)
Semantic α=0.1
73.69 (0.84)
70.43 (4.20)
72.99 (1.30)
Semantic α=1.0
73.56 (0.97)
71.32 (3.31)
72.31 (1.98)
Geometric
67.41 (7.12)
64.84 (9.79)
69.24 (5.05)
Table 3: Downstream FL performance under FedAvg across datasets and client counts. Parentheses report degradation relative to random partitioning. Lower non-random performance within each dataset/client-count block in bold. Label Dirichlet baselines are omitted for dense prediction tasks because they do not provide a single sample-level class label.
Random initialization
Pretrained weights
Method
Label Dir.
Geometric
Semantic
Label Dir.
Geometric
Semantic
FedAvg
32.97 ± 0.09 (3)
24.89 ± 0.22 (4)
35.14 ± 0.35 (2)
54.73 ± 0.23 (3)
42.11 ± 0.16 (4)
55.97 ± 0.34 (5)
FedDyn
30.92 ± 0.13 (6)
25.11 ± 0.04 (2)
33.91 ± 0.08 (5)
53.06 ± 0.10 (7)
36.41 ± 0.33 (6)
56.73 ± 0.02 (3)
FedNova
32.91 ± 0.36 (4)
20.03 ± 7.69 (7)
33.76 ± 0.06 (6)
54.22 ± 0.05 (4)
26.32 ± 1.86 (7)
55.72 ± 0.39 (7)
FedProx
33.07 ± 0.09 (2)
24.92 ± 0.24 (3)
34.98 ± 0.09 (4)
54.17 ± 0.12 (5)
42.95 ± 0.08 (2)
55.98 ± 0.08 (4)
FedSAM
30.16 ± 0.18 (7)
25.97 ± 0.10 (1)
32.66 ± 0.26 (7)
55.45 ± 0.03 (2)
45.66 ± 0.08 (1)
57.27 ± 0.01 (2)
Table 4: Accuracy (%) of seven aggregation methods on Tiny ImageNet with K=10 , from random and from ImageNet initializations. Settings differ from Table 3 (Appendix B ). Mean ± standard deviation over three seeds, rank in parentheses. Geometric partitioning is the most challenging protocol for every method.
Dataset
Partition
Label JSD
Feature MMD
k -NN consistency
Fed-ISIC2019
Natural
0.115
0.027
0.260
Geometric
0.028
0.192
0.530
Office-Caltech10
Natural
0.010
0.016
0.446
Geometric
0.684
0.334
0.982
FEMNIST
Natural
0.130
0.043
0.127
Geometric
0.643
0.465
0.654
Table 5: Natural versus simulated heterogeneity. For each naturally federated dataset, we compare the original client split to a geometric partition with the same number of clients. Higher values indicate greater heterogeneity.
Figure 1: NMI between k -means partitions ( K=10 ) of Clevr-4 and each of its taxonomies after removing the top p principal components of DINOv2 (left) or MAE (right) embeddings. The k -means seed standard deviation at p=0 is at most 0.021 (DINOv2, three seeds). No partition collapses, since every client holds at least 1.2% of the images, and the coefficient of variation of client sizes ranges from 0.22 to 0.37 for MAE and from 0.27 to 0.61 for DINOv2.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Criterion
Clients
FID ↑
MMD ↑
k NN sep. ↑
Centroid ↑
Tiny ImageNet
main object
7
115.3
0.0368
0.840
13.7
location
5
73.2
0.0237
0.781
10.9
size
4
60.2
0.0160
0.621
8.6
color
7
41.3
0.0092
0.482
7.0
time
4
29.8
0.0073
0.583
6.2
ADE20K
scene
4
122.1
0.0396
0.853
14.2
Appendix
Table 6: Pure-criterion ablation for semantic partitioning. For each criterion, one client is created per criterion value and all samples with that value are assigned to the corresponding client. Higher values indicate stronger feature-space separation.
(a) Criterion (rows) against taxonomy (columns), NMI
Texture
Shape
Color
Count
Texture
0.524
0.063
0.012
0.003
Shape
0.004
0.860
0.002
0.005
Color
0.025
0.008
0.793
0.002
Count
0.002
0.016
0.003
0.297
Appendix
Table 7: Clevr-4 validation of the semantic pipeline (InternVL3.5-38B and Llama 3.3-70B). (a) NMI between the partition produced for each criterion and each annotated taxonomy. (b) AMI between the four criterion partitions. (c) AMI between the four annotated taxonomies.
Dataset
Criterion
K=5
K=10
K=20
Tiny ImageNet
main object c
0.483
0.515
0.486
size s
0.073
0.084
0.085
time
0.080
0.081
0.075
color s
0.053
0.059
0.072
location
0.241
0.237
0.233
Food101
cuisine c
0.117
0.148
0.255
Appendix
Table 8: AMI between each geometric partition and the pure-criterion partition of each criterion, for K∈{5,10,20} . c Content criteria. s Criteria of lighting, color, camera, and size.
Table 9: Client signatures of the geometric partitions at K=10 : for each client, the three criterion values with the largest lift (client rate over global rate) among values covering at least 5% of the client.
Biodata Mining Group, Faculty of Technology, University of Bielefeld, Germany · Computer Science Research Centre, University of Surrey, United Kingdom · Phycology Group, Faculty of Biology, University of Duisburg-Essen, Germany