Self-Supervised Federated Learning under Data Heterogeneity for Label-Scarce Diatom Classification
Authors: Mingkun Tan, Xilu Wang, Michael Kloster, Tim W. Nattkemper
Organizations: Biodata Mining Group, Faculty of Technology, University of Bielefeld, Germany · Computer Science Research Centre, University of Surrey, United Kingdom · Phycology Group, Faculty of Biology, University of Duisburg-Essen, Germany
Label-scarce visual classification under decentralized and heterogeneous data is a fundamental challenge in pattern recognition, especially when sites exhibit partially overlapping class sets. While self-supervised federated learning (SSFL) offers a promising solution, existing studies commonly assume the same data heterogeneity pattern throughout pre-training and fine-tuning. Moreover, current partitioning schemes often fail to generate pure partially class-disjoint data settings, limiting controllable simulation of real-world label-space heterogeneity. In this work, we introduce SSFL for diatom classification as a representative real-world instance and systematically investigate stage-specific data heterogeneity. We study cross-site variation in unlabeled data volume during pre-training and label-space misalignment during downstream fine-tuning. To study the latter in a controllable setting, we propose PreDi, a partitioning scheme that disentangles label-space heterogeneity into two orthogonal dimensions, namely class Prevalence and class-set size Disparity, enabling separate analysis of their effects. Guided by the resulting insights, we further propose PreP-WFL (Prevalence-based Personalized Weighted Federated Learning) to adaptively strengthen rare-class representations in low-prevalence scenarios. Extensive experiments show that SSFL consistently outperforms local-only training under both homogeneous and heterogeneous settings. The pronounced heterogeneity in unlabeled data volume is associated with improved representation pre-training, whereas under label-space heterogeneity, prevalence dominates performance and disparity has a smaller effect. PreP-WFL effectively mitigates this degradation, with gains increasing as prevalence decreases. These findings provide a mechanistic basis for characterizing label-space heterogeneity in decentralized recognition systems.
Figures & tables
Figure 1: Overview of the self-supervised federated learning framework with PreP-WFL. In the pre-training stage, masked image reconstruction is adopted as the self-supervised task: at each communication round t , every client k∈{1,…,K} updates its local encoder fθk and decoder gϕk on the unlabeled dataset Dku , and uploads the parameters (θkt,ϕkt) to the server. The server performs FedAvg to obtain a global encoder (fθ(∗),gϕ(∗)) and broadcasts it to all clients. In the fine-tuning stage, the final global encoder fθ(∗) initializes the local encoders, and a classifier head hψk is attached on each client and trained on its labeled dataset Dkℓ . The PreP - weight module (red box) performs a one-shot aggregation of the clients’ label sets Ck (label IDs only, no images) to compute the global class prevalence ρc and the corresponding weights wc . These weights are then broadcast to all clients and incorporated into a weighted cross-entropy loss during local fine-tuning, thereby emphasizing rare, site-specific taxa in the federated model.
Figure 2: Overview of the dataset reconstruction pipeline. From the original dataset, we first derive the filtered labeled set Df , which is partitioned into training and test sets Dtrain and Dtest ( 80/20 ). The remaining taxa form Dr . From Dtrain , we sample balanced (50 images per class) labeled subsets {Dkℓ}k=14 to serve as the labeled dataset on each client for downstream fine-tuning. The unlabeled pool Du is constructed by combining Dtrain with Dr and discarding all labels.
Splits
Client 1
Client 2
Client 3
Client 4
SplitIIDu
18,115
18,115
18,115
18,116
Split1u(α=1.0)
19,988
23,405
14,258
14,810
Split2u(α=0.5)
18,696
32,209
10,653
10,903
Split3u(α=0.2)
22,558
35,750
6,546
7,607
Split4u(α=0.1)
18,982
39,156
6,975
7,348
Table 1: Unlabeled data splits. Each cell reports the number of unlabeled samples held by a given client.
Figure 3: Heterogeneity in unlabeled data volume. Bars show the number of unlabeled samples per client for the IID Split IIDu and the four non-IID partitions Split 1u –Split 4u (corresponding to α∈{1.0,0.5,0.2,0.1} ). See also Table 1 .
Figure 4: Illustration of class prevalence and class-set size disparity. The x-axis indicates prevalence (low to high), the y-axis indicates disparity (low to high), and each colored circle denotes a class (A–D).
Figure 5: Heatmap of the number of classes per client under different labeled-data splits. The x-axis enumerates the IID split SplitIIDℓ and PreDi non-IID splits Splitρˉ∗,σ∗ℓ obtained by varying ρˉ∗∈{3.5,3.0,2.5,2.0,1.5} and σ∗∈{0.0,1.0,2.0,3.0} . The y-axis lists the clients, and each cell indicates the number of classes assigned to that client under the corresponding split.
Figure 6: SSFL performance under IID settings (see also Table 2 ).
Method
Pre-train data
Fine-tune data
Test Accuracy (%)
Client 1
SSL
D1u in SplitIIDu
D1ℓ in SplitIIDℓ
77.57
ImageNet
78.85
Client 2
SSL
D2u in SplitIIDu
D2ℓ in SplitIIDℓ
77.58
ImageNet
78.28
Client 3
SSL
D3u in SplitIIDu
D3ℓ in SplitIIDℓ
77.18
ImageNet
78.84
Table 2: Accuracy results under IID settings. Dku and Dkℓ denote the unlabeled and labeled datasets of client k , respectively.
Figure 7: Results on unlabeled splits. Bars show mean accuracy and error bars denote standard deviation over three runs. More severe quantity skew in the unlabeled data (smaller α ) consistently yields higher accuracy.
Pre-train data
Fine-tune data
Test Accuracy (%)
Client 1
D1u in Split4u
D1ℓ in SplitIIDℓ
79.69
Client 2
D2u in Split4u
D2ℓ in Split IIDℓ
81.33
Client 3
D3u in Split4u
D3ℓ in Split IIDℓ
70.64
Client 4
D4u in Split4u
D4ℓ in Split IIDℓ
73.86
Federated
Split4u
SplitIIDℓ
86.33
Table 3: Accuracy results on the most heterogeneous unlabeled split Split 4u . For each client k , Dku and Dkℓ denote its local unlabeled and labeled datasets, respectively.
Figure 8: Results on non-IID labeled data.The x-axis shows the target average class prevalence ρˉ∗ and the colored curves correspond to different levels of class-set size disparity σ∗ . Accuracy is primarily driven by ρˉ , whereas varying σ leads to only minor differences across settings. See also Table 4 .
Test Accuracy (%)
ρˉ∗=3.5
ρˉ∗=3.0
ρˉ∗=2.5
ρˉ∗=2.0
ρˉ∗=1.5
σ∗=0.0
82.47
79.56
77.00
75.89
67.82
σ∗=1.0
82.53
79.64
77.47
75.61
67.50
σ∗=2.0
82.34
79.84
77.35
74.43
65.86
σ∗=3.0
82.81
79.67
77.97
74.71
66.94
Table 4: Accuracy results on non-IID labeled data under different ( ρˉ∗,σ∗ ) configurations.
σ∗=0.0
Test Accuracy (%)
ρˉ∗=3.5
ρˉ∗=3.0
ρˉ∗=2.5
ρˉ∗=2.0
ρˉ∗=1.5
Client 1
69.55
59.79
50.18
42.08
11.04
Client 2
71.12
58.74
54.46
43.11
33.71
Client 3
68.63
67.66
50.28
38.30
30.50
Client 4
69.04
62.43
50.54
42.41
30.10
Federated
82.47
79.56
77.00
75.89
67.82
Table 5: Fine-tuning accuracy results under fixed disparity σ∗=0.0 for varying prevalence ρˉ∗ .
ρˉ∗=3.5
Test Accuracy (%)
σ∗=0.0
σ∗=1.0
σ∗=2.0
σ∗=3.0
Client 1
69.55
69.37
71.96
72.24
Client 2
71.12
65.85
64.40
62.38
Client 3
68.63
68.71
66.34
66.48
Client 4
69.04
74.16
75.60
77.90
Federated
82.47
82.53
82.34
82.81
Table 6: Fine-tuning accuracy results under fixed prevalence ρˉ∗=3.5 for varying disparity σ∗ .
Figure 9: Accuracy gain (percentage points) of PreP-WFL over the Fed-MAE baseline across (ρˉ∗,σ∗) configurations of labeled data.
Test Accuracy (%)
ρˉ∗=3.5
ρˉ∗=3.0
ρˉ∗=2.5
ρˉ∗=2.0
ρˉ∗=1.5
σ∗=0.0
Fed-MAE
82.47
79.56
77.00
75.89
67.82
Fed-MAE + PreP-WFL (ours)
82.62
81.22
79.01
78.11
71.16
σ∗=1.0
Fed-MAE
82.53
79.64
77.47
75.61
67.50
Fed-MAE + PreP-WFL (ours)
82.56
81.00
79.51
77.30
70.21
σ∗=2.0
Fed-MAE
82.34
79.84
77.35
74.43
65.86
Table 7: Accuracy results of Fed-MAE (baseline) and Fed-MAE with PreP-WFL (ours) across (ρˉ∗,σ∗) configurations.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Pre-train data
Fine-tune data
Test F1-score
Client 1
SSL
D1u in SplitIIDu
D1ℓ in SplitIIDℓ
0.7054
ImageNet
0.7234
Client 2
SSL
D2u in SplitIIDu
D2ℓ in SplitIIDℓ
0.7100
ImageNet
0.7186
Client 3
SSL
D3u in SplitIIDu
D3ℓ in SplitIIDℓ
0.7051
ImageNet
0.7552
Appendix
Table 8: F1-score results under IID settings. Dku and Dkℓ denote the unlabeled and labeled datasets of client k , respectively.
Pre-train data
Test F1-score
SplitIIDu
0.8008
Split1u(α=1.0)
0.7934
Split2u(α=0.5)
0.7975
Split3u(α=0.2)
0.8063
Split4u(α=0.1)
0.8168
Appendix
Table 9: F1-score results on unlabeled splits.
Test F1-score
ρˉ∗=3.5
ρˉ∗=3.0
ρˉ∗=2.5
ρˉ∗=2.0
ρˉ∗=1.5
σ∗=0.0
0.7853
0.7570
0.7310
0.7217
0.6277
σ∗=1.0
0.7833
0.7589
0.7357
0.7163
0.6244
σ∗=2.0
0.7827
0.7611
0.7312
0.7059
0.6010
σ∗=3.0
0.7882
0.7579
0.7404
0.7038
0.6158
Appendix
Table 10: F1-score results on non-IID labeled data under different ( ρˉ∗,σ∗ ) configurations.
Test F1-score
ρˉ∗=3.5
ρˉ∗=3.0
ρˉ∗=2.5
ρˉ∗=2.0
ρˉ∗=1.5
σ∗=0.0
Fed-MAE
0.7853
0.7570
0.7310
0.7217
0.6277
Fed-MAE + PreP-WFL (ours)
0.7822
0.7696
0.7426
0.7364
0.6611
σ∗=1.0
Fed-MAE
0.7833
0.7589
0.7357
0.7163
0.6244
Fed-MAE + PreP-WFL (ours)
0.7838
0.7687
0.7492
0.7279
0.6473
σ∗=2.0
Fed-MAE
0.7827
0.7611
0.7312
0.7059
0.6010
Appendix
Table 11: F1-score results of Fed-MAE (baseline) and Fed-MAE with PreP-WFL (ours) across (ρˉ∗,σ∗) configurations.
Federated learning has emerged as the foremost approach for decentralized model training with privacy preservation. The global class imbalance and cross-client data heterogeneity naturally coexist, and the mismatch between local and global imbalances exacerbates the performance degradation of the aggregated model. The agnosticism of global class distribution poses significant challenges for data-level methods, especially under extreme conditions with severe class absence across clients. In this paper, we propose FedReLa, a novel data-level approach that tackles the coexistence of data heterogeneity and class imbalance in federated learning. By re-labeling samples with a feature-dependent label re-allocator, FedReLa corrects biased global decision boundaries without requiring knowledge of the global class distribution. This modular, model-agnostic approach can be integrated with algorithmic methods to deliver consistent improvements without additional communication overhead. Through extensive experiments, our method significantly improves the accuracy of minority classes and the overall accuracy on stepwise-imbalanced and long-tailed datasets, outperforming the previous state of the art.
Guangzheng Hu, Patricia Menéndez, Feng Liu +3
School of Mathematics and Statistics, University of Melbourne, Victoria, Australia · School of Computing and Information Systems, University of Melbourne, Victoria, Australia · School of Statistics and Data Science, LPMC, KLMDASR, and LEBPS, Nankai University
Federated Learning (FL) enables distributed clients to collaboratively train models without sharing raw data, making it promising for leveraging massive devices in communication networks. In distillation-based FL, each client applies its local model on an unlabeled public dataset, and shares only prediction results with the server. While heterogeneous local data introduces label distribution skew, thus biasing client models toward majority classes and leading to potentially inaccurate predictions. The lack of ground-truth labels in the public dataset hampers the server's ability to calibrate predictions, which ultimately degrades overall performance. To address this, we propose FedCC, a simple and effective algorithm for mitigating client misclassification. Instead of being forced to classify and risking error propagation, clients are allowed to tag ambiguous samples as 'unknown'. This additional class, together with calibrated pseudo-labels on the public data, balances confidence in majority classes against uncertainty in under-represented ones. Extensive experiments demonstrate that FedCC significantly outperforms existing methods, especially under severe label skew. In the extreme scenario where each client holds samples from only one of ten classes, FedCC achieves 67.3% accuracy, while baselines collapse to near-random results.
Wenxuan Ye, Onur Ayan, Xueli An +1
Huawei Heisenberg Research Center, Huawei Technologies Duesseldorf GmbH · TUM School of Computation, Information and Technology, Technical University of Munich
Federated learning enables privacy-preserving collaboration across distributed devices without centralizing local data. However, clients may differ not only in data distributions but also in domain knowledge and annotation capabilities. In this paper, we introduce label granularity skew, a new form of statistical heterogeneity in federated hierarchical classification, in which clients provide taxonomy-consistent labels at different levels of detail within a shared class hierarchy. To model this heterogeneity, we generate client-specific local label hierarchies using a probabilistic relational neighbor classifier and construct a WordNet-guided hierarchy via silhouette score-based coarsening. Our analysis shows that strongly coupled hierarchical models are sensitive to incomplete supervision, while the conditional softmax classifier is more robust. Based on this insight, we propose Branch-wise Decoupled Fine-Tuning (BDFT) and its federated version, FedBDFT, which fine-tune branch-wise classifiers and aggregate them through federated optimization. Experiments on CIFAR-100, TinyImageNet, and ImageNet show that FedBDFT substantially improves robustness under severe label granularity skew, with average gains of 27.9% and 56.4% at skewness levels of 0.6 and 0.9, respectively. Zero-shot results further indicate that FedBDFT better preserves hierarchical representations for unseen fine-grained classes. These findings demonstrate its effectiveness for federated hierarchical classification with heterogeneous label granularities.
Jaeheon Kim, Hokeun Kim, Bong Jun Choi
School of Computer Science and Engineering, Soongsil University, Seoul, Republic of Korea · School of Computing and Augmented Intelligence, Arizona State University, Tempe, United States