Federated learning systems are increasingly deployed to facilitate collaborative model training across a heterogeneous client population. Existing practice mostly implicitly assumes that the aggregated client data distribution is representative of the learner's target distribution or that learning from all available clients is uniformly beneficial for the learner distribution. However, such an assumption often does not hold in reality. Traditional client selection strategies in FL literature largely overlook such misalignment, while most existing work on multi-source transfer learning either requires direct access to local data or uses one-shot model/feature aggregation. In this paper, we take the initiative to understand and mitigate the impacts of such learner-client population misalignment. In particular, we consider the practical setting where the learner keeps a small proxy dataset. We observe that client contributions vary significantly across training rounds, and traditional technology is insufficient to identify beneficial sources under multi-source transfer diversity. Then, we propose a dynamic, influence-aware client selection framework that estimates each client's potential utility to the learner's optimization objective using proxy influence signals on a learner-specific proxy set. Via using leave-one-out evaluations, we prioritize the most informative sources of knowledge while controlling the negative impacts of statistical noise and data heterogeneity. Experiments on CIFAR-10 under heterogeneous data partitions demonstrate that our approach consistently outperforms static and dynamic baselines, achieving faster convergence and higher accuracy.
Figures & tables
Fig. 1: Mismatch of client population data and target distribution. The left panel illustrates model training, where a learner receives updates from heterogeneous client devices. The empirical client population induces a training data distribution shown at the bottom left. The right panel illustrates the learner side, where the trained model is deployed to target users or applications, whose data distribution can differ substantially from the training distribution.
Method
Multi-Source
Target Data Scarcity
Learner-Centric
Target Label
Static / Dynamic
FedAvg [ 1 ]
✓
✗
✗
Supervised
Static
FL+HC [ 29 ]
✓
✗
✗
Supervised
Static
MCW [ 30 ]
✗
✓
✗
Supervised
Static
LogME [ 31 ]
✗
✓
✗
Unsupervised
Static
H-Ensemble [ 10 ]
✓
✗
✗
Unsupervised
Static
OTQMS [ 32 ]
✓
✓
✗
Unsupervised
Dynamic
TABLE I: Comparison of representative client/model selection criteria in federated and multi-source transfer learning. Methods are categorized as static or dynamic based on whether their selection signal is updated across training rounds.
Fig. 2: Label distribution heatmaps across four experimental cases. The learner is shown with circular markers, while other clients use square markers. Color indicates class probability.
Fig. 3: Runtime accuracy across the four cases.
Fig. 4: Selected client IDs per round across the four cases.
Ratio s
α
FedAvg
KMeans
FL+HC
H-Ensemble
Power-of-Choice
DIC-KT(Ours)
(A) Label-exclusive case
s=0.15
α=0.1
19.91 (± 11.0)
12.40 (± 12.4)
78.67 (± 0.34)
79.31 (± 1.32)
61.67 (± 13.86)
79.52 (± 1.0)
s=0.2
α=0.1
14.76 (± 12.1)
41.20 (± 30.3)
80.12 (± 0.42)
79.31 (± 1.07)
62.76 (± 15.67)
79.32 (± 0.62)
s=0.35
α=0.1
42.72 (± 30.9)
37.14 (± 34.2)
65.67 (± 0.34)
66.31 (± 2.62)
65.41 (± 11.89)
66.78 (± 8.38)
s=0.5
α=0.1
46.70 (± 31.0)
47.73 (± 28.9)
54.67 (± 0.54)
51.31 (± 8.12)
51.67 (±16.67)
56.05 (± 22.0)
(B) Mismatch levels under the fixed selection ratio
TABLE II: Learner test accuracy (%, mean ± std over the last 10 rounds). We evaluate (A) label-exclusive client selection under different numbers, (B) Dirichlet heterogeneity levels, and (C) selection ratio under α=0.1 . Best results in each row are shown in bold . The selection ratio is denoted as s and the Dirichlet concentration is denoted as α .
Fig. 5: Our influence-aware selection yields consistently higher accuracy and more stable convergence than FedAvg, FL+HC, KMeans, H-Ensemble, and Power-of-Choice.
Fig. 6: Selected client IDs across communication rounds for different client selection strategies. Each marker represents a selected client at a given round.
Fig. 7: Illustration of the label-exclusive setting and the corresponding distribution distance. (a) The left plot shows that the learner (yellow star) and clients containing labels {0,1,2,3} form the left cluster, while the remaining clients containing labels {4,5,6,7,8,9} form the right cluster. (b) The right plot shows the JSD distance between each client distribution and the learner target distribution.
Fig. 8: Effect of the learner proxy set size on knowledge transfer performance. The plot shows the average accuracy over the last ten rounds, with error bars indicating the standard deviation. Even a small learner proxy set provides sufficient guidance for effective knowledge transfer.
Federated learning (FL) is a popular distributed learning framework where multiple clients perform local training and a server aggregates the locally updated models. FL enables decentralized training while preserving the privacy of clients' datasets. However, non-independent and identically distributed (non-IID) or noisy datasets can lead to low model accuracy or high convergence latency. Precluding these clients through client selection may mitigate the problem, but heavily biased client selections may also degrade the learning performance. In this study, we first experimentally measure the impact of non-IID data (including skews in data quantity and label distribution), noisy data, and fairness in client selection on model accuracy and convergence. We then propose a privacy-preserving scoring method to assess each client's contribution in FL, with experiments conducted to demonstrate the effectiveness of the proposed assessment.
Yuan-Heng Tsai, Li-Hsing Yen, Yan-Wei Chen
Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan.
Federated Learning enables collaborative model training across decentralized data sources without data transfer. Averaging-based FL is limited by the presence of non-IID data, which negatively impacts convergence speed and final model accuracy. Conventional alternatives suffer from significant inefficiency. Clients with noisy or highly heterogeneous data contribute expensive gradient computations that are either discarded or heavily down-weighted before aggregation. These reactive approaches waste computational resources, require more communication rounds and result in unnecessary privacy exposure. In this paper, we propose a proactive client selection framework that aims to find an optimal federation of clients whose combined data match utility and fairness requirements before training begins. Our method relies on mutual information computed from differentially private contingency tables to quantify the relevance of cross-feature correlations in the union dataset. We introduce a Potential Federation Loss (PFL) over the set of fixed-size federations, which balances two objectives. Maximizing collective data utility while ensuring fair cross-features correlations to prevent group unfairness. Client selection is expressed as an optimal subset search problem over the PFL objective, which we solve using simulated annealing under strong differential privacy guarantees for clients' local statistics. Experimental results on four benchmarks show faster, fairer, and more accurate models trained on optimally found federations, compared to uniform sampling, even when state-of-the-art adaptive aggregation or sampling strategies are employed.
Federated learning (FL) is a promising paradigm of machine learning, which preserves user privacy by enabling learning without sharing raw data with a cloud server. Straggling clients have been a problem for FL as they introduce delays in aggregating the local models and hence, the convergence of the global model. Therefore, it is important to have a mechanism that ensures fast convergence of the global model as well as good FL participation rate. Another issue for the convergence of a model in FL is the non-independent and identically distributed (non-iid) data across the clients. Prior approaches based on probabilistic client selection do not work well under non-iid data especially when the number of clients is small. We show scenarios where such approaches fail and propose a joint client-training data selection algorithm for fast convergence of FL models. Our experiments on CIFAR-100 dataset show that convergence of the FL model can be significantly improved over prior works that can consider non-iid data and heterogeneous computation and higher model accuracy.
Muzaffer Citir, Hiroki Nishikawa, Sangyoung Park
Smart Mobility Systems, Technical University of Berlin, Germany · Graduate School of Information Science and Technology, The University of Osaka, Japan