Effective Unbounded Drift Control in Heterogeneous Federated Learning: Bayesian Fine-Tuning Using a Reference Model
Authors: Taehwan Yoon, Wesley De Neve, Bongjun Choi
Organizations: School of Computer Science and Engineering, Soongsil University, Seoul, Republic of Korea · Center for Biosystems and Biotech Data Science, Ghent University, Global Campus, Incheon, Republic of Korea
Federated learning (FL) enables collaborative model training across distributed clients while preserving data privacy. However, data and system heterogeneity often induce unbounded drift in model updates due to each clients' different multiple local steps, partial participation, and unbounded dynamics in model updates. These challenges severely degrade predictive performance, exacerbate catastrophic forgetting, and hinder global optimization forward to optimal updates. Related works (AdaBest, FedEve, A-FedPD) defined dual drift problems and proposed each regularization term to client optimization process. However, in aspect of temporal unbounded drift, they generally reflect last round temporal drift to client optimization while forgets prior temporal unbounded drift. To address these limitations, we propose FedRef, a robust optimization framework that leverages a reference model generalized from prior global models via Bayesian fine-tuning. FedRef integrates a Maximum A Posteriori (MAP)-based regularization mechanism that calibrates global updates toward a temporally aggregated reference model, thereby effectively controlling unbounded drift and maximizing empirical stability. Moreover, unlike prior approaches that rely on complex client-side optimization heuristics, FedRef achieves superior generalization to reflect recency weighting by seamlessly blending a robust probabilistic prior with global likelihood optimization. By maintaining a temporal moving average centered on previous global trajectories, the proposed framework provides a highly reliable optimization anchor.
Figures & tables
Symbol
Description
A
Aggregation function
s
Local training step
k
Client index, k={1,2,…,K}
Bks
Mini-batch sampled from local dataset of client k
Dk
Dataset of client k
Dref
Synthetic data of reference model
TABLE I : List of Symbols and Abbreviations
Fig. 1 : Overview of our FedRef system.
Hyperparameter
Value
Number of clients ( K )
10
Number of epochs per round
3
FedProx ( 3 )
μ=0.5
FedOpt ( 4 )
β1=0.9 , β2=0.99 , τ=10−4
AdaBest ( 5 )
α:0.1 , β:1.0
FedEve ( 6 )
β:0.9 , μ:0.01 , ρ:0.9
TABLE II : Hyperparameters
Fig. 2 : Number of samples per client for each dataset: CINIC-10 (high IID), FEMNIST (low non-IID), and FeTS2022 (high non-IID).
UDP
λ
ρ
KLD
Accuracy ( % )
F1score ( % )
0.2
0.01
1
2.050
25.09
25.67
4
25.11
25.68
8
25.06
25.64
0.1
1
25.13
25.76
4
25.08
25.66
8
25.08
25.69
TABLE III : FedRef Hyperparameter Sensitivity in CINIC-10
UDP
λ
ρ
KLD
Accuracy ( % )
F1score ( % )
0.2
0.01
1
3.987
65.30
70.62
4
3.992
65.25
70.61
8
3.991
65.31
70.62
0.1
1
3.975
65.38
70.69
4
4.002
65.17
70.50
8
3.974
65.31
70.62
TABLE IV : FedRef Hyperparameter Sensitivity in FEMNIST
UDP
λ
ρ
KLD
Dice Coefficient ( % )
Hausdorff Distance
0.2
0.01
1
6556286
32.39
10.51
4
6556006
9.19
8
6562821
10.54
0.1
1
6558238
9.51
4
6553981
11.24
8
6550471
8.43
TABLE V : FedRef Hyperparameter Sensitivity in FeTS
Dataset
Method
UDP
KLD
Accuracy (%)
F1score (%)
CINIC-10
FedAvg
0.2
2.050
25.13
25.72
FedProx
2.050
25.09
25.71
FedOpt
2.050
25.09
25.73
Adabest
2.325
13.82
14.70
FedEve
2.064
24.50
25.12
A-FedPD
2.051
25.09
25.72
TABLE VI : Predictive Performance Table in Cinic-10
Dataset
Method
UDP
KLD
Accuracy (%)
F1score (%)
FEMNIST
FedAvg
0.2
3.995
65.32
70.65
FedProx
3.989
65.32
70.64
FedOpt
3.994
65.27
70.60
Adabest
5.101
35.68
43.04
FedEve
3.197
61.75
66.42
A-FedPD
4.095
64.81
70.24
TABLE VII : Predictive Performance Table in FEMNIST
Dataset
Method
UDP
KLD
DC (%)
Hausdorff Distance
FeTS
FedAvg
0.2
6561239
32.39
10.02
FedProx
6554945
32.38
9.30
FedOpt
6564171
32.39
9.15
Adabest
6564214
32.09
9.08
FedEve
6550415
32.34
11.32
A-FedPD
6562726
32.14
9.84
TABLE VIII : Predictive Performance Table in Image Segmentation
Method
Optimization Paradigm
Drift Control Strategy
Generalization Framework
FedAvg [ 3 ]
Standard Empirical Risk Minimization
Raw Model Aggregation
No Prior Alignment
FedProx [ 9 ]
Local Proximal Regularization
Static Norm Constraints
Immediate Global Anchor
FedOpt [ 10 ]
Server-Side Adaptive Optimization
Momentum-based Smoothing
Gradient-driven Tracking
AdaBest [ 6 ]
Client-Side Adaptive Bias Estimation
Dynamic Update Norm Constraints
Local Bias Compensation
FedEve [ 7 ]
Predict-Observe Filtering
Variance Reduction Heuristics
Dual Drift Filtering
A-FedPD [ 8 ]
Primal-Dual Optimization
Aligning Dual-Drift
Dual Drift Balancing
TABLE IX : Comparison of Optimization Mechanisms and Generalization Attributes
In real-world Federated Learning (FL) deployments, data distributions on devices that participate in training evolve over time. This leads to asynchronous data drift, where different devices shift at different times and toward different distributions. Mitigating such drift is challenging: frequent retraining incurs high computational cost on resource-constrained devices, while infrequent retraining degrades performance on drifting devices. We propose DriftGuard, a federated continual learning framework that efficiently adapts to asynchronous data drift. DriftGuard adopts a Mixture-of-Experts (MoE) inspired architecture that separates shared parameters, which capture globally transferable knowledge, from local parameters that adapt to group-specific distributions. This design enables two complementary retraining strategies: (i) global retraining, which updates the shared parameters when system-wide drift is identified, and (ii) group retraining, which selectively updates local parameters for clusters of devices identified via MoE gating patterns, without sharing raw data. Experiments across multiple datasets and models show that DriftGuard matches or exceeds state-of-the-art accuracy while reducing total retraining cost by up to 83%. As a result, it achieves the highest accuracy per unit retraining cost, improving over the strongest baseline by up to 2.3x. DriftGuard is available for download from https://github.com/blessonvar/DriftGuard.
Yizhou Han, Di Wu, Blesson Varghese
School of Computer Science, University of St Andrews, UK
Federated Learning enables collaborative model training across distributed edge devices ("clients") while preserving data privacy. However, real-world deployments involve clients that observe continuous data streams from evolving environments while operating under strict resource limitations. Existing solutions still face two major challenges: local concept drift, where each client's data distribution shifts over time, and global client drift, arising from asynchronous engagement and imbalanced contributions across heterogeneous training participants, formally described as dual client heterogeneity. We propose STORMGUIDE-FL, a federated optimization and data-mining framework that mitigates both types of drift using a server-guided pipeline for data acquisition and local training. For data acquisition, a lightweight client-side sampling strategy incrementally selects informative and diverse instances from non-stationary, non-IID local streams under bounded memory, compute, and labeling budget. For local training, the server's remote guidance is embedded directly into local optimization as an update-level correction, counteracting client drift without discarding scarce labeled samples. We empirically verify the effectiveness of the proposed sampling strategy on federated data streams across gradual, sudden, and recurring concept drift, and demonstrate that its gains compound with the update-level drift correction. Experimental results further show improved speed and robustness of global model convergence under resource-constrained, heterogeneous conditions.
Manuel Röder, Frank-Michael Schleif
Technical University of Applied Sciences Würzburg-Schweinfurt, Würzburg, Germany.
This paper addresses the challenging problem of dynamic feature drift in federated learning, where data distributions evolve across clients and over time -- a common scenario in real-world applications like financial technology. Existing approaches often assume static drift, limiting their effectiveness in non-stationary environments. To overcome this, we propose FedCausal-Dyn, a novel federated learning framework built on a causal-dynamic paradigm. Its key innovation is causal-domain feature separation, which disentangles domain-invariant causal features from spurious, domain-specific variations via specialized projection heads and adversarial training. This enables reliable and dynamic prototype aggregation, weighting local class prototypes by estimated reliability before global aggregation. We further introduce causal-feature guided collaborative regularization, unifying prototype contrastive alignment and domain invariance into a cohesive objective. Extensive experiments on three federated domain generalization benchmarks demonstrate that FedCausal-Dyn consistently achieves state-of-the-art performance, with the highest average accuracy and the most stable results. Ablation studies confirm each component's critical contribution. Our work provides a robust and principled solution for federated learning under dynamic feature drift.