Effective Unbounded Drift Control in Heterogeneous Federated Learning: Bayesian Fine-Tuning Using a Reference Model
Authors: Taehwan Yoon, Wesley De Neve, Bongjun Choi
Organizations: School of Computer Science and Engineering, Soongsil University, Seoul, Republic of Korea · Center for Biosystems and Biotech Data Science, Ghent University, Global Campus, Incheon, Republic of Korea
Federated learning (FL) enables collaborative model training across distributed clients while preserving data privacy. However, data and system heterogeneity often induce unbounded drift in model updates due to each clients' different multiple local steps, partial participation, and unbounded dynamics in model updates. These challenges severely degrade predictive performance, exacerbate catastrophic forgetting, and hinder global optimization forward to optimal updates. Related works (AdaBest, FedEve, A-FedPD) defined dual drift problems and proposed each regularization term to client optimization process. However, in aspect of temporal unbounded drift, they generally reflect last round temporal drift to client optimization while forgets prior temporal unbounded drift. To address these limitations, we propose FedRef, a robust optimization framework that leverages a reference model generalized from prior global models via Bayesian fine-tuning. FedRef integrates a Maximum A Posteriori (MAP)-based regularization mechanism that calibrates global updates toward a temporally aggregated reference model, thereby effectively controlling unbounded drift and maximizing empirical stability. Moreover, unlike prior approaches that rely on complex client-side optimization heuristics, FedRef achieves superior generalization to reflect recency weighting by seamlessly blending a robust probabilistic prior with global likelihood optimization. By maintaining a temporal moving average centered on previous global trajectories, the proposed framework provides a highly reliable optimization anchor.
Figures & tables
Symbol
Description
A
Aggregation function
s
Local training step
k
Client index, k={1,2,…,K}
Bks
Mini-batch sampled from local dataset of client k
Dk
Dataset of client k
Dref
Synthetic data of reference model
TABLE I : List of Symbols and Abbreviations
Fig. 1 : Overview of our FedRef system.
Hyperparameter
Value
Number of clients ( K )
10
Number of epochs per round
3
FedProx ( 3 )
μ=0.5
FedOpt ( 4 )
β1=0.9 , β2=0.99 , τ=10−4
AdaBest ( 5 )
α:0.1 , β:1.0
FedEve ( 6 )
β:0.9 , μ:0.01 , ρ:0.9
TABLE II : Hyperparameters
Fig. 2 : Number of samples per client for each dataset: CINIC-10 (high IID), FEMNIST (low non-IID), and FeTS2022 (high non-IID).
UDP
λ
ρ
KLD
Accuracy ( % )
F1score ( % )
0.2
0.01
1
2.050
25.09
25.67
4
25.11
25.68
8
25.06
25.64
0.1
1
25.13
25.76
4
25.08
25.66
8
25.08
25.69
TABLE III : FedRef Hyperparameter Sensitivity in CINIC-10
UDP
λ
ρ
KLD
Accuracy ( % )
F1score ( % )
0.2
0.01
1
3.987
65.30
70.62
4
3.992
65.25
70.61
8
3.991
65.31
70.62
0.1
1
3.975
65.38
70.69
4
4.002
65.17
70.50
8
3.974
65.31
70.62
TABLE IV : FedRef Hyperparameter Sensitivity in FEMNIST
UDP
λ
ρ
KLD
Dice Coefficient ( % )
Hausdorff Distance
0.2
0.01
1
6556286
32.39
10.51
4
6556006
9.19
8
6562821
10.54
0.1
1
6558238
9.51
4
6553981
11.24
8
6550471
8.43
TABLE V : FedRef Hyperparameter Sensitivity in FeTS
Dataset
Method
UDP
KLD
Accuracy (%)
F1score (%)
CINIC-10
FedAvg
0.2
2.050
25.13
25.72
FedProx
2.050
25.09
25.71
FedOpt
2.050
25.09
25.73
Adabest
2.325
13.82
14.70
FedEve
2.064
24.50
25.12
A-FedPD
2.051
25.09
25.72
TABLE VI : Predictive Performance Table in Cinic-10
Dataset
Method
UDP
KLD
Accuracy (%)
F1score (%)
FEMNIST
FedAvg
0.2
3.995
65.32
70.65
FedProx
3.989
65.32
70.64
FedOpt
3.994
65.27
70.60
Adabest
5.101
35.68
43.04
FedEve
3.197
61.75
66.42
A-FedPD
4.095
64.81
70.24
TABLE VII : Predictive Performance Table in FEMNIST
Dataset
Method
UDP
KLD
DC (%)
Hausdorff Distance
FeTS
FedAvg
0.2
6561239
32.39
10.02
FedProx
6554945
32.38
9.30
FedOpt
6564171
32.39
9.15
Adabest
6564214
32.09
9.08
FedEve
6550415
32.34
11.32
A-FedPD
6562726
32.14
9.84
TABLE VIII : Predictive Performance Table in Image Segmentation
Method
Optimization Paradigm
Drift Control Strategy
Generalization Framework
FedAvg [ 3 ]
Standard Empirical Risk Minimization
Raw Model Aggregation
No Prior Alignment
FedProx [ 9 ]
Local Proximal Regularization
Static Norm Constraints
Immediate Global Anchor
FedOpt [ 10 ]
Server-Side Adaptive Optimization
Momentum-based Smoothing
Gradient-driven Tracking
AdaBest [ 6 ]
Client-Side Adaptive Bias Estimation
Dynamic Update Norm Constraints
Local Bias Compensation
FedEve [ 7 ]
Predict-Observe Filtering
Variance Reduction Heuristics
Dual Drift Filtering
A-FedPD [ 8 ]
Primal-Dual Optimization
Aligning Dual-Drift
Dual Drift Balancing
TABLE IX : Comparison of Optimization Mechanisms and Generalization Attributes