Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalization

Li Shen; Yan Sun; Dacheng Tao

cs.LGApr 14, 2026

Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalization

Authors: Li Shen, Yan Sun, Dacheng Tao

Organizations: School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University · School of Computer Science, Faculty of Engineering, The University of Sydney · Generative AI Lab, College of Computing and Data Science, Nanyang Technological University

Abstract

Federated learning (FL) is a distributed paradigm that coordinates massive local clients to collaboratively train a global model via stage-wise local training processes on the heterogeneous dataset. Previous works have implicitly studied that FL suffers from the client-drift'' problem, which is caused by the inconsistent optimum across local clients. However, till now it still lacks solid theoretical analysis to explain the impact of this local inconsistency. To alleviate the negative impact of client drift'' and explore its substance in FL, in this paper, we first propose an efficient FL algorithm FedInit, which allows employing the personalized relaxed initialization state at the beginning of each local training stage. Specifically, FedInit initializes the local state by moving away from the current global state towards the reverse direction of the latest local state. Moreover, to further understand how inconsistency disrupts performance in FL, we introduce the excess risk analysis and study the divergence term to investigate the test error in FL. Our studies show that optimization error is not sensitive to this local inconsistency, while it mainly affects the generalization error bound. Extensive experiments are conducted to validate its efficiency. The proposed FedInit method could achieve comparable results compared to several advanced benchmarks without any additional training or communication costs. Meanwhile, the stage-wise personalized relaxed initialization could also be incorporated into several current advanced algorithms to achieve higher generalization performance in the FL paradigm.

Figures & tables

Symbol	Definition
$\mathcal{N},N$	set / number of active clients
$\mathcal{C},C$	set / number of total clients
$K,k$	number / index of local interval
$T,t$	number / index of communication rounds
$w,w^{\star}$	model parameters / optimum parameters
$\mathcal{S},S$	local dataset / size of local dataset

Table 1: Notation tables.

Assumption	Convergence bound	Convergence rate
1 , 3 , 5	$\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}\\|\nabla f(w^{t})\\|^{2}\leq\frac{D}{\lambda\eta KT}+\eta\frac{\kappa L\sigma}{\lambda N}-\frac{13\beta^{2}\kappa_{\beta}L^{2}}{\lambda\eta NKT}\Delta^{T}$	$\mathcal{O}\left(\frac{1}{\sqrt{NKT}}\right)$
1 , 2 , 3 , 5	$\mathbb{E}\left[f(w^{T})-f(w^{\star})\right]\leq e^{-\lambda\mu\eta KT}D+\eta\frac{\kappa L\sigma}{\lambda\mu N}+\mathcal{O}\left(\frac{\eta}{T}+\eta^{2}\right)$	$\mathcal{O}\left(\frac{1}{NKT}\right)$
1 , 4 , 6	$\frac{1}{T}\sum_{t=0}^{T-1}\mathbb{E}\\|\nabla f(w^{t})\\|^{2}\leq\frac{D}{\zeta\eta KT}-\frac{2\gamma_{\beta}\beta^{2}L}{\zeta\eta KT}\Delta^{T}$	$\mathcal{O}\left(\frac{1}{NKT}\right)$
1 , 2 , 4 , 6	$\mathbb{E}\left[f(w^{T})-f(w^{\star})\right]\leq e^{-\zeta\mu\eta KT}D+R_{\beta}\eta^{2}K^{2}LD$	$\mathcal{O}\left(\frac{1}{T^{2}}\right)$

Table 2: Summary of the convergence bound of our proposed FedInit method.

	SGD	FedAvg
number of samples	$\mathcal{O}\left(\frac{1}{CS}\right)$	$\mathcal{O}\left(\frac{N^{\frac{cKL}{1+cKL}}}{CS}\right)$
number of iterations	$\mathcal{O}\left(T^{\frac{cL}{1+cL}}\right)$	$\mathcal{O}\left((TK)^{\frac{cKL}{1+cKL}}\right)$

Table 3: Comparison between vanilla SGD and FedAvg.

Method	CIFAR-10
	$10\%$ -100 clients		$5\%$ -200 clients
	Dir-0.6	Dir-0.1	Dir-0.6	Dir-0.1
FedAvg	$78.77_{\pm.11}$	$72.53_{\pm.17}$	$74.81_{\pm.18}$	$70.65_{\pm.21}$
FedAdam	$76.52_{\pm.14}$	$70.44_{\pm.22}$	$73.28_{\pm.18}$	$68.87_{\pm.26}$
FedSAM	$79.23_{\pm.22}$	$72.89_{\pm.23}$	$75.45_{\pm.19}$	$71.23_{\pm.26}$
SCAFFOLD	$81.37_{\pm.17}$	$75.06_{\pm.16}$	$78.17_{\pm.28}$	$74.24_{\pm.22}$

Table 4: Test accuracy on the CIFAR-10 dataset. We test two active ratios on each dataset. On each setup, we test two Dirichlet splittings, and each result tests 3 times. The table reports results on ResNet-18-GN (upper) and VGG-11 (lower).

Method	CIFAR-100
	$10\%$ -100 clients		$5\%$ -200 clients
	Dir-0.6	Dir-0.1	Dir-0.6	Dir-0.1
FedAvg	$46.35_{\pm.15}$	$42.62_{\pm.22}$	$44.70_{\pm.22}$	$40.41_{\pm.33}$
FedAdam	$48.35_{\pm.17}$	$40.77_{\pm.31}$	$44.33_{\pm.26}$	$38.04_{\pm.25}$
FedSAM	$47.51_{\pm.26}$	$43.43_{\pm.12}$	$45.98_{\pm.27}$	$40.22_{\pm.27}$
SCAFFOLD	$51.98_{\pm.23}$	$\textbf{44.41}_{\pm.15}$	$50.70_{\pm.29}$	$41.83_{\pm.29}$

Table 5: Test accuracy on the CIFAR-10

/

100 dataset. We test two active ratios on each dataset. On each setup, we test two Dirichlet splittings, and each result tests 3 times. The table reports results on ResNet-18-GN (upper) and VGG-11 (lower).

Panel of Figure 1 — Figure 1: Sensitivity studies of local intervals $K$ and relaxed coefficient $\beta$ of the FedInit method on CIFAR-10. To fairly compare their efficiency.

Panel of Figure 2 — Figure 2: The accuracy and divergence of different $\beta$ . (a) and (b) are tested with FedAvg + RI ( FedInit ), while (c) and (d) for SCAFFOLD + RI. (a) and (c) are tested on the Dir-0.6 10%-100 setups, while (b) and (d) for Dir-0.1 5%-200 setups.

Method	$10\%$ -100 clients				$5\%$ -200 clients
	Dir-0.6		Dir-0.1		Dir-0.6		Dir-0.1
	-	+RI	-	+RI	-	+RI	-	+RI
FedAvg	78.77	83.11	72.53	75.95	74.81	80.58	70.65	74.92
FedAdam	76.52	78.33	70.44	72.55	73.28	78.33	68.87	71.34
FedSAM	79.23	83.36	72.89	76.34	75.45	80.66	71.23	75.08
SCAFFOLD	81.37	83.27	75.06	77.30	78.17	81.02	74.24	76.22

Table 6: We incorporate the relaxed initialization (RI) into the different benchmarks to test their improvements on the ResNet-18-GN model on the CIFAR-10 dataset under the same settings and hyperparameters selections.

	FedAvg	FedAdam	FedSAM	SCAFFOLD	FedDyn	FedCM	FedInit
10 $\%$ -100	19.38	23.22	30.23	28.61	23.84	22.63	20.41
ratio	1 $\times$	1.19 $\times$	1.56 $\times$	1.47 $\times$	1.23 $\times$	1.17 $\times$	1.05 $\times$
5 $\%$ -200	15.87	17.50	22.18	24.49	20.61	18.19	16.14
ratio	1 $\times$	1.10 $\times$	1.40 $\times$	1.54 $\times$	1.30 $\times$	1.15 $\times$	1.02 $\times$

Table 7: Averaged wall-clock time cost (s

/

round) of the ResNet-18-GN model in the training process on CIFAR-10 dataset.

	Round		Time (s)
		Speed Ratio		Speed Ratio
FedAvg	371	1 $\times$	7189	1 $\times$
FedAdam	489	0.76 $\times$	11354	0.63 $\times$
FedSAM	377	0.98 $\times$	11396	0.63 $\times$
SCAFFOLD	248	1.50 $\times$	7095	1.01 $\times$
FedDyn	192	1.93 $\times$	4577	1.57 $\times$

Table 8: Acceleration ratio of the communication rounds and wall-clock time of the ResNet-18-GN model on CIFAR-10 dataset.

Method	Communication	ratio	Storage	ratio
FedAvg	$Nd$	1 $\times$	$Cd$	1 $\times$
FedAdam	$Nd$	1 $\times$	$2Cd$	2 $\times$
FedSAM	$Nd$	1 $\times$	$2Cd$	2 $\times$
SCAFFOLD	$2Nd$	2 $\times$	$2Cd$	2 $\times$
FedDyn	$Nd$	1 $\times$	$2Cd$	2 $\times$
FedCM	$2Nd$	2 $\times$	$2Cd$	2 $\times$

Table 9: Communication and storage costs.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Formulation	Description
$w_{i,k}^{t}$	-	parameters at $k$ -th iteration in round $t$ on client $i$
$w^{t}$	-	global parameters in round $t$
$V_{1}^{t}$	$\frac{1}{C}\sum_{i\in\mathcal{C}}\sum_{k=0}^{K-1}\mathbb{E}\\|w_{i,k}^{t}-w^{t}\\|^{2}$	averaged norm of the local updates in round $t$
$V_{2}^{t}$	$\mathbb{E}\\|w^{t+1}-w^{t}\\|^{2}$	norm of the global updates in round $t$
$\Delta^{t}$	$\frac{1}{C}\sum_{i\in\mathcal{C}}\mathbb{E}\\|w_{i,K}^{t-1}-w^{t}\\|^{2}$	inconsistency/divergence term in round $t$
$D$	$f(w^{0})-f(w^{\star})$	bias between the initialization state and optimal

Appendix

Table 10: Some abbreviations of the used terms in the proof of the optimization process.

Notation	Formulation	Description
$w$	-	parameters trained with set $\mathcal{C}$
$\widetilde{w}$	-	parameters trained with set $\widetilde{\mathcal{C}}$
$\Delta^{t}$	$\frac{1}{C}\sum_{i\in\mathcal{C}}\mathbb{E}\\|w_{i,K}^{t-1}-w^{t}\\|^{2}$	inconsistency/divergence term in round $t$
$\delta_{k}^{t}$	$\frac{1}{C}\sum_{i\in\mathcal{C}}\mathbb{E}\\|w_{i,k}^{t}-\widetilde{w}_{i,k}^{t}\\|^{2}$	stability difference at $k$ -iteration on $t$ -round

Appendix

Table 11: Some abbreviations of the used terms in the proof of bounded stability error.

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 943 papers

Is High Local Accuracy Enough? Geometric Consensus with Support-Adaptive Statistical Personalization for Federated Learning
Oct 22, 2026Junyu Lin, Bosong Qi, Pengcheng Shi +1Federated Learning Heterogeneous Federated Learning
School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430072, China
Global Federated Learning Strategies for Building Efficient Personalized Models
Aug 15, 2026Seongyoon KimFederated Learning Heterogeneous Federated Learning
한국과학기술원 Korea Advanced Institute of Science and Technology 산업및시스템공학과
FedAvg for HAR: Exploring the Tradeoff Between Personalized and Generalization Accuracy
Jul 3, 2026Andrea De Luna, Susanna Peretti, Chiara Contoli +1Federated Learning Aggregation Human Activity Recognition
DiSPeA University of Urbino

Oct 22, 2026J/K move · Enter open · S save

Is High Local Accuracy Enough? Geometric Consensus with Support-Adaptive Statistical Personalization for Federated Learning

Junyu Lin, Bosong Qi, Pengcheng Shi, Jiayuan Li

School of Remote Sensing and Information Engineering, Wuhan University, Wuhan 430072, China

Federated Learning Heterogeneous Federated Learning

Open paper AskPDF arXiv

Real-world deployment of personalized federated learning (PFL) requires adapting to heterogeneous clients while preserving recognition performance beyond local distributions. However, high local accuracy can conceal failures on locally rare or absent classes. Effective personalization should therefore strengthen global recognition while raising the personalization floor. We propose an evaluation protocol that complements local accuracy with global, rare-class, and unseen-class evaluations to reveal generalization obscured by local scores, and assesses personalization floors among the worst-performing clients to expose weaknesses hidden by aggregate performance. To meet these deployment challenges, we propose FedGRC, a unified PFL framework integrating generalization and personalization through (i) Consensus-Guided Representation Learning (CGRL), which learns discriminative representations through shared directional and magnitude guidance; (ii) Multilayer Metric Synthesis (MMS), which constructs a shared covariance metric from these multilayer representations for statistical classification; and (iii) Support-Adaptive Shrinkage Personalization (SASP), which adapts class means within this metric by shrinking local estimates toward shared means according to local sample support, enabling adaptation while retaining shared knowledge. Extensive experiments, including benchmarking against 21 methods across 5 heterogeneity settings on 3 datasets and comprehensive ablation studies, validate the effectiveness of FedGRC. On CIFAR-10 and CIFAR-100, FedGRC surpasses state-of-the-art (SOTA) baselines in global recognition and personalization floors by at least 2.17% and 2.13%, respectively. On the real-world heterogeneous FedRS dataset, it outperforms the respective best baselines by 2.15%, 3.30%, and 4.61% in local adaptation, global recognition, and the personalization floor.

Abstract

Federated Learning

Explore similar work

Is High Local Accuracy Enough? Geometric Consensus with Support-Adaptive Statistical Personalization for Federated Learning

Global Federated Learning Strategies for Building Efficient Personalized Models

FedAvg for HAR: Exploring the Tradeoff Between Personalized and Generalization Accuracy