Global Federated Learning Strategies for Building Efficient Personalized Models

Seongyoon Kim

cs.LGAug 15, 2026

Global Federated Learning Strategies for Building Efficient Personalized Models

Authors: Seongyoon Kim

Organizations: 한국과학기술원 Korea Advanced Institute of Science and Technology 산업및시스템공학과

Abstract

Federated learning (FL) is a practical framework that can train models on distributed user data while guaranteeing data privacy; however, due to heterogeneity in which each user has a different data distribution, problems frequently arise where both global and personalization performance deteriorate simultaneously. This dissertation presents methodologies for building efficient personalized models by identifying which strategies are effective in the global training stage and by showing how to preserve global knowledge while securing user-specific performance during local adaptation. First, we show that as data heterogeneity increases, the collapse of feature vectors is a more fundamental bottleneck than classifier weights, and propose a method that directly mitigates the discrepancy in representation magnitude between local and global models. Second, we analyze that a training approach that strengthens local alignment can induce forgetting of global knowledge (e.g., categories not observed locally), and propose a method that achieves both local alignment and global knowledge preservation by combining feature distillation based on the global model's feature vectors. Third, in federated personalized reward model learning with preference heterogeneity, we empirically verify the conventional belief that "increasing the number of global models yields better initialization," and we show that when sufficient local fine-tuning is allowed, a single global initialization can instead provide stronger personalization performance. This study redefines the role of global initialization under data and preference heterogeneity and provides practical training strategies that simultaneously satisfy global knowledge preservation and personalization.

Figures & tables

Figure 1.1 : An overview of dissertation

Indices
$c\in[C]$	index for a class
$r\in[R]$	index for FL round
$n\in[N]$	index for a client
Dataset
$(x,y)$	(input image , true class label of $x$ )
$D_{train},D_{test}$	total train and test dataset

Table 2.1 : Main Notations Throughout the Paper.

Figure 2.1 : 4-Factor Analysis of FedAVG: Results for $s=10$ (Top Row) and $s=2$ (Bottom Row).

Panel of Figure 2 — Figure 2.2 : Means of Classifier Weight and Feature Norms for Local and Global Models with $s$ =2.

Figure 2.3 : Feature Norm Means of Global and Local Models across Varying Data Heterogeneity.

	FedAvg	FedFN
$s$ =10	81.97	83.80
$s$ =2	74.24	77.77

Table 2.2 : Accuracy of FedAvg and FedFN.

Figure 2.5 : 4-Factor Analysis of FedFN; Results for $s$ =10 (Top Row) and $s$ =2 (Bottom Row).

Algorithm	Module	VGG11 on CIFAR-10				MobileNet on CIFAR-100
Algorithm	Module	$s$ =2	$s$ =3	$s$ =5	$s$ =10	$s$ =10	$s$ =50	$s$ =100
	Baseline	73.62	77.70	81.62	82.13	37.25	42.90	43.36
FedAVG (2017)	+ BABU	73.11	76.57	81.22	81.89	43.20	39.70	39.59
	+ FN	76.47	78.66	82.07	83.09	44.67	48.17	49.67
	Baseline	77.07	( Failed )	84.30	84.98	41.86	43.59	41.74
Scaffold (2020)	+ BABU	77.17	( Failed )	83.54	84.43	46.41	41.61	42.55

Table 2.3 : FL Accuracy Comparison for Baseline, +BABU, and +FN.

Algorithm	Pretrained= False				Pretrained= True
Algorithm	$s$ =2	$s$ =3	$s$ =5	$s$ =10	$s$ =2	$s$ =3	$s$ =5	$s$ =10
FedAVG	41.50	55.31	67.64	73.39	37.87	58.31	71.75	84.50
FedBABU	49.21	58.44	68.84	73.82	49.78	49.61	66.46	84.22
FedFN	55.17	60.47	77.12	81.26	56.84	76.84	80.02	84.99

Table 2.4 : Accuracy Comparison on CIFAR-10 on ResNet18.

$\eta$ =0.01	$E$ =1			$E$ =5			$E$ =10			$E$ =15			$E$ =20
$\eta$ =0.01	BABU	AVG	FN	BABU	AVG	FN	BABU	AVG	FN	BABU	AVG	FN	BABU	AVG	FN
$s$ =10	49.51	52.98	52.02	82.43	82.68	82.82	82.56	83.03	82.7	82.16	81.97	82.11	81.82	81.66	82.42
$s$ =5	42.47	46.28	50.26	79.39	80.42	79.76	81.13	81.36	82.09	81.04	81.08	81.74	80.82	81.35	81.3
$s$ =3	29.45	38.32	48.24	71.64	73.83	72.94	78.39	77.39	77.49	77.73	77.29	78.37	78.00	78.03	78.48
$s$ =2	24.21	30.61	46.24	56.37	63.78	62.5	73.31	73.24	73.79	75.05	74.24	75.34	75.25	74.98	76.33
$\eta$ =0.03	$E$ =1			$E$ =5			$E$ =10			$E$ =15			$E$ =20

Table 2.5 : Grid Search Results for VGG11 on CIFAR-10.

$\eta$ =0.1	$E$ =1			$E$ =5			$E$ =10			$E$ =15			$E$ =20
$\eta$ =0.1	BABU	AVG	FN	BABU	AVG	FN	BABU	AVG	FN	BABU	AVG	FN	BABU	AVG	FN
$s$ =100	41.32	41.57	76.61	40.49	43.19	51.18	37.03	38.13	45.26	36.16	36.10	44.03	35.91	36.19	42.25
$s$ =50	40.86	38.35	7.51	40.76	40.63	51.11	36.79	37.77	46.74	36.10	37.54	44.47	34.93	36.05	43.61
$s$ =10	35.95	27.02	4.85	45.56	36.60	43.18	41.01	34.67	47.63	37.87	35.53	45.50	38.28	35.94	47.41
$\eta$ =0.3	$E$ =1			$E$ =5			$E$ =10			$E$ =15			$E$ =20
$\eta$ =0.3	BABU	AVG	FN	BABU	AVG	FN	BABU	AVG	FN	BABU	AVG	FN	BABU	AVG	FN

Table 2.6 : Grid Search Results for MobileNet on CIFAR-100.

$\eta$ =0.01	$E$ =5			$E$ =10			$E$ =15			$E$ =20
$\eta$ =0.01	BABU	AVG	FN	BABU	AVG	FN	BABU	AVG	FN	BABU	AVG	FN
$s$ =10	75.32	76.07	78.00	74.53	74.17	77.72	73.55	73.85	77.80	73.16	72.32	77.18
$s$ =5	68.02	69.34	72.14	67.54	68.13	71.17	68.36	67.71	73.06	68.56	66.48	74.74
$s$ =3	58.60	58.93	64.94	57.87	58.45	66.45	57.07	57.38	64.23	60.93	55.13	66.58
$s$ =2	46.79	41.15	45.50	45.50	41.92	47.22	49.21	45.92	50.81	47.50	45.23	47.99
$\eta$ =0.03	$E$ =5			$E$ =10			$E$ =15			$E$ =20

Table 2.7 : Grid Search Results for ResNet18 on CIFAR-10.

$\eta$ =0.01	$E$ =5			$E$ =10			$E$ =15			$E$ =20
$\eta$ =0.01	BABU	AVG	FN	BABU	AVG	FN	BABU	AVG	FN	BABU	AVG	FN
$s$ =10	83.85	83.80	84.65	84.06	84.29	84.20	84.02	83.96	84.43	84.43	84.33	84.81
$s$ =5	68.83	73.31	74.90	69.73	69.24	74.72	69.44	70.77	75.26	68.58	70.89	76.75
$s$ =3	54.21	58.52	63.87	51.85	48.52	61.76	48.96	57.85	60.72	47.84	51.11	62.06
$s$ =2	48.21	33.84	47.74	43.02	35.91	47.30	49.00	39.04	50.82	44.91	36.95	49.98
$\eta$ =0.03	$E$ =5			$E$ =10			$E$ =15			$E$ =20

Table 2.8 : Grid Search Results for Pretrained ResNet18 on CIFAR-10.

$\eta$ =0.1	$E$ =1		$E$ =5		$E$ =10
$\eta$ =0.1	CE	MSE	CE	MSE	CE	MSE
$s$ =100	12.96	1.37	44.92	2.16	39.50	4.25
$s$ =50	12.21	1.30	43.38	1.73	38.65	3.84
$s$ =10	6.67	1.46	33.32	2.34	30.02	3.81
$\eta$ =0.3	$E$ =1		$E$ =5		$E$ =10
$\eta$ =0.3	CE	MSE	CE	MSE	CE	MSE

Table 2.9 : Grid Search Results for SphereFed with MobileNet on CIFAR-100.

$\eta$ =1.5	$E$ =1	$E$ =5	$E$ =10
$\eta$ =1.5	MSE	MSE	MSE
$s$ =100	2.84	44.40	40.32
$s$ =50	2.81	41.91	42.18
$s$ =10	2.69	33.68	35.66
$\eta$ =3.0	$E$ =1	$E$ =5	$E$ =10
$\eta$ =3.0	MSE	MSE	MSE

Table 2.10 : Additional Grid Search Results for SphereFed (MSE) with MobileNet on CIFAR-100.

Figure 2.6 : Discrepancy in Weight Norms between Local and Global Models in the

s

=2 Setting.

Figure 2.7 : Discrepancy in Weight Norms between Local and Global Models in Various Heterogeneity Settings.

Algorithm	Module	VGG11 on CIFAR-10								MobileNet on CIFAR-100
Algorithm	Module	$s$ =2	$s$ =3	$s$ =5	$s$ =10	$\alpha$ =0.1	$\alpha$ =0.3	$\alpha$ =0.5	$\alpha$ =1.0	$s$ =10	$s$ =50	$s$ =100	$\alpha$ =0.1	$\alpha$ =0.3	$\alpha$ =0.5	$\alpha$ =1.0
	Baseline	73.83	78.99	81.40	82.79	73.16	80.77	81.63	82.45	26.64	40.18	41.50	42.66	42.68	43.82	41.57
FedAVG	+ BABU	74.31	78.88	81.17	82.37	72.42	80.59	80.94	82.93	43.96	39.94	40.37	42.99	39.61	39.53	39.58
	+ FN	77.23	81.46	82.80	84.16	75.58	81.85	83.06	83.65	45.16	48.83	49.54	47.87	47.66	47.86	47.80
	Baseline	77.07	( Failed )	84.50	85.28	( Failed )	83.24	83.54	85.21	34.64	42.56	43.67	45.18	45.62	44.42	43.16
Scaffold	+ BABU	76.89	82.24	84.26	85.02	( Failed )	82.95	84.12	85.06	46.80	42.73	44.50	43.28	43.76	44.37	44.20

Table 2.11 : FL Accuracy Comparison for Baseline, +BABU, and +FN.

Algorithm	$s$ =10	$s$ =50	$s$ =100
Local only	58.64 $\pm$ 7.21	25.38 $\pm$ 4.12	18.52 $\pm$ 3.15
FedPer (2019)	70.92 $\pm$ 6.93	33.73 $\pm$ 4.68	22.77 $\pm$ 4.30
Per-FedAVG (2020)	32.57 $\pm$ 11.02	43.09 $\pm$ 7.36	45.00 $\pm$ 7.05
FedRep (2021)	62.69 $\pm$ 7.26	34.66 $\pm$ 5.34	26.53 $\pm$ 4.52
FedAVG (2017)	36.30	41.97	42.51
FedAVG-FT (0.1 $\times\eta$ )	77.39 $\pm$ 6.40	50.57 $\pm$ 5.31	46.95 $\pm$ 4.90

Table 2.12 : PFL Accuracy Comparison for MobileNet on CIFAR-100.

Algorithm	$s$ =10	$s$ =50	$s$ =100
FedAVG (2017)	36.30	41.97	42.51
FedAVG-FT ( $\eta$ )	67.23 $\pm$ 6.32	35.98 $\pm$ 5.56	38.25 $\pm$ 5.29
FedAVG-FT (0.1 $\times\eta$ )	77.39 $\pm$ 6.40	50.57 $\pm$ 5.31	46.95 $\pm$ 4.90
FedAVG-FT (0.01 $\times\eta$ )	70.96 $\pm$ 6.81	44.78 $\pm$ 4.88	44.00 $\pm$ 4.93
FedBABU (2022)	45.73	39.57	40.7
FedBABU-FT ( $\eta$ )	13.32 $\pm$ 4.51	4.72 $\pm$ 1.85	2.85 $\pm$ 1.27

Table 2.13 : Search for Finetuning Learning Rates in Two-Step Methods.

Algorithm	Accuracy
SphereFed (CE), $\tau$ =1	40.39
SphereFed (CE), $\tau$ =10	42.84
SphereFed (CE), $\tau$ =15	45.78
SphereFed (CE), $\tau$ =20	44.95
SphereFed (CE), $\tau$ =25	44.46
SphereFed (CE), $\tau$ =30	39.62

Table 2.14 : Comparison of Modified SphereFed (CE) with SphereFed (MSE), FedBABU, and FedFN on

s

=10 Setting of CIFAR-100.

SphereFed (MSE)	$\eta$ =0.1	$\eta$ =0.3	$\eta$ =0.5	$\eta$ =1.0	$\eta$ =1.5	$\eta$ =3.0	$\eta$ =4.5	$\eta$ =5.0	Original Result ( $\eta$ =0.5 )
$\alpha$ =0.5	2.77	9.73	18.01	43.26	52.19	52.69	48.20	40.87	71.85
$\alpha$ =0.1	2.88	9.65	20.19	41.48	45.41	46.34	43.62	40.56	-
SphereFed (CE)	$\eta$ =0.1	$\eta$ =0.3	$\eta$ =0.5	$\eta$ =1.0	$\eta$ =1.5	$\eta$ =3.0	$\eta$ =4.5	$\eta$ =5.0	Original Result
$\alpha$ =0.5	37.57	49.39	48.18	52.51	52.99	51.09	44.87	44.23	-
$\alpha$ =0.1	22.37	40.58	42.85	40.92	47.72	42.11	35.30	33.83	-
FedAVG	$\eta$ =0.1	$\eta$ =0.3	$\eta$ =0.5	$\eta$ =1.0	$\eta$ =1.5	$\eta$ =3.0	$\eta$ =4.5	$\eta$ =5.0	Original Result ( $\eta$ =0.1 )

Table 2.15 : Reproduced Results for SphereFed, FedAVG, and FedFN under the Same Settings, Utilizing MobileNet (as Described in [ 1 ] ) on CIFAR-100.

Algorithm	VGG11 on CIFAR-10
Algorithm	$s$ =2	$s$ =3	$s$ =5	$s$ =10
FedAVG	74.24	77.29	81.08	81.97
FedBABU	75.05	77.73	81.04	82.16
FedFR	76.14	77.89	81.61	82.18
FedFN	77.77	78.93	82.43	83.80

Table 2.16 : Accuracy Comparison on CIFAR-10.

FedFR	$\mu$ =0.5	$\mu$ =0.1	$\mu$ =0.05	$\mu$ =0.01	$\mu$ =0.005	$\mu$ =0.001	$\mu$ =0.0005	$\mu$ =0.0001
FedFR	(Failed)	37.74	39.50	36.72	36.51	36.77	37.04	37.30
FedAVG	36.30 ( $\mu$ =0.0 )
FedBABU	45.73
FedFN	46.98

Table 2.17 : Accuracy Comparison on the

s=10

Setting of MobileNet on CIFAR-100.

Figure 2.8 : Module Comparison in Centralized Learning

Algorithm	shard 10	shard 50	shard 100	lda 0.1	lda 0.5	lda 1.0
Local only	58.64	25.38	18.52	43.63	21.03	14.88
FedPer (2019)	70.92	33.73	22.77	55.93	27.32	18.93
Per-FedAVG (2020)	32.57	43.09	45.00	35.12	42.95	42.82
FedRep (2021)	62.69	34.66	26.53	48.52	30.94	23.93
FedAVG (2017)	36.63	40.51	43.18	33.70	39.91	41.36
FedAVG-FT (2017)	70.39	43.54	44.08	55.06	45.07	43.66

Table 27

Figure 3.1: Overview of the proposed method, FedDr+ trained with $\mathcal{L}_{\text{Dr+}}$ . To enhance the local alignment, we employ dot-regression loss $\mathcal{L}_{\text{DR}}$ , which discards the pushing term of cross-entropy loss, and propose a feature distillation $\mathcal{L}_{\text{FD}}$ to preserve the knowledge imbued in the global model.

Indices
$c\in[C]$	Index for a class
$r\in[R]$	Index for FL round
$i\in[N]$	Index for a client
Dataset
$D_{\text{train}}^{i}$	Training dataset for client $i$
$D_{\text{test}}^{i}$	Test dataset for client $i$

Table 3.1 : Main used throughout the paper.

Panel of Figure 3 — Figure 3.2 : Comparison of (a) feature-classifier alignment gap and (b) feature-classifier alignment on the observed and unobserved classes test data for $\bm{\theta}_{r}^{i}$ trained with $\mathcal{L}_{\text{CE}}$ and $\mathcal{L}_{\text{DR}}$ .

Figure 3.3 : Comparison of the global test accuracy of

\bm{\theta}_{r}^{g}

on all classes trained using

\mathcal{L}_{\text{CE}}

and

\mathcal{L}_{\text{DR}}

	Sharding ( $s=10$ )							LDA ( $\alpha=0.1$ )
Algorithm	Baseline	+Prox	+KD	+NTD	+LD	+MOON	+FD	Baseline	+Prox	+KD	+NTD	+LD	+MOON	+FD
FedAvg	37.22	36.87	36.25	37.71	37.17	37.43	37.82	42.52	43.22	44.21	43.39	43.43	44.79	43.76
FedBABU	46.20	46.03	46.37	47.22	46.71	46.49	46.95	47.37	46.62	47.60	46.48	45.78	46.27	46.49
SphereFed	43.90	41.96	44.94	43.47	43.95	43.13	45.21	46.98	43.77	47.76	47.25	47.01	46.81	49.74
FedETF	32.42	31.87	32.76	32.65	32.25	34.30	32.77	46.27	45.71	46.67	46.16	45.91	45.98	46.47
FedGELA	29.17	28.69	29.11	28.84	29.36	28.80	30.33	27.11	29.03	28.45	29.62	29.41	28.09	29.75

Table 3.2 : Synergy of various FL algorithms and regularizers. Baseline indicates training FL models without a regularizer. FD denotes feature distillation, which is the regularizer we use in FedDr+ .

NIID Partition Strategy: Sharding
	CIFAR-100				CIFAR-10			ImageNet-100
	$s$ =10	$s$ =20	$s$ =50	$s$ =100	$s$ =2	$s$ =5	$s$ =10	ImageNet-100
FedAvg	36.63 $\pm$ 0.22	42.25 $\pm$ 1.42	45.57 $\pm$ 0.22	48.20 $\pm$ 1.36	72.08 $\pm$ 0.67	81.53 $\pm$ 0.35	82.38 $\pm$ 0.40	67.78 $\pm$ 0.41
FedProx	37.07 $\pm$ 0.21	42.35 $\pm$ 0.83	45.18 $\pm$ 1.07	47.78 $\pm$ 0.79	71.92 $\pm$ 0.51	81.29 $\pm$ 0.40	82.45 $\pm$ 0.35	67.81 $\pm$ 0.65
SCAFFOLD ( $\times 2$ )	46.08 $\pm$ 0.37	48.15 $\pm$ 1.21	49.31 $\pm$ 0.62	50.73 $\pm$ 0.42	75.49 $\pm$ 0.42	84.14 $\pm$ 0.13	85.11 $\pm$ 0.29	70.47 $\pm$ 0.46
MOON	36.95 $\pm$ 0.37	43.05 $\pm$ 0.27	43.95 $\pm$ 0.12	46.92 $\pm$ 0.08	67.55 $\pm$ 1.16	80.90 $\pm$ 0.26	82.62 $\pm$ 0.31	68.19 $\pm$ 0.32

Table 3.3: Accuracy comparison in the GFL setting. The entries are based on results obtained from three different seeds, indicating the mean and standard deviation of the accuracy of the global model, represented as X

\pm

Y . The best performance in each case is highlighted in bold .

	Local update without global model						Local update with global model
	FedAvg	FedBABU	FedETF	SphereFed	FedGELA	FedExP	FedProx	SCAFFOLD	FedNTD	FedSOL	MOON	FedDr+ (Ours)
Elapsed time	20.4	20.5	20.7	20.4	20.5	20.1	22.6	21.4	21.2	27.2	56.7	24.9

Table 3.4: Elapsed time per round (in seconds) for various GFL algorithms.

Algorithm	$s$ =10	$s$ =20	$s$ =100	$\alpha$ =0.05	$\alpha$ =0.1	$\alpha$ =0.3
Local only ( $\mathcal{L}_{\text{CE}}$ )	58.42 $\pm$ 0.22	42.37 $\pm$ 0.29	19.02 $\pm$ 0.34	55.71 $\pm$ 0.19	44.02 $\pm$ 0.39	27.98 $\pm$ 0.08
Local only ( $\mathcal{L}_{\text{CE}}$ +ETF)	58.05 $\pm$ 0.25	41.72 $\pm$ 0.26	19.06 $\pm$ 0.18	55.41 $\pm$ 0.21	43.56 $\pm$ 0.31	27.69 $\pm$ 0.17
Local only ( $\mathcal{L}_{\text{DR}}$ )	61.05 $\pm$ 0.37	44.28 $\pm$ 0.21	21.06 $\pm$ 0.21	58.56 $\pm$ 0.14	47.05 $\pm$ 0.13	31.16 $\pm$ 0.15
FedPer	70.62 $\pm$ 0.71	55.65 $\pm$ 1.35	25.57 $\pm$ 0.59	63.35 $\pm$ 1.96	51.90 $\pm$ 2.13	35.84 $\pm$ 2.16
Per-FedAvg	31.71 $\pm$ 1.08	38.64 $\pm$ 0.40	45.71 $\pm$ 0.81	28.85 $\pm$ 0.27	36.00 $\pm$ 0.42	42.41 $\pm$ 0.32
FedRep	62.59 $\pm$ 0.30	51.18 $\pm$ 1.00	26.51 $\pm$ 0.27	57.73 $\pm$ 0.41	49.59 $\pm$ 0.40	36.22 $\pm$ 0.86

Table 3.5 : PFL accuracy comparison with MobileNet on CIFAR-100. Results are reported in the format X

\pm

Y , representing the mean and standard deviation of the average personalized accuracies across all clients, computed over five seeds. The best performance in each case is highlighted in bold .

Indices
$c\in[C]$	Index for a class
$r\in[R]$	Index for FL round
$i\in[N]$	Index for a client
Dataset
$D_{\text{train}}^{i}$	Training dataset for client $i$
$D_{\text{test}}^{i}$	Test dataset for client $i$

Table 3.6 : Notations used throughout the paper.

Figure 40

Figure 3.8 : Feature gradient of $\mathcal{L}_{\text{DR}}$ . The gradient update rotates $f(x;\bm{\theta}_{f})$ toward $V_{y}$ while increasing its norm. As training progresses, the update magnitude decreases, leading to convergence.

Datasets	$C$	$\|D_{\text{train}}\|$	$\|D_{\text{test}}\|$	$N$	$R$	$r$	$E$	$B$	$m$	$\lambda$
CIFAR-10	10	50000	10000	100	320	0.1	10	50	0.9	1e-5
CIFAR-100	100	50000	10000	100	320	0.1	3	50	0.9	1e-5
ImageNet-100	100	130000	5000	100	320	0.1	5	50	0.9	1e-5

Table 3.7: Summary of Dataset, Model, FL System, and Optimizer Specifications

Feature un-normalized algorithms

Feature normalized algorithms

Hyperparameters

FedAvg

FedBABU

FedProx

SCAFFOLD

MOON

FedNTD

FedExP

FedSOL

FedGELA

FedETF

SphereFed

FedDr+ (Ours)

\eta

0.01

0.05

0.55

0.35

Additional

None

\mu

=0.001

None

(\mu,\tau)

=(1,0.5)

(\beta,\tau)

=(1,3)

\epsilon

=0.001

\rho

=2.0

None

(\beta,\tau)

=(1,1)

None

\beta

=0.9

Table 3.8: Hyperparameters for VGG11 training on CIFAR-10.

Feature un-normalized algorithms

Feature normalized algorithms

Hyperparameters

FedAvg

FedBABU

FedProx

SCAFFOLD

MOON

FedNTD

FedExP

FedSOL

FedGELA

FedETF

SphereFed

FedDr+ (Ours)

\eta

0.1

0.5

6.5

5.0

Additional

None

\mu

=0.001

None

(\mu,\tau)

=(1,0.5)

(\beta,\tau)

=(1,3)

\epsilon

=0.001

\rho

=2.0

None

(\beta,\tau)

=(1,1)

None

\beta

=0.9

Table 3.9: Hyperparameters for MobileNet training on CIFAR-100.

Feature un-normalized algorithms

Feature normalized algorithms

Hyperparameters

FedAvg

FedBABU

FedProx

SCAFFOLD

MOON

FedNTD

FedExP

FedSOL

FedGELA

FedETF

SphereFed

FedDr+ (Ours)

\eta

0.1

1.0

Additional

None

\mu

=0.001

None

(\mu,\tau)

=(1,0.5)

(\beta,\tau)

=(1,3)

\epsilon

=0.001

\rho

=2.0

None

(\beta,\tau)

=(1,1)

None

\beta

=0.9

Table 3.10: Hyperparameters for ResNet-18 training on ImageNet-100.

	Sharding ( $s=10$ )							LDA ( $\alpha=0.1$ )
Algorithm	Baseline	+Prox	+MOON	+KD	+NTD	+LD	+FD	Baseline	+Prox	+MOON	+KD	+NTD	+LD	+FD
FedAvg	37.22	36.87	37.43	36.25	37.71	37.17	37.82	42.52	43.22	44.79	44.21	43.39	43.43	43.76
FedBABU	46.20	46.03	46.49	46.37	47.22	46.71	46.95	47.37	46.62	46.27	47.60	46.48	45.78	46.49
SphereFed	43.90	41.96	43.13	44.94	43.47	43.95	45.21	46.98	43.77	46.81	47.76	47.25	47.01	49.74
FedETF	32.42	31.87	34.30	32.76	32.65	32.25	32.77	46.27	45.71	45.98	46.67	46.16	45.91	46.47
FedGELA	29.17	28.69	28.80	29.11	28.84	29.36	30.33	27.11	29.03	28.09	28.45	29.62	29.41	29.75

Table 3.11 : Synergy of various FL algorithms and regularizers. Baseline indicates training FL models without a regularizer. FD denotes feature distillation, which is the regularizer we use in FedDr+ .

	Sharding ( $s=10$ )							LDA ( $\alpha=0.1$ )
Algorithm	Baseline	+Prox	+MOON	+KD	+NTD	+LD	+FD	Baseline	+Prox	+MOON	+KD	+NTD	+LD	+FD
FedAvg	None	0.999	0.5	0.9999	0.9999	0.999	0.9	None	0.999	0.99	0.999	0.99	0.999	0.9999
FedBABU	None	0.9999	0.9	0.999	0.99	0.999	0.999	None	0.999	0.999	0.999	0.999	0.99	0.9999
SphereFed	None	0.9999	0.9999	0.9999	0.9	0.99	0.9	None	0.9999	0.999	0.999	0.9999	0.999	0.99
FedETF	None	0.999	0.3	0.5	0.999	0.9	0.9	None	0.9999	0.9999	0.5	0.999	0.99	0.99
FedGELA	None	0.9999	0.7	0.5	0.7	0.5	0.7	None	0.99	0.9	0.5	0.5	0.5	0.3

Table 3.12 : Optimal

\beta

value selected through grid search to achieve the best synergy of various FL algorithms and regularizers.

	Sharding ( $s=10$ )							LDA ( $\alpha=0.1$ )
Algorithm	Baseline	+Prox	+MOON	+KD	+NTD	+LD	+FD	Baseline	+Prox	+MOON	+KD	+NTD	+LD	+FD
FedAvg	37.22	30.27	36.67	35.14	35.56	34.83	37.82	42.52	36.09	42.09	41.48	41.34	43.36	43.10
FedBABU	46.20	36.71	46.49	45.50	45.09	45.81	45.31	47.37	39.04	45.92	45.58	45.56	46.46	44.77
SphereFed	43.90	1.36	1.89	41.01	43.47	41.73	45.21	46.98	1.46	2.21	45.22	46.25	43.84	48.61
FedETF	32.42	25.18	32.58	32.76	31.98	32.25	32.77	46.27	34.92	45.38	44.94	45.77	44.36	45.92
FedGELA	29.17	25.52	28.57	28.84	28.67	28.37	29.07	27.11	26.84	28.09	27.78	28.27	27.97	27.60

Table 3.13 : Synergy of various FL algorithms and regularizers at

\beta=0.9

Algorithm	$s$ =10	$s$ =20	$s$ =100	$\alpha$ =0.05	$\alpha$ =0.1	$\alpha$ =0.3
Dot-Regression	42.52	49.02	52.86	30.31 $\pm$ 7.95	37.52 $\pm$ 5.60	47.08 $\pm$ 3.69
Dot-Regression FT ( $\mathcal{L}_{\text{DR}}$ )	80.84 $\pm$ (5.99)	74.18 $\pm$ (5.78)	56.84 $\pm$ (5.04)	72.02 $\pm$ (6.80)	66.96 $\pm$ (5.36)	60.34 $\pm$ (3.66)
Dot-Regression FT ( $\mathcal{L}_{\text{Dr+}}$ )	80.82 $\pm$ (6.12)	73.73 $\pm$ (5.75)	56.69 $\pm$ (4.95)	71.85 $\pm$ (7.03)	66.59 $\pm$ (5.32)	59.87 $\pm$ (3.65)
FedDr+ (ours)	48.69	51.00	53.23	39.63 $\pm$ 9.12	45.83 $\pm$ 6.18	48.04 $\pm$ 3.44
FedDr+ FT ( $\mathcal{L}_{\text{DR}}$ ) (ours)	84.23 $\pm$ (5.44)	75.73 $\pm$ (4.79)	56.90 $\pm$ (4.85)	78.65 $\pm$ (6.17)	74.86 $\pm$ (4.77)	62.47 $\pm$ (3.72)
FedDr+ FT ( $\mathcal{L}_{\text{Dr+}}$ ) (ours)	84.10 $\pm$ (5.43)	75.42 $\pm$ (4.80)	56.76 $\pm$ (4.91)	78.55 $\pm$ (6.16)	74.75 $\pm$ (4.75)	62.16 $\pm$ (3.73)

Table 3.14 : PFL accuracy comparison with MobileNet on CIFAR-100. For PFL, we denote the entries in the form of X

\pm

(Y) , representing the mean and standard deviation of personalized accuracies across all clients derived from a single seed.

Algorithm	Accuracy (mean $\pm$ std)
FedAvg	47.19 $\pm$ 1.06
FedBABU	45.18 $\pm$ 0.61
Dot-regression	51.48 $\pm$ 0.99
FedDr+	51.10 $\pm$ 0.61

Table 3.15 : Global model accuracy (%) in IID data settings.

Algorithm	Static Setting	Stochastic Setting
CE	46.20	43.59
Dot-regression	42.52	38.13
FedDr+	48.69	44.96

Table 3.16 : Global model accuracy (%) in static and stochastic client data settings.

Algorithm	N=100	N=200	N=500	N=1,000
FedAvg	50.50 $\pm$ 0.57	42.51 $\pm$ 1.47	33.02 $\pm$ 0.74	26.63 $\pm$ 1.31
FedBABU	58.19 $\pm$ 1.07	48.75 $\pm$ 1.99	37.40 $\pm$ 0.41	25.10 $\pm$ 1.08
FedDr+	64.21 $\pm$ 1.24	59.78 $\pm$ 0.71	43.27 $\pm$ 0.31	28.99 $\pm$ 0.98

Table 3.17 : Global model accuracy (%) for different numbers of clients with 1,000 communication rounds.

Panel of Figure 4 — Figure 4.1 : Comparison of (a) mean preference-prediction accuracy of $\theta_{m}^{\text{PFL}}$ on $\mathcal{D}_{\text{test}}^{m}$ as a function of fine-tuning steps for the real and synthetic datasets, comparing a single with hard ( $K\in\{3,4,5\}$ ), and (b) hard-cluster indices assigned to each client on the synthetic dataset for the same hard settings.

Figure 4.2 : Mean preference-prediction accuracy of

\theta_{m}^{\text{PFL}}

\mathcal{D}_{\text{test}}^{m}

as a function of fine-tuning steps on the synthetic dataset, averaged over clients within each preference group (G1–G4), comparing Single and Hard .

Figure 4.3 : Mean preference-prediction accuracy on the real (left) and synthetic (right) datasets as a function of fine-tuning steps, comparing global initialization strategies Single , Hard , Hard-Oracle , Soft , Base , and Centralized .

Figure 4.4 : Sensitivity of Single vs. Centralized on the synthetic dataset under matched compute. We vary local iteration steps

\tau\in\{1,5,15,30\}

and the learning rate (default vs.

10\times

smaller), and report mean accuracy before personalization (

\tau=0

) and after

80

fine-tuning updates.

Explore similar work

May 4, 2026cs.LG

Personalized Federated Learning for Gradient Alignment

Personalized federated learning (pFL) aims to adapt models to client specific data distributions, yet it often fails to reliably preserve personalized information. Local training is hindered by high variance gradients induced by limited and heterogeneous client data, while aggregation further distorts client specific optimization directions. To address these challenges, we propose pFLAlign, a gradient alignment framework to maintain client specific information during both local training and aggregation. pFLAlign consists of two complementary mechanisms: one adapts local gradient directions to reduce variance during client side optimization, and the other mitigates aggregation induced distortion by realigning the global model with each client's personalized direction. Theoretically, we derive pFLAlign from a PAC Bayesian analysis, which reveals how personalized gradient alignment preserves client specific information. Our experiments and ablation studies show that pFLAlign consistently improves personalization performance and training stability, achieving state of the art results.

Dongwon Kim, Gyuejeong Lee

SAKAK Inc., Korea · Div. of Artificial Intelligence and Data Science, Korea Cyber University, Korea

Federated Learning Aggregation Non-IID Federated Learning Federated Learning Gradient Alignment

Ask PDF

Sep 15, 2026cs.LG

Personalized Federated Learning through Global Knowledge Distillation and Local Head Adaptation

Statistical heterogeneity limits federated learning when a single global classifier cannot represent client-specific label distributions. In this work, we propose Personalized Federated Knowledge Distillation with Head Adaptation (pFedKDH), which aggregates only the shared backbone, keeps persistent client-specific heads, and uses a recalibrated global head as a teacher during local training. Across MNIST, Fashion-MNIST, CIFAR10, and CIFAR100 under class-wise Dirichlet partitions, pFedKDH obtains the best accuracy in most settings, with accuracy gaps up to 37.67% over the weakest baseline and consistently low standard deviation across repetitions. Component-wise diagnostics and convergence results support the role of persistent heads and distillation-guided local optimization under label-skewed data.

Polycarpo Souza Neto, José Mairton Barros da Silva Júnior, Charles Casimiro Cavalcante

Universidade Federal do Cear´a, Fortaleza, CE, Brazil · Uppsala University, Uppsala, Sweden

Federated Knowledge Distillation Non-IID Federated Learning Knowledge Distillation Federated Learning

Ask PDF

Jul 23, 2024cs.LG

Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformation

Federated Learning (FL) faces challenges due to data heterogeneity, which limits the global model's performance across diverse client distributions. Personalized Federated Learning (PFL) addresses this by enabling each client to possess an individual model adapted to its local distribution. Many existing methods assume that certain global model parameters are difficult to train effectively in a collaborative manner under heterogeneous data. Consequently, they localize or fine-tune these parameters to obtain personalized models. In this paper, we reveal that both the feature extractor and classifier of the global model are inherently strong, and the primary cause of its suboptimal performance is the mismatch between local features and the global classifier. Although existing methods alleviate this mismatch to some extent and improve performance, we find that they either (1) fail to fully resolve the mismatch while degrading the feature extractor, or (2) address the mismatch only post-training, allowing it to persist during training. This increases inter-client gradient divergence, hinders model aggregation, and ultimately leaves the feature extractor suboptimal for client data. To address this issue, we propose FedPFT, a novel framework that resolves the mismatch during training using personalized prompts. These prompts, along with local features, are processed by a shared self-attention-based transformation module, ensuring alignment with the global classifier. Additionally, this prompt-driven approach offers strong flexibility, enabling task-specific prompts to incorporate additional training objectives (e.g., contrastive learning) to further enhance the feature extractor. Extensive experiments show that FedPFT outperforms state-of-the-art methods by up to 5.07%, with further gains of up to 7.08% when collaborative contrastive learning is incorporated.

Xinghao Wu, Xuefeng Liu, Jianwei Niu +4

School of Computer Science and Engineering, Beihang University, Beijing, China · Zhongguancun Laboratory, Beijing, China · Sichuan University, Sichuan, China +1

Prompt Learning Federated Learning Heterogeneous Federated Learning

Ask PDF