Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning

Minghui Chen; Meirui Jiang; Xin Zhang; Qi Dou; Zehua Wang; Xiaoxiao Li

cs.LGOct 31, 2024

Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learning

Authors: Minghui Chen, Meirui Jiang, Xin Zhang, Qi Dou, Zehua Wang, Xiaoxiao Li

Organizations: University of British Columbia · Vector Institute · Chinese University of Hong Kong · Meta

Abstract

Federated learning (FL) is a learning paradigm that enables collaborative training of models using decentralized data. Recently, the utilization of pre-trained weight initialization in FL has been demonstrated to effectively improve model performance. However, the evolving complexity of current pre-trained models, characterized by a substantial increase in parameters, markedly intensifies the challenges associated with communication rounds required for their adaptation to FL. To address these communication cost issues and increase the performance of pre-trained model adaptation in FL, we propose an innovative model interpolation-based local training technique called ``Local Superior Soups.'' Our method enhances local training across different clients, encouraging the exploration of a connected low-loss basin within a few communication rounds through regularized model interpolation. This approach acts as a catalyst for the seamless adaptation of pre-trained models in in FL. We demonstrated its effectiveness and efficiency across diverse widely-used FL datasets. Our code is available at https://github.com/ubc-tea/Local-Superior-Soups.

Figures & tables

Figure 1: Illustration on isolated (left) and connected low-loss valley with larger regions in dark red (right).

Figure 2: Illustration on diversity (left) and affinity (right) regularization.

	FMNIST		CIFAR-10
Method	Accuracy ( $R=1$ ) $\uparrow$	Accuracy ( $R=3$ ) $\uparrow$	Accuracy ( $R=1$ ) $\uparrow$	Accuracy ( $R=3$ ) $\uparrow$
FedAvg [ 35 ]	$35.54(1.71)$	$90.04(0.32)$	$58.34(0.86)$	$66.74(0.76)$
FedProx [ 28 ]	$33.48(1.52)$	$89.28(0.36)$	$56.74(0.92)$	$63.21(0.83)$
MOON [ 27 ]	$36.01(1.66)$	$91.28(0.30)$	$58.96(1.24)$	$67.04(1.12)$
FedBN [ 30 ]	$34.20(1.73)$	$89.87(0.47)$	$57.04(0.75)$	$64.51(0.67)$
FedFomo [ 57 ]	$33.94(1.65)$	$88.41(0.69)$	$55.01(0.89)$	$62.69(0.75)$

Table 1: Label shift test accuracy after

R=1

and

R=3

communication rounds. We primarily compared two categories of methods: conventional FL methods and state-of-the-art local weight averaging-based fine-tuning methods that enhance domain generalization.

	Digit-5		DomainNet
Method	Accuracy ( $R=1$ ) $\uparrow$	Accuracy ( $R=3$ ) $\uparrow$	Accuracy ( $R=1$ ) $\uparrow$	Accuracy ( $R=3$ ) $\uparrow$
FedAvg [ 35 ]	$46.36(2.08)$	$80.48(0.81)$	$18.76(3.52)$	$29.43(2.01)$
FedProx [ 28 ]	$44.01(1.92)$	$77.83(0.68)$	$17.27(3.22)$	$27.18(2.29)$
MOON [ 27 ]	$50.11(1.72)$	$83.02(0.64)$	$19.61(3.54)$	$31.27(2.34)$
FedBN [ 30 ]	$46.02(1.93)$	$81.42(0.71)$	$18.16(3.09)$	$28.65(1.89)$
FedFomo [ 57 ]	$41.87(2.13)$	$76.21(0.98)$	$15.10(3.82)$	$25.69(2.38)$

Table 2: Feature shift test accuracy after

R=1

and

R=3

communication rounds. LSS consistently outperforms other methods on both datasets across under feature shift settings.

Panel of Figure 3 — Figure 3: Convergence comparison of our proposed LSS with FedAvg. LSS achieves high accuracy much earlier (around 6 to 8 rounds) than FedAvg, which takes hundreds of communication rounds.

Panel of Figure 4 — Figure 4: Evaluation on ViT fine-tuned with LoRA (Digit5 dataset).

Panel of Figure 5 — Figure 5: Ablation on the affinity & diversity.

Panel of Figure 6 — Figure 6: Ablation studies on the impact of the number of averaged models on communication efficiency and performance variance. We evaluated the influence of varied model quantities on global and averaged local model performance, as well as generalization on the worst client.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Notation	Description
Data and local training
$M,m,n$	Number of clients; parameter dimension; samples per client.
$\mathcal{X},\mathcal{Y}$	Input and label spaces; $(x,y)$ is a labeled example, also denoted $\xi$ in the empirical loss.
$\mathcal{D}_{i},\mathcal{D}$	Client $i$ ’s training set; collection $\{\mathcal{D}_{i}\}_{i=1}^{M}$ .
$\ell_{i},\mathcal{L}_{i},\mathcal{L}$	Per-example loss; client empirical objective; global objective.
$p_{i}$	Client aggregation weight, with $\sum_{i}p_{i}=1$ .

Appendix

Table 3: Important notations used in the paper.

Figure 7: Comparison on model ensemble, model soups, and superior soups.

Method	Accuracy ( $\tau=1$ ) $\uparrow$	Accuracy ( $\tau=4$ ) $\uparrow$	Accuracy ( $\tau=8$ ) $\uparrow$	Accuracy ( $\tau=12$ ) $\uparrow$	Accuracy ( $\tau=16$ ) $\uparrow$
FedAvg [ 35 ]	$34.03(2.84)$	$49.08(1.51)$	$\textbf{58.34}(0.86)$	$55.76(0.82)$	$53.21(0.80)$

Appendix

Table 4: FedAvg with different local steps: Label shift test accuracy after

R=1

communication rounds (CIFAR-10 with 5 Clients).

Costs	FedAvg [ 35 ]	SWA [ 21 ]	Soups [ 51 ]	LSS (2 models)	LSS (4 models)
MACs (G)	1.82	1.82	1.82	2.73	4.55
Train Time Per Epoch (s)	2.66	2.73	2.66	12.27	20.43
Train Time Per Round (s)	21.28	433.31	683.52	100.98	169.77

Appendix

Table 5: Computational costs of different methods (ResNet-18). Peak training memory is not reported in this table.

	CIFAR-10 ( $4/255$ )		CIFAR-10 ( $8/255$ )
Method	Accuracy drop ( $R=1$ ) $\downarrow$	Accuracy drop ( $R=3$ ) $\downarrow$	Accuracy drop ( $R=1$ ) $\downarrow$	Accuracy drop ( $R=3$ ) $\downarrow$
FedAvg [ 35 ]	$1.30$	$1.17$	$3.06$	$2.93$
LSS	$0.89$	$0.76$	$2.37$	$1.85$

Appendix

Table 6: Accuracy drop under random input noise at the displayed amplitudes, after

R=1

and

R=3

communication rounds on CIFAR-10.

	FedAvg $\downarrow$	LSS (2 models) $\downarrow$	LSS (3 models) $\downarrow$	LSS (4 models) $\downarrow$
Hessian Eigenvalue	193.18	147.20	136.67	119.14

Appendix

Table 7: Median estimated minibatch Hessian curvature under the reported power-iteration protocol. This is a local empirical proxy.

	CIFAR-10 (5 Clients)		CIFAR-10 (50 Clients)
Method	Accuracy ( $R=1$ ) $\uparrow$	Accuracy ( $R=3$ ) $\uparrow$	Accuracy ( $R=1$ ) $\uparrow$	Accuracy ( $R=3$ ) $\uparrow$
FedAvg [ 35 ]	$58.34(0.86)$	$66.74(0.76)$	$49.32(0.93)$	$68.39(0.61)$
LSS	$\textbf{65.96}(1.50)$	$\textbf{75.16}(1.07)$	$\textbf{56.72}(0.53)$	$\textbf{73.32}(0.46)$

Appendix

Table 8: Different client numbers (5 Clients and 50 Clients): Label shift test accuracy after

R=1

and

R=3

communication rounds.

	CIFAR-10 (ResNet-18)		CIFAR-10 (ViT Base)
Method	Accuracy ( $R=1$ ) $\uparrow$	Accuracy ( $R=3$ ) $\uparrow$	Accuracy ( $R=1$ ) $\uparrow$	Accuracy ( $R=3$ ) $\uparrow$
FedAvg [ 35 ]	$58.34(0.86)$	$66.74(0.76)$	$60.35(0.82)$	$69.38(0.51)$
LSS	$\textbf{65.96}(1.50)$	$\textbf{75.16}(1.07)$	$\textbf{67.48}(0.70)$	$\textbf{76.81}(0.47)$

Appendix

Table 9: Different Network Architecture (ResNet-18 and ViT): Label shift test accuracy after

R=1

and

R=3

communication rounds.

	CIFAR-10 ( $\alpha=1.0$ )		CIFAR-10 ( $\alpha=0.1$ )
Method	Accuracy ( $R=1$ ) $\uparrow$	Accuracy ( $R=3$ ) $\uparrow$	Accuracy ( $R=1$ ) $\uparrow$	Accuracy ( $R=3$ ) $\uparrow$
FedAvg [ 35 ]	$58.34(0.86)$	$66.74(0.76)$	$18.30(2.25)$	$45.85(1.24)$
LSS	$\textbf{65.96}(1.50)$	$\textbf{75.16}(1.07)$	$\textbf{26.70}(1.62)$	$\textbf{50.02}(0.82)$

Appendix

Table 10: Different Non-IID level (Dirichlet distribution

\alpha=1.0

and

\alpha=0.1

): Label shift test accuracy after

R=1

and

R=3

communication rounds.

Figure 8: FedAya Evaluation Comparison with FedAvg and LSS. Our method, LSS, when applied to large language models for instruction tuning, achieves higher scores than the common FedAvg. This suggests that LSS is a promising approach for improving performance and convergence in federated learning settings for large language models, in addition to its success in image classification. Exploring the use of our method in a diverse set of complex LLM tasks is an interesting direction for future research.

	CIFAR-10 (Pre-trained)		CIFAR-10 (Random)
Method	Accuracy ( $R=1$ ) $\uparrow$	Accuracy ( $R=3$ ) $\uparrow$	Accuracy ( $R=1$ ) $\uparrow$	Accuracy ( $R=3$ ) $\uparrow$
FedAvg [ 35 ]	$58.34(0.86)$	$66.74(0.76)$	$14.83(2.03)$	$25.42(0.71)$
LSS	$\textbf{65.96}(1.50)$	$\textbf{75.16}(1.07)$	$\textbf{30.64}(1.73)$	$\textbf{37.86}(1.33)$

Appendix

Table 11: Different model initialization (Pre-trained v.s. Random): Label shift test accuracy after

R=1

and

R=3

communication rounds. Result: It shows that our method still maintains a significant advantage with random initialization, but it does not achieve the near-optimal performance seen with pre-trained initialization.

Panel of Figure 9 — Figure 9: Convergence comparison of our proposed LSS with FedProx. LSS also achieves high accuracy much earlier (around 6 to 8 rounds) than FedProx, which takes hundreds of communication rounds.

Explore similar work

FedPLT: Scalable, Resource-Efficient, and Heterogeneity-Aware Federated Learning via Partial Layer Training
May 4, 2026Ahmad Dabaja, Rachid El-AzouziHeterogeneous Federated Learning Federated Learning
Laboratoire Informatique d’Avignon (LIA), Avignon University, France
Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients
Jul 31, 2026Shengkun Zhu, Jinshan Zeng, Zhihua Allen-Zhao +5Fedavg Parameter-Efficient Fine-Tuning Methods
The Hong Kong Polytechnic University · OceanBase, Ant Group · School of Management, Xi’an Jiaotong University +3
Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning
May 27, 2026Yuting Ma, Lechao Cheng, Xiaohua XuFederated Learning Vision-Language Model Adaptation
School of Computer Science and Technology, University of Science and Technology of China, Hefei, Anhui, China · School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, Anhui, China.

May 4, 2026, cs.DCJ/K move · Enter open · S save

FedPLT: Scalable, Resource-Efficient, and Heterogeneity-Aware Federated Learning via Partial Layer Training

Ahmad Dabaja, Rachid El-Azouzi

Laboratoire Informatique d’Avignon (LIA), Avignon University, France

Heterogeneous Federated Learning Federated Learning Heterogeneity Parameter-Efficient Fine-Tuning Methods Resource-Efficient

Open paper AskPDF arXiv

Federated Learning (FL) has gained significant attention in distributed machine learning by enabling collaborative model training across decentralized system while preserving data privacy. Although extensive research has addressed statistical data heterogeneity, FL still faces several challenges, including high communication and computation overheads and severe device heterogeneity, which require further investigation. Prior work has addressed these issues through sub-model training and partial parameter training. However, such methods often suffer from inconsistent parameter distributions across clients, inaccurate global loss estimation, and increased bias and variance. Guided by our empirical analysis, we propose FedPLT (Federated Learning with Partial Layer Training), an innovative and structured partial parameter training approach that exhibits training behavior similar to full model training while assigning client-specific portions of the model according to their communication and computational capabilities. In addition, we evaluate the performance of FedPLT when combined with optimal client sampling under communication constraints. We show that this integration improves FL performance by reducing sampling variance under the same communication budget. Through extensive experiments, we demonstrate that FedPLT achieves performance comparable to, or even surpassing, that of full-model training (i.e., FedAvg), while requiring significantly fewer trainable parameters per client. Moreover, FedPLT outperforms existing methods in highly heterogeneous environments, effectively adapts to client resource constraints, and reduces the number of straggling clients. In particular, FedPLT reduces the number of trainable parameters by 71%-82% while achieving performance on par with full-model training.

Abstract

Explore similar work

FedPLT: Scalable, Resource-Efficient, and Heterogeneity-Aware Federated Learning via Partial Layer Training

Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients

Decoupled Training with Local Reinforcement Fine-Tuning in Federated Learning