Communication-Efficient Personalized Adaptation via Federated-Local Model Merging

Yinan Zou; Md Kamran Chowdhury Shisher; Christopher G. Brinton; Vishrant Tripathi

cs.LGFeb 20, 2026

Communication-Efficient Personalized Adaptation via Federated-Local Model Merging

Authors: Yinan Zou, Md Kamran Chowdhury Shisher, Christopher G. Brinton, Vishrant Tripathi

Organizations: Elmore Family School of Electrical and Computer Engineering, Purdue University.

Abstract

Parameter-efficient fine-tuning methods, such as LoRA, offer a practical way to adapt large vision and language models to client tasks. However, this becomes particularly challenging under task-level heterogeneity in federated deployments. In this regime, personalization requires balancing general knowledge with personalized knowledge, yet existing approaches largely rely on heuristic mixing rules and lack theoretical justification. Moreover, prior model merging approaches are also computation and communication intensive, making the process inefficient in federated settings. In this work, we propose Potara, a principled framework for federated personalization that constructs a personalized model for each client by merging two complementary models: (i) a federated model capturing general knowledge, and (ii) a local model capturing personalized knowledge. Through the construct of linear mode connectivity, we show that the expected task loss admits a variance trace upper bound, whose minimization yields closed-form optimal mixing weights that guarantee a tighter bound for the merged model than for either the federated or local model alone. Experiments on vision and language benchmarks show that Potara consistently improves personalization while reducing communication, leading to a strong performance-communication trade-off.

Figures & tables

\mathbf{B}^{(i),t+1}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{B}^{(i),t},\mathbf{A}^{(i),t+1}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{A}^{(i),t}.

Algorithm 1 FedIT

Figure 1: Illustration of Potara . We train a FedIT model via FL and train local fine-tuning models. We obtain the final personalized model by weighted merging these FedIT and local models.

Method	Communication Overhead
FedIT	$\mathcal{O}(r(m+n)T)$
FedSA	$\mathcal{O}(rnT)$
FFA-LoRA	$\mathcal{O}(rmT)$
FedALT	$\mathcal{O}(r(m+n)T)$
FedDPA	$\mathcal{O}(r(m+n)T)$
Potara	$\mathcal{O}(r(m+n)T_{\text{FedIT}})$

Table 1: Communication Overhead Comparison.

Table 4

Panel of Figure 2 — Figure 2: Performance-communication trade-off across different benchmarks. Potara ( $t$ ) denotes our method where the FedIT model obtained at communication round $t$ is used for merging. Potara achieves strong performance with substantially lower communication overhead than baselines.

Table 6

Panel of Figure 3 — Figure 3: Ablation studies demonstrating the performance of Potara .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

FLAN benchmark (Qwen3-4B-Instruct model)
Methods	Closed-book QA	Reading Comp.	Coreference	Paraphrase	Commonsense	NLI	Reading	Sentiment	Avg.
Local	69.53 $\pm$ 0.75	69.38 $\pm$ 2.81	88.01 $\pm$ 0.70	80.00 $\pm$ 3.56	87.62 $\pm$ 0.78	85.78 $\pm$ 2.18	73.87 $\pm$ 0.66	76.16 $\pm$ 1.69	78.79 $\pm$ 0.99
FFA-LoRA	71.00 $\pm$ 0.66	70.54 $\pm$ 0.66	77.18 $\pm$ 1.34	79.89 $\pm$ 0.57	85.65 $\pm$ 1.18	86.33 $\pm$ 1.18	73.57 $\pm$ 0.39	77.06 $\pm$ 0.74	77.65 $\pm$ 0.70
FedSA	69.54 $\pm$ 0.95	69.54 $\pm$ 1.09	87.78 $\pm$ 2.61	81.89 $\pm$ 0.87	84.05 $\pm$ 1.16	86.00 $\pm$ 1.19	72.72 $\pm$ 0.65	76.47 $\pm$ 0.98	78.50 $\pm$ 0.25
FedIT	69.96 $\pm$ 0.57	71.37 $\pm$ 0.63	83.19 $\pm$ 0.51	80.44 $\pm$ 1.40	85.37 $\pm$ 0.96	86.33 $\pm$ 1.25	73.78 $\pm$ 0.85	76.68 $\pm$ 0.56	78.39 $\pm$ 0.63
FedDPA	70.03 $\pm$ 0.19	70.57 $\pm$ 0.34	81.89 $\pm$ 1.37	81.44 $\pm$ 0.79	86.93 $\pm$ 1.71	87.55 $\pm$ 0.88	74.01 $\pm$ 0.02	76.45 $\pm$ 0.88	78.61 $\pm$ 0.29

Appendix

Table 6: ROUGE-1 performance on FLAN benchmark (Qwen3-4B-Instruct model).

Task	Dataset	Description
Closed-book QA	arc_challenge	Answer multiple-choice questions.
Reading Comp. w/ Commonsense	cosmos_qa	Reading comprehension with commonsense.
Coreference Resolution	definite_pronoun_resolution	Resolve ambiguous pronouns.
Paraphrase Detection	glue_qqp	Identify semantic equivalence.
Commonsense Reasoning	hellaswag	Choose the most plausible continuation.
Natural Language Inference	mnli	Infer relations between sentence pairs.

Appendix

Table 7: FLAN benchmark.

Client	Local	FFA-LoRA	FedSA	FedIT	FedDPA	Fisher Merging	Potara
Client 1	63.53 $\pm$ 2.22	44.37 $\pm$ 1.97	61.28 $\pm$ 2.27	59.52 $\pm$ 0.73	67.41 $\pm$ 2.92	62.00 $\pm$ 2.46	62.39 $\pm$ 3.91
Client 2	62.28 $\pm$ 5.39	49.25 $\pm$ 6.36	61.16 $\pm$ 5.72	61.35 $\pm$ 6.12	65.95 $\pm$ 6.74	61.39 $\pm$ 9.09	70.18 $\pm$ 5.12
Client 3	60.40 $\pm$ 0.23	45.24 $\pm$ 1.43	58.25 $\pm$ 2.07	64.25 $\pm$ 2.42	65.39 $\pm$ 3.34	59.64 $\pm$ 2.42	70.09 $\pm$ 1.34
Client 4	58.45 $\pm$ 2.62	39.74 $\pm$ 0.34	57.29 $\pm$ 1.19	56.98 $\pm$ 2.56	59.52 $\pm$ 0.55	62.54 $\pm$ 0.71	63.63 $\pm$ 1.26
Client 5	62.07 $\pm$ 2.18	50.88 $\pm$ 4.72	62.73 $\pm$ 2.31	66.00 $\pm$ 3.74	68.64 $\pm$ 1.89	66.22 $\pm$ 0.01	67.94 $\pm$ 4.36
Client 6	64.14 $\pm$ 0.67	47.09 $\pm$ 2.28	61.07 $\pm$ 0.85	65.06 $\pm$ 2.94	65.34 $\pm$ 2.65	59.56 $\pm$ 0.25	67.00 $\pm$ 0.09

Appendix

Table 8: Testing accuracy on personalized CIFAR-100 benchmark with 20 clients.

Client	Local	FFA-LoRA	FedSA	FedIT	FedALT	FedDPA	Potara
Client 1 (ARC-c)	71.80 $\pm$ 0.20	73.00 $\pm$ 0.80	70.60 $\pm$ 0.40	73.60 $\pm$ 0.00	73.50 $\pm$ 0.70	74.50 $\pm$ 0.90	74.80 $\pm$ 0.20
Client 2 (ARC-c)	70.50 $\pm$ 0.50	73.10 $\pm$ 0.30	72.20 $\pm$ 1.20	73.60 $\pm$ 0.00	72.40 $\pm$ 0.00	74.50 $\pm$ 0.10	75.20 $\pm$ 1.00
Client 3 (ARC-e)	85.80 $\pm$ 0.20	87.50 $\pm$ 0.50	86.20 $\pm$ 1.20	87.30 $\pm$ 0.70	85.90 $\pm$ 0.70	86.90 $\pm$ 0.30	88.50 $\pm$ 0.30
Client 4 (ARC-e)	86.70 $\pm$ 0.50	87.10 $\pm$ 0.50	85.90 $\pm$ 1.30	87.30 $\pm$ 0.70	85.00 $\pm$ 0.20	87.40 $\pm$ 0.60	87.90 $\pm$ 0.30
Client 5 (BoolQ)	65.20 $\pm$ 3.00	67.00 $\pm$ 0.60	61.60 $\pm$ 5.60	64.30 $\pm$ 3.50	66.70 $\pm$ 0.10	66.90 $\pm$ 0.10	65.60 $\pm$ 0.60
Client 6 (BoolQ)	67.90 $\pm$ 1.50	63.50 $\pm$ 1.70	64.90 $\pm$ 0.70	64.30 $\pm$ 3.50	65.50 $\pm$ 0.30	64.20 $\pm$ 3.00	66.70 $\pm$ 1.70

Appendix

Table 9: Testing accuracy on commonsense reasoning benchmark with 16 clients.

Panel of Figure 4 — Figure 4: Client performance under various mixing weights. For each client, we construct a personalized model by merging a FedIT checkpoint at round 50,150, or 300 with a fixed Local model at round 300, and sweep the local mixing weight $\lambda_{\text{FedIT}}$ (x-axis). Curves report the resulting client accuracy. The red dashed line marks the mixing weight calculated by our method for each client.

Task	Pre-trained Model	Mixing Weights Computation Time	FedIT + Local Training Time
CIFAR-100	ViT-B/16	52.06s	0.83h
Commonsense Reasoning	LLaMA-3.2-3B	0.46h	5.92h
FLAN	Qwen3-4B-Instruct-2507	1.18h	6.83h

Appendix

Table 10: Computation time and training time.

Hyperparameter	ViT-B/16 & CIFAR-100	LLaMA-3.2-3B-Instruct & commonsense reasoning	Qwen3-4B-Instruct-2507 & FLAN
Batch size	10	4	4
Learning rate	0.001	0.0001	0.0001
LoRA dropout rate	0.1	0.05	0.05
Local iteration number	5	4	4
Target module	[“query”, “value”]	[“q_proj”, “k_proj”, “v_proj”]	[“q_proj”, “k_proj”, “v_proj”]
Number of data samples to compute mixing weights	30	30	30

Appendix

Table 11: Hyperparameter setting.

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 943 papers

FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning
Sep 29, 2026Mengjun Yi, Huaian Gu, Yinghao Ai +2Low-Rank Adaptation Federated Fine-Tuning
State Key Laboratory for Novel Software Technology · School of Artificial Intelligence · School of Computer Science +1
Task-Centric Personalized Federated Fine-Tuning of Language Models
Mar 30, 2026Gabriel U. Talasso, Meghdad Kurmanji, Allan M. de Souza +2Federated Learning Federated Fine-Tuning
Universidade Estadual de Campinas · University of Cambridge · Flower Labs
Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models
Jun 4, 2026Sunny Gupta, Shambhavi Shanker, Amit SethiFederated Learning Aggregation Non-IID Federated Learning
Indian Institute of Technology, Bombay

Sep 29, 2026J/K move · Enter open · S save

FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning

Mengjun Yi, Huaian Gu, Yinghao Ai, Furao Shen+1

State Key Laboratory for Novel Software Technology · School of Artificial Intelligence · School of Computer Science+1

Low-Rank Adaptation Federated Fine-Tuning Federated Learning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Federated parameter-efficient fine-tuning enables clients to adapt pre-trained models without sharing raw data or communicating the full model, but statistical heterogeneity makes a single global adapter insufficient for personalized prediction. Existing personalized methods typically use the same low-rank structure for both shared and private adaptation, overlooking their distinct requirements for aggregation and personalization. We propose FedLAFP, a role-aware framework that couples a compact, globally aggregated LoRA branch with a client-private, full-rank-capable RandLoRA branch. The shared branch provides an efficient interface for transferring common knowledge, whereas the private branch combines fixed random low-rank bases with learned scaling coefficients to provide expressive client-specific adaptation without additional communication. Client- and layer-specific mixing coefficients jointly fuse the two branches, and only the shared LoRA parameters are exchanged. A controlled linear study supports this role assignment: LoRA yields more aligned client updates and lower aggregation error, while RandLoRA more accurately recovers client-specific residuals. Experiments across four visual recognition benchmarks show that FedLAFP consistently outperforms local-only and federated LoRA baselines, achieving an average personalized accuracy of

86.93\%

and exceeding the best baseline average by

1.30

percentage points.

Abstract

Federated Learning

Explore similar work

FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning

Task-Centric Personalized Federated Fine-Tuning of Language Models

Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models