Stabilized Fine-Tuning with LoRA in Federated Learning: Mitigating the Side Effect of Client Size and Rank via the Scaling Factor

Jiayu Huang; Xiaohu Wu; Tiantian He; Qicheng Lao

cs.LGMar 9, 2026

Stabilized Fine-Tuning with LoRA in Federated Learning: Mitigating the Side Effect of Client Size and Rank via the Scaling Factor

Authors: Jiayu Huang, Xiaohu Wu, Tiantian He, Qicheng Lao

Organizations: Beijing University of Posts and Telecommunications, Beijing, China · Agency for Science, Technology and Research, Singapore, Singapore

Abstract

Large Language Models (LLMs) are pivotal in natural language processing. The impracticality of full fine-tuning has prompted Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA), optimizing low-rank matrices A and B. In distributed scenarios where privacy constraints necessitate Federated Learning (FL), however, the integration of LoRA is often unstable. Specifically, we identify that aggregating updates from multiple clients introduces statistical variance that scales with the client count, causing gradient collapse when using high-rank adapters. Existing scaling factor candidates, such as the one used by Rank-Stabilized LoRA, ignore the interaction caused by the aggregation process. To bridge this gap, this paper introduces Stabilized Federated LoRA (SFed-LoRA), a framework that theoretically characterizes the interaction between adapter rank and federated aggregation. We derive an optimal scaling factor designed to effectively mitigate the aggregation error accumulating across N clients. By correcting the scaling mismatch inherent in previous approaches, SFed-LoRA restores the efficacy of high-rank adaptation without altering the original model architecture or increasing inference latency. Extensive experiments in diverse tasks, model architectures, and heterogeneous data distributions are conducted to validate our results. We demonstrate that SFed-LoRA prevents high-rank collapse, and achieves significantly improved stability and faster convergence compared with state-of-the-art baselines for high-rank adaptation.

Figures & tables

Figure 1: The framework of SFed-LoRA. It adopts a split aggregation strategy where clients upload only matrix $A$ while maintaining $B$ locally to protect privacy. A novel scaling factor $\gamma_{z}=\alpha\sqrt{N/r}$ is integrated into the local computation (as shown in the equation) to counteract aggregation interference, ensuring stable training performance across varying client numbers.

Figure 2: Convergence of Perplexity (PPL) on the Alpaca dataset using the LLaMA2-7B model under an IID federated learning setting. The subplots compare the training trajectories of four methods across ranks $r\in\{4,8,32,128,512\}$ : (a) RoLoRA (purple), (b) FedSA-LoRA (copper), (c) FedSA-rsLoRA (blue), and (d) SFed-LoRA (green). Darker curves correspond to higher ranks. The figure displays the evolution of validation perplexity over 100 communication rounds.

Figure 3: Evolution of average parameter gradient norms on the Alpaca dataset (IID). The subplots display the training trajectories for (a) RoLoRA (purple), (b) FedSA-LoRA (copper), (c) FedSA-rsLoRA (blue), and (d) SFed-LoRA (green) across ranks $r\in\{4,8,32,128,512\}$ . Darker curves correspond to higher ranks.

Figure 4: Comparative analysis of perplexity with a fixed rank $r=512$ using the LLaMA2-7B-hf model on the Alpaca dataset in an IID federated learning setting. The subplots correspond to varying client counts: (a) $N=5$ , (b) $N=10$ , (c) $N=15$ , and (d) $N=20$ . Within each plot, the curves represent RoLoRA (purple), FedSA-LoRA (orange), FedSA-rsLoRA (blue), and SFed-LoRA (green).

Method	GSM8K Accuracy (%)
	Rank = 4	Rank = 8	Rank = 32
RoLoRA	$14.16\pm 1.18$	$15.84\pm 1.18$	$15.00\pm 0.00$
FedSA-LoRA	$13.33\pm 0.00$	$13.89\pm 0.96$	$15.56\pm 0.96$
FedSA-rsLoRA	$14.44\pm 0.96$	$16.67\pm 3.34$	$16.11\pm 0.96$
SFed-LoRA	14.44 $\pm$ 1.93	17.08 $\pm$ 2.08	16.66 $\pm$ 1.67
	Rank = 128	Rank = 512

Table 1: Accuracy (%) comparison on the GSM8K benchmark across ranks. Results are reported as mean

\pm

standard deviation.

Method	MNLI-m Accuracy (%)
	Rank = 4	Rank = 8	Rank = 32
RoLoRA	$57.72\pm 1.51$	$69.02\pm 1.37$	$69.26\pm 0.89$
FedSA-LoRA	$73.89\pm 10.97$	$81.31\pm 5.16$	$81.88\pm 3.80$
FedSA-rsLoRA	$81.24\pm 4.08$	$85.83\pm 1.39$	$87.23\pm 1.65$
SFed-LoRA	83.95 $\pm$ 2.59	86.51 $\pm$ 1.34	87.64 $\pm$ 1.83
	Rank = 128	Rank = 512

Table 2: Accuracy (%) comparison on the GLUE benchmark (MNLI-m). Results are reported as mean

\pm

standard deviation.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Evolution of average parameter gradient norms on the GSM8K dataset using the LLaMA2-7B model. The subplots display the training trajectories for (a) RoLoRA, (b) FedSA-LoRA, (c) FedSA-rsLoRA, and (d) SFed-LoRA across ranks $r\in\{4,8,32,128,512\}$ . Darker curves correspond to higher ranks.

Figure 6: Evolution of average parameter gradient norms on the GLUE benchmark using the RoBERTa-large model with the AdamW optimizer. The subplots compare (a) RoLoRA, (b) FedSA-LoRA, (c) FedSA-rsLoRA, and (d) SFed-LoRA across ranks $r\in\{4,8,32,128,512\}$ . Darker curves correspond to higher ranks.

Figure 7: Perplexity (top row) and gradient norms (bottom row) of LLaMA-2-7B fine-tuned on the GSM8K dataset. Adapters are applied to all attention modules ( $\texttt{W}_{\text{q}},\texttt{W}_{\text{k}},\texttt{W}_{\text{v}},\texttt{W}_{\text{o}}$ ) across ranks $r\in\{4,8,32,128,512\}$ . The curves represent FedSA-LoRA (copper), FedSA-rsLoRA (blue), and SFed-LoRA (green), with darker shades indicating higher ranks.

Figure 8: Perplexity trajectories of LLaMA-2-7B on the GSM8K dataset with a fixed high rank $r=2048$ . The plot compares the proposed SFed-LoRA (green) against FedSA-LoRA (red), FedSA-rsLoRA (blue), and alternative heuristics $\gamma_{za}$ (brown) and $\gamma_{zb}$ (yellow). Note the broken y-axis used to visualize the extreme initial perplexity of $\gamma_{zb}$ .

Figure 9: Mean (top row) and variance (bottom row) of post-adapter pre-LayerNorm activations for LLaMA-2-7B on the GSM8K dataset. The plots compare FedSA-LoRA (red), FedSA-rsLoRA (blue), and SFed-LoRA (green) across ranks $r\in\{32,128,512,2048\}$ , with darker shades indicating higher ranks.

Federated Fine-Tuning

Federated fine-tuning adapts pretrained models across decentralized data holders without pooling their private data.

See all 115 papers

FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA
Aug 15, 2026Juseok Jeon, Ramy E. Ali, Doyun Kwon +3Federated Fine-Tuning Federated Learning
DGIST Daegu, Republic of Korea · Samsung San Diego, USA
CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
Sep 27, 2026Yanan Ma, Qiyuan Chen, Zihan Fang +2Federated Fine-Tuning Federated Learning
City University of Hong Kong · The University of Hong Kong
FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning
Mar 9, 2026Peishen Yan, Yang Hua, Hao Wang +4Federated Fine-Tuning Federated Learning Aggregation
Shanghai Jiao Tong University · Queen’s University Belfast · Stevens Institute of Technology +2

Aug 15, 2026J/K move · Enter open · S save

FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA

Juseok Jeon, Ramy E. Ali, Doyun Kwon, Myungbeom Her+2

DGIST Daegu, Republic of Korea · Samsung San Diego, USA

Federated Fine-Tuning Federated Learning Federated Learning Aggregation Low-Rank Adaptation Heterogeneous Federated Learning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorized parameterization creates a tension between accurate aggregation of local updates and continuity of locally optimized factors. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch from newly reconstructed factors. We propose FedPA-LoRA, a product-aligned federated LoRA framework that jointly addresses these limitations and provably converges under both homogeneous and heterogeneous client ranks. Each client preserves its local factors across communication rounds and aligns its product toward a rank-specific global reference, maintaining local optimization continuity while promoting global consistency under data heterogeneity. The server aggregates heterogeneous-rank updates in the common product space and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. This design supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show that FedPA-LoRA consistently outperforms representative baselines across varying levels of data heterogeneity and homogeneous- and heterogeneous-rank settings, with up to a

6.82

percentage-point improvement in average GLUE accuracy under heterogeneous client ranks.

Abstract

Federated Fine-Tuning

Explore similar work

FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning