Large Language Models (LLMs) are pivotal in natural language processing. The impracticality of full fine-tuning has prompted Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA), optimizing low-rank matrices A and B. In distributed scenarios where privacy constraints necessitate Federated Learning (FL), however, the integration of LoRA is often unstable. Specifically, we identify that aggregating updates from multiple clients introduces statistical variance that scales with the client count, causing gradient collapse when using high-rank adapters. Existing scaling factor candidates, such as the one used by Rank-Stabilized LoRA, ignore the interaction caused by the aggregation process. To bridge this gap, this paper introduces Stabilized Federated LoRA (SFed-LoRA), a framework that theoretically characterizes the interaction between adapter rank and federated aggregation. We derive an optimal scaling factor designed to effectively mitigate the aggregation error accumulating across N clients. By correcting the scaling mismatch inherent in previous approaches, SFed-LoRA restores the efficacy of high-rank adaptation without altering the original model architecture or increasing inference latency. Extensive experiments in diverse tasks, model architectures, and heterogeneous data distributions are conducted to validate our results. We demonstrate that SFed-LoRA prevents high-rank collapse, and achieves significantly improved stability and faster convergence compared with state-of-the-art baselines for high-rank adaptation.
Figures & tables
Figure 1: The framework of SFed-LoRA. It adopts a split aggregation strategy where clients upload only matrix A while maintaining B locally to protect privacy. A novel scaling factor γz=αN/r is integrated into the local computation (as shown in the equation) to counteract aggregation interference, ensuring stable training performance across varying client numbers.
Figure 2: Convergence of Perplexity (PPL) on the Alpaca dataset using the LLaMA2-7B model under an IID federated learning setting. The subplots compare the training trajectories of four methods across ranks r∈{4,8,32,128,512} : (a) RoLoRA (purple), (b) FedSA-LoRA (copper), (c) FedSA-rsLoRA (blue), and (d) SFed-LoRA (green). Darker curves correspond to higher ranks. The figure displays the evolution of validation perplexity over 100 communication rounds.
Figure 3: Evolution of average parameter gradient norms on the Alpaca dataset (IID). The subplots display the training trajectories for (a) RoLoRA (purple), (b) FedSA-LoRA (copper), (c) FedSA-rsLoRA (blue), and (d) SFed-LoRA (green) across ranks r∈{4,8,32,128,512} . Darker curves correspond to higher ranks.
Figure 4: Comparative analysis of perplexity with a fixed rank r=512 using the LLaMA2-7B-hf model on the Alpaca dataset in an IID federated learning setting. The subplots correspond to varying client counts: (a) N=5 , (b) N=10 , (c) N=15 , and (d) N=20 . Within each plot, the curves represent RoLoRA (purple), FedSA-LoRA (orange), FedSA-rsLoRA (blue), and SFed-LoRA (green).
Method
GSM8K Accuracy (%)
Rank = 4
Rank = 8
Rank = 32
RoLoRA
14.16±1.18
15.84±1.18
15.00±0.00
FedSA-LoRA
13.33±0.00
13.89±0.96
15.56±0.96
FedSA-rsLoRA
14.44±0.96
16.67±3.34
16.11±0.96
SFed-LoRA
14.44 ± 1.93
17.08 ± 2.08
16.66 ± 1.67
Rank = 128
Rank = 512
Table 1: Accuracy (%) comparison on the GSM8K benchmark across ranks. Results are reported as mean ± standard deviation.
Method
MNLI-m Accuracy (%)
Rank = 4
Rank = 8
Rank = 32
RoLoRA
57.72±1.51
69.02±1.37
69.26±0.89
FedSA-LoRA
73.89±10.97
81.31±5.16
81.88±3.80
FedSA-rsLoRA
81.24±4.08
85.83±1.39
87.23±1.65
SFed-LoRA
83.95 ± 2.59
86.51 ± 1.34
87.64 ± 1.83
Rank = 128
Rank = 512
Table 2: Accuracy (%) comparison on the GLUE benchmark (MNLI-m). Results are reported as mean ± standard deviation.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Evolution of average parameter gradient norms on the GSM8K dataset using the LLaMA2-7B model. The subplots display the training trajectories for (a) RoLoRA, (b) FedSA-LoRA, (c) FedSA-rsLoRA, and (d) SFed-LoRA across ranks r∈{4,8,32,128,512} . Darker curves correspond to higher ranks.
Figure 6: Evolution of average parameter gradient norms on the GLUE benchmark using the RoBERTa-large model with the AdamW optimizer. The subplots compare (a) RoLoRA, (b) FedSA-LoRA, (c) FedSA-rsLoRA, and (d) SFed-LoRA across ranks r∈{4,8,32,128,512} . Darker curves correspond to higher ranks.
Figure 7: Perplexity (top row) and gradient norms (bottom row) of LLaMA-2-7B fine-tuned on the GSM8K dataset. Adapters are applied to all attention modules ( Wq,Wk,Wv,Wo ) across ranks r∈{4,8,32,128,512} . The curves represent FedSA-LoRA (copper), FedSA-rsLoRA (blue), and SFed-LoRA (green), with darker shades indicating higher ranks.
Figure 8: Perplexity trajectories of LLaMA-2-7B on the GSM8K dataset with a fixed high rank r=2048 . The plot compares the proposed SFed-LoRA (green) against FedSA-LoRA (red), FedSA-rsLoRA (blue), and alternative heuristics γza (brown) and γzb (yellow). Note the broken y-axis used to visualize the extreme initial perplexity of γzb .
Figure 9: Mean (top row) and variance (bottom row) of post-adapter pre-LayerNorm activations for LLaMA-2-7B on the GSM8K dataset. The plots compare FedSA-LoRA (red), FedSA-rsLoRA (blue), and SFed-LoRA (green) across ranks r∈{32,128,512,2048} , with darker shades indicating higher ranks.
Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorized parameterization creates a tension between accurate aggregation of local updates and continuity of locally optimized factors. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch from newly reconstructed factors. We propose FedPA-LoRA, a product-aligned federated LoRA framework that jointly addresses these limitations and provably converges under both homogeneous and heterogeneous client ranks. Each client preserves its local factors across communication rounds and aligns its product toward a rank-specific global reference, maintaining local optimization continuity while promoting global consistency under data heterogeneity. The server aggregates heterogeneous-rank updates in the common product space and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. This design supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show that FedPA-LoRA consistently outperforms representative baselines across varying levels of data heterogeneity and homogeneous- and heterogeneous-rank settings, with up to a 6.82 percentage-point improvement in average GLUE accuracy under heterogeneous client ranks.
Juseok Jeon, Ramy E. Ali, Doyun Kwon +3
DGIST Daegu, Republic of Korea · Samsung San Diego, USA
Federated low-rank adaptation (LoRA) enables collaborative fine-tuning of large language models without centralizing private client data. Its factorized update, however, creates a structural mismatch in federated averaging: averaging the two LoRA factors separately does not equal averaging their products. Existing exact methods resolve this issue mainly by freezing an entire factor or alternating factors across rounds, but none can update factors simultaneously without aggregation errors or expanding communication ranks. To address this fundamental problem, we present CFLoRA, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round. By ensuring that columns and rows are complementary across factors, we eliminate bilinear terms in matrix multiplications, making federated aggregation exact. Crucially, our framework also supports clients with heterogeneous rank budgets. Convergence analysis validates CFLoRA achieves O(1/T) convergence rate of the original LoRA objective in homogeneous-rank cases. Extensive experiments with RoBERTa on the GLUE benchmark and with LLaMA-3.2-3B-Instruct on commonsense reasoning tasks demonstrate that CFLoRA achieves superior performance and training efficiency compared to state-of-the-art federated LoRA baselines.
Yanan Ma, Qiyuan Chen, Zihan Fang +2
City University of Hong Kong · The University of Hong Kong
Federated fine-tuning of large language models (LLMs) with low-rank adaptation (LoRA) offers a communication-efficient and privacy-preserving solution for task-specific adaptation. Naive aggregation of LoRA modules introduces noise due to mathematical incorrectness when averaging the downsampling and upsampling matrices independently. However, existing noise-free aggregation strategies inevitably compromise the structural expressiveness of LoRA, limiting its ability to retain client-specific adaptations by either improperly reconstructing the low-rank structure or excluding partially trainable components. We identify this problem as loss of training momentum, where LoRA updates fail to accumulate effectively across rounds, resulting in slower convergence and suboptimal performance. To address this, we propose FedMomentum, a novel framework that enables structured and momentum-preserving LoRA aggregation via singular value decomposition (SVD). Specifically, after aggregating low-rank updates in a mathematically correct manner, FedMomentum applies SVD to extract the dominant components that capture the main update directions. These components are used to reconstruct the LoRA modules with the same rank, while residual components can be retained and later merged into the backbone to preserve semantic information and ensure robustness. Extensive experiments across multiple tasks demonstrate that FedMomentum consistently outperforms prior state-of-the-art methods in convergence speed and final accuracy.
Peishen Yan, Yang Hua, Hao Wang +4
Shanghai Jiao Tong University · Queen’s University Belfast · Stevens Institute of Technology +2