Federated fine-tuning of large language models (LLMs) with low-rank adaptation (LoRA) offers a communication-efficient and privacy-preserving solution for task-specific adaptation. Naive aggregation of LoRA modules introduces noise due to mathematical incorrectness when averaging the downsampling and upsampling matrices independently. However, existing noise-free aggregation strategies inevitably compromise the structural expressiveness of LoRA, limiting its ability to retain client-specific adaptations by either improperly reconstructing the low-rank structure or excluding partially trainable components. We identify this problem as loss of training momentum, where LoRA updates fail to accumulate effectively across rounds, resulting in slower convergence and suboptimal performance. To address this, we propose FedMomentum, a novel framework that enables structured and momentum-preserving LoRA aggregation via singular value decomposition (SVD). Specifically, after aggregating low-rank updates in a mathematically correct manner, FedMomentum applies SVD to extract the dominant components that capture the main update directions. These components are used to reconstruct the LoRA modules with the same rank, while residual components can be retained and later merged into the backbone to preserve semantic information and ensure robustness. Extensive experiments across multiple tasks demonstrate that FedMomentum consistently outperforms prior state-of-the-art methods in convergence speed and final accuracy.
Figures & tables
Method
Communication Efficiency
Aggregation Correctness
Training Momentum
FedIT ( Zhang et al., 2024 )
✓
✗
✗
FLoRA ( Wang et al., 2024 )
✓
✓
✗
FFA‑LoRA ( Sun et al., 2024 )
✓
✓
✗
RoLoRA ( Chen et al., 2025 )
✓
✓
✗
FedEx-LoRA ( Singhal et al., 2025 )
✗
✓
✗
Ours
✓
✓
✓
Table 1: Taxonomy of representative federated LoRA fine-tuning methods. Only our method satisfies all three desired properties: communication efficiency, noise-free aggregation, and training momentum preservation.
Figure 1: Training loss curves of fine-tuning LLaMA2-7B on MetaMathQA with 10 clients under different LoRA update strategies. We compare various centralized fine-tuning and federated fine-tuning strategies under a unified setting: LoRA rank = 16, batch size = 16, local update steps = 10, and 100 iterations. The same experimental settings are adopted in Figure 1 .
Figure 2: (Left ) Normalized singular value spectrum for the first LoRA module of the first training round. The elbow point indicates the effective rank r of the matrix. (Right ) Residual rank statistics across all LoRA modules throughout training rounds. The solid line represents the average residual rank, while the shaded area reflects the range between the maximum and minimum values.
Figure 3: Overview of the SVD-based aggregation process.
Method
GSM8K
MATH
Avg.
Pre-trained
13.87±0.95
2.50±0.22
8.19
FedIT
10.72±0.85
3.84±0.27
7.28
FLoRA
29.06±1.25
3.86±0.27
16.53
FFA-LoRA
25.40±1.20
4.18±0.28
14.91
RoLoRA
27.29±1.23
4.42±0.29
15.86
FedEx-LoRA
26.18±1.14
3.97±0.26
9.15
Table 2: Experiment results on math reasoning. The highest accuracy for each task is highlighted in bold, and the second-best result is underlined. Avg indicates the average result of corresponding metrics.
Figure 4: Training loss for math reasoning task.
Figure 5: Statistical analysis for the aggregated updates of FedMomentum .
Method
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Avg.
Pre-trained
80.34
79.54
50.72
78.37
73.72
79.76
49.32
45.20
67.12
FedIT
82.10
78.35
53.23
76.63
74.30
80.52
50.64
47.70
67.93
FLoRA
76.53
77.15
48.72
73.54
70.72
76.64
43.43
42.10
63.60
FFA-LoRA
82.88
78.67
51.85
76.30
74.27
81.00
49.79
45.20
67.49
RoLoRA
81.89
77.15
48.56
76.12
69.61
78.87
50.68
42.20
65.64
FedEx-LoRA
80.02
77.40
51.90
72.30
72.57
79.55
47.99
47.90
66.20
Table 3: Experiment results on eight commonsense reasoning tasks.
Table 9
Method
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Avg.
Pre-trained
80.34
79.54
50.72
78.37
73.72
79.76
49.32
45.20
67.12
FedMomentum
82.77
79.03
54.48
77.54
75.81
82.16
51.37
49.00
69.02
w/o balance
82.13
78.56
53.15
76.54
73.88
80.37
49.74
46.90
67.67
w/o residual
81.71
77.91
52.64
76.28
74.78
80.93
49.36
48.40
67.75
Table 6: Results of ablation study on commonsense reasoning tasks.
Method
HumanEval
MBPP
Avg.
Pre-trained
12.19
23.40
17.80
FedMomentum
17.07
25.60
21.34
w/o balance
14.63
23.80
19.22
w/o residual
16.46
24.80
20.63
Table 7: Results of ablation study on code generation.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Aggregation Time (s)
Updating Time (s)
FedIT
0.10
0.02
FLoRA
0.05
0.03
FFA-LoRA
0.06
0.02
RoLoRA
0.09
0.03
FedEx-LoRA
0.15
0.03
FedMomentum
0.60
0.03
Appendix
Table 8: Average runtime (in seconds) of different methods.
Figure 6: Comparison of different methods under different LoRA ranks on the math reasoning task.
Figure 7: Loss landscape and optimization trajectories under different federated aggregation methods
Category
Representative Works
Where Decomposition Is Applied
FL Aggregation
Momentum-aware
Centralized LoRA optimization
PiSSA ( Meng et al., 2024 ) SORSA ( Cao and Song, 2024 ) NoRM ( Jiang et al., 2025 ) CorDA ( Yang et al., 2024 ) GOAT ( Fan et al., 2025 ) AdaLoRA ( Zhang et al., 2023a )
Decomposition applied to centralized LoRA parameterization, including adapter initialization, structure-aware conditioning/regularization, and adaptive rank or budget allocation during training.
✗
✗
Federated LoRA initializationn
FeDeRA ( Yan et al., 2026 )
Decomposition of backbone weights and then use for federated fine-tuning
✗
✗
Federated rank adaptation
FlexLoRA ( Bai et al., 2024 ) HierFedLoRA ( Liu et al., 2025 ) AFLoRA ( Zhou et al., 2025 ) FedARA ( Wu et al., 2025 )
Decomposition used to reconcile heterogeneous LoRA ranks in FL, enabling aggregation and broadcast under client/resource heterogeneity.
✓
✗
Robust / orthogonal aggregation
FedSVD ( Lee et al., 2025 ) FedRPCA ( Jhunjhunwala et al., 2025 )
Decomposition applied at the aggregation stage to orthogonalize or separate aggregated updates, aiming to suppress noise and improve robustness of federated LoRA aggregation.
✓
✗
Momentum-preserving aggregation
FedMomentum (ours)
Decomposition is used to reconstruct a consistent low-rank update structure across communication rounds, explicitly preserving cross-round update directions and training momentum.
✓
✓
Appendix
Table 9: Taxonomy of SVD-/decomposition-based LoRA and federated fine-tuning methods. FedMomentum uniquely focuses on preserving the structural consistency of LoRA updates across communication rounds, which enables momentum preservation during federated optimization.