Align Before Aggregation: Basis-Consistent Federated LoRA under Heterogeneous Ranks
Authors: Pengpeng Qiao, Yang Cao, Lingling Zhang, Guo Cheng, Junwei Chen, Manjiang Yu, Wei Yang Bryan Lim, Masatoshi Yoshikawa
Organizations: Institute of Science Tokyo · Capital Normal University · Beijing Institute of Technology · The University of Queensland · Nanyang Technological University · Osaka Seikei University
Federated fine-tuning of large language models (LLMs) under heterogeneous client capacities is increasingly important, and low-rank adaptation (LoRA) with heterogeneous ranks makes it practical. Existing server-side aggregators compose local matrix updates ΔWi from client-specific LoRA factors (Ai,Bi) before aggregating them. Although these matrix updates lie in the same weight space Rm×n and can be averaged directly, the LoRA factors that generate them may expressed client-specific local basis. In the typical compose--aggregate--factorize--dispatch loop, this basis inconsistency can make the rank-ri reference prefix dispatched to lower-rank clients unstable after server-side refactorization. In this work, we propose FedAbA, a basis-consistent federated LoRA aggregation framework for heterogeneous ranks. FedAbA first extracts each client's local basis, aligns it to the corresponding prefix of the global reference basis, and constructs a basis-consistent reconstruction of each local matrix update before aggregation. It then aggregates these reconstructions with weights that combine client data size and consistency score, followed by exact coefficient-space SVD refactorization and gauge fixing before rank-prefix dispatch. Our theoretical analysis clarifies the role of basis consistency and supports the proposed consistency-aware aggregation mechanism. Extensive experiments show that FedAbA outperforms representative federated LoRA baselines, with the largest gains under the strongest rank heterogeneity.
Figures & tables
Figure 1: Illustration of federated LoRA under heterogeneous ranks. Compose–aggregate–factorize–dispatch training process (top). Two challenges: basis inconsistency and biased aggregation (bottom).
Figure 2: (a) Distribution of cit at t=10 shifts downward as rank heterogeneity grows. (b) Under strong heterogeneity, cit is weakly correlated with the data-size weight pi .
Figure 3: Overview of FedAbA.
Method
NLU, RoBERTa-large
NLG-style, LLaMA-2-7B
NI, LLaMA-1B
MRPC
CoLA
QNLI
RTE
Wizard
Dolly-15K
Alpaca
Alpaca
Natural Instructions v2
F1
MCC
ACC
ACC
(MMLU)
(MMLU)
(MMLU)
(MT-Bench)
Rouge-L
Rouge-1
HetLORA
86.6
59.8
93.8
67.5
34.5
43.8
32.3
3.57
57.0
65.3
FLoRA
87.2
61.8
93.5
71.8
35.4
44.9
34.6
3.80
57.1
65.3
LoRA-A 2
88.9
62.5
93.8
79.5
35.6
45.4
35.6
4.03
57.4
65.4
FlexLoRA
88.7
62.4
93.8
78.3
36.2
45.9
36.2
4.13
57.6
65.5
Table 1: Performance comparison of six baselines across various models and tasks.
Method
NLG-style, LLaMA-2-7B, Dolly-15K
NI, LLaMA-1B, Natural Instructions v2
Overall (Rouge-L)
r=4
r=8
r=16
r=32
r=64
Overall
r=4
r=8
r=16
r=32
HetLORA
29.2
27.6
28.5
29.4
30.2
31.0
57.0
54.8
56.5
57.8
59.1
FLoRA
28.7
26.8
27.9
28.8
30.1
30.9
57.1
54.2
56.1
58.0
59.5
LoRA-A 2
30.0
28.5
29.4
30.1
30.8
31.4
57.4
55.0
56.7
58.2
59.8
FlexLoRA
30.2
28.0
29.0
30.4
31.3
32.0
57.6
55.1
56.9
58.4
60.0
ECLoRA
30.2
28.2
29.1
30.4
31.4
31.9
57.7
55.5
57.2
58.6
60.2
Table 2: Performance comparison across different LoRA ranks on Dolly and Natural Instructions v2.
Task Family
Rank Regime
Rank Setting
HetLoRA
FLoRA
LoRA-A 2
FlexLoRA
ECLoRA
Fed-PLoRA
FedAbA
NLU Avg.
Homo.
[8]×10
76.33
77.98
80.00
79.80
80.80
80.75
80.97
Mild
{4,8,16}
76.65
78.43
80.73
80.17
81.35
81.28
81.72
High
{4,8,16,32,64}
76.93
78.58
81.18
80.80
82.20
82.10
82.90
NLG Avg. (MMLU)
Homo.
[8]×10
36.37
37.70
38.07
38.77
38.86
39.14
39.30
Mild
{4,8,16}
36.60
38.17
38.57
39.00
39.16
39.51
39.83
High
{4,8,16,32,64}
36.87
38.30
38.87
39.43
39.63
40.37
40.90
Table 3: Performance comparison across different rank heterogeneity regimes on NLU, NLG, and NI.
Figure 4: Sensitivity analysis under high rank heterogeneity. (a) Basis diagnostics comparing FlexLoRA, ECLoRA. (b)–(d) Sensitivity of FedAbA to warm-start rounds Tw , consistency weighting strength γc , and gauge-fixing threshold τsvd , respectively.
Table 8
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
N , St
total number of clients; active client set at round t
ni , pit=ni/∑j∈Stnj
client i ’s sample count; data-size weight
ri , rmax
client i ’s LoRA rank; maximum rank used by the server
(Ait,Bit)
client i ’s LoRA factors at round t , shapes (ri,n) and (m,ri)
ΔWit=sBitAit
client i ’s local matrix update
(Agt,Bgt)
global LoRA factors at round t , shapes (rmax,n) and (m,rmax)
Appendix
Table 6: Notation used throughout the paper.
Method
MRPC F1
RTE Acc.
Dolly-15K MMLU
Alpaca MT-Bench
NI ROUGE-L
NI ROUGE-1
Fed-PLoRA
89.42±0.45
81.63±1.10
47.17±0.58
4.21±0.12
57.98±0.46
65.66±0.48
FedAbA
90.28±0.64
83.28±0.82
47.87±0.40
4.31±0.14
58.37±0.30
66.05±0.63
Appendix
Table 7: Repeated-run comparison: mean ± sample standard deviation over three independent matched runs. Dolly-15K and Alpaca denote training datasets, evaluated with MMLU and MT-Bench, respectively.
Method
All clients
Lowest-consistency 20%
ROUGE-L
ROUGE-1
ROUGE-L
ROUGE-1
Fed-PLoRA
58.1
65.8
54.8
62.4
FedAbA
58.4
66.1
55.1
62.6
Appendix
Table 8: Natural Instructions results over all clients and the fixed lowest-consistency 20% subset.
Method
LLaMA-2-7B
Qwen3-4B
LLaMA-2-13B
FlexLoRA
45.9
57.0
51.7
ECLoRA
46.1
57.2
51.9
Fed-PLoRA
47.2
58.3
53.0
FedAbA w/o basis-consistent reconstruction
46.9
58.0
52.7
FedAbA
47.8
59.0
53.7
Appendix
Table 9: Backbone comparison after federated fine-tuning on Dolly-15K, evaluated with MMLU.