Align Before Aggregation: Basis-Consistent Federated LoRA under Heterogeneous Ranks
Authors: Pengpeng Qiao, Yang Cao, Lingling Zhang, Guo Cheng, Junwei Chen, Manjiang Yu, Wei Yang Bryan Lim, Masatoshi Yoshikawa
Organizations: Institute of Science Tokyo · Capital Normal University · Beijing Institute of Technology · The University of Queensland · Nanyang Technological University · Osaka Seikei University
Federated fine-tuning of large language models (LLMs) under heterogeneous client capacities is increasingly important, and low-rank adaptation (LoRA) with heterogeneous ranks makes it practical. Existing server-side aggregators compose local matrix updates ΔWi from client-specific LoRA factors (Ai,Bi) before aggregating them. Although these matrix updates lie in the same weight space Rm×n and can be averaged directly, the LoRA factors that generate them may expressed client-specific local basis. In the typical compose--aggregate--factorize--dispatch loop, this basis inconsistency can make the rank-ri reference prefix dispatched to lower-rank clients unstable after server-side refactorization. In this work, we propose FedAbA, a basis-consistent federated LoRA aggregation framework for heterogeneous ranks. FedAbA first extracts each client's local basis, aligns it to the corresponding prefix of the global reference basis, and constructs a basis-consistent reconstruction of each local matrix update before aggregation. It then aggregates these reconstructions with weights that combine client data size and consistency score, followed by exact coefficient-space SVD refactorization and gauge fixing before rank-prefix dispatch. Our theoretical analysis clarifies the role of basis consistency and supports the proposed consistency-aware aggregation mechanism. Extensive experiments show that FedAbA outperforms representative federated LoRA baselines, with the largest gains under the strongest rank heterogeneity.
Figures & tables
Figure 1: Illustration of federated LoRA under heterogeneous ranks. Compose–aggregate–factorize–dispatch training process (top). Two challenges: basis inconsistency and biased aggregation (bottom).
Figure 2: (a) Distribution of cit at t=10 shifts downward as rank heterogeneity grows. (b) Under strong heterogeneity, cit is weakly correlated with the data-size weight pi .
Figure 3: Overview of FedAbA.
Method
NLU, RoBERTa-large
NLG-style, LLaMA-2-7B
NI, LLaMA-1B
MRPC
CoLA
QNLI
RTE
Wizard
Dolly-15K
Alpaca
Alpaca
Natural Instructions v2
F1
MCC
ACC
ACC
(MMLU)
(MMLU)
(MMLU)
(MT-Bench)
Rouge-L
Rouge-1
HetLORA
86.6
59.8
93.8
67.5
34.5
43.8
32.3
3.57
57.0
65.3
FLoRA
87.2
61.8
93.5
71.8
35.4
44.9
34.6
3.80
57.1
65.3
LoRA-A 2
88.9
62.5
93.8
79.5
35.6
45.4
35.6
4.03
57.4
65.4
FlexLoRA
88.7
62.4
93.8
78.3
36.2
45.9
36.2
4.13
57.6
65.5
Table 1: Performance comparison of six baselines across various models and tasks.
Method
NLG-style, LLaMA-2-7B, Dolly-15K
NI, LLaMA-1B, Natural Instructions v2
Overall (Rouge-L)
r=4
r=8
r=16
r=32
r=64
Overall
r=4
r=8
r=16
r=32
HetLORA
29.2
27.6
28.5
29.4
30.2
31.0
57.0
54.8
56.5
57.8
59.1
FLoRA
28.7
26.8
27.9
28.8
30.1
30.9
57.1
54.2
56.1
58.0
59.5
LoRA-A 2
30.0
28.5
29.4
30.1
30.8
31.4
57.4
55.0
56.7
58.2
59.8
FlexLoRA
30.2
28.0
29.0
30.4
31.3
32.0
57.6
55.1
56.9
58.4
60.0
ECLoRA
30.2
28.2
29.1
30.4
31.4
31.9
57.7
55.5
57.2
58.6
60.2
Table 2: Performance comparison across different LoRA ranks on Dolly and Natural Instructions v2.
Task Family
Rank Regime
Rank Setting
HetLoRA
FLoRA
LoRA-A 2
FlexLoRA
ECLoRA
Fed-PLoRA
FedAbA
NLU Avg.
Homo.
[8]×10
76.33
77.98
80.00
79.80
80.80
80.75
80.97
Mild
{4,8,16}
76.65
78.43
80.73
80.17
81.35
81.28
81.72
High
{4,8,16,32,64}
76.93
78.58
81.18
80.80
82.20
82.10
82.90
NLG Avg. (MMLU)
Homo.
[8]×10
36.37
37.70
38.07
38.77
38.86
39.14
39.30
Mild
{4,8,16}
36.60
38.17
38.57
39.00
39.16
39.51
39.83
High
{4,8,16,32,64}
36.87
38.30
38.87
39.43
39.63
40.37
40.90
Table 3: Performance comparison across different rank heterogeneity regimes on NLU, NLG, and NI.
Figure 4: Sensitivity analysis under high rank heterogeneity. (a) Basis diagnostics comparing FlexLoRA, ECLoRA. (b)–(d) Sensitivity of FedAbA to warm-start rounds Tw , consistency weighting strength γc , and gauge-fixing threshold τsvd , respectively.
Table 8
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
N , St
total number of clients; active client set at round t
ni , pit=ni/∑j∈Stnj
client i ’s sample count; data-size weight
ri , rmax
client i ’s LoRA rank; maximum rank used by the server
(Ait,Bit)
client i ’s LoRA factors at round t , shapes (ri,n) and (m,ri)
ΔWit=sBitAit
client i ’s local matrix update
(Agt,Bgt)
global LoRA factors at round t , shapes (rmax,n) and (m,rmax)
Appendix
Table 6: Notation used throughout the paper.
Method
MRPC F1
RTE Acc.
Dolly-15K MMLU
Alpaca MT-Bench
NI ROUGE-L
NI ROUGE-1
Fed-PLoRA
89.42±0.45
81.63±1.10
47.17±0.58
4.21±0.12
57.98±0.46
65.66±0.48
FedAbA
90.28±0.64
83.28±0.82
47.87±0.40
4.31±0.14
58.37±0.30
66.05±0.63
Appendix
Table 7: Repeated-run comparison: mean ± sample standard deviation over three independent matched runs. Dolly-15K and Alpaca denote training datasets, evaluated with MMLU and MT-Bench, respectively.
Method
All clients
Lowest-consistency 20%
ROUGE-L
ROUGE-1
ROUGE-L
ROUGE-1
Fed-PLoRA
58.1
65.8
54.8
62.4
FedAbA
58.4
66.1
55.1
62.6
Appendix
Table 8: Natural Instructions results over all clients and the fixed lowest-consistency 20% subset.
Method
LLaMA-2-7B
Qwen3-4B
LLaMA-2-13B
FlexLoRA
45.9
57.0
51.7
ECLoRA
46.1
57.2
51.9
Fed-PLoRA
47.2
58.3
53.0
FedAbA w/o basis-consistent reconstruction
46.9
58.0
52.7
FedAbA
47.8
59.0
53.7
Appendix
Table 9: Backbone comparison after federated fine-tuning on Dolly-15K, evaluated with MMLU.
Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorization creates a tension between aggregation fidelity and local optimization continuity. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch. We propose FedPA-LoRA, a product-aligned framework that addresses both mismatches and establishes convergence bounds for homogeneous and heterogeneous ranks. Clients preserve local factors across rounds to eliminate initialization mismatch and align their products with rank-specific global references to reduce client drift. The server aggregates heterogeneous-rank updates in product space to mitigate aggregation mismatch and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. By decoupling local and reference ranks, FedPA-LoRA supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show consistent gains under data and resource heterogeneity, including a 6.82 percentage-point improvement in average GLUE accuracy under heterogeneous ranks. The code is available at https://github.com/Juseok-Jeon/FedPA-LoRA.
Juseok Jeon, Ramy E. Ali, Doyun Kwon +3
DGIST Daegu, Republic of Korea · Samsung San Diego, USA
Federated low-rank adaptation (LoRA) enables collaborative fine-tuning of large language models without centralizing private client data. Its factorized update, however, creates a structural mismatch in federated averaging: averaging the two LoRA factors separately does not equal averaging their products. Existing exact methods resolve this issue mainly by freezing an entire factor or alternating factors across rounds, but none can update factors simultaneously without aggregation errors or expanding communication ranks. To address this fundamental problem, we present CFLoRA, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round. By ensuring that columns and rows are complementary across factors, we eliminate bilinear terms in matrix multiplications, making federated aggregation exact. Crucially, our framework also supports clients with heterogeneous rank budgets. Convergence analysis validates CFLoRA achieves O(1/T) convergence rate of the original LoRA objective in homogeneous-rank cases. Extensive experiments with RoBERTa on the GLUE benchmark and with LLaMA-3.2-3B-Instruct on commonsense reasoning tasks demonstrate that CFLoRA achieves superior performance and training efficiency compared to state-of-the-art federated LoRA baselines.
Yanan Ma, Qiyuan Chen, Zihan Fang +2
City University of Hong Kong · The University of Hong Kong
Federated LoRA enables parameter-efficient adaptation of large language models under decentralized data and limited client resources.However, directly averaging LoRA factors is representation-dependent: the same intrinsic update admits infinitely many gauge-equivalent factorizations, so factor-level aggregation can change under arbitrary coordinate choices while the underlying update remains unchanged. This reveals a semantic mismatch in existing federated LoRA aggregation rules. We propose GLoRA, a gauge-aware server representation for federated LoRA.Instead of aggregating raw factors, GLoRA estimates a consensus update subspace from client projectors and aggregates client updates in shared reference coordinates, thereby representing semantic update aggregation entirely in low-rank form. To support heterogeneous client capacities, GLoRA further provides a rank-compatible readout that instantiates adapters of different ranks from the same server state without dense update reconstruction. Experiments on GLUE and SuperNI show that GLoRA consistently outperforms federated LoRA baselines under data, resource, and task heterogeneity, including heterogeneous client ranks, sparse participation, larger backbones, and unseen-task evaluation. GLoRA also achieves a favorable efficiency--performance trade-off, suggesting that effective federated LoRA requires not merely averaging low-rank factors, but defining a semantically meaningful server-side representation for aggregation.
Jinqian Chen, Chang Liu, Jihua Zhu
School of Software Engineering Xi’an Jiaotong University Xi’an, China 710000