CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

Yanan Ma; Qiyuan Chen; Zihan Fang; Xianhao Chen; Yuguang Fang

cs.LGSep 27, 2026

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

Authors: Yanan Ma, Qiyuan Chen, Zihan Fang, Xianhao Chen, Yuguang Fang

Organizations: City University of Hong Kong · The University of Hong Kong

Abstract

Federated low-rank adaptation (LoRA) enables collaborative fine-tuning of large language models without centralizing private client data. Its factorized update, however, creates a structural mismatch in federated averaging: averaging the two LoRA factors separately does not equal averaging their products. Existing exact methods resolve this issue mainly by freezing an entire factor or alternating factors across rounds, but none can update factors simultaneously without aggregation errors or expanding communication ranks. To address this fundamental problem, we present \texttt{CFLoRA}, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round. By ensuring that columns and rows are complementary across factors, we eliminate bilinear terms in matrix multiplications, making federated aggregation exact. Crucially, our framework also supports clients with heterogeneous rank budgets. Convergence analysis validates \texttt{CFLoRA} achieves $\mathcal{O}(1/\sqrt{T})$ convergence rate of the \textit{original} LoRA objective in homogeneous-rank cases. Extensive experiments with RoBERTa on the GLUE benchmark and with LLaMA-3.2-3B-Instruct on commonsense reasoning tasks demonstrate that \texttt{CFLoRA} achieves superior performance and training efficiency compared to state-of-the-art federated LoRA baselines.

Figures & tables

Figure 1: The illustration of the proposed CFLoRA framework. CFLoRA separates two decisions: which factor may change on each latent channel and which channels a client can afford to train. A shared complementary selection mask ensures exact factor aggregation, while client-specific supports accommodate heterogeneous ranks.

Method	QNLI	MRPC	CoLA	MNLI-M	MNLI-MM	QQP	Avg.
FedIT	86.61 $\pm$ 0.26	72.27 $\pm$ 2.32	70.49 $\pm$ 0.90	79.10 $\pm$ 2.21	79.89 $\pm$ 2.28	71.75 $\pm$ 1.35	76.69
FFA-LoRA	88.49 $\pm$ 0.23	70.58 $\pm$ 1.56	73.81 $\pm$ 1.74	72.42 $\pm$ 3.27	73.39 $\pm$ 3.08	73.26 $\pm$ 1.38	75.33
RoLoRA	87.60 $\pm$ 2.21	73.16 $\pm$ 2.11	74.40 $\pm$ 0.89	72.01 $\pm$ 1.55	72.50 $\pm$ 3.56	73.47 $\pm$ 0.41	75.52
AS-LoRA	88.26 $\pm$ 0.67	72.30 $\pm$ 1.06	73.57 $\pm$ 2.15	71.41 $\pm$ 1.88	72.60 $\pm$ 1.51	75.23 $\pm$ 2.57	75.56
FLoRA	89.27 $\pm$ 0.20	71.46 $\pm$ 1.24	76.81 $\pm$ 2.74	70.56 $\pm$ 0.82	71.52 $\pm$ 0.21	76.43 $\pm$ 2.74	76.01
CFLoRA	90.61 $\pm$ 0.25	73.53 $\pm$ 1.59	78.95 $\pm$ 1.48	79.16 $\pm$ 1.22	79.85 $\pm$ 1.01	76.30 $\pm$ 0.36	79.73

Table 1: Test accuracy for fine-tuning the RoBERTa model on the GLUE benchmark under homogeneous client ranks.

Method	QNLI	MRPC	CoLA	MNLI-M	MNLI-MM	QQP	Avg.
FedIT	70.34 $\pm$ 2.01	69.60 $\pm$ 2.43	73.56 $\pm$ 3.67	58.34 $\pm$ 1.03	58.22 $\pm$ 0.93	73.12 $\pm$ 1.41	67.20
FFA-LoRA	86.86 $\pm$ 2.15	70.67 $\pm$ 2.31	73.09 $\pm$ 2.97	63.74 $\pm$ 5.02	64.04 $\pm$ 4.88	75.91 $\pm$ 1.01	72.39
RoLoRA	84.87 $\pm$ 3.10	68.46 $\pm$ 3.12	73.14 $\pm$ 3.40	51.19 $\pm$ 3.95	51.60 $\pm$ 2.84	74.73 $\pm$ 1.29	67.33
AS-LoRA	80.71 $\pm$ 2.52	71.12 $\pm$ 1.33	73.09 $\pm$ 2.97	63.60 $\pm$ 4.86	63.95 $\pm$ 5.94	70.73 $\pm$ 1.94	70.53
FLoRA	87.35 $\pm$ 1.84	69.80 $\pm$ 1.71	75.75 $\pm$ 1.34	72.22 $\pm$ 1.26	70.97 $\pm$ 1.14	76.61 $\pm$ 0.53	75.45
FlexLoRA	86.66 $\pm$ 1.55	71.92 $\pm$ 1.19	77.60 $\pm$ 0.58	71.35 $\pm$ 1.49	72.25 $\pm$ 1.48	76.40 $\pm$ 1.32	76.03

Table 2: Testing accuracy for fine-tuning the RoBERTa model on the GLUE benchmark under heterogeneous client ranks.

Schedule	QNLI	MRPC	CoLA	MNLI-M	MNLI-MM	QQP	Avg.
$p=0.3$	87.80 $\pm$ 0.02	70.83 $\pm$ 1.73	73.33 $\pm$ 2.37	71.15 $\pm$ 1.07	72.73 $\pm$ 1.03	72.45 $\pm$ 1.39	74.72
$p=0.5$	90.64 $\pm$ 0.25	72.43 $\pm$ 2.50	78.72 $\pm$ 1.73	77.82 $\pm$ 0.49	78.00 $\pm$ 0.12	74.57 $\pm$ 0.43	78.70
$p=0.7$	90.69 $\pm$ 0.24	72.55 $\pm$ 2.06	79.81 $\pm$ 1.52	77.47 $\pm$ 0.61	78.02 $\pm$ 0.65	74.52 $\pm$ 1.43	78.84
$p=0.9$	90.61 $\pm$ 0.25	73.53 $\pm$ 1.59	78.95 $\pm$ 1.48	79.16 $\pm$ 1.22	79.85 $\pm$ 1.01	76.30 $\pm$ 0.36	79.73
$p=1.0$	88.61 $\pm$ 0.27	70.47 $\pm$ 1.54	73.89 $\pm$ 1.77	72.36 $\pm$ 3.23	73.54 $\pm$ 3.13	73.17 $\pm$ 1.39	75.34

Table 3: Ablation study on selection probability

p

for RoBERTa on GLUE tasks under homogeneous client ranks.

Schedule	QNLI	MRPC	CoLA	MNLI-M	MNLI-MM	QQP	Avg.
$p=0.3$	88.31 $\pm$ 0.99	68.79 $\pm$ 0.35	71.46 $\pm$ 1.27	68.78 $\pm$ 0.49	69.66 $\pm$ 1.29	73.21 $\pm$ 1.15	73.37
$p=0.5$	91.16 $\pm$ 0.10	73.98 $\pm$ 2.26	76.90 $\pm$ 2.38	76.37 $\pm$ 1.02	76.85 $\pm$ 1.30	77.16 $\pm$ 1.28	78.74
$p=0.7$	91.41 $\pm$ 0.87	74.13 $\pm$ 1.04	77.32 $\pm$ 0.85	76.74 $\pm$ 1.37	77.51 $\pm$ 0.73	79.61 $\pm$ 1.07	79.45
$p=0.9$	91.29 $\pm$ 0.45	74.86 $\pm$ 0.69	77.80 $\pm$ 1.26	76.82 $\pm$ 0.80	77.32 $\pm$ 1.14	79.34 $\pm$ 0.87	79.57
$p=1.0$	89.90 $\pm$ 1.39	73.00 $\pm$ 2.02	74.51 $\pm$ 1.42	75.25 $\pm$ 1.57	75.81 $\pm$ 0.91	73.63 $\pm$ 1.95	77.02

Table 4: Ablation study on selection probability

p

for RoBERTa on GLUE tasks under heterogeneous client ranks.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Standard federated LoRA	CFLoRA
Active adapter parameters	$1$	$1$
Trainable adapter parameters	$1$	$1/2$
Forward MACs	$1$	$1$
Input-gradient MACs	$1$	$1$
Factor-gradient MACs	$1$	$1/2$
Total adapter MACs per step	$1$	$5/6$

Appendix

Table C.1: Normalized cost comparison at the same local rank for square projections between standard federated LoRA and our CFLoRA . MAC counts assume that input gradients are required and only selected factor gradients are computed. Communication assumes compact transmission at equal precision. Control information is negligible, which is excluded.

Panel of Figure 7 — Figure C.1: Test accuracy versus communication costs of CFLoRA (Ours) and federated LoRA baselines on the MNLI and QNLI datasets.

Setting	RoBERTa / GLUE	LLaMA-3.2-3B-Instruct / commonsense reasoning
Total clients	$25$	$25$
Client participation ratio $q$	$0.4$	$0.4$
Communication rounds $T$	$200$	$100$
Local steps $K$	$50$	$40$
Batch size	$8$	$8$
Learning rate	$1\times 10^{-4}$	$2\times 10^{-4}$

Appendix

Table D.2: parameter settings.

Method	ARC-c	ARC-e	BoolQ	OBQA	PIQA	SIQA	Avg.
FedIT	67.72 $\pm$ 0.45	84.48 $\pm$ 1.10	62.44 $\pm$ 0.27	74.53 $\pm$ 0.47	70.51 $\pm$ 2.39	66.47 $\pm$ 1.89	71.03
FFA-LoRA	67.45 $\pm$ 0.10	80.62 $\pm$ 0.29	64.17 $\pm$ 0.04	63.37 $\pm$ 1.93	70.39 $\pm$ 3.89	71.69 $\pm$ 0.54	69.62
RoLoRA	69.60 $\pm$ 1.93	86.78 $\pm$ 0.12	64.18 $\pm$ 0.07	73.73 $\pm$ 2.45	69.61 $\pm$ 3.88	69.73 $\pm$ 2.27	72.27
AS-LoRA	67.53 $\pm$ 0.06	80.75 $\pm$ 0.26	64.12 $\pm$ 0.01	63.60 $\pm$ 1.70	70.30 $\pm$ 3.03	71.95 $\pm$ 0.69	69.71
FLoRA	70.65 $\pm$ 0.16	83.67 $\pm$ 0.03	65.26 $\pm$ 1.10	72.07 $\pm$ 0.75	72.67 $\pm$ 1.34	72.67 $\pm$ 1.41	72.83
CFLoRA	70.67 $\pm$ 1.39	85.55 $\pm$ 1.05	64.24 $\pm$ 0.11	75.60 $\pm$ 0.90	75.73 $\pm$ 1.27	73.88 $\pm$ 0.58	74.28

Appendix

Table D.3: Testing accuracy for fine-tuning the LLaMA-3.2-3B-Instruct on the commonsense reasoning tasks under homogeneous client ranks.

Method	ARC-c	ARC-e	BoolQ	OBQA	PIQA	SIQA	Avg.
FedIT	67.04 $\pm$ 0.32	80.13 $\pm$ 0.97	62.37 $\pm$ 0.81	71.33 $\pm$ 1.40	70.24 $\pm$ 2.13	68.32 $\pm$ 1.37	69.91
FLoRA	66.98 $\pm$ 1.20	84.18 $\pm$ 0.42	63.55 $\pm$ 1.94	76.00 $\pm$ 0.57	74.27 $\pm$ 1.63	70.63 $\pm$ 1.88	72.60
FlexLoRA	67.45 $\pm$ 0.74	82.84 $\pm$ 1.03	62.35 $\pm$ 0.97	74.02 $\pm$ 0.55	73.31 $\pm$ 0.86	69.59 $\pm$ 1.64	71.59
FSLoRA	69.88 $\pm$ 0.27	84.51 $\pm$ 0.49	64.21 $\pm$ 0.75	73.80 $\pm$ 0.92	74.96 $\pm$ 1.50	71.83 $\pm$ 1.69	73.20
CFLoRA	69.51 $\pm$ 0.12	85.82 $\pm$ 0.77	66.51 $\pm$ 0.35	75.40 $\pm$ 0.39	78.73 $\pm$ 1.45	76.20 $\pm$ 1.52	75.36

Appendix

Table D.4: Testing accuracy for fine-tuning the LLaMA-3.2-3B-Instruct on the commonsense reasoning benchmark under heterogeneous client ranks.

Method	ARC-c	ARC-e	BoolQ	OBQA	PIQA	SIQA	Avg.
$p=0.3$	65.98 $\pm$ 1.67	84.71 $\pm$ 0.44	62.54 $\pm$ 0.50	70.40 $\pm$ 0.57	72.33 $\pm$ 1.79	72.25 $\pm$ 2.07	71.37
$p=0.5$	69.51 $\pm$ 0.96	84.87 $\pm$ 0.15	63.62 $\pm$ 0.67	74.63 $\pm$ 0.65	74.53 $\pm$ 1.48	73.26 $\pm$ 0.62	73.40
$p=0.7$	68.65 $\pm$ 0.14	85.64 $\pm$ 1.24	64.17 $\pm$ 0.13	75.09 $\pm$ 0.41	75.74 $\pm$ 2.00	73.63 $\pm$ 1.31	73.82
$p=0.9$	70.67 $\pm$ 1.39	85.55 $\pm$ 1.05	64.24 $\pm$ 0.11	75.60 $\pm$ 0.90	75.73 $\pm$ 1.27	73.88 $\pm$ 0.58	74.28
$p=1.0$	67.56 $\pm$ 0.13	80.54 $\pm$ 0.24	64.31 $\pm$ 0.08	64.25 $\pm$ 1.39	70.15 $\pm$ 2.44	71.63 $\pm$ 0.73	69.74

Appendix

Table D.5: Ablation study on selection probability

p

for LLaMA-3.2-3B-Instruct on the commonsense reasoning benchmark under homogeneous client ranks.

Method	ARC-c	ARC-e	BoolQ	OBQA	PIQA	SIQA	Avg.
$p=0.3$	68.81 $\pm$ 0.16	82.78 $\pm$ 0.76	62.25 $\pm$ 0.11	74.97 $\pm$ 0.61	71.47 $\pm$ 3.24	73.44 $\pm$ 0.20	72.29
$p=0.5$	68.89 $\pm$ 0.45	83.76 $\pm$ 0.71	65.25 $\pm$ 0.44	75.50 $\pm$ 1.13	76.93 $\pm$ 0.46	75.61 $\pm$ 0.07	74.32
$p=0.7$	69.10 $\pm$ 0.60	84.53 $\pm$ 0.66	66.18 $\pm$ 0.50	75.82 $\pm$ 0.57	78.88 $\pm$ 4.90	76.01 $\pm$ 0.17	75.09
$p=0.9$	69.51 $\pm$ 0.12	85.82 $\pm$ 0.77	66.51 $\pm$ 0.35	75.40 $\pm$ 0.39	78.73 $\pm$ 1.45	76.20 $\pm$ 1.52	75.36
$p=1.0$	69.13 $\pm$ 0.38	83.65 $\pm$ 0.73	62.34 $\pm$ 0.23	67.73 $\pm$ 0.94	73.33 $\pm$ 0.63	74.12 $\pm$ 0.34	71.72

Appendix

Table D.6: Ablation study on selection probability

p

for LLaMA-3.2-3B-Instruct on the commonsense reasoning benchmark under heterogeneous client ranks.

Method	$q=0.2$	$q=0.4$	$q=0.6$	$q=0.8$	$q=1.0$	Avg.
FedIT	85.01 $\pm$ 0.23	86.61 $\pm$ 0.26	88.90 $\pm$ 0.26	93.06 $\pm$ 0.90	90.02 $\pm$ 0.46	88.72
FFA-LoRA	86.13 $\pm$ 0.35	88.49 $\pm$ 0.23	90.94 $\pm$ 0.33	94.09 $\pm$ 0.32	91.42 $\pm$ 0.19	90.21
RoLoRA	86.52 $\pm$ 0.27	87.60 $\pm$ 2.21	92.60 $\pm$ 0.46	93.23 $\pm$ 0.14	93.10 $\pm$ 0.46	90.61
AS-LoRA	87.13 $\pm$ 0.35	88.26 $\pm$ 0.67	91.28 $\pm$ 0.38	92.71 $\pm$ 0.17	92.46 $\pm$ 1.03	90.37
FLoRA	88.83 $\pm$ 0.24	89.27 $\pm$ 0.20	90.69 $\pm$ 0.41	92.93 $\pm$ 0.44	91.41 $\pm$ 0.35	90.63
CFLoRA	88.95 $\pm$ 0.06	90.61 $\pm$ 0.25	92.98 $\pm$ 0.31	93.59 $\pm$ 0.06	93.93 $\pm$ 0.34	92.01

Appendix

Table D.7: Testing accuracy for fine-tuning the RoBERTa model on the QNLI dataset under homogeneous client ranks with varying client participation.

Method	$q=0.2$	$q=0.4$	$q=0.6$	$q=0.8$	$q=1.0$	Avg.
FedIT	68.01 $\pm$ 4.58	70.34 $\pm$ 2.01	85.89 $\pm$ 0.37	90.03 $\pm$ 3.26	88.22 $\pm$ 2.36	80.50
FLoRA	70.97 $\pm$ 4.32	87.35 $\pm$ 1.84	87.75 $\pm$ 3.21	92.67 $\pm$ 0.72	91.77 $\pm$ 2.35	86.10
FlexLoRA	78.41 $\pm$ 5.10	86.66 $\pm$ 1.55	86.81 $\pm$ 1.36	93.11 $\pm$ 1.25	94.02 $\pm$ 0.95	87.80
FSLoRA	73.68 $\pm$ 4.01	86.68 $\pm$ 2.27	86.92 $\pm$ 1.48	89.40 $\pm$ 2.45	88.54 $\pm$ 1.60	85.04
CFLoRA	76.84 $\pm$ 3.73	91.29 $\pm$ 0.45	91.46 $\pm$ 2.10	93.60 $\pm$ 0.38	93.39 $\pm$ 0.91	89.32

Appendix

Table D.8: Testing accuracy for fine-tuning the RoBERTa model on the QNLI dataset under heterogeneous client ranks with varying client participation.

Method	QNLI	MRPC	CoLA	MNLI-M	MNLI-MM	QQP	Avg.
FedIT	90.02 $\pm$ 0.46	70.30 $\pm$ 0.39	75.16 $\pm$ 0.15	82.40 $\pm$ 0.44	82.19 $\pm$ 0.33	80.98 $\pm$ 1.05	80.18
FFA-LoRA	91.42 $\pm$ 0.19	73.20 $\pm$ 0.42	74.16 $\pm$ 0.45	84.79 $\pm$ 0.16	84.78 $\pm$ 0.16	81.89 $\pm$ 1.05	81.71
RoLoRA	93.10 $\pm$ 0.46	73.86 $\pm$ 1.26	73.93 $\pm$ 0.29	85.76 $\pm$ 0.43	85.84 $\pm$ 0.53	81.25 $\pm$ 0.38	82.29
AS-LoRA	92.46 $\pm$ 1.03	72.44 $\pm$ 0.67	75.04 $\pm$ 0.94	82.79 $\pm$ 0.33	86.76 $\pm$ 0.35	80.95 $\pm$ 1.67	81.74
FLoRA	91.41 $\pm$ 0.35	76.34 $\pm$ 0.80	77.26 $\pm$ 0.25	83.66 $\pm$ 0.77	83.88 $\pm$ 0.66	82.79 $\pm$ 0.32	82.56
CFLoRA	93.93 $\pm$ 0.34	80.83 $\pm$ 0.85	76.74 $\pm$ 0.46	86.54 $\pm$ 0.40	86.71 $\pm$ 0.26	84.15 $\pm$ 0.53	84.82

Appendix

Table D.9: Testing accuracy for fine-tuning the RoBERTa model on the GLUE benchmark under homogeneous client ranks with full participation.

Explore similar work

FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA
Aug 15, 2026Juseok Jeon, Ramy E. Ali, Doyun Kwon +3Federated Fine-Tuning Federated Learning
DGIST Daegu, Republic of Korea · Samsung San Diego, USA
CF-LoRA: Decoupled Factor Aggregation and Adaptation-Aware Client Clustering for Federated LoRA Fine-Tuning
Sep 29, 2026Mengjun Yi, Langxing Yang, Suhan Guo +2Federated Learning Aggregation Federated Fine-Tuning
State Key Laboratory for Novel Software Technology · School of Artificial Intelligence School of Electronic Science and Engineering Nanjing University
Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment
May 20, 2026Shuaida He, Liwen Chen, Long FengLLM Fine-Tuning Low-Rank Adaptation
School of Computing & Data Science, The University of Hong Kong

Aug 15, 2026, cs.AIJ/K move · Enter open · S save

FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA

Juseok Jeon, Ramy E. Ali, Doyun Kwon, Myungbeom Her+2

DGIST Daegu, Republic of Korea · Samsung San Diego, USA

Federated Fine-Tuning Federated Learning Federated Learning Aggregation Low-Rank Adaptation Heterogeneous Federated Learning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorized parameterization creates a tension between accurate aggregation of local updates and continuity of locally optimized factors. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch from newly reconstructed factors. We propose FedPA-LoRA, a product-aligned federated LoRA framework that jointly addresses these limitations and provably converges under both homogeneous and heterogeneous client ranks. Each client preserves its local factors across communication rounds and aligns its product toward a rank-specific global reference, maintaining local optimization continuity while promoting global consistency under data heterogeneity. The server aggregates heterogeneous-rank updates in the common product space and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. This design supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show that FedPA-LoRA consistently outperforms representative baselines across varying levels of data heterogeneity and homogeneous- and heterogeneous-rank settings, with up to a

6.82

percentage-point improvement in average GLUE accuracy under heterogeneous client ranks.

Abstract

Explore similar work

FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA

CF-LoRA: Decoupled Factor Aggregation and Adaptation-Aware Client Clustering for Federated LoRA Fine-Tuning

Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment