FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment

Chuiyang Meng; Ming Tang; Vincent W. S. Wong

cs.LGFeb 19, 2026

FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment

Authors: Chuiyang Meng, Ming Tang, Vincent W. S. Wong

Organizations: The University of British Columbia · Southern University of Science and Technology

Abstract

Parameter-efficient fine-tuning techniques such as low-rank adaptation (LoRA) enable large language models (LLMs) to adapt to downstream tasks efficiently. Federated learning (FL) further facilitates this process by enabling collaborative fine-tuning across distributed clients without sharing private data. However, the use of two separate low-rank matrices in LoRA for federated fine-tuning introduces two types of challenges. First, aggregation error can arise from separately aggregating the two low-rank matrices. Second, even if the server aggregates the product of two low-rank matrices, it needs to decompose the aggregated matrix back into low-rank matrices. Since the decomposition is not unique, it can lead to decomposition drift. To tackle the aforementioned challenges, we propose federated low-rank Gram-matrix aggregation (FLoRG), a federated fine-tuning framework which employs a single low-rank matrix for fine-tuning and aggregates its Gram matrix (i.e., the matrix of inner products of its column vectors). FLoRG can eliminate the aggregation error and reduce the communication overhead. It also minimizes the decomposition drift by introducing a Procrustes alignment approach which aligns the decomposed matrix between consecutive fine-tuning rounds for consistent updates. We theoretically analyze the convergence of FLoRG and prove that adopting the Procrustes alignment results in a tighter convergence bound. Experimental results across multiple LLM fine-tuning benchmarks demonstrate that FLoRG outperforms five state-of-the-art baseline schemes by providing higher downstream task accuracy and can reduce the communication overhead by up to 2041 $\times$ .

Figures & tables

Figure 1: System model of our proposed FLoRG.

Base Model	Dataset	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA	FedEx-LoRA
OPT-125M	MNLI	87.35	79.42	81.15	83.54	84.61	85.83
	QNLI	89.52	84.18	86.71	87.93	88.69	89.88
	WNLI	65.28	58.45	59.34	62.61	62.83	64.15
	RTE	68.92	61.08	64.51	66.02	67.39	68.27
RoBERTa-large	MNLI	91.27	84.91	88.06	89.28	90.75	90.96
	QNLI	92.58	87.49	90.07	90.96	91.54	92.13

Table 1: Comparison of the testing accuracy across different baseline schemes.

Base Model	Target Acc.	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA	FedEx-LoRA
OPT-125M	80.00	$\mathbf{8.2\times 10^{6}}$	$3.78\times 10^{7}$	$2.46\times 10^{7}$	$1.59\times 10^{7}$	$2.10\times 10^{7}$	$1.25\times 10^{10}$
OPT-125M	85.00	$\mathbf{1.07\times 10^{7}}$	$-$	$4.2\times 10^{7}$	$2.75\times 10^{7}$	$3.61\times 10^{7}$	$1.77\times 10^{10}$
RoBERTa-large	80.00	$\mathbf{8.7\times 10^{6}}$	$4.68\times 10^{7}$	$3.02\times 10^{7}$	$1.85\times 10^{7}$	$2.59\times 10^{7}$	$2.08\times 10^{10}$
RoBERTa-large	85.00	$\mathbf{1.45\times 10^{7}}$	$8.12\times 10^{7}$	$5.17\times 10^{7}$	$3.35\times 10^{7}$	$4.42\times 10^{7}$	$2.96\times 10^{10}$

Table 2: Comparison of the total number of transmitted parameters (upload and download) to achieve the target accuracy. Symbol “

-

” means that the target accuracy cannot be achieved.

Base Model	FLoRG	MRPC	QQP	MNLI	QNLI	WNLI	RTE
OPT-125M	w/ Procrustes alignment	86.54	88.71	87.20	89.69	65.41	68.77
OPT-125M	w/o Procrustes alignment	83.14	85.85	80.93	86.72	59.81	64.32
RoBERTa-large	w/ Procrustes alignment	89.87	91.27	91.39	92.48	66.41	71.40
RoBERTa-large	w/o Procrustes alignment	86.50	88.43	88.93	88.62	62.07	67.09

Table 3: Comparison of the testing accuracy of FLoRG with and without Procrustes alignment.

Rank	Dataset	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA	FedEx-LoRA
$r=2$	WNLI	60.55	55.57	56.30	57.50	59.14	58.32
$r=2$	RTE	65.82	58.41	61.19	62.88	64.30	62.79
$r=4$	WNLI	66.34	59.19	61.97	63.55	63.61	66.12
$r=4$	RTE	71.41	64.11	66.97	68.62	70.10	71.28
$r=8$	WNLI	68.83	61.70	63.52	65.10	66.47	68.02
$r=8$	RTE	72.10	64.78	66.99	68.02	70.61	71.30

Table 4: Comparison of the testing accuracy under different ranks.

Non-IIDness	Dataset	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA	FedEx-LoRA
$\rho=0.1$	WNLI	60.12	53.07	54.21	56.14	57.74	58.84
$\rho=0.1$	RTE	65.30	55.60	59.19	61.20	60.75	61.67
$\rho=0.5$	WNLI	66.34	59.19	61.97	63.55	63.61	66.12
$\rho=0.5$	RTE	71.41	64.11	66.97	68.62	70.10	71.28
$\rho=1$	WNLI	67.83	61.70	63.52	64.33	65.61	68.31
$\rho=1$	RTE	72.21	66.90	68.71	70.40	71.78	72.55

Table 5: Comparison of the testing accuracy under different degrees of data heterogeneity.

Base Model	Initialization	MRPC	QQP	MNLI	QNLI	WNLI	RTE
OPT-125M	Semi-orthogonal	86.54	88.71	87.20	89.69	65.41	68.77
	Kaiming	84.35	88.90	85.23	87.73	62.29	69.31
	SVD	86.41	87.69	83.19	88.74	64.37	67.69
RoBERTa-large	Semi-orthogonal	89.87	91.27	91.39	92.48	66.41	71.40
	Kaiming	87.68	92.34	89.11	91.57	64.19	70.32
	SVD	88.70	90.37	91.49	91.45	65.08	71.40

Table 6: Comparison of the testing accuracy under different initialization schemes.

Client participation ratio	Dataset	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA	FedEx-LoRA
$0.2$	WNLI	58.42	54.15	55.20	56.30	57.10	56.46
$0.2$	RTE	63.25	57.80	59.45	60.92	61.85	61.35
$0.5$	WNLI	64.50	58.10	60.35	61.80	62.40	63.58
$0.5$	RTE	69.15	62.50	65.20	66.75	68.20	68.61
$1$	WNLI	66.34	59.19	61.97	63.55	63.61	66.12
$1$	RTE	71.41	64.11	66.97	68.62	70.10	71.28

Table 7: Comparison of the testing accuracy under different client participation ratios.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Metrics	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA	FedEx-LoRA
Client-side FLOPs	$1.18\times 10^{12}$	$1.24\times 10^{12}$	$1.24\times 10^{12}$	$\mathbf{8.25\times 10^{11}}$	$1.24\times 10^{12}$	$1.18\times 10^{12}$
Server-side FLOPs	$8.50\times 10^{10}$	“ $-$ ”	$2.22\times 10^{11}$	“ $-$ ”	“ $-$ ”	$2.09\times 10^{9}$
Memory Usage	2815	2117	2117	2110	2117	3117

Appendix

Table 8: Comparison of the per-round client-side computation overhead (in FLOPs), per-round server-side computation overhead (in FLOPs), and memory usage (in MB). Symbol “

-

” means that the overhead is negligible.

Base Model	Metric	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA	FedEx-LoRA
OPT-125M	EM	68.52	62.31	64.78	63.19	67.14	66.85
OPT-125M	F1	78.23	72.08	74.52	73.04	77.31	76.18
RoBERTa-large	EM	86.34	81.53	83.12	81.69	84.47	86.02
RoBERTa-large	F1	91.15	88.09	90.03	88.68	90.23	90.84
Llama-3.2-3B	EM	89.83	84.21	86.58	85.37	87.12	88.25
Llama-3.2-3B	F1	92.74	89.53	91.08	90.31	91.35	91.82

Appendix

Table 9: Comparison of the exact match (EM) and F1 scores on SQuAD v1.1.

Base Model	Dataset	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA
OPT-125M	MRPC	88.30	82.09	85.70	84.78	86.51
OPT-125M	QQP	89.72	84.18	87.09	86.30	87.92
RoBERTa-large	MRPC	90.80	85.16	88.30	87.50	89.21
RoBERTa-large	QQP	91.52	87.89	89.30	88.51	90.01

Appendix

Table 10: Additional comparison of the testing accuracy across different baseline schemes.

Rank	Dataset	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA
$r=2$	MRPC	85.91	79.66	83.17	82.32	83.00
	QQP	87.62	81.88	84.90	84.09	85.65
	MNLI	86.14	80.70	83.44	82.64	84.22
	QNLI	89.70	86.67	87.22	86.40	88.10
$r=4$	MRPC	90.80	85.16	88.30	87.50	89.21
	QQP	91.52	87.89	89.30	88.51	90.01

Appendix

Table 11: Additional comparison of the testing accuracy under different ranks.

Non-IIDness	Dataset	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA
$\rho=0.1$	MRPC	85.06	78.19	81.91	81.00	83.01
	QQP	86.95	78.80	84.15	83.21	85.00
	MNLI	81.35	75.44	78.42	77.57	79.36
	QNLI	88.88	83.31	86.20	85.33	87.19
$\rho=0.5$	MRPC	90.80	85.16	88.30	87.50	89.21
	QQP	91.52	87.89	89.30	88.51	90.01

Appendix

Table 12: Additional comparison of the testing accuracy under different degrees of data heterogeneity.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 746 papers

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
Sep 27, 2026Yanan Ma, Qiyuan Chen, Zihan Fang +2Federated Fine-Tuning Federated Learning
City University of Hong Kong · The University of Hong Kong
FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning
Mar 9, 2026Peishen Yan, Yang Hua, Hao Wang +4Federated Fine-Tuning Federated Learning Aggregation
Shanghai Jiao Tong University · Queen’s University Belfast · Stevens Institute of Technology +2
FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptation
May 28, 2026Zehao Wang, Guanglei Yang, Yihan Zeng +4Low-Rank Adaptation Federated Fine-Tuning
1Harbin Institute of Technology · 2Huawei Noah’s Ark Lab · University College Dublin

Sep 27, 2026J/K move · Enter open · S save

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

Yanan Ma, Qiyuan Chen, Zihan Fang, Xianhao Chen+1

City University of Hong Kong · The University of Hong Kong

Federated Fine-Tuning Federated Learning Low-Rank Adaptation Federated Learning Aggregation

Open paper AskPDF arXiv

Federated low-rank adaptation (LoRA) enables collaborative fine-tuning of large language models without centralizing private client data. Its factorized update, however, creates a structural mismatch in federated averaging: averaging the two LoRA factors separately does not equal averaging their products. Existing exact methods resolve this issue mainly by freezing an entire factor or alternating factors across rounds, but none can update factors simultaneously without aggregation errors or expanding communication ranks. To address this fundamental problem, we present CFLoRA, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round. By ensuring that columns and rows are complementary across factors, we eliminate bilinear terms in matrix multiplications, making federated aggregation exact. Crucially, our framework also supports clients with heterogeneous rank budgets. Convergence analysis validates CFLoRA achieves

\mathcal{O}(1/\sqrt{T})

convergence rate of the original LoRA objective in homogeneous-rank cases. Extensive experiments with RoBERTa on the GLUE benchmark and with LLaMA-3.2-3B-Instruct on commonsense reasoning tasks demonstrate that CFLoRA achieves superior performance and training efficiency compared to state-of-the-art federated LoRA baselines.

Abstract

LLM Fine-Tuning

Explore similar work

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning

FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptation