FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment

Chuiyang Meng; Ming Tang; Vincent W. S. Wong

cs.LGFeb 19, 2026

FLoRG: Federated Fine-tuning with Low-rank Gram Matrices and Procrustes Alignment

Authors: Chuiyang Meng, Ming Tang, Vincent W. S. Wong

Organizations: The University of British Columbia · Southern University of Science and Technology

Abstract

Parameter-efficient fine-tuning techniques such as low-rank adaptation (LoRA) enable large language models (LLMs) to adapt to downstream tasks efficiently. Federated learning (FL) further facilitates this process by enabling collaborative fine-tuning across distributed clients without sharing private data. However, the use of two separate low-rank matrices in LoRA for federated fine-tuning introduces two types of challenges. First, aggregation error can arise from separately aggregating the two low-rank matrices. Second, even if the server aggregates the product of two low-rank matrices, it needs to decompose the aggregated matrix back into low-rank matrices. Since the decomposition is not unique, it can lead to decomposition drift. To tackle the aforementioned challenges, we propose federated low-rank Gram-matrix aggregation (FLoRG), a federated fine-tuning framework which employs a single low-rank matrix for fine-tuning and aggregates its Gram matrix (i.e., the matrix of inner products of its column vectors). FLoRG can eliminate the aggregation error and reduce the communication overhead. It also minimizes the decomposition drift by introducing a Procrustes alignment approach which aligns the decomposed matrix between consecutive fine-tuning rounds for consistent updates. We theoretically analyze the convergence of FLoRG and prove that adopting the Procrustes alignment results in a tighter convergence bound. Experimental results across multiple LLM fine-tuning benchmarks demonstrate that FLoRG outperforms five state-of-the-art baseline schemes by providing higher downstream task accuracy and can reduce the communication overhead by up to 2041 $\times$ .

Figures & tables

Figure 1: System model of our proposed FLoRG.

Base Model	Dataset	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA	FedEx-LoRA
OPT-125M	MNLI	87.35	79.42	81.15	83.54	84.61	85.83
	QNLI	89.52	84.18	86.71	87.93	88.69	89.88
	WNLI	65.28	58.45	59.34	62.61	62.83	64.15
	RTE	68.92	61.08	64.51	66.02	67.39	68.27
RoBERTa-large	MNLI	91.27	84.91	88.06	89.28	90.75	90.96
	QNLI	92.58	87.49	90.07	90.96	91.54	92.13

Table 1: Comparison of the testing accuracy across different baseline schemes.

Base Model	Target Acc.	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA	FedEx-LoRA
OPT-125M	80.00	$\mathbf{8.2\times 10^{6}}$	$3.78\times 10^{7}$	$2.46\times 10^{7}$	$1.59\times 10^{7}$	$2.10\times 10^{7}$	$1.25\times 10^{10}$
OPT-125M	85.00	$\mathbf{1.07\times 10^{7}}$	$-$	$4.2\times 10^{7}$	$2.75\times 10^{7}$	$3.61\times 10^{7}$	$1.77\times 10^{10}$
RoBERTa-large	80.00	$\mathbf{8.7\times 10^{6}}$	$4.68\times 10^{7}$	$3.02\times 10^{7}$	$1.85\times 10^{7}$	$2.59\times 10^{7}$	$2.08\times 10^{10}$
RoBERTa-large	85.00	$\mathbf{1.45\times 10^{7}}$	$8.12\times 10^{7}$	$5.17\times 10^{7}$	$3.35\times 10^{7}$	$4.42\times 10^{7}$	$2.96\times 10^{10}$

Table 2: Comparison of the total number of transmitted parameters (upload and download) to achieve the target accuracy. Symbol “

-

” means that the target accuracy cannot be achieved.

Base Model	FLoRG	MRPC	QQP	MNLI	QNLI	WNLI	RTE
OPT-125M	w/ Procrustes alignment	86.54	88.71	87.20	89.69	65.41	68.77
OPT-125M	w/o Procrustes alignment	83.14	85.85	80.93	86.72	59.81	64.32
RoBERTa-large	w/ Procrustes alignment	89.87	91.27	91.39	92.48	66.41	71.40
RoBERTa-large	w/o Procrustes alignment	86.50	88.43	88.93	88.62	62.07	67.09

Table 3: Comparison of the testing accuracy of FLoRG with and without Procrustes alignment.

Rank	Dataset	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA	FedEx-LoRA
$r=2$	WNLI	60.55	55.57	56.30	57.50	59.14	58.32
$r=2$	RTE	65.82	58.41	61.19	62.88	64.30	62.79
$r=4$	WNLI	66.34	59.19	61.97	63.55	63.61	66.12
$r=4$	RTE	71.41	64.11	66.97	68.62	70.10	71.28
$r=8$	WNLI	68.83	61.70	63.52	65.10	66.47	68.02
$r=8$	RTE	72.10	64.78	66.99	68.02	70.61	71.30

Table 4: Comparison of the testing accuracy under different ranks.

Non-IIDness	Dataset	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA	FedEx-LoRA
$\rho=0.1$	WNLI	60.12	53.07	54.21	56.14	57.74	58.84
$\rho=0.1$	RTE	65.30	55.60	59.19	61.20	60.75	61.67
$\rho=0.5$	WNLI	66.34	59.19	61.97	63.55	63.61	66.12
$\rho=0.5$	RTE	71.41	64.11	66.97	68.62	70.10	71.28
$\rho=1$	WNLI	67.83	61.70	63.52	64.33	65.61	68.31
$\rho=1$	RTE	72.21	66.90	68.71	70.40	71.78	72.55

Table 5: Comparison of the testing accuracy under different degrees of data heterogeneity.

Base Model	Initialization	MRPC	QQP	MNLI	QNLI	WNLI	RTE
OPT-125M	Semi-orthogonal	86.54	88.71	87.20	89.69	65.41	68.77
	Kaiming	84.35	88.90	85.23	87.73	62.29	69.31
	SVD	86.41	87.69	83.19	88.74	64.37	67.69
RoBERTa-large	Semi-orthogonal	89.87	91.27	91.39	92.48	66.41	71.40
	Kaiming	87.68	92.34	89.11	91.57	64.19	70.32
	SVD	88.70	90.37	91.49	91.45	65.08	71.40

Table 6: Comparison of the testing accuracy under different initialization schemes.

Client participation ratio	Dataset	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA	FedEx-LoRA
$0.2$	WNLI	58.42	54.15	55.20	56.30	57.10	56.46
$0.2$	RTE	63.25	57.80	59.45	60.92	61.85	61.35
$0.5$	WNLI	64.50	58.10	60.35	61.80	62.40	63.58
$0.5$	RTE	69.15	62.50	65.20	66.75	68.20	68.61
$1$	WNLI	66.34	59.19	61.97	63.55	63.61	66.12
$1$	RTE	71.41	64.11	66.97	68.62	70.10	71.28

Table 7: Comparison of the testing accuracy under different client participation ratios.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Metrics	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA	FedEx-LoRA
Client-side FLOPs	$1.18\times 10^{12}$	$1.24\times 10^{12}$	$1.24\times 10^{12}$	$\mathbf{8.25\times 10^{11}}$	$1.24\times 10^{12}$	$1.18\times 10^{12}$
Server-side FLOPs	$8.50\times 10^{10}$	“ $-$ ”	$2.22\times 10^{11}$	“ $-$ ”	“ $-$ ”	$2.09\times 10^{9}$
Memory Usage	2815	2117	2117	2110	2117	3117

Appendix

Table 8: Comparison of the per-round client-side computation overhead (in FLOPs), per-round server-side computation overhead (in FLOPs), and memory usage (in MB). Symbol “

-

” means that the overhead is negligible.

Base Model	Metric	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA	FedEx-LoRA
OPT-125M	EM	68.52	62.31	64.78	63.19	67.14	66.85
OPT-125M	F1	78.23	72.08	74.52	73.04	77.31	76.18
RoBERTa-large	EM	86.34	81.53	83.12	81.69	84.47	86.02
RoBERTa-large	F1	91.15	88.09	90.03	88.68	90.23	90.84
Llama-3.2-3B	EM	89.83	84.21	86.58	85.37	87.12	88.25
Llama-3.2-3B	F1	92.74	89.53	91.08	90.31	91.35	91.82

Appendix

Table 9: Comparison of the exact match (EM) and F1 scores on SQuAD v1.1.

Base Model	Dataset	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA
OPT-125M	MRPC	88.30	82.09	85.70	84.78	86.51
OPT-125M	QQP	89.72	84.18	87.09	86.30	87.92
RoBERTa-large	MRPC	90.80	85.16	88.30	87.50	89.21
RoBERTa-large	QQP	91.52	87.89	89.30	88.51	90.01

Appendix

Table 10: Additional comparison of the testing accuracy across different baseline schemes.

Rank	Dataset	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA
$r=2$	MRPC	85.91	79.66	83.17	82.32	83.00
	QQP	87.62	81.88	84.90	84.09	85.65
	MNLI	86.14	80.70	83.44	82.64	84.22
	QNLI	89.70	86.67	87.22	86.40	88.10
$r=4$	MRPC	90.80	85.16	88.30	87.50	89.21
	QQP	91.52	87.89	89.30	88.51	90.01

Appendix

Table 11: Additional comparison of the testing accuracy under different ranks.

Non-IIDness	Dataset	FLoRG	FedIT	FeDeRA	FFA-LoRA	FedSA-LoRA
$\rho=0.1$	MRPC	85.06	78.19	81.91	81.00	83.01
	QQP	86.95	78.80	84.15	83.21	85.00
	MNLI	81.35	75.44	78.42	77.57	79.36
	QNLI	88.88	83.31	86.20	85.33	87.19
$\rho=0.5$	MRPC	90.80	85.16	88.30	87.50	89.21
	QQP	91.52	87.89	89.30	88.51	90.01

Appendix

Table 12: Additional comparison of the testing accuracy under different degrees of data heterogeneity.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 746 papers

Sep 27, 2026

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

Federated low-rank adaptation (LoRA) enables collaborative fine-tuning of large language models without centralizing private client data. Its factorized update, however, creates a structural mismatch in federated averaging: averaging the two LoRA factors separately does not equal averaging their products. Existing exact methods resolve this issue mainly by freezing an entire factor or alternating factors across rounds, but none can update factors simultaneously without aggregation errors or expanding communication ranks. To address this fundamental problem, we present CFLoRA, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round. By ensuring that columns and rows are complementary across factors, we eliminate bilinear terms in matrix multiplications, making federated aggregation exact. Crucially, our framework also supports clients with heterogeneous rank budgets. Convergence analysis validates CFLoRA achieves

\mathcal{O}(1/\sqrt{T})

convergence rate of the original LoRA objective in homogeneous-rank cases. Extensive experiments with RoBERTa on the GLUE benchmark and with LLaMA-3.2-3B-Instruct on commonsense reasoning tasks demonstrate that CFLoRA achieves superior performance and training efficiency compared to state-of-the-art federated LoRA baselines.

Yanan Ma, Qiyuan Chen, Zihan Fang +2

City University of Hong Kong · The University of Hong Kong

Federated Fine-Tuning Federated Learning Low-Rank Adaptation Federated Learning Aggregation

Ask PDF

Mar 9, 2026

FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning

Federated fine-tuning of large language models (LLMs) with low-rank adaptation (LoRA) offers a communication-efficient and privacy-preserving solution for task-specific adaptation. Naive aggregation of LoRA modules introduces noise due to mathematical incorrectness when averaging the downsampling and upsampling matrices independently. However, existing noise-free aggregation strategies inevitably compromise the structural expressiveness of LoRA, limiting its ability to retain client-specific adaptations by either improperly reconstructing the low-rank structure or excluding partially trainable components. We identify this problem as loss of training momentum, where LoRA updates fail to accumulate effectively across rounds, resulting in slower convergence and suboptimal performance. To address this, we propose FedMomentum, a novel framework that enables structured and momentum-preserving LoRA aggregation via singular value decomposition (SVD). Specifically, after aggregating low-rank updates in a mathematically correct manner, FedMomentum applies SVD to extract the dominant components that capture the main update directions. These components are used to reconstruct the LoRA modules with the same rank, while residual components can be retained and later merged into the backbone to preserve semantic information and ensure robustness. Extensive experiments across multiple tasks demonstrate that FedMomentum consistently outperforms prior state-of-the-art methods in convergence speed and final accuracy.

Peishen Yan, Yang Hua, Hao Wang +4

Shanghai Jiao Tong University · Queen’s University Belfast · Stevens Institute of Technology +2

Federated Fine-Tuning Federated Learning Aggregation Federated Learning Low-Rank Adaptation

Ask PDF

May 28, 2026

FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptation

Federated fine-tuning of foundation models with Low-Rank Adaptation (LoRA) provides an efficient solution for reducing communication and computation costs while preserving data locality. However, the direct combination of FedAvg and LoRA suffers from three key issues: limited update space, which restricts the model's effective learning capacity; inter-round state mismatch, which disrupts cross-round local optimization continuity; and a client-agnostic starting state, which slows local convergence on clients. Although recent methods mitigate the limited update space issue by merging LoRA updates into the backbone across communication rounds, inter-round state mismatch and the client-agnostic starting state remain insufficiently addressed. To address these issues, we propose FedSmoothLoRA, a federated LoRA tuning framework that preserves the enlarged update space, improves cross-round local optimization continuity, and provides a client-aware starting state for local training. At each communication round, FedSmoothLoRA constructs the local LoRA initialization using two matrices: a Round-Matching matrix that preserves cross-round local state continuity, and a Gradient-Aligned matrix that provides client-specific optimization guidance from gradient signals estimated on local data. Together, these designs enable smoother and faster convergence. Extensive experiments on image classification and natural language generation tasks demonstrate that FedSmoothLoRA consistently outperforms existing federated LoRA tuning methods. Code: https://github.com/wangzehao0704/FedSmoothLoRA

Zehao Wang, Guanglei Yang, Yihan Zeng +4

1Harbin Institute of Technology · 2Huawei Noah’s Ark Lab · University College Dublin

Low-Rank Adaptation Federated Fine-Tuning Federated Learning Parameter-Efficient Fine-Tuning

Ask PDF

Abstract

LLM Fine-Tuning

Explore similar work

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning

FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptation