Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models

Jin Liu; Yinbin Miao; Ning Xi; Junkang Liu

cs.LGFeb 23, 2026

Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models

Authors: Jin Liu, Yinbin Miao, Ning Xi, Junkang Liu

Organizations: School of Cyber Engineering, Xidian University · College of Intelligence and Computing, Tianjin University

Abstract

Fine-tuning large vision models (LVMs) and large language models (LLMs) under differentially private federated learning (DPFL) is hindered by a fundamental privacy-utility trade-off. Low-Rank Adaptation (LoRA), a promising parameter-efficient fine-tuning (PEFT) method, reduces computational and communication costs by introducing two trainable low-rank matrices while freezing pre-trained weights. However, directly applying LoRA in DPFL settings leads to performance degradation, especially in LVMs. Our analysis reveals three previously underexplored challenges: (1) gradient coupling caused by the simultaneous update of two asymmetric low-rank matrices, (2) compounded noise amplification under differential privacy, and (3) sharpness of the global aggregated model in the parameter space. To address these issues, we propose LA-LoRA (Local Alternating LoRA), a novel approach that decouples gradient interactions and aligns update directions across clients to enhance robustness under stringent privacy constraints. Theoretically, LA-LoRA strengthens convergence guarantees in noisy federated environments. Extensive experiments demonstrate that LA-LoRA achieves state-of-the-art (SOTA) performance on Swin Transformer and RoBERTa models, showcasing robustness to DP noise and broad applicability across both LVMs and LLMs. For example, when fine-tuning the Swin-B model on the Tiny-ImageNet dataset under a strict privacy budget ( $ε= 1$ ), LA-LoRA outperforms the best baseline, RoLoRA, by 16.83% in test accuracy. Code is provided in \repolink.

Figures & tables

Figure 1: The illustration of DP-LoRA, FFA-LoRA, RoLoRA, and LA-LORA. DP-LoRA updates both noisy $A$ and $B$ simultaneously and sends them to the server for aggregation. FFA-LoRA freezes $A$ , updates only the noisy $B$ , and sends it to the server. RoLoRA alternately updates noisy $A$ and $B$ across rounds. Our LA-LoRA alternately updates noisy $A$ and $B$ within each local round.

Method	DP	LVMs	Exp.	Speed
DP-LoRA	✓	✕	mid	slow
FFA-LoRA	✓	✕	low	slow
RoLoRA	✕	✕	mid	mid
LA-LoRA	✓	✓	high	fast

Table 1: Comparison of LoRA-based methods in DPFL. ✓ denotes support, ✕ denotes not considered. Exp. indicates the effective expression ability under DP.

Panel of Figure 2 — Figure 2: Comparison of cosine similarity between $\nabla_{A}\mathcal{L}$ and $\nabla_{B}\mathcal{L}$ , test loss and test accuracy for Swin-T on CIFAR-100 ( $\epsilon=3$ ). LA-LoRA(-filter) uses local alternating updates without smoothing.

Figure 3: Scaling of perturbation Frobenius norms with $\sigma$ on QNLI.

Panel of Figure 4 — Figure 4: Comparison of global loss landscapes for fine-tuning Swin-T model on CIFAR-100.

Method	CIFAR-100	Tiny-ImageNet
DP-LoRA $\epsilon{=}\infty$	42.45	44.80
LA-LoRA $\epsilon{=}\infty$	30.12	33.25
DP-LoRA $\epsilon{=}1$	101.62	115.36
LA-LoRA $\epsilon{=}1$	64.77	69.53

Table 2: Maximum Hessian eigenvalue of Swin-B on CIFAR-100 and Tiny-ImageNet.

Panel of Figure 6 — Figure 6: Test accuracy of Swin-T and Swin-B on CIFAR-100 and Tiny-ImageNet with $\epsilon=1$ .

Privacy Budget	Method	Swin-T Model (%)		Swin-B Model (%)
Privacy Budget	Method	CIFAR-100	Tiny-ImageNet	CIFAR-100	Tiny-ImageNet
$\epsilon=3$	DP-LoRA	$45.40_{\pm 0.40}$	$32.27_{\pm 0.91}$	$56.52_{\pm 0.51}$	$30.64_{\pm 0.30}$
	FFA-LoRA	$52.09_{\pm 0.36}$	$44.62_{\pm 0.55}$	$62.10_{\pm 0.39}$	$39.84_{\pm 0.19}$
	RoLoRA	$55.19_{\pm 0.42}$	$50.87_{\pm 0.56}$	$67.96_{\pm 0.64}$	$44.18_{\pm 0.66}$
	LA-LoRA	$\mathbf{60.07}_{\pm 0.41}$	$\mathbf{60.97}_{\pm 0.44}$	$\mathbf{75.29}_{\pm 0.35}$	$\mathbf{61.97}_{\pm 0.56}$
$\epsilon=2$	DP-LoRA	$44.82_{\pm 0.57}$	$32.14_{\pm 1.39}$	$56.31_{\pm 0.28}$	$30.31_{\pm 0.55}$

Table 3: Test accuracy of Swin-T and Swin-B on CIFAR-100 and Tiny-ImageNet for different

\epsilon

Privacy Budget	Method	SST-2	QNLI	QQP	MNLI
$\epsilon=3$	DP-LoRA	$92.36_{\pm 0.75}$	$86.31_{\pm 0.32}$	$84.56_{\pm 0.83}$	$80.98_{\pm 0.44}$
	FFA-LoRA	$92.32_{\pm 0.49}$	$87.20_{\pm 0.37}$	$85.12_{\pm 0.34}$	$81.71_{\pm 0.69}$
	RoLoRA	$92.70_{\pm 0.52}$	$88.23_{\pm 0.49}$	$85.35_{\pm 0.42}$	$82.12_{\pm 0.44}$
	LA-LoRA	$\mathbf{93.12}_{\pm 0.67}$	$\mathbf{89.83}_{\pm 0.41}$	$\mathbf{85.83}_{\pm 0.49}$	$\mathbf{82.99}_{\pm 0.42}$
$\epsilon=2$	DP-LoRA	$92.20_{\pm 0.64}$	$86.03_{\pm 0.57}$	$84.26_{\pm 0.57}$	$80.62_{\pm 0.35}$
	FFA-LoRA	$92.39_{\pm 0.51}$	$87.30_{\pm 0.60}$	$84.73_{\pm 0.47}$	$81.94_{\pm 0.58}$

Table 4: Test accuracy (%) of RoBERTa-Base on SST-2, QNLI, QQP and MNLI under different

\epsilon

Method	GLUE tasks (RoBERTa-Base)				Image Classification (Swin-B)
Method	SST-2	QNLI	QQP	MNLI	CIFAR-100	Tiny-ImageNet
DP-LoRA	$92.36_{\pm 0.75}$	$86.31_{\pm 0.32}$	$84.56_{\pm 0.83}$	$80.98_{\pm 0.44}$	$56.52_{\pm 0.51}$	$30.64_{\pm 0.30}$
DP-LoRA(+filter)	$92.52_{\pm 0.55}$	$87.06_{\pm 0.66}$	$84.79_{\pm 0.42}$	$81.43_{\pm 0.57}$	$69.08_{\pm 0.52}$	$49.85_{\pm 0.55}$
LA-LoRA(-filter)	$92.74_{\pm 0.67}$	$88.92_{\pm 0.50}$	$84.98_{\pm 0.51}$	$82.40_{\pm 0.53}$	$70.38_{\pm 0.48}$	$53.07_{\pm 0.60}$
LA-LoRA	$\mathbf{93.12}_{\pm 0.67}$	$\mathbf{89.83}_{\pm 0.41}$	$\mathbf{85.83}_{\pm 0.49}$	$\mathbf{82.99}_{\pm 0.42}$	$\mathbf{75.29}_{\pm 0.35}$	$\mathbf{61.97}_{\pm 0.56}$

Table 5: Impact of local alternating updates and low-pass smoothing filter on federated LoRA performance (%) across GLUE (RoBERTa-Base) and image classification (Swin-B) benchmarks.

Strategy	Dirichlet $\beta=0.6$		Dirichlet $\beta=0.1$
Strategy	CIFAR-100	Tiny-ImageNet	CIFAR-100	Tiny-ImageNet
1-step $B$ / 1-step $A$ (default)	$\mathbf{89.62}_{\pm 0.37}$	$\mathbf{80.77}_{\pm 0.50}$	$75.29_{\pm 0.35}$	$\mathbf{61.97}_{\pm 0.56}$
2-step $B$ / 2-step $A$	$89.29_{\pm 0.47}$	$80.58_{\pm 0.39}$	$75.15_{\pm 0.38}$	$61.36_{\pm 0.42}$
5-step $B$ / 5-step $A$	$89.31_{\pm 0.42}$	$80.72_{\pm 0.40}$	$\mathbf{75.47}_{\pm 0.46}$	$61.78_{\pm 0.51}$
5-step $A$ / 5-step $B$	$89.45_{\pm 0.26}$	$80.70_{\pm 0.42}$	$75.34_{\pm 0.44}$	$61.79_{\pm 0.58}$

Table 6: Effect of different local alternating strategies on CIFAR-100 and Tiny-ImageNet (Swin-B) under

\epsilon=3

and different Dirichlet distributions. Strategy “

k

-step

B

k

-step

A

” denotes performing

k

local gradient steps on

B

followed by

k

steps on

A

in each local round.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Test accuracy of Swin-T and Swin-B on CIFAR-100 and Tiny-ImageNet with $\epsilon=2$ .

Panel of Figure 8 — Figure 8: Test accuracy of Swin-T and Swin-B on CIFAR-100 and Tiny-ImageNet with $\epsilon=3$ .

Method	Time Cost (s)		Memory Cost (MB)		Test Accuracy (%)
Method	CIFAR-100	Tiny-ImageNet	CIFAR-100	Tiny-ImageNet	CIFAR-100	Tiny-ImageNet
DP-LoRA	$30.35$	$28.02$	$3524$	$3524$	$55.98$	$30.20$
DP-LoRA(+filter)	$30.72$	$28.51$	$3524$	$3524$	$67.95$	$48.09$
FFA-LoRA	$17.85$	$16.54$	$1762$	$1762$	$61.94$	$39.33$
RoLoRA	$16.64$	$16.32$	$1762$	$1762$	$67.88$	$43.85$
LA-LoRA(-filter)	$17.30$	$17.16$	$1762$	$1762$	$69.87$	$52.72$

Appendix

Table 7: Per-round computation cost, memory cost and performance comparison of Swin-B at

\epsilon=1

Rank	Method	CIFAR-100	Tiny-ImageNet
$r=8$	DP-LoRA	$54.30_{\pm 0.52}$	$29.57_{\pm 0.53}$
	FFA-LoRA	$59.78_{\pm 0.44}$	$36.58_{\pm 0.35}$
	RoLoRA	$65.74_{\pm 0.61}$	$40.89_{\pm 0.56}$
	LA-LoRA	$\mathbf{72.88}_{\pm 0.57}$	$\mathbf{59.03}_{\pm 0.45}$
$r=16$	DP-LoRA	$55.98_{\pm 0.56}$	$30.20_{\pm 0.46}$
	FFA-LoRA	$61.94_{\pm 0.37}$	$39.33_{\pm 0.48}$

Appendix

Table 8: Impact of rank of Swin-B on CIFAR-100 and Tiny-ImageNet at

\epsilon=1

, averaged over 5 runs.

Setting	Method	Test Acc. (%)	$\Delta$ Acc	Grad. Cos. (late)	$\Delta$ Cos
Centralized	DP-LoRA	$76.11_{\pm 0.38}$	-	0.681	-
	LA-LoRA(-filter)	$77.29_{\pm 0.43}$	$\uparrow$ 1.18	0.713	$\uparrow$ 0.032
Federated	DP-LoRA	$45.40_{\pm 0.40}$	-	0.337	-
	LA-LoRA(-filter)	$56.62_{\pm 0.54}$	$\uparrow$ 11.22	0.445	$\uparrow$ 0.108

Appendix

Table 9: Centralized and federated DP training for Swin-T on CIFAR-100 with

\epsilon=3

. “Grad. Cos. (late)” denotes the average cosine similarity between

\nabla_{A}\mathcal{L}

and

\nabla_{B}\mathcal{L}

over the last 10% of training steps. Federated DP-LoRA has lower test accuracy and gradient cosine than centralized DP-LoRA, while LA-LoRA(-filter) improves both settings.

	Method	Test Acc. (%)	$\Delta$ Acc	Grad. Cos. (late)	$\Delta$ Cos
Non-private	Fed-LoRA	$90.56_{\pm 0.22}$	-	0.694	-
Non-private	LA-LoRA(-filter)	$91.25_{\pm 0.15}$	$\uparrow$ 0.69	0.783	$\uparrow$ 0.089

Appendix

Table 10: Non-private federated training for Swin-T on CIFAR-100. “Grad. Cos. (late)” denotes the average cosine similarity between

\nabla_{A}\mathcal{L}

and

\nabla_{B}\mathcal{L}

over the last 10% of training steps.

	Method	CIFAR-100	Tiny-ImageNet
Non-private	DP-LoRA	$83.21$	$81.37$
	FFA-LoRA	$81.65$	$80.76$
	RoLoRA	$83.25$	$81.03$
	LA-LoRA	$\mathbf{84.21}$	$\mathbf{82.58}$

Appendix

Table 11: Non-private federated training for Swin-B on CIFAR-100 and Tiny-ImageNet.

Panel of Figure 9 — Figure 9: Test accuracy of RoBERTa-Base on SST-2, QNLI, QQP, and MNLI with $\epsilon=1$ .

Panel of Figure 10 — Figure 10: Test accuracy of RoBERTa-Base on SST-2, QNLI, QQP, and MNLI with $\epsilon=2$ .

Panel of Figure 11 — Figure 11: Test accuracy of RoBERTa-Base on SST-2, QNLI, QQP, and MNLI with $\epsilon=3$ .

Model	Method	SST-2	QNLI	QQP	MNLI	Avg
Llama-2-7B	DP-LoRA	$91.56$	$88.22$	$85.56$	$86.86$	$88.05$
	FFA-LoRA	$92.53$	$89.23$	$85.56$	$86.98$	$88.58$
	RoLoRA	$92.12$	$89.34$	$85.98$	$87.21$	$88.66$
	LA-LoRA	$93.36$	$89.78$	$86.75$	$87.56$	$89.36$

Appendix

Table 12: Test accuracy of Llama-2-7B on SST-2, QNLI, QQP and MNLI with

\epsilon=1

Privacy	Method	SST-2	QNLI	QQP	MNLI	Avg.
Non-private	DP-LoRA	92.07	86.25	84.02	81.22	85.89
	FFA-LoRA	92.56	87.53	85.36	81.54	86.75
	RoLoRA	93.65	88.65	85.52	82.22	87.51
	LA-LoRA	93.94	89.96	86.41	83.32	88.41
$\epsilon=1$	DP-LoRA	90.13	83.79	83.28	79.80	84.25
	FFA-LoRA	90.62	84.63	84.10	80.98	85.08

Appendix

Table 13: Test accuracy(%) of RoBERTa-Base on SST-2, QNLI, QQP and MNLI, Dirichlet

\beta=0.3

Method	CIFAR-100	Tiny-ImageNet
DP-LoRA	$45.40_{\pm 0.40}$	$32.27_{\pm 0.91}$
DP-LoRA(+filter)	$55.75_{\pm 0.63}$	$50.69_{\pm 0.67}$
LA-LoRA(-filter)	$56.62_{\pm 0.54}$	$52.18_{\pm 0.37}$
LA-LoRA	$\mathbf{60.07}_{\pm 0.41}$	$\mathbf{60.97}_{\pm 0.44}$

Appendix

Table 14: Effect of local alternating updates and low-pass smoothing filter for Swin-T.

Method	GLUE tasks (RoBERTa-Base)		Image Classification (Swin-B)
Method	QQP	MNLI	CIFAR-100	Tiny-ImageNet
DP-LoRA	$84.56_{\pm 0.83}$	$80.98_{\pm 0.44}$	$56.52_{\pm 0.51}$	$30.64_{\pm 0.30}$
DP-LoRA(+filter)	$84.79_{\pm 0.42}$	$81.43_{\pm 0.57}$	$69.08_{\pm 0.52}$	$49.85_{\pm 0.55}$
DP-LoRA(+Doppler)	$84.82_{\pm 0.49}$	$81.04_{\pm 0.57}$	$66.12_{\pm 0.71}$	$48.53_{\pm 0.57}$
LA-LoRA(-filter)	$84.98_{\pm 0.51}$	$82.40_{\pm 0.53}$	$70.38_{\pm 0.48}$	$53.07_{\pm 0.60}$
LA-LoRA	$\mathbf{85.83}_{\pm 0.49}$	$\mathbf{82.99}_{\pm 0.42}$	$\mathbf{75.29}_{\pm 0.35}$	$\mathbf{61.97}_{\pm 0.56}$

Appendix

Table 15: Impact of different low-pass filters on federated LoRA performance (%) across GLUE and image classification benchmarks,

\epsilon=3

Method	GLUE tasks (RoBERTa-Base)		Image Classification (Swin-B)
Method	QQP	$\lambda_{\max}(H)$	CIFAR-100	$\lambda_{\max}(H)$
DP-LoRA	$84.02$	$43.74\phantom{{}_{(\downarrow 00.00)}}$	$55.98$	$101.62\phantom{{}_{(\downarrow 00.00)}}$
DP-LoRA(+filter)	$85.63$	$41.36_{(\downarrow 2.38)}$	$67.95$	$80.33_{(\downarrow 21.29)}$
LA-LoRA(-filter)	$85.95$	$40.82_{(\downarrow 2.92)}$	$69.87$	$64.77_{(\downarrow 36.85)}$
LA-LoRA	$\mathbf{86.41}$	$40.22_{(\downarrow 3.52)}$	$\mathbf{74.56}$	$55.76_{(\downarrow 45.86)}$

Appendix

Table 16: Ablation of LA-LoRA components and the three challenges (noise amplification, gradient coupling, and loss sharpness) on GLUE (

\beta=0.3

) and image classification (

\beta=0.1

\epsilon=1

. We report test accuracy (%) and the maximum Hessian eigenvalue

\lambda_{\max}(H)

on LoRA parameters.

$K$	$\sigma$	10	20	30	50
CIFAR-100(%)	$\sigma=0.56$	$72.13$	$74.56$	$75.21$	$75.26$
Tiny-ImageNet(%)	$\sigma=0.283$	$58.54$	$60.68$	$61.24$	$61.32$

Appendix

Table 17: LA-LoRA test accuracy (%) with different local steps

K

(Swin-B).

Smoothing parameter $\sigma_{s}$	GLUE tasks (RoBERTa-Base)				Image Classification (Swin-B)
Smoothing parameter $\sigma_{s}$	SST-2	QNLI	QQP	MNLI	CIFAR-100	Tiny-ImageNet
$\sigma_{s}=0.000$	$92.51_{\pm 0.45}$	$87.95_{\pm 0.56}$	$84.72_{\pm 0.49}$	$81.83_{\pm 0.55}$	$70.34_{\pm 0.32}$	$54.05_{\pm 0.52}$
$\sigma_{s}=0.001$	$\mathbf{92.66}_{\pm 0.47}$	$88.73_{\pm 0.42}$	$\mathbf{85.34}_{\pm 0.35}$	$82.35_{\pm 0.46}$	$71.80_{\pm 0.26}$	$56.14_{\pm 0.45}$
$\sigma_{s}=0.005$	$92.60_{\pm 0.44}$	$\mathbf{88.75}_{\pm 0.53}$	$85.26_{\pm 0.42}$	$\mathbf{82.42}_{\pm 0.55}$	$72.67_{\pm 0.49}$	$58.75_{\pm 0.38}$
$\sigma_{s}=0.010$	$92.62_{\pm 0.47}$	$87.97_{\pm 0.42}$	$85.03_{\pm 0.48}$	$82.03_{\pm 0.32}$	$\mathbf{74.56}_{\pm 0.52}$	$60.68_{\pm 0.55}$
$\sigma_{s}=0.050$	$92.50_{\pm 0.49}$	$88.02_{\pm 0.54}$	$84.96_{\pm 0.36}$	$82.03_{\pm 0.57}$	$74.33_{\pm 0.40}$	$\mathbf{61.27}_{\pm 0.56}$

Appendix

Table 18: Impact of the smoothing parameter

\sigma_{s}

on federated LoRA performance (%) across GLUE (RoBERTa-Base) language tasks and image classification (Swin-B) benchmarks (

\epsilon=1

Kernel	Size	Coefficients	CIFAR-100	Tiny-ImageNet
$G_{s}^{(3)}$	3	$\tfrac{1}{4}[1,2,1]$	$73.59_{\pm 0.60}$	$59.92_{\pm 0.51}$
$G_{s}^{(5)}$	5	$\tfrac{1}{16}[1,4,6,4,1]$	$\mathbf{74.56}_{\pm 0.52}$	$60.68_{\pm 0.55}$
$G_{S}^{(7)}$	7	$\tfrac{1}{64}[1,6,15,20,15,6,1]$	$73.88_{\pm 0.46}$	$\mathbf{60.74}_{\pm 0.63}$

Appendix

Table 19: Sensitivity of LA-LoRA to the choice of 1D binomial kernel size. Results are reported for Swin-B on CIFAR-100 and Tiny-ImageNet at

\epsilon=1

Notation	Description
$W_{0}\in\mathbb{R}^{m\times n}$	Pre-trained backbone weight matrix
$A,B$	LoRA down-projection/up-projection matrices
$r$	LoRA rank, $r\ll\min\{m,n\}$
$\alpha$	LoRA scaling hyperparameter
$s=\alpha/r$	LoRA scaling factor
$\nabla_{A}\mathcal{L},\ \nabla_{B}\mathcal{L}$	Gradients of loss w.r.t. $A$ and $B$

Appendix

Table 20: Summary of the main notations.

Federated Fine-Tuning

Federated fine-tuning adapts pretrained models across decentralized data holders without pooling their private data.

See all 115 papers

May 7, 2026

Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning

Differentially private federated fine-tuning of large models with LoRA suffers from aggregation error caused by LoRA's multiplicative structure, which is further amplified by DP noise and degrades both stability and accuracy. Existing remedies apply a single update mode uniformly across all layers and all communication rounds (or alternate them on a fixed schedule), ignoring both the structural asymmetry between the two LoRA factors and the round-wise dynamics of training. We propose AS-LoRA, an adaptive framework defined by three axes (i) layer-wise freedom, in which each layer independently selects its active component, (ii) round-wise adaptivity, in which the selection updates over communication rounds, and (iii) a curvature-aware score derived from a second-order approximation of the loss. Theoretically, AS-LoRA eliminates the reconstruction-error floor of layer-tied schedules, accelerates convergence, implicitly biases solutions toward flatter minima, and incurs no additional privacy cost. Across GLUE, SQuAD, CIFAR-100, and Tiny-ImageNet under strict DP budgets and non-IID partitions, AS-LoRA improves over the federated LoRA baselines by up to

+7.5

pp on GLUE and

+12.5

pp on MNLI-mm for example, while matching or exceeding SVD-based aggregation methods at

33\text{--}180 \times

lower aggregation cost and with negligible communication overhead. Code for the proposed method is available at https://anonymous.4open.science/r/as_lora-F75F/.

Myoungjun Kim, Sangwoo Park, Yoseob Han +1

Myongji University · King’s College London · Soongsil University

Differentially Private Federated Learning Federated Learning Low-Rank Adaptation Federated Learning Aggregation Differential Privacy Federated Fine-Tuning+1

Ask PDF

May 11, 2026

DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models

Federated learning (FL) enables the collaborative training of large-scale language models (LLMs) across edge devices while keeping user data on-device. However, FL still exposes sensitive information through client-provided gradients. Differentially private stochastic gradient descent (DP-SGD) mitigates this risk by clipping each client's contribution to a threshold

C

and adding noise proportional to

C

. Existing adaptive clipping techniques dynamically adjust

C

but demand tedious hyperparameter tuning, which can erode the privacy budget. In this paper, we introduce DP-LAC, a method that first estimates an initial clipping threshold within an order of magnitude of the optimum using private histogram estimation, and then adapts this threshold during training without consuming additional privacy budget or introducing new hyperparameters. Empirical results show that DP-LAC outperforms both state-of-the-art adaptive clipping methods and vanilla DP-SGD, achieving an average accuracy gain of

6.6\%

Haaris Mehmood, Jie Xu, Karthikeyan Saravanan +2

Samsung R&D Institute UK (SRUK) · Samsung AI Centre Cambridge

Privacy-Preserving Language Models Differentially Private Stochastic Gradient Descent LLM Fine-Tuning Differentially Private Federated Learning Federated Fine-Tuning Federated Learning

Ask PDF

May 20, 2026

Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment

Low-rank adaptation (LoRA) has emerged as a powerful tool for parameter-efficient fine-tuning of large language models (LLMs). This paper studies LoRA under a federated learning setting, enabling collaborative fine-tuning across clients while preserving parameter efficiency. We focus on a highly heterogeneous regime in which clients share only partial structure and a substantial subset may be contaminated. We propose Collaborative Low-rank Alignment and Identifiable Recovery (CLAIR), a contamination-aware framework that relies only on preliminary local estimators. Its formulation applies broadly, from linear regression to neural network and LLM modules, whenever local adaptation can be represented by matrix-valued updates. CLAIR recovers the shared LoRA subspace and detects contaminated clients via a structured low-rank plus block-sparse decomposition. We prove exact recovery of the shared LoRA subspace in the noiseless case, stable recovery under preliminary estimation error, and consistent collaborative-set recovery under mild separation conditions. We further quantify the gain from CLAIR refinement: it reduces off-subspace estimation error through cross-client averaging while preserving client-specific variation within the shared LoRA subspace, thus improves over local fine-tuning whenever this oracle gain outweighs the costs of subspace estimation and benign-client heterogeneity. Empirically, we demonstrate the benefits of CLAIR by fine-tuning a Transformer architecture on a text-copying task. The results show accurate contamination detection and improved benign-client performance compared with local fine-tuning and non-robust federated averaging.

Shuaida He, Liwen Chen, Long Feng

School of Computing & Data Science, The University of Hong Kong

LLM Fine-Tuning Low-Rank Adaptation Federated Fine-Tuning Federated Learning Heterogeneous Federated Learning

Ask PDF

Abstract

Federated Fine-Tuning

Explore similar work

Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning

DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models

Federated LoRA Fine-Tuning for LLMs via Collaborative Alignment