DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models

Jin Liu; Yinbin Miao; Ning Xi; Junkang Liu

cs.LGFeb 23, 2026

DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models

Authors: Jin Liu, Yinbin Miao, Ning Xi, Junkang Liu

Organizations: School of Cyber Engineering, Xidian University, Xi’an, China · College of Intelligence and Computing, Tianjin University, Tianjin, China

Abstract

Balancing convergence efficiency and robustness under Differential Privacy (DP) is a central challenge in Federated Learning (FL). While AdamW accelerates training and fine-tuning in large-scale models, we find that directly applying it to Differentially Private FL (DPFL) suffers from three major issues: (i) data heterogeneity and privacy noise jointly amplify the variance of second-moment estimator, (ii) DP perturbations bias the second-moment estimator, and (iii) DP amplify AdamW sensitivity to local overfitting, worsening client drift. We propose DP-FedAdamW, the first AdamW-based optimizer for DPFL. It restores AdamW under DP by stabilizing second-moment variance, removing DP-induced bias, and aligning local updates to the global descent to curb client drift. Theoretically, we establish an unbiased second-moment estimator and prove a linearly accelerated convergence rate without any heterogeneity assumption, while providing tighter $(\varepsilon,δ)$ -DP guarantees. Our empirical results demonstrate the effectiveness of DP-FedAdamW across language and vision Transformers and ResNet-18. On Tiny-ImageNet (Swin-Base, $\varepsilon=1$ ), DP-FedAdamW outperforms the state-of-the-art (SOTA) by 5.83%. The code is available in Appendix.

Figures & tables

Figure 1: Illustration of our DP-FedAdamW. It aggregates the mean of block-wise Bias-Corrected second moment estimates and performs local–global alignment to stabilize DPFL optimization.

Figure 2: An illustration of local update in DP-FedAdamW, which corrects client drift caused through global update guidance.

Method	DP-FedAvg	DP-SCAFFOLD	DP-FedSAM	DP-LocalAdamW	DP-FedAdamW
Optimizer Type	SGD	SGD	SAM	AdamW	AdamW
Scalability to Large Models	Limited	Limited	Moderate	Moderate	High
Client Drift	Strong	Weakened	Slightly weakened	Slightly weakened	Weakened
Communication Cost	1 $\times$	2 $\times$	2 $\times$	1 $\times$	1 $\times$

Table 1: Comparison with existing DPFL optimizers.

Panel of Figure 3 — Figure 3: Training on CIFAR-100, Swin-Tiny, $\sigma{=}1$ , $\alpha{=}0.1$ . (a) Non-IID (DPFL) causes high variance in second-moment estimator across clients of DP-LocalAdamW. (b) DP-LocalAdamW suffers from more severe client drift than FedAvg and LocalAdamW.

Panel of Figure 4 — Figure 4: Histogram for DP-LocalAdamW, CIFAR-10, Swin-Tiny, $\sigma{=}1$ , $\alpha{=}0.1$ . (a) The distribution centers of $\bm{m}^{t}$ are aligned with or without DP, but the variance is larger with DP. (b) The distribution of $\sqrt{\bm{v}^{t}}$ shows a significant difference, with the center of the distribution shifting approximately by $\sqrt{\sigma^{2}C^{2}/(sR)^{2}}$ .

Method	CIFAR-10 (ResNet-18)		CIFAR-100 (ResNet-18)		CIFAR-10 (ViT-Base)		CIFAR-100 (ViT-Base)		Privacy Budget
Method	$\alpha{=}0.6$	$\alpha{=}0.1$	$\alpha{=}0.6$	$\alpha{=}0.1$	$\alpha{=}0.6$	$\alpha{=}0.1$	$\alpha{=}0.6$	$\alpha{=}0.1$	Privacy Budget
DP-FedAvg	$71.69_{\pm 0.37}$	$60.38_{\pm 0.21}$	$31.13_{\pm 0.57}$	$25.04_{\pm 0.72}$	$90.51_{\pm 0.37}$	$78.85_{\pm 0.21}$	$45.86_{\pm 0.57}$	$13.41_{\pm 0.72}$	$(\varepsilon,\delta)$
DP-SCAFFOLD	$71.79_{\pm 0.44}$	$59.04_{\pm 0.39}$	$31.52_{\pm 0.36}$	$24.94_{\pm 0.58}$	$90.88_{\pm 0.52}$	$80.69_{\pm 0.64}$	$62.45_{\pm 0.57}$	$64.69_{\pm 0.76}$	$(\varepsilon,\delta)$
DP-FedAvg-LS	$72.27_{\pm 0.57}$	$60.84_{\pm 0.45}$	$35.91_{\pm 0.54}$	$28.68_{\pm 0.52}$	$91.48_{\pm 0.47}$	$88.48_{\pm 0.50}$	$67.76_{\pm 0.58}$	$65.85_{\pm 0.63}$	$(\varepsilon,\delta)$
DP-FedSAM	$67.39_{\pm 0.41}$	$57.04_{\pm 0.46}$	$30.36_{\pm 0.50}$	$24.03_{\pm 0.66}$	$91.76_{\pm 0.55}$	$89.35_{\pm 0.65}$	$70.24_{\pm 0.63}$	$66.27_{\pm 0.65}$	$(2\varepsilon,\delta)$
DP-LocalAdamW	$71.73_{\pm 0.47}$	$60.55_{\pm 0.53}$	$34.92_{\pm 0.69}$	$28.70_{\pm 0.81}$	$91.16_{\pm 0.47}$	$89.85_{\pm 0.53}$	$68.21_{\pm 0.69}$	$65.24_{\pm 0.81}$	$(\varepsilon,\delta)$

Table 2: Averaged test accuracy (%) and privacy budget comparison on CIFAR-10 and CIFAR-100 using ResNet-18 and ViT-Base. Assuming DP-FedAvg is

(\varepsilon,\delta)

-DP, all other methods share this budget except DP-FedSAM, which is approximately

(2\varepsilon,\delta)

-DP.

Method	CIFAR-100 (Swin-T)		CIFAR-100 (Swin-B)		Tiny-ImageNet (Swin-T)		Tiny-ImageNet (Swin-B)		Privacy Budget
Method	$\alpha{=}0.6$	$\alpha{=}0.1$	$\alpha{=}0.6$	$\alpha{=}0.1$	$\alpha{=}0.6$	$\alpha{=}0.1$	$\alpha{=}0.6$	$\alpha{=}0.1$	Privacy Budget
DP-FedAvg	$36.96_{\pm 0.50}$	$28.07_{\pm 0.31}$	$65.61_{\pm 0.27}$	$60.77_{\pm 0.32}$	$18.82_{\pm 0.94}$	$10.12_{\pm 1.36}$	$52.04_{\pm 0.66}$	$36.19_{\pm 0.74}$	$(\varepsilon,\delta)$
DP-SCAFFOLD	$41.68_{\pm 0.64}$	$33.52_{\pm 0.65}$	$68.03_{\pm 0.55}$	$65.34_{\pm 0.49}$	$21.91_{\pm 0.78}$	$14.54_{\pm 0.72}$	$53.78_{\pm 0.79}$	$43.55_{\pm 0.60}$	$(\varepsilon,\delta)$
DP-FedAvg-LS	$41.95_{\pm 0.53}$	$36.72_{\pm 0.65}$	$68.80_{\pm 0.52}$	$66.08_{\pm 0.56}$	$23.03_{\pm 0.64}$	$15.99_{\pm 0.68}$	$55.83_{\pm 0.66}$	$47.46_{\pm 0.57}$	$(\varepsilon,\delta)$
DP-FedSAM	$44.19_{\pm 0.48}$	$38.43_{\pm 0.62}$	$70.44_{\pm 0.57}$	$66.28_{\pm 0.64}$	$22.75_{\pm 0.65}$	$16.86_{\pm 0.77}$	$56.12_{\pm 0.74}$	$48.08_{\pm 0.62}$	$(2\varepsilon,\delta)$
DP-LocalAdamW	$42.04_{\pm 0.63}$	$35.51_{\pm 0.58}$	$68.33_{\pm 0.33}$	$65.28_{\pm 0.39}$	$23.41_{\pm 0.87}$	$15.81_{\pm 0.73}$	$54.58_{\pm 0.60}$	$46.52_{\pm 0.64}$	$(\varepsilon,\delta)$

Table 3: Averaged test accuracy (%) and privacy budget comparison on CIFAR-100 and Tiny-ImageNet using Swin-Tiny and Swin-Base.

Method	SST-2	QQP	QNLI	MNLI
DP-FedAvg	$49.08_{\pm 0.12}$	$63.18_{\pm 0.50}$	$49.46_{\pm 0.44}$	$35.45_{\pm 0.20}$
DP-SCAFFOLD	$52.35_{\pm 0.18}$	$66.87_{\pm 0.42}$	$52.62_{\pm 0.52}$	$38.74_{\pm 0.32}$
DP-FedAvg-LS	$68.38_{\pm 0.50}$	$72.59_{\pm 0.48}$	$69.22_{\pm 0.68}$	$50.46_{\pm 0.55}$
DP-FedSAM	$70.95_{\pm 0.62}$	$75.66_{\pm 0.53}$	$70.54_{\pm 0.63}$	$56.88_{\pm 0.56}$
DP-LocalAdamW	$91.04_{\pm 0.53}$	$82.33_{\pm 0.59}$	$86.07_{\pm 0.47}$	$75.20_{\pm 0.44}$
DP-FedAdamW	$\color[rgb]{1,0,0}{91.17}_{\pm 0.32}$	$\color[rgb]{1,0,0}{83.34}_{\pm 0.39}$	$\color[rgb]{1,0,0}{86.33}_{\pm 0.34}$	$\color[rgb]{1,0,0}{78.68}_{\pm 0.38}$

Table 4: Averaged test accuracy (%) on RoBERTa-Base.

Method	CIFAR-10			CIFAR-100			Tiny-ImageNet
Method	$\varepsilon{=}1$	$\varepsilon{=}2$	$\varepsilon{=}3$	$\varepsilon{=}1$	$\varepsilon{=}2$	$\varepsilon{=}3$	$\varepsilon{=}1$	$\varepsilon{=}2$	$\varepsilon{=}3$
DP-FedAvg	$66.52_{\pm 0.89}$	$72.63_{\pm 0.55}$	$88.02_{\pm 0.76}$	$19.66_{\pm 1.52}$	$42.98_{\pm 0.60}$	$57.73_{\pm 0.55}$	$20.65_{\pm 0.77}$	$37.48_{\pm 0.54}$	$45.32_{\pm 0.47}$
DP-SCAFFOLD	$66.39_{\pm 0.68}$	$72.81_{\pm 0.53}$	$88.07_{\pm 0.66}$	$22.38_{\pm 0.67}$	$44.52_{\pm 0.68}$	$59.85_{\pm 0.52}$	$20.93_{\pm 0.66}$	$38.40_{\pm 0.51}$	$47.06_{\pm 0.30}$
DP-FedAvg-LS	$71.57_{\pm 1.02}$	$75.02_{\pm 0.59}$	$88.96_{\pm 0.54}$	$25.77_{\pm 0.61}$	$49.54_{\pm 0.59}$	$62.46_{\pm 0.53}$	$26.89_{\pm 0.46}$	$46.57_{\pm 0.44}$	$48.83_{\pm 0.21}$
DP-FedSAM	$53.51_{\pm 1.39}$	$59.33_{\pm 0.86}$	$76.82_{\pm 0.97}$	$12.33_{\pm 0.64}$	$28.32_{\pm 0.49}$	$46.93_{\pm 0.50}$	$10.88_{\pm 0.95}$	$26.76_{\pm 0.78}$	$41.69_{\pm 0.45}$
DP-LocalAdamW	$66.57_{\pm 1.35}$	$73.04_{\pm 0.58}$	$88.83_{\pm 0.71}$	$24.69_{\pm 0.88}$	$47.26_{\pm 0.59}$	$62.18_{\pm 0.61}$	$28.40_{\pm 0.72}$	$46.62_{\pm 0.37}$	$48.55_{\pm 0.28}$

Table 5: Effect of privacy budgets (

\varepsilon

) on test accuracy (%) using Swin-Base,

\alpha=0.1

Panel of Figure 5 — Figure 5: Test accuracy (%) on CIFAR-100 using ResNet-18 and Swin-Tiny under the Dirichlet $\alpha=0.6$ and $\alpha=0.1$ settings.

Method	CIFAR-100	Tiny-ImageNet
DP-LocalAdamW	$65.28$	$46.52$
w/o Agg	$66.41$	$47.35$
w/o BC	$67.02$	$48.11$
w/o Align	$66.37$	$47.82$
DP-FedAdamW (full)	$\color[rgb]{1,0,0}{67.42}$	$\color[rgb]{1,0,0}{50.73}$

Table 6: Ablation on the main components of DP-FedAdamW on Swin-Base. We report test accuracy (%) on CIFAR-100 and Tiny-ImageNet under non-IID setting (

\alpha=0.1

). “Agg” denotes block-wise second-moment aggregation, “BC” the DP bias correction, and “Align” the local–global alignment term.

Strategy	CIFAR-100	Tiny-ImageNet	Comm(↑)
NoAgg	$66.21_{\pm 0.11}$	$49.12_{\pm 0.12}$	$5.7$ M
Agg- $\bm{v}$	$67.52_{\pm 0.12}$	$50.88_{\pm 0.12}$	$11.4$ M
Agg- mean - $\bm{v}$	$\color[rgb]{1,0,0}{67.45}_{\pm 0.10}$	$\color[rgb]{1,0,0}{50.76}_{\pm 0.10}$	5.7M

Table 7: Ablation study of moment aggregation strategies of DP-LocalAdamW on CIFAR-100 with Swin-Base under

\alpha{=}0.1

$\gamma$	$0.00$	$0.25$	$\color[rgb]{1,0,0}0.50$	$0.75$	$1.00$
CIFAR-100 (%)	$66.37$	$67.01$	$\color[rgb]{1,0,0}67.45$	$67.10$	$66.52$
Tiny-ImageNet (%)	$47.82$	$49.36$	$\color[rgb]{1,0,0}50.76$	$50.21$	$49.18$

Table 8: Impact of

\gamma

using Swin-Base (

\alpha{=}0.1

Method	CIFAR-100	Tiny-ImageNet
DP-FedOpt(Adam) [ 57 ]	${64.54}_{\pm 0.56}$	${44.81}_{\pm 0.62}$
DP-FAFED [ 69 ]	$65.75_{\pm 0.54}$	$46.10_{\pm 0.67}$
DP-FedLADA [ 65 ]	$66.03_{\pm 0.61}$	$47.85_{\pm 0.57}$
DP-FedAdamW (ours)	$\color[rgb]{1,0,0}{67.45}_{\pm 0.26}$	$\color[rgb]{1,0,0}{50.76}_{\pm 0.51}$

Table 9: Comparison with DP variants of federated adaptive optimizers under the same DP settings (Swin-Base,

\alpha{=}0.1

\sigma{=}1

Method	Swin-Tiny		Swin-Base		Privacy Budget
Method	$\alpha{=}0.6$	$\alpha{=}0.1$	$\alpha{=}0.6$	$\alpha{=}0.1$	Privacy Budget
DP-FedAvg	$82.93_{\pm 0.53}$	$78.24_{\pm 0.59}$	$89.56_{\pm 0.47}$	$88.23_{\pm 0.44}$	$(\varepsilon,\delta)$
DP-SCAFFOLD	$82.85_{\pm 0.50}$	$78.53_{\pm 0.54}$	$90.07_{\pm 0.48}$	$89.08_{\pm 0.62}$	$(\varepsilon,\delta)$
DP-FedAvg-LS	$83.94_{\pm 0.62}$	$79.67_{\pm 0.68}$	$90.97_{\pm 0.52}$	$90.14_{\pm 0.56}$	$(\varepsilon,\delta)$
DP-FedSAM	$84.05_{\pm 0.51}$	$80.36_{\pm 0.57}$	$91.25_{\pm 0.54}$	$89.82_{\pm 0.60}$	$(2\varepsilon,\delta)$
DP-LocalAdamW	$83.73_{\pm 0.58}$	$79.28_{\pm 0.64}$	$90.84_{\pm 0.39}$	$89.08_{\pm 0.41}$	$(\varepsilon,\delta)$

Table 10: Averaged test accuracy (%) comparison on CIFAR-10. Assuming DP-FedAvg is

(\varepsilon,\delta)

-DP, all other methods share this budget except DP-FedSAM, which is approximately

(2\varepsilon,\delta)

-DP.

Method	$\alpha=0.6$				$\alpha=0.1$
Method	$\sigma{=}0.5$	$\sigma{=}1$	$\sigma{=}1.5$	$\sigma{=}2$	$\sigma{=}0.5$	$\sigma{=}1$	$\sigma{=}1.5$	$\sigma{=}2$
DP-FedAvg	$67.23_{\pm 0.33}$	$65.61_{\pm 0.27}$	$63.78_{\pm 0.34}$	$60.22_{\pm 0.64}$	$62.49_{\pm 0.26}$	$60.77_{\pm 0.32}$	$57.85_{\pm 0.48}$	$54.34_{\pm 0.79}$
DP-SCAFFOLD	$70.41_{\pm 0.25}$	$68.03_{\pm 0.55}$	$64.64_{\pm 0.48}$	$63.14_{\pm 0.58}$	$66.72_{\pm 0.38}$	$65.34_{\pm 0.49}$	$61.73_{\pm 0.44}$	$58.23_{\pm 0.66}$
DP-FedAvg-LS	$70.93_{\pm 0.40}$	$68.80_{\pm 0.52}$	$66.52_{\pm 0.43}$	$64.77_{\pm 0.63}$	$68.36_{\pm 0.39}$	$66.08_{\pm 0.56}$	$62.87_{\pm 0.52}$	$59.98_{\pm 0.58}$
DP-FedSAM	$72.55_{\pm 0.22}$	$70.44_{\pm 0.57}$	$67.45_{\pm 0.33}$	$65.75_{\pm 0.69}$	$67.94_{\pm 0.51}$	$66.28_{\pm 0.64}$	$62.21_{\pm 0.60}$	$60.84_{\pm 0.71}$
DP-LocalAdamW	$70.74_{\pm 0.13}$	$68.33_{\pm 0.33}$	$66.46_{\pm 0.53}$	$64.10_{\pm 0.60}$	$67.35_{\pm 0.31}$	$65.28_{\pm 0.39}$	$60.55_{\pm 0.43}$	$58.78_{\pm 0.74}$

Table 11: Test accuracy (%) on CIFAR-100 with Swin-Base under different noise multipliers

\sigma

Dataset	#Classes	Image Size	Train / Test
CIFAR-10 [ 28 ]	10	$32\times 32$	50,000 / 10,000
CIFAR-100 [ 28 ]	100	$32\times 32$	50,000 / 10,000
Tiny-ImageNet [ 30 ]	200	$64\times 64$	100,000 / 10,000

Table 12: Summary of the vision datasets used in our experiments.

Dataset	Task Type	#Classes	Train/Test
SST-2	Sentiment Classification (binary)	2	67,349 / 1,821
QQP	Duplicate Question Detection (binary)	2	363,846 / 390,965
QNLI	Question-Answer NLI (binary)	2	104,743 / 5,463
MNLI	Natural Language Inference (entailment)	3	392,702 / 9,815

Table 13: Summary of GLUE datasets used in our experiments.

Model	Family	Depth	Stages	Width / Dim	Heads
ResNet-18 (GN)	CNN	18	$[2,2,2,2]$	$[64,128,256,512]$	–
ViT-Base	ViT Transformer	12	–	768 hidden, 3072 MLP	12 / layer
Swin-Tiny	Swin Transformer	12	$[2,2,6,2]$	$[96,192,384,768]$	$[3,6,12,24]$
Swin-Base	Swin Transformer	24	$[2,2,18,2]$	$[128,256,512,1024]$	$[4,8,16,32]$
RoBERTa-Base	NLP Transformer	12	–	768 hidden, 3072 FFN	12 / layer

Table 14: Overview of the model architectures used in our experiments. “Depth” denotes the total number of layers (blocks for ResNet/Swin, encoder layers for ViT/RoBERTa), and “Stages” denotes the number of blocks per stage for hierarchical models.

Dataset	Model	$N$	$T$	$K$	$l$	$sR$	$\sigma$	$\delta$	$r$ (LoRA)	$\alpha_{L}$ (LoRA)	Dropout
CIFAR-10	ResNet-18 (GN)	50	300	50	0.2	50	1.0	$10^{-5}$	16	32	0.1
CIFAR-10	ViT-Base	50	100	20	0.1	16	1.0	$10^{-5}$	16	32	0.1
	Swin-Tiny/Base	50	100	20	0.1	16	1.0	$10^{-5}$	16	32	0.1
CIFAR-100	ResNet-18 (GN)	50	300	50	0.2	50	1.0	$10^{-5}$	16	32	0.1
CIFAR-100	ViT-Base	50	100	20	0.1	16	1.0	$10^{-5}$	16	32	0.1
	Swin-Tiny/Base	50	100	20	0.1	16	1.0	$10^{-5}$	16	32	0.1

Table 15: Federated configurations for vision models across CIFAR-10/100 and Tiny-ImageNet. We report the number of clients

N

, communication rounds

T

, local steps

K

, client participation rate

l

, local batch size

sR

, DP noise multiplier

\sigma

, failure probability

\delta

, and LoRA hyperparameters

(r,\alpha_{L})

and dropout rate.

Dataset	Model	$N$	$T$	$K$	$l$	$sR$	$\sigma$	$\delta$	$r$ (LoRA)	$\alpha_{L}$ (LoRA)	Length_max	Dropout
SST-2	RoBERTa-Base	4	100	20	1.0	16	1.0	$10^{-5}$	16	32	128	0.1
QQP		4	100	20	1.0	16	1.0	$10^{-6}$	16	32	128	0.1
QNLI		4	100	20	1.0	16	1.0	$10^{-5}$	16	32	128	0.1
MNLI		4	100	20	1.0	16	1.0	$10^{-6}$	16	32	128	0.1

Table 16: Federated configurations for RoBERTa-Base with LoRA on GLUE tasks. We report the number of clients

N

, communication rounds

T

, local steps

K

, client participation rate

l

, local batch size

sR

, DP noise multiplier

\sigma

, failure probability

\delta

, and LoRA hyperparameters

(r,\alpha_{L})

, as well as the maximum sequence length and dropout rate.

Method	Local Optimizer	$\eta$	$\beta_{1}$	$\beta_{2}$	Weight Decay
DP-FedAvg	SGD	$10^{-1}$	-	-	0.001
DP-SCAFFOLD	SGD	$10^{-1}$	-	-	0.001
DP-FedAvg-LS	SGD	$10^{-1}$	-	-	0.001
DP-FedSAM	SAM	$10^{-1}$	-	-	0.001
DP-LocalAdamW	AdamW	$3{\times}10^{-4}$	0.9	0.999	0.01
DP-FedAdamW	AdamW	$3{\times}10^{-4}$	0.9	0.999	0.01

Table 17: Hyperparameter configuration of ResNet-18 (CIFAR-10/100) across different algorithms.

Differentially Private Federated Learning

Differentially private federated learning trains shared models across data-holding clients while limiting what updates reveal about individual records or users.

See all 64 papers

Feb 23, 2026

Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models

Fine-tuning large vision models (LVMs) and large language models (LLMs) under differentially private federated learning (DPFL) is hindered by a fundamental privacy-utility trade-off. Low-Rank Adaptation (LoRA), a promising parameter-efficient fine-tuning (PEFT) method, reduces computational and communication costs by introducing two trainable low-rank matrices while freezing pre-trained weights. However, directly applying LoRA in DPFL settings leads to performance degradation, especially in LVMs. Our analysis reveals three previously underexplored challenges: (1) gradient coupling caused by the simultaneous update of two asymmetric low-rank matrices, (2) compounded noise amplification under differential privacy, and (3) sharpness of the global aggregated model in the parameter space. To address these issues, we propose LA-LoRA (Local Alternating LoRA), a novel approach that decouples gradient interactions and aligns update directions across clients to enhance robustness under stringent privacy constraints. Theoretically, LA-LoRA strengthens convergence guarantees in noisy federated environments. Extensive experiments demonstrate that LA-LoRA achieves state-of-the-art (SOTA) performance on Swin Transformer and RoBERTa models, showcasing robustness to DP noise and broad applicability across both LVMs and LLMs. For example, when fine-tuning the Swin-B model on the Tiny-ImageNet dataset under a strict privacy budget (

ε= 1

), LA-LoRA outperforms the best baseline, RoLoRA, by 16.83% in test accuracy. Code is provided in \repolink.

Jin Liu, Yinbin Miao, Ning Xi +1

School of Cyber Engineering, Xidian University · College of Intelligence and Computing, Tianjin University

Federated Fine-Tuning Low-Rank Adaptation Differentially Private Federated Learning Parameter-Efficient Fine-Tuning LLM Fine-Tuning

Ask PDF

May 11, 2026

DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models

Federated learning (FL) enables the collaborative training of large-scale language models (LLMs) across edge devices while keeping user data on-device. However, FL still exposes sensitive information through client-provided gradients. Differentially private stochastic gradient descent (DP-SGD) mitigates this risk by clipping each client's contribution to a threshold

C

and adding noise proportional to

C

. Existing adaptive clipping techniques dynamically adjust

C

but demand tedious hyperparameter tuning, which can erode the privacy budget. In this paper, we introduce DP-LAC, a method that first estimates an initial clipping threshold within an order of magnitude of the optimum using private histogram estimation, and then adapts this threshold during training without consuming additional privacy budget or introducing new hyperparameters. Empirical results show that DP-LAC outperforms both state-of-the-art adaptive clipping methods and vanilla DP-SGD, achieving an average accuracy gain of

6.6\%

Haaris Mehmood, Jie Xu, Karthikeyan Saravanan +2

Samsung R&D Institute UK (SRUK) · Samsung AI Centre Cambridge

Privacy-Preserving Language Models Differentially Private Stochastic Gradient Descent LLM Fine-Tuning Differentially Private Federated Learning Federated Fine-Tuning Federated Learning

Ask PDF

May 7, 2026

Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning

Differentially private federated fine-tuning of large models with LoRA suffers from aggregation error caused by LoRA's multiplicative structure, which is further amplified by DP noise and degrades both stability and accuracy. Existing remedies apply a single update mode uniformly across all layers and all communication rounds (or alternate them on a fixed schedule), ignoring both the structural asymmetry between the two LoRA factors and the round-wise dynamics of training. We propose AS-LoRA, an adaptive framework defined by three axes (i) layer-wise freedom, in which each layer independently selects its active component, (ii) round-wise adaptivity, in which the selection updates over communication rounds, and (iii) a curvature-aware score derived from a second-order approximation of the loss. Theoretically, AS-LoRA eliminates the reconstruction-error floor of layer-tied schedules, accelerates convergence, implicitly biases solutions toward flatter minima, and incurs no additional privacy cost. Across GLUE, SQuAD, CIFAR-100, and Tiny-ImageNet under strict DP budgets and non-IID partitions, AS-LoRA improves over the federated LoRA baselines by up to

+7.5

pp on GLUE and

+12.5

pp on MNLI-mm for example, while matching or exceeding SVD-based aggregation methods at

33\text{--}180 \times

lower aggregation cost and with negligible communication overhead. Code for the proposed method is available at https://anonymous.4open.science/r/as_lora-F75F/.

Myoungjun Kim, Sangwoo Park, Yoseob Han +1

Myongji University · King’s College London · Soongsil University

Differentially Private Federated Learning Federated Learning Low-Rank Adaptation Federated Learning Aggregation Differential Privacy Federated Fine-Tuning+1

Ask PDF

Abstract

Differentially Private Federated Learning

Explore similar work

Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models

DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Models

Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning