Breaking the Privacy Promise: Gradient Inversion Attacks on Federated Low-Rank Adapted Vision Transformers

Ritik K. Badiya; Siladittya Manna; Ashmit Sinha; M Yashwanth; Anirban Chakraborty

cs.CVNov 6, 2026

Breaking the Privacy Promise: Gradient Inversion Attacks on Federated Low-Rank Adapted Vision Transformers

Authors: Ritik K. Badiya, Siladittya Manna, Ashmit Sinha, M Yashwanth, Anirban Chakraborty

Organizations: Indian Institute of Science, Bengaluru, India

Abstract

Parameter-efficient fine-tuning (PEFT) has been increasingly adopted to fine-tune pretrained models with low computational cost and is therefore also preferred in the federated learning paradigm to reduce computational and communication overhead. In this work, we investigate the sparsely explored problem of the risk of private data leakage in LoRA-adapted Vision Transformers (ViTs), where low-rank updates weaken the gradient signal available to a server with access to individual client updates. We show that quality of reconstruction is limited by two coupled ambiguities: reduced information in LoRA gradients and missing spatial cues from frozen positional embeddings. To address this, we propose a spatially anchored generative inversion framework that incorporates server-prescribed trainable positional embeddings into ViT blocks during fine-tuning at clients. The resulting positional gradients act as layer-wise spatial anchors, providing additional positional cues, while the generative prior keeps reconstructions on a realistic image manifold, reducing duplicate-data and patch-permutation artifacts, {which produce similar gradients as those produced when the target client images are used}. Adding our proposed spatial anchoring to GIAS improves PSNR by 10.14 dB, SSIM by 0.45, and LPIPS by 0.098 on CIFAR-10, as well as PSNR gains of 7.83 and 11.74 dB on high-resolution CelebA and CARS datasets, respectively, in the FedSGD setting. Such gains in reconstructed image quality metrics are also maintained at higher batch sizes and in FedAvg settings. These results indicate that LoRA-based PEFT does not, by itself, prevent data leakage in FL, and carefully incorporated positional embeddings can help in exposing private information.

Figures & tables

Figure 1: Conceptual illustration of duplicate sample issue in GIAs. Multiple local minima in the gradient-matching objective lead to reconstruction ambiguity, where multiple images yield nearly identical gradients to the target sample. All initializations and reconstructed images are real samples from our experiments.

Figure 2: Comparison of intrinsic dimensionality of the gradients obtained from full fine-tuning and the low-rank-adapted base model (ViT-Small), which supports our hypothesis that the dimensionality of the manifold on which the shared gradient lies decreases with reduction in the number of trainable parameters. The metrics are discussed in detail in Supp. Sec. A .

Panel of Figure 3 — Figure 3: Examples of jumbled patches in both optimization and generative GIAs for different initializations on low rank-adapted ( $r=128$ ) ViT-Small on the CIFAR-10 dataset.

Figure 4: A high-level overview of our method showing how the baseline generative GIA framework works. $g^{C}$ and $g^{S}$ denote gradients from the client and server models.

Figure 5: Schematic of the proposed GIA. The overall optimization proceeds in two stages, from client gradients to image reconstruction and subsequent updates of the latent code and generator parameters. The gradients of the trainable positional embeddings enable improved reconstruction quality. $g^{s}_{\cdot}=\nabla_{\cdot}\ell(\hat{y},y)$

Met.	APRIL	+Ours	$\mathbf{\Delta}$	GradViT	+Ours	$\mathbf{\Delta}$	GIAS	+Ours	$\mathbf{\Delta}$
P $\uparrow$	12.75 (2.19)	12.03 (2.39)	-0.72	15.03 (4.68)	16.62 (4.5)	+1.59	15.77 (4.52)	25.91 (6.29)	+10.14
S $\uparrow$	0.40 (0.152)	0.45 (0.11)	+0.05	0.397 (0.226)	0.48 (0.266)	+0.083	0.34 (0.221)	0.79 (0.17)	+0.45
L $\downarrow$	0.24 (0.077)	0.19 (0.224)	-0.05	0.124 (0.05)	0.12 (0.051)	-0.004	0.12 (0.075)	0.022 (0.033)	-0.098

Table 1: Performance on CIFAR-10 for Batch Size (BS) = 1.

\Delta

indicates the change brought by (+Ours). In the case of LPIPS (L), a negative difference means improvement, while a positive difference indicates improvement for PSNR (P) and SSIM (S).

Panel of Figure 6 — Figure 6: Qualitative comparison on the CIFAR-10 dataset. We can observe that the spatial information obscuration issue is solved after applying the proposed modification in the ViT-Small model.

Panel of Figure 7 — Figure 7: Qualitative comparison for GIAS and GIAS+Ours for batch sizes 4 and 16 in the single-step setting with rank 128.

Panel of Figure 8 — Figure 8: Qualitative comparison on CelebA and CARS datasets. We observe significant improvements in reconstruction quality with trainable positional embeddings applied to GIAS, resolving distorted patches in high-resolution images. Notably, instance-level identities of humans and cars are accurately retrieved, unlike competing techniques, highlighting the attack’s potential impact with minimal computational overhead.

Datasets	CelebA			CARS
Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
GIAS	11.71 (2.8)	0.24 (0.10)	0.31 (0.08)	9.04 (1.90)	0.07 (0.08)	0.36 (0.07)
GIAS +Ours	19.54 (3.72)	0.585 (0.125)	0.125 (0.082)	20.783 (3.166)	0.72 (0.128)	0.089 (0.058)

Table 2: Quantitative comparison on CelebA and CARS datasets. Evidently, we can observe the significant jump in performance when trainable positional embedding is applied to GIAS.

			APRIL			GradViT			GIAS
E	B	Met.	w/o Ours	+Ours	$\mathbf{\Delta}$	w/o Ours	+Ours	$\mathbf{\Delta}$	w/o Ours	+Ours	$\mathbf{\Delta}$
		P $\uparrow$	13.41 (0.585)	13.56 (0.44)	+0.15	14.01 (0.290)	13.91 (0.517)	-0.1	11.04 (0.648)	13.12 (1.58)	+2.08
1	8	S $\uparrow$	0.31 (0.028)	0.333 (0.02)	+0.023	0.243 (0.033)	0.243 (0.037)	0.0	0.075 (0.0173)	0.22 (0.105)	+0.15
		L $\downarrow$	0.265 (0.013)	0.263 (0.014)	-2e-3	0.199 (0.018)	0.199 (0.022)	0.0	0.133 (0.014)	0.106 (0.024)	-0.027
		P $\uparrow$	13.34 (0.583)	13.79 (0.352)	+0.45	14.09 (0.948)	13.86 (0.527)	-0.23	10.82 (0.557)	13.21 (1.59)	+2.39
1	4	S $\uparrow$	0.30 (2.5e-2)	0.33 (2.2e-2)	+0.03	0.257 (0.058)	0.244 (0.021)	-0.013	0.076 (0.014)	0.225 (0.095)	+0.15

Table 3: Evaluation of performance on CIFAR-10 for multiple local updates. Parentheses denote standard deviation.

\Delta

tracks difference in performance. Metrics (Met.) P, S, and L refer to PSNR, SSIM, and LPIPS, respectively.

Figure 9: Performance comparison across multiple ranks

r\in\{16,32,64,128\}

for GIAS, GradViT and APRIL.

Figure 10: Performance comparison across multiple batch sizes

BS\in\{1,4,16,32\}

for GIAS, GradViT and APRIL.

Panel of Figure 11 — Figure 11: Qualitative comparison of GIAS and GIAS+Ours under gradient-level defence strategies. Numbers within parentheses denote the standard deviation of the noise applied to the gradients.

Metric	GIAS				GIAS+Ours
Metric	$\mathcal{N}$ ( $0.01$ )	$\mathcal{N}$ ( $0.1$ )	GC	$\mathcal{N}$ + GC	$\mathcal{N}$ ( $0.01$ )	$\mathcal{N}$ ( $0.1$ )	GC	$\mathcal{N}$ + GC
P	13.11 (3.73)	9.86 (1.92)	13.9 (4.12)	9.72 (1.88)	14.19 (4.33)	9.28 (1.38)	25.39 (4.81)	9.31 (1.41)
S	0.23 (0.19)	0.04 (0.06)	0.30 (0.24)	0.05 (0.06)	0.31 (0.21)	0.03 (0.05)	0.77 (0.13)	0.03 (0.06)
L	0.13 (0.08)	0.17 (0.06)	0.13 (0.06)	0.15 (0.07)	0.11 (0.07)	0.19 (0.06)	0.02 (0.02)	0.19 (0.06)

Table 4: Quantitative comparison of GIAS and GIAS+Ours under gradient-level defence strategies.

\mathcal{N}

, GC, P, S and L denote additive noise, gradient clipping, PSNR, SSIM and LPIPS, respectively. Numbers within brackets indicate the standard deviation.

Figure 12: Lower PSNR indicates reduced privacy leakage, while higher test accuracy reflects better model utility. PEFTLeak drops to zero in both attack and utility as the poisoned weight changes, and it fails to reconstruct any information. See Supp. Sec. R for SSIM and LPIPS plots.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	LoRA	PE Present (Layers)		PE Trainable (Layers)		Patch Size	Full Params Trainable	Accuracy (%) $\mu(\sigma)$
		$1^{\text{st}}$	$2^{\text{nd}}$ - $\mathrm{L}^{\text{th}}$	$1^{\text{st}}$	$2^{\text{nd}}$ - $\mathrm{L}^{\text{th}}$
Trained From Scratch
CIFAR-10
ViT	No	Yes	No	Yes	N/A	4	Yes	95.6 (0.1)
Ours	No	Yes	Yes	Yes	Yes	4	Yes	95.7 (0.2)
Finetuned on ImageNet Pre-trained weights

Appendix

Table 5: Comparison of the performance of the proposed backbone used in our experiments.

Method	GLR	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
GIAS	False	15.22 (4.78)	0.313 (0.226)	0.094 (0.056)
GIAS	True	15.77 (4.52)	0.340 (0.221)	0.120 (0.075)
$p$ -value		0.649	0.642	0.134
GIAS + Ours	False	25.35 (8.00)	0.784 (0.235)	0.021 (0.033)
GIAS + Ours	True	25.91 (6.29)	0.790 (0.170)	0.022 (0.033)
$p$ -value		0.764	0.910	0.907

Appendix

Table 6: Effect of enabling group lazy regularization (

\lambda_{G}=10^{-4}

). Parentheses denote standard deviation. Welch’s

t

-test (

N=30

) compares enabling group lazy regularization for both methods.

Panel of Figure 13 — Figure 13: Visual comparison of reconstructions with and without group lazy regularization (GLR) using $\lambda_{G}=10^{-4}$ . Results are shown for both the baseline GIAS and GIAS+Ours. Enabling GLR produces comparable reconstruction quality for the baseline while yielding slightly improved visual fidelity for GIAS+Ours, consistent with the quantitative results in Tab. 6 .

Method	$\lambda_{G}$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
GIAS	$10^{-4}$	15.77 (4.52)	0.340 (0.221)	0.120 (0.075)
GIAS	$10^{-3}$	14.01 (3.29)	0.268 (0.164)	0.102 (0.052)
$p$ -value		0.090	0.158	0.285
GIAS + Ours	$10^{-4}$	25.91 (6.29)	0.790 (0.170)	0.022 (0.033)
GIAS + Ours	$10^{-3}$	27.64 (6.31)	0.850 (0.163)	0.011 (0.016)
$p$ -value		0.292	0.168	0.108

Appendix

Table 7: Effect of the group lazy-regularization coefficient value when GLR optimization is enabled. Parentheses denote standard deviation. Welch’s

t

-test (

N=30

) compares the two

\lambda_{G}

values.

Panel of Figure 14 — Figure 14: Visual comparison of reconstructions obtained using different group lazy regularization coefficients, $\lambda_{G}\in\{10^{-4},10^{-3}\}$ . While a larger coefficient ( $10^{-3}$ ) produces slightly sharper reconstructions for GIAS+Ours, the qualitative differences are modest. We therefore adopt $\lambda_{G}=10^{-4}$ in the remaining experiments to maintain identical optimization settings with the baseline GIAS.

Panel of Figure 15 — Figure 15: Qualitative reconstruction for GradViT [ 23 ] comparison across 8 samples on the CIFAR-10 dataset. We can observe that the proposed framework also improves the spatial information obscuration issue for optimization-based gradient inversion attacks too, on the backbone ViT-Small model.

Panel of Figure 16 — Figure 16: Qualitative reconstruction comparison for GIAS and GIAS+Ours for batch sizes 4 and 16 in the single-step gradient setting with rank 128.

Panel of Figure 17 — Figure 17: Qualitative comparison on samples from the CelebA dataset with image resolution $128\times 128$ .

	CARS			CIFAR-100
Method	PSNR	SSIM	LPIPS	PSNR	SSIM	LPIPS
GIAS	9.04 (1.90)	0.07 (0.08)	0.357 (0.074)	13.90 (4.64)	0.307 (0.238)	0.099 (0.073)
GIAS+Ours	20.783 (3.166)	0.72 (0.128)	0.089 (0.058)	24.16 (5.293)	0.794 (0.201)	0.024 (0.041)

Appendix

Table 8: Quantitative comparison on CIFAR-100 and CARS datasets.

Panel of Figure 18 — Figure 18: Qualitative comparison on the Stanford CARS datasets.

Panel of Figure 19 — Figure 19: Qualitative comparison on the CIFAR-100 datasets.

$E$	$B$	Metric	APRIL	GradViT	GIAS
1	8	P	0.548	0.622	0.004
1	8	S	0.064	1.000	0.003
1	8	L	0.758	1.000	0.011
1	4	P	0.069	0.536	0.002
1	4	S	0.016	0.541	0.001
1	4	L	0.272	1.000	0.003

Appendix

Table 9: Statistical significance analysis of the improvements reported in Tab. 3 . P-values are computed using a two-sided Welch’s t-test over nine independent random initializations. Bold values denote statistically significant differences (

p<0.05

). Metrics P, S, and L refer to PSNR, SSIM, and LPIPS, respectively.

Panel of Figure 20 — Figure 20: Qualitative reconstruction comparison on the CelebA dataset. Top: Batch Size $B=4$ . Bottom: Batch Size $B=16$ .

Method	Batch Size
	1			4
	PSNR	SSIM	LPIPS	PSNR	SSIM	LPIPS
GIAS	12.48	0.300	0.198	11.00	0.192	0.238
GIAS+Ours	22.44	0.752	0.064	17.25	0.470	0.139
	16			32
	PSNR	SSIM	LPIPS	PSNR	SSIM	LPIPS

Appendix

Table 10: Comparison of performance for multiple batch sizes on

64\times 64

CelebA dataset.

Initialization	GIAS			GIAS+Ours
Initialization	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Standard	13.48 (3.74)	0.24 (0.19)	0.126 (0.077)	23.47 (6.04)	0.77 (0.20)	0.025 (0.031)
Uniform	15.77 (4.52)	0.34 (0.221)	0.12 (0.075)	25.91 (6.29)	0.79 (0.17)	0.022 (0.033)
LoRA-One [ 57 ]	10.32 (1.62)	0.08 (0.05)	0.198 (0.064)	10.32 (1.44)	0.07 (0.05)	0.197 (0.064)

Appendix

Table 11: Effect of different initialization strategies on reconstruction performance of GIAS.

ViT Init.	Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
CIFAR	GIAS	15.08 (7.51)	0.356 (0.312)	0.187 (0.145)
CIFAR	GIAS+Ours	17.67 (9.195)	0.468 (0.382)	0.174 (0.161)
ImageNet	GIAS	15.77 (4.52)	0.34 (0.221)	0.12 (0.075)
ImageNet	GIAS+Ours	25.91 (6.29)	0.79 (0.17)	0.022 (0.033)

Appendix

Table 12: Ablation on the ViT backbone initialization. The generator initialization is kept fixed across all experiments.

Figure 21: Ablation on the effect of the number of steps in the server. Parentheses denote standard deviation.

\Delta

tracks performance change from (s) to (m). ”Ep

x

y

” denotes that the optimization in the user was conducted for

x

epochs with a batch size of

y

, before the gradient was sent to the server.

Panel of Figure 22 — Figure 22: Qualitative reconstruction comparison across 8 samples on the CIFAR-10 dataset with LoRA rank $r=64$ and $r=32$ . We can observe that the proposed method is effective even at lower ranks and is capable of solving the spatial information obscuration issue for the backbone ViT-Small model.

	Batch Size 1			Batch Size 4
Method	PSNR	SSIM	LPIPS	PSNR	SSIM	LPIPS
GIAS	11.56 (1.49)	0.132 (0.063)	0.135 (0.029)	11.08 (0.598)	0.08 (0.022)	0.136 (0.012)
GIAS+Ours	19.44 (4.28)	0.593 (0.222)	0.046 (0.036)	13.516 (1.72)	0.247 (0.106)	0.1 (0.023)

Appendix

Table 13: Quantitative comparison on CIFAR-10 under Secure Aggregation. The proposed trainable positional embeddings clearly benefit reconstruction even under secure aggregation.

Panel of Figure 23 — Figure 23: Qualitative reconstruction comparison for Secure Aggregation scenarios. Top: Batch Size $B=1$ (in each client). Bottom: Batch Size $B=4$ (in each client).

Method	Batch Size
	1			4
	PSNR	SSIM	LPIPS	PSNR	SSIM	LPIPS
PEFTLeak	16.25	0.798	0.241	15.57	0.765	0.252
GIAS+Ours	25.91	0.79	0.022	20.97	0.628	0.042
	16			32
	PSNR	SSIM	LPIPS	PSNR	SSIM	LPIPS

Appendix

Table 14: Comparison of performance with PEFTLeak on the CIFAR-10 dataset for multiple batch sizes in the single-step update setting.

Panel of Figure 24 — Figure 24: Qualitative evidence of reconstructions using LoRA gradients from ViT in SigLIP VLM client models.

Figure 25: Higher SSIM and lower LPIPS indicate better reconstruction. We can observe that as the FedAvg training progresses, the attack performance with no defence actually drops, while the LPIPS remains the same over the training duration. The behaviour of PEFTLeak remains the same as PSNR and Accuracy for both SSIM and LPIPS over the whole training duration.

Panel of Figure 26 — Figure 26: Qualitative comparison of reconstruction on FEMNIST dataset between GIAS and GIAS + Ours.

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
GIAS	11.31 (3.27)	0.180 (0.149)	0.211 (0.082)
GIAS + Ours	22.11 (5.31)	0.662 (0.201)	0.033 (0.045)

Appendix

Table 15: Comparison of performance on the FEMNIST dataset in the FedSGD setting.

Gradient Inversion Attacks

Gradient inversion attacks infer private training examples or labels from gradients shared during distributed model training.

See all 22 papers

MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning
Aug 2, 2026Hasin Us Sami, Swapneel Sen, Basak GulerGradient Inversion Attacks Privacy Leakage in Language Models
University of California, Riverside, CA
Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models
Feb 23, 2026Jin Liu, Yinbin Miao, Ning Xi +1Federated Fine-Tuning Low-Rank Adaptation
School of Cyber Engineering, Xidian University · College of Intelligence and Computing, Tianjin University
Practical Feasibility of Gradient Inversion Attacks in Federated Learning
Aug 27, 2025Viktor Valadi, Lucas Beerens, Mattias Åkesson +4Gradient Inversion Attacks Adversarial Attacks
Scaleout Systems Sweden · Recorded Future Sweden · AI Sweden Sweden +1

Aug 2, 2026J/K move · Enter open · S save

MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning

Hasin Us Sami, Swapneel Sen, Basak Guler

University of California, Riverside, CA

Gradient Inversion Attacks Privacy Leakage in Language Models Federated Fine-Tuning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Parameter-efficient fine-tuning (PEFT), such as low-rank adaptation (LoRA), has recently been adopted in federated learning to reduce communication and computation costs. In this setup, users download a pretrained model from the server prior to fine-tuning, and then fine-tune lightweight LoRA modules locally while keeping the pretrained model frozen, sharing only the gradients of the fine-tuning parameters with the server. Despite its growing popularity, robustness of federated fine-tuning against an adversarial server remains underexplored, where the server maliciously tampers with the training protocol to breach the privacy of users' data. In this work, we investigate gradient inversion attacks on LoRA fine-tuning. We propose an analytical attack that enables a malicious server to recover private user data by leveraging a poisoned pretrained model and fine-tuning parameters. Our design embeds fine-tuning data within the shared gradients, to allow the server to analytically reconstruct user data. Unlike prior works, our attack is applicable to both language and vision tasks, does not rely on computationally expensive (adversarial) pretraining with public datasets or require the number of training tokens to be less than the rank of LoRA modules. Experimental results on both language and vision tasks demonstrate high-fidelity data recovery across multiple baselines, revealing several critical vulnerabilities.

Abstract

Gradient Inversion Attacks

Explore similar work

MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning

Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models

Practical Feasibility of Gradient Inversion Attacks in Federated Learning