Parameter-efficient fine-tuning (PEFT) has been increasingly adopted to fine-tune pretrained models with low computational cost and is therefore also preferred in the federated learning paradigm to reduce computational and communication overhead. In this work, we investigate the sparsely explored problem of the risk of private data leakage in LoRA-adapted Vision Transformers (ViTs), where low-rank updates weaken the gradient signal available to a server with access to individual client updates. We show that quality of reconstruction is limited by two coupled ambiguities: reduced information in LoRA gradients and missing spatial cues from frozen positional embeddings. To address this, we propose a spatially anchored generative inversion framework that incorporates server-prescribed trainable positional embeddings into ViT blocks during fine-tuning at clients. The resulting positional gradients act as layer-wise spatial anchors, providing additional positional cues, while the generative prior keeps reconstructions on a realistic image manifold, reducing duplicate-data and patch-permutation artifacts, {which produce similar gradients as those produced when the target client images are used}. Adding our proposed spatial anchoring to GIAS improves PSNR by 10.14 dB, SSIM by 0.45, and LPIPS by 0.098 on CIFAR-10, as well as PSNR gains of 7.83 and 11.74 dB on high-resolution CelebA and CARS datasets, respectively, in the FedSGD setting. Such gains in reconstructed image quality metrics are also maintained at higher batch sizes and in FedAvg settings. These results indicate that LoRA-based PEFT does not, by itself, prevent data leakage in FL, and carefully incorporated positional embeddings can help in exposing private information.
Figures & tables
Figure 1: Conceptual illustration of duplicate sample issue in GIAs. Multiple local minima in the gradient-matching objective lead to reconstruction ambiguity, where multiple images yield nearly identical gradients to the target sample. All initializations and reconstructed images are real samples from our experiments.
Figure 2: Comparison of intrinsic dimensionality of the gradients obtained from full fine-tuning and the low-rank-adapted base model (ViT-Small), which supports our hypothesis that the dimensionality of the manifold on which the shared gradient lies decreases with reduction in the number of trainable parameters. The metrics are discussed in detail in Supp. Sec. A .
Figure 3: Examples of jumbled patches in both optimization and generative GIAs for different initializations on low rank-adapted ( r=128 ) ViT-Small on the CIFAR-10 dataset.
Figure 4: A high-level overview of our method showing how the baseline generative GIA framework works. gC and gS denote gradients from the client and server models.
Figure 5: Schematic of the proposed GIA. The overall optimization proceeds in two stages, from client gradients to image reconstruction and subsequent updates of the latent code and generator parameters. The gradients of the trainable positional embeddings enable improved reconstruction quality. g⋅s=∇⋅ℓ(y^,y)
Met.
APRIL
+Ours
Δ
GradViT
+Ours
Δ
GIAS
+Ours
Δ
P ↑
12.75 (2.19)
12.03 (2.39)
-0.72
15.03 (4.68)
16.62 (4.5)
+1.59
15.77 (4.52)
25.91 (6.29)
+10.14
S ↑
0.40 (0.152)
0.45 (0.11)
+0.05
0.397 (0.226)
0.48 (0.266)
+0.083
0.34 (0.221)
0.79 (0.17)
+0.45
L ↓
0.24 (0.077)
0.19 (0.224)
-0.05
0.124 (0.05)
0.12 (0.051)
-0.004
0.12 (0.075)
0.022 (0.033)
-0.098
Table 1: Performance on CIFAR-10 for Batch Size (BS) = 1. Δ indicates the change brought by (+Ours). In the case of LPIPS (L), a negative difference means improvement, while a positive difference indicates improvement for PSNR (P) and SSIM (S).
Figure 6: Qualitative comparison on the CIFAR-10 dataset. We can observe that the spatial information obscuration issue is solved after applying the proposed modification in the ViT-Small model.
Figure 7: Qualitative comparison for GIAS and GIAS+Ours for batch sizes 4 and 16 in the single-step setting with rank 128.
Figure 8: Qualitative comparison on CelebA and CARS datasets. We observe significant improvements in reconstruction quality with trainable positional embeddings applied to GIAS, resolving distorted patches in high-resolution images. Notably, instance-level identities of humans and cars are accurately retrieved, unlike competing techniques, highlighting the attack’s potential impact with minimal computational overhead.
Datasets
CelebA
CARS
Method
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
GIAS
11.71 (2.8)
0.24 (0.10)
0.31 (0.08)
9.04 (1.90)
0.07 (0.08)
0.36 (0.07)
GIAS +Ours
19.54 (3.72)
0.585 (0.125)
0.125 (0.082)
20.783 (3.166)
0.72 (0.128)
0.089 (0.058)
Table 2: Quantitative comparison on CelebA and CARS datasets. Evidently, we can observe the significant jump in performance when trainable positional embedding is applied to GIAS.
APRIL
GradViT
GIAS
E
B
Met.
w/o Ours
+Ours
Δ
w/o Ours
+Ours
Δ
w/o Ours
+Ours
Δ
P ↑
13.41 (0.585)
13.56 (0.44)
+0.15
14.01 (0.290)
13.91 (0.517)
-0.1
11.04 (0.648)
13.12 (1.58)
+2.08
1
8
S ↑
0.31 (0.028)
0.333 (0.02)
+0.023
0.243 (0.033)
0.243 (0.037)
0.0
0.075 (0.0173)
0.22 (0.105)
+0.15
L ↓
0.265 (0.013)
0.263 (0.014)
-2e-3
0.199 (0.018)
0.199 (0.022)
0.0
0.133 (0.014)
0.106 (0.024)
-0.027
P ↑
13.34 (0.583)
13.79 (0.352)
+0.45
14.09 (0.948)
13.86 (0.527)
-0.23
10.82 (0.557)
13.21 (1.59)
+2.39
1
4
S ↑
0.30 (2.5e-2)
0.33 (2.2e-2)
+0.03
0.257 (0.058)
0.244 (0.021)
-0.013
0.076 (0.014)
0.225 (0.095)
+0.15
Table 3: Evaluation of performance on CIFAR-10 for multiple local updates. Parentheses denote standard deviation. Δ tracks difference in performance. Metrics (Met.) P, S, and L refer to PSNR, SSIM, and LPIPS, respectively.
Figure 9: Performance comparison across multiple ranks r∈{16,32,64,128} for GIAS, GradViT and APRIL.
Figure 10: Performance comparison across multiple batch sizes BS∈{1,4,16,32} for GIAS, GradViT and APRIL.
Figure 11: Qualitative comparison of GIAS and GIAS+Ours under gradient-level defence strategies. Numbers within parentheses denote the standard deviation of the noise applied to the gradients.
Metric
GIAS
GIAS+Ours
N ( 0.01 )
N ( 0.1 )
GC
N + GC
N ( 0.01 )
N ( 0.1 )
GC
N + GC
P
13.11 (3.73)
9.86 (1.92)
13.9 (4.12)
9.72 (1.88)
14.19 (4.33)
9.28 (1.38)
25.39 (4.81)
9.31 (1.41)
S
0.23 (0.19)
0.04 (0.06)
0.30 (0.24)
0.05 (0.06)
0.31 (0.21)
0.03 (0.05)
0.77 (0.13)
0.03 (0.06)
L
0.13 (0.08)
0.17 (0.06)
0.13 (0.06)
0.15 (0.07)
0.11 (0.07)
0.19 (0.06)
0.02 (0.02)
0.19 (0.06)
Table 4: Quantitative comparison of GIAS and GIAS+Ours under gradient-level defence strategies. N , GC, P, S and L denote additive noise, gradient clipping, PSNR, SSIM and LPIPS, respectively. Numbers within brackets indicate the standard deviation.
Figure 12: Lower PSNR indicates reduced privacy leakage, while higher test accuracy reflects better model utility. PEFTLeak drops to zero in both attack and utility as the poisoned weight changes, and it fails to reconstruct any information. See Supp. Sec. R for SSIM and LPIPS plots.
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
Backbone
LoRA
PE Present (Layers)
PE Trainable (Layers)
Patch Size
Full Params Trainable
Accuracy (%) μ(σ)
1st
2nd - Lth
1st
2nd - Lth
Trained From Scratch
CIFAR-10
ViT
No
Yes
No
Yes
N/A
4
Yes
95.6 (0.1)
Ours
No
Yes
Yes
Yes
Yes
4
Yes
95.7 (0.2)
Finetuned on ImageNet Pre-trained weights
Appendix
Table 5: Comparison of the performance of the proposed backbone used in our experiments.
Method
GLR
PSNR ↑
SSIM ↑
LPIPS ↓
GIAS
False
15.22 (4.78)
0.313 (0.226)
0.094 (0.056)
True
15.77 (4.52)
0.340 (0.221)
0.120 (0.075)
p -value
0.649
0.642
0.134
GIAS + Ours
False
25.35 (8.00)
0.784 (0.235)
0.021 (0.033)
True
25.91 (6.29)
0.790 (0.170)
0.022 (0.033)
p -value
0.764
0.910
0.907
Appendix
Table 6: Effect of enabling group lazy regularization ( λG=10−4 ). Parentheses denote standard deviation. Welch’s t -test ( N=30 ) compares enabling group lazy regularization for both methods.
Figure 13: Visual comparison of reconstructions with and without group lazy regularization (GLR) using λG=10−4 . Results are shown for both the baseline GIAS and GIAS+Ours. Enabling GLR produces comparable reconstruction quality for the baseline while yielding slightly improved visual fidelity for GIAS+Ours, consistent with the quantitative results in Tab. 6 .
Method
λG
PSNR ↑
SSIM ↑
LPIPS ↓
GIAS
10−4
15.77 (4.52)
0.340 (0.221)
0.120 (0.075)
10−3
14.01 (3.29)
0.268 (0.164)
0.102 (0.052)
p -value
0.090
0.158
0.285
GIAS + Ours
10−4
25.91 (6.29)
0.790 (0.170)
0.022 (0.033)
10−3
27.64 (6.31)
0.850 (0.163)
0.011 (0.016)
p -value
0.292
0.168
0.108
Appendix
Table 7: Effect of the group lazy-regularization coefficient value when GLR optimization is enabled. Parentheses denote standard deviation. Welch’s t -test ( N=30 ) compares the two λG values.
Figure 14: Visual comparison of reconstructions obtained using different group lazy regularization coefficients, λG∈{10−4,10−3} . While a larger coefficient ( 10−3 ) produces slightly sharper reconstructions for GIAS+Ours, the qualitative differences are modest. We therefore adopt λG=10−4 in the remaining experiments to maintain identical optimization settings with the baseline GIAS.
Figure 15: Qualitative reconstruction for GradViT [ 23 ] comparison across 8 samples on the CIFAR-10 dataset. We can observe that the proposed framework also improves the spatial information obscuration issue for optimization-based gradient inversion attacks too, on the backbone ViT-Small model.
Figure 16: Qualitative reconstruction comparison for GIAS and GIAS+Ours for batch sizes 4 and 16 in the single-step gradient setting with rank 128.
Figure 17: Qualitative comparison on samples from the CelebA dataset with image resolution 128×128 .
CARS
CIFAR-100
Method
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
GIAS
9.04 (1.90)
0.07 (0.08)
0.357 (0.074)
13.90 (4.64)
0.307 (0.238)
0.099 (0.073)
GIAS+Ours
20.783 (3.166)
0.72 (0.128)
0.089 (0.058)
24.16 (5.293)
0.794 (0.201)
0.024 (0.041)
Appendix
Table 8: Quantitative comparison on CIFAR-100 and CARS datasets.
Figure 18: Qualitative comparison on the Stanford CARS datasets.
Figure 19: Qualitative comparison on the CIFAR-100 datasets.
E
B
Metric
APRIL
GradViT
GIAS
1
8
P
0.548
0.622
0.004
1
8
S
0.064
1.000
0.003
1
8
L
0.758
1.000
0.011
1
4
P
0.069
0.536
0.002
1
4
S
0.016
0.541
0.001
1
4
L
0.272
1.000
0.003
Appendix
Table 9: Statistical significance analysis of the improvements reported in Tab. 3 . P-values are computed using a two-sided Welch’s t-test over nine independent random initializations. Bold values denote statistically significant differences ( p<0.05 ). Metrics P, S, and L refer to PSNR, SSIM, and LPIPS, respectively.
Figure 20: Qualitative reconstruction comparison on the CelebA dataset. Top: Batch Size B=4 . Bottom: Batch Size B=16 .
Method
Batch Size
1
4
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
GIAS
12.48
0.300
0.198
11.00
0.192
0.238
GIAS+Ours
22.44
0.752
0.064
17.25
0.470
0.139
16
32
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
Appendix
Table 10: Comparison of performance for multiple batch sizes on 64×64 CelebA dataset.
Initialization
GIAS
GIAS+Ours
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Standard
13.48 (3.74)
0.24 (0.19)
0.126 (0.077)
23.47 (6.04)
0.77 (0.20)
0.025 (0.031)
Uniform
15.77 (4.52)
0.34 (0.221)
0.12 (0.075)
25.91 (6.29)
0.79 (0.17)
0.022 (0.033)
LoRA-One [ 57 ]
10.32 (1.62)
0.08 (0.05)
0.198 (0.064)
10.32 (1.44)
0.07 (0.05)
0.197 (0.064)
Appendix
Table 11: Effect of different initialization strategies on reconstruction performance of GIAS.
ViT Init.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
CIFAR
GIAS
15.08 (7.51)
0.356 (0.312)
0.187 (0.145)
GIAS+Ours
17.67 (9.195)
0.468 (0.382)
0.174 (0.161)
ImageNet
GIAS
15.77 (4.52)
0.34 (0.221)
0.12 (0.075)
GIAS+Ours
25.91 (6.29)
0.79 (0.17)
0.022 (0.033)
Appendix
Table 12: Ablation on the ViT backbone initialization. The generator initialization is kept fixed across all experiments.
Figure 21: Ablation on the effect of the number of steps in the server. Parentheses denote standard deviation. Δ tracks performance change from (s) to (m). ”Ep x BS y ” denotes that the optimization in the user was conducted for x epochs with a batch size of y , before the gradient was sent to the server.
Figure 22: Qualitative reconstruction comparison across 8 samples on the CIFAR-10 dataset with LoRA rank r=64 and r=32 . We can observe that the proposed method is effective even at lower ranks and is capable of solving the spatial information obscuration issue for the backbone ViT-Small model.
Batch Size 1
Batch Size 4
Method
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
GIAS
11.56 (1.49)
0.132 (0.063)
0.135 (0.029)
11.08 (0.598)
0.08 (0.022)
0.136 (0.012)
GIAS+Ours
19.44 (4.28)
0.593 (0.222)
0.046 (0.036)
13.516 (1.72)
0.247 (0.106)
0.1 (0.023)
Appendix
Table 13: Quantitative comparison on CIFAR-10 under Secure Aggregation. The proposed trainable positional embeddings clearly benefit reconstruction even under secure aggregation.
Figure 23: Qualitative reconstruction comparison for Secure Aggregation scenarios. Top: Batch Size B=1 (in each client). Bottom: Batch Size B=4 (in each client).
Method
Batch Size
1
4
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PEFTLeak
16.25
0.798
0.241
15.57
0.765
0.252
GIAS+Ours
25.91
0.79
0.022
20.97
0.628
0.042
16
32
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
Appendix
Table 14: Comparison of performance with PEFTLeak on the CIFAR-10 dataset for multiple batch sizes in the single-step update setting.
Figure 24: Qualitative evidence of reconstructions using LoRA gradients from ViT in SigLIP VLM client models.
Figure 25: Higher SSIM and lower LPIPS indicate better reconstruction. We can observe that as the FedAvg training progresses, the attack performance with no defence actually drops, while the LPIPS remains the same over the training duration. The behaviour of PEFTLeak remains the same as PSNR and Accuracy for both SSIM and LPIPS over the whole training duration.
Figure 26: Qualitative comparison of reconstruction on FEMNIST dataset between GIAS and GIAS + Ours.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
GIAS
11.31 (3.27)
0.180 (0.149)
0.211 (0.082)
GIAS + Ours
22.11 (5.31)
0.662 (0.201)
0.033 (0.045)
Appendix
Table 15: Comparison of performance on the FEMNIST dataset in the FedSGD setting.
Parameter-efficient fine-tuning (PEFT), such as low-rank adaptation (LoRA), has recently been adopted in federated learning to reduce communication and computation costs. In this setup, users download a pretrained model from the server prior to fine-tuning, and then fine-tune lightweight LoRA modules locally while keeping the pretrained model frozen, sharing only the gradients of the fine-tuning parameters with the server. Despite its growing popularity, robustness of federated fine-tuning against an adversarial server remains underexplored, where the server maliciously tampers with the training protocol to breach the privacy of users' data. In this work, we investigate gradient inversion attacks on LoRA fine-tuning. We propose an analytical attack that enables a malicious server to recover private user data by leveraging a poisoned pretrained model and fine-tuning parameters. Our design embeds fine-tuning data within the shared gradients, to allow the server to analytically reconstruct user data. Unlike prior works, our attack is applicable to both language and vision tasks, does not rely on computationally expensive (adversarial) pretraining with public datasets or require the number of training tokens to be less than the rank of LoRA modules. Experimental results on both language and vision tasks demonstrate high-fidelity data recovery across multiple baselines, revealing several critical vulnerabilities.
Fine-tuning large vision models (LVMs) and large language models (LLMs) under differentially private federated learning (DPFL) is hindered by a fundamental privacy-utility trade-off. Low-Rank Adaptation (LoRA), a promising parameter-efficient fine-tuning (PEFT) method, reduces computational and communication costs by introducing two trainable low-rank matrices while freezing pre-trained weights. However, directly applying LoRA in DPFL settings leads to performance degradation, especially in LVMs. Our analysis reveals three previously underexplored challenges: (1) gradient coupling caused by the simultaneous update of two asymmetric low-rank matrices, (2) compounded noise amplification under differential privacy, and (3) sharpness of the global aggregated model in the parameter space. To address these issues, we propose LA-LoRA (Local Alternating LoRA), a novel approach that decouples gradient interactions and aligns update directions across clients to enhance robustness under stringent privacy constraints. Theoretically, LA-LoRA strengthens convergence guarantees in noisy federated environments. Extensive experiments demonstrate that LA-LoRA achieves state-of-the-art (SOTA) performance on Swin Transformer and RoBERTa models, showcasing robustness to DP noise and broad applicability across both LVMs and LLMs. For example, when fine-tuning the Swin-B model on the Tiny-ImageNet dataset under a strict privacy budget (ε=1), LA-LoRA outperforms the best baseline, RoLoRA, by 16.83% in test accuracy. Code is provided in \repolink.
Jin Liu, Yinbin Miao, Ning Xi +1
School of Cyber Engineering, Xidian University · College of Intelligence and Computing, Tianjin University
Gradient inversion attacks are often presented as a serious privacy threat in federated learning, with recent work reporting increasingly strong reconstructions under favorable experimental settings. However, it remains unclear whether such attacks are feasible in modern, performance-optimized systems deployed in practice. In this work, we evaluate the practical feasibility of gradient inversion for image-based federated learning. We conduct a systematic study across multiple datasets and tasks, including image classification and object detection, using canonical vision architectures at contemporary resolutions. Our results show that while gradient inversion remains possible for certain legacy or transitional designs under highly restrictive assumptions, modern, performance-optimized models consistently resist meaningful reconstruction visually. We further demonstrate that many reported successes rely on upper-bound settings, such as inference mode operation or architectural simplifications which do not reflect realistic training pipelines. Taken together, our findings indicate that, under an honest-but-curious server assumption, high-fidelity image reconstruction via gradient inversion does not constitute a critical privacy risk in production-optimized federated learning systems, and that practical risk assessments must carefully distinguish diagnostic attack settings from real-world deployments.
Viktor Valadi, Lucas Beerens, Mattias Åkesson +4
Scaleout Systems Sweden · Recorded Future Sweden · AI Sweden Sweden +1