VeilTune: Secure Fine-Tuning of Private LoRA Adapters on Public Pretrained Language Models
Authors: Huiyu Xie, Tanping Zhou, Tao Hu, Xinlei Liu, Hailong Ma
Organizations: Information Engineering University, Zhengzhou, China · Key Laboratory of Cyberspace Endogenous Safety & Security of Henan Province, Zhengzhou, China · Chinese People’s Armed Police Force Engineering University, Xi’an, China
Fine-tuning public pretrained language models with lightweight adapters such as LoRA has become the mainstream way to adapt them to downstream tasks. A service provider often trains its adapter on a data owner's samples, so two assets need protection: the data owner's samples and the provider's adapter. Existing secure fine-tuning protocols protect the samples but reveal to the data owner intermediate results of every training step, such as the hidden representations or logits of its inputs. We show that a single step suffices to copy the adapter: from the hidden representations of one mini-batch of 32 samples, the data owner fits an adapter that agrees with the provider's on 99.2% of test inputs, and logits alone yield about 94%. Protecting both assets therefore requires that no value that depends on the adapter be revealed throughout training. We present VeilTune, to the best of our knowledge the first protocol that fine-tunes a pretrained language model with backpropagation entirely in secure multi-party computation (MPC). VeilTune computes only what the loss depends on, replaces generic MPC operators with training-aware nonlinear operators that stay accurate on the values that training produces, and obliviously skips the steps corrupted by truncation failures. On four few-shot tasks with RoBERTa-large, VeilTune nearly matches plaintext training and is 5.3 points more accurate on average than SecP-Tuning, the state-of-the-art secure fine-tuning protocol. On SST-2, it achieves a 3.5x end-to-end speedup and a 3.6x reduction in communication over SecP-Tuning, and 26x and 17x over full fine-tuning in MPC.
Figures & tables
Figure 1. Attack performance (agreement with θT , % ↑ ) of AEA against existing protocols, compared with the local model, on four tasks (mean of three seeds). “+act.” marks the active attacker, the other groups are passive; SecP is SecP-Tuning.
Figure 2. Workflow of VeilTune and one secure training step. All values inside the middle box are secret-shared, and the Transformer block is repeated for the L−s adapted blocks.
Figure 3. A secure training step built from generic operators. (a) Communication and rounds by operator. (b) Arguments of three generic operators during fine-tuning; the shaded band is the range on which the operator is accurate.
Figure 4. Token packing for a batch of three examples. M marks the [MASK] token; hatched cells are computed but wasted. (a) Padding. (b) Packed tokens for the per-token layers. (c) Attention scores over all pairs of packed tokens (left) and within each example after regrouping (right).
SST-2
Yelp
AG News
RTE
Stage
LAN
WAN
Comm.
LAN
WAN
Comm.
LAN
WAN
Comm.
LAN
WAN
Comm.
Offline
Correlated randomness
–
–
6.8
–
–
36.4
–
–
19.0
–
–
30.4
Forward
Linear layers
4.9
34.6
0.7
22.1
132
3.2
12.3
75.0
1.8
18.2
89.9
3.1
Softmax
8.4
99.7
3.1
57.9
660
30.4
24.8
295
12.6
55.8
658
30.4
GELU
26.5
322
15.3
139
1,644
79.8
72.5
901
43.7
102
1,231
60.1
LayerNorm
3.8
39.9
0.6
9.5
91.0
3.0
5.7
61.9
1.7
12.7
66.6
2.3
Table 1. Cost of one secure training step by stage, per party, on the four tasks (batch size 32). Time is in seconds on a LAN (3 Gbps, 0.8 ms) and a WAN (200 Mbps, 40 ms); communication is in GB. The offline phase generates the correlated randomness and is independent of the inputs. The last row gives a full training run (150 steps; 100 on RTE) in hours and TB.
Method
Test accuracy (%) ↑
SST-2
Yelp
AG News
RTE
Avg.
SFT
85.4 ± 2.8
91.8 ± 0.8
86.4 ± 1.9
58.6 ± 6.2
80.5
SecP-Tuning
88.1 ± 0.6
85.2 ± 0.5
81.3 ± 0.7
53.0 ± 0.3
76.9
LoRA + CMA-ES
85.1 ± 3.4
91.1 ± 1.4
86.6 ± 2.0
63.9 ± 2.9
81.7
LoRA + SPSA
85.6
91.1
86.0
64.9
81.9
LoRA + Adam
86.7 ± 1.1
92.0 ± 0.9
87.6 ± 1.0
66.5 ± 1.7
83.2
Table 2. Few-shot test accuracy (%, mean ± standard deviation over three seeds; higher is better). SFT and SecP-Tuning are reported in ( Luo et al., 2026 ) ; the other rows train our adapter, in plaintext except for the last row; LoRA + SPSA is zeroth-order and given as a mean. The best result in each column is in bold and the second best is underlined.
Variant
GB ↓
Rounds ↓
Grad. error ↓
VeilTune
22.6
1,208
0.54%
w/o token packing
42.0
1,208
–
w/o [MASK] -only rows
25.5
1,208
–
w/o loss scaling
–
–
15.7%
generic operators
28.6
1,665
8.0×108
all removed
55.3
1,665
2.0×1011
Table 3. Effect of removing one technique at a time (one training step on SST-2 without the optimizer, batch size 32, per party). The last column is the relative error of the adapter gradient against float64.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Protocol (opened values)
θT acc.
Local
K=1
K=100
SST-2 (local model accuracy 86.4%)
SAP (hidden repr.)
90.4
90.6
90.6 / 99.2
90.6 / 99.1
PrivTuner (logits)
90.4
90.6
87.4 / 93.2
87.4 / 92.7
+ active
90.4
90.6
90.9 / 97.8
90.9 / 97.7
SecP-Tuning (logits)
87.5
88.1
86.7 / 91.0
88.0 / 90.6
+ active
87.5
88.1
87.9 / 94.3
87.4 / 94.5
Appendix
Table 4. Adapter leakage: test accuracy / agreement with θT (%) of the adapter θ^ extracted by AEA from the last K steps the protocol opens; rows without “+ active” are passive attackers. The second column gives the test accuracy of θT , and the third the agreement of the local model.
Figure 5. GELU and its derivative as computed by VeilTune in 16-bit fixed point, against the exact erf-based GELU. The polynomial is used between the dashed lines; the bottom row shows the absolute error.
Loss scale
1
16
64
256
1,024
4,096
Median
13.3
1.03
0.59
0.48
0.62
0.60
Range
3.5–19.1
0.33–1.08
0.24–0.60
0.22–0.71
0.21–0.64
0.20–0.60
Appendix
Table 5. Relative error (%) of the secure gradient for different loss scales: median and range over three batches of SST-2.
Setting
Secure
Plaintext
Skipped
Default ( τ=10 , scale 256, S=213 )
87.6
87.7
18
τ=1
87.8
87.7
13
τ=1,000
87.3
87.7
12
Loss scale 64
88.9
88.8
28
Loss scale 1,024
89.0
88.7
19
S=211
87.6
87.7
13
Appendix
Table 6. Secure training on SST-2 (seed 42) with one hyperparameter changed at a time: test accuracy (%) of the secure run and of plaintext training that skips the same steps, and the number of skipped steps out of 150.
Symbol
Meaning
P0 , P1
data owner, model owner
[[x]]
additive sharing of x over Z264
f
fractional bits of the fixed-point encoding ( f=16 )
Fine-tuning large vision models (LVMs) and large language models (LLMs) under differentially private federated learning (DPFL) is hindered by a fundamental privacy-utility trade-off. Low-Rank Adaptation (LoRA), a promising parameter-efficient fine-tuning (PEFT) method, reduces computational and communication costs by introducing two trainable low-rank matrices while freezing pre-trained weights. However, directly applying LoRA in DPFL settings leads to performance degradation, especially in LVMs. Our analysis reveals three previously underexplored challenges: (1) gradient coupling caused by the simultaneous update of two asymmetric low-rank matrices, (2) compounded noise amplification under differential privacy, and (3) sharpness of the global aggregated model in the parameter space. To address these issues, we propose LA-LoRA (Local Alternating LoRA), a novel approach that decouples gradient interactions and aligns update directions across clients to enhance robustness under stringent privacy constraints. Theoretically, LA-LoRA strengthens convergence guarantees in noisy federated environments. Extensive experiments demonstrate that LA-LoRA achieves state-of-the-art (SOTA) performance on Swin Transformer and RoBERTa models, showcasing robustness to DP noise and broad applicability across both LVMs and LLMs. For example, when fine-tuning the Swin-B model on the Tiny-ImageNet dataset under a strict privacy budget (ε=1), LA-LoRA outperforms the best baseline, RoLoRA, by 16.83% in test accuracy. Code is provided in \repolink.
Jin Liu, Yinbin Miao, Ning Xi +1
School of Cyber Engineering, Xidian University · College of Intelligence and Computing, Tianjin University
Differentially private federated fine-tuning of large models with LoRA suffers from aggregation error caused by LoRA's multiplicative structure, which is further amplified by DP noise and degrades both stability and accuracy. Existing remedies apply a single update mode uniformly across all layers and all communication rounds (or alternate them on a fixed schedule), ignoring both the structural asymmetry between the two LoRA factors and the round-wise dynamics of training. We propose AS-LoRA, an adaptive framework defined by three axes (i) layer-wise freedom, in which each layer independently selects its active component, (ii) round-wise adaptivity, in which the selection updates over communication rounds, and (iii) a curvature-aware score derived from a second-order approximation of the loss. Theoretically, AS-LoRA eliminates the reconstruction-error floor of layer-tied schedules, accelerates convergence, implicitly biases solutions toward flatter minima, and incurs no additional privacy cost. Across GLUE, SQuAD, CIFAR-100, and Tiny-ImageNet under strict DP budgets and non-IID partitions, AS-LoRA improves over the federated LoRA baselines by up to +7.5 pp on GLUE and +12.5 pp on MNLI-mm for example, while matching or exceeding SVD-based aggregation methods at 33–180× lower aggregation cost and with negligible communication overhead. Code for the proposed method is available at https://anonymous.4open.science/r/as_lora-F75F/.
Myoungjun Kim, Sangwoo Park, Yoseob Han +1
Myongji University · King’s College London · Soongsil University
Parameter-efficient fine-tuning (PEFT), such as low-rank adaptation (LoRA), has recently been adopted in federated learning to reduce communication and computation costs. In this setup, users download a pretrained model from the server prior to fine-tuning, and then fine-tune lightweight LoRA modules locally while keeping the pretrained model frozen, sharing only the gradients of the fine-tuning parameters with the server. Despite its growing popularity, robustness of federated fine-tuning against an adversarial server remains underexplored, where the server maliciously tampers with the training protocol to breach the privacy of users' data. In this work, we investigate gradient inversion attacks on LoRA fine-tuning. We propose an analytical attack that enables a malicious server to recover private user data by leveraging a poisoned pretrained model and fine-tuning parameters. Our design embeds fine-tuning data within the shared gradients, to allow the server to analytically reconstruct user data. Unlike prior works, our attack is applicable to both language and vision tasks, does not rely on computationally expensive (adversarial) pretraining with public datasets or require the number of training tokens to be less than the rank of LoRA modules. Experimental results on both language and vision tasks demonstrate high-fidelity data recovery across multiple baselines, revealing several critical vulnerabilities.