VeilTune: Secure Fine-Tuning of Private LoRA Adapters on Public Pretrained Language Models
Authors: Huiyu Xie, Tanping Zhou, Tao Hu, Xinlei Liu, Hailong Ma
Organizations: Information Engineering University, Zhengzhou, China · Key Laboratory of Cyberspace Endogenous Safety & Security of Henan Province, Zhengzhou, China · Chinese People’s Armed Police Force Engineering University, Xi’an, China
Fine-tuning public pretrained language models with lightweight adapters such as LoRA has become the mainstream way to adapt them to downstream tasks. A service provider often trains its adapter on a data owner's samples, so two assets need protection: the data owner's samples and the provider's adapter. Existing secure fine-tuning protocols protect the samples but reveal to the data owner intermediate results of every training step, such as the hidden representations or logits of its inputs. We show that a single step suffices to copy the adapter: from the hidden representations of one mini-batch of 32 samples, the data owner fits an adapter that agrees with the provider's on 99.2% of test inputs, and logits alone yield about 94%. Protecting both assets therefore requires that no value that depends on the adapter be revealed throughout training. We present VeilTune, to the best of our knowledge the first protocol that fine-tunes a pretrained language model with backpropagation entirely in secure multi-party computation (MPC). VeilTune computes only what the loss depends on, replaces generic MPC operators with training-aware nonlinear operators that stay accurate on the values that training produces, and obliviously skips the steps corrupted by truncation failures. On four few-shot tasks with RoBERTa-large, VeilTune nearly matches plaintext training and is 5.3 points more accurate on average than SecP-Tuning, the state-of-the-art secure fine-tuning protocol. On SST-2, it achieves a 3.5x end-to-end speedup and a 3.6x reduction in communication over SecP-Tuning, and 26x and 17x over full fine-tuning in MPC.
Figures & tables
Figure 1. Attack performance (agreement with θT , % ↑ ) of AEA against existing protocols, compared with the local model, on four tasks (mean of three seeds). “+act.” marks the active attacker, the other groups are passive; SecP is SecP-Tuning.
Figure 2. Workflow of VeilTune and one secure training step. All values inside the middle box are secret-shared, and the Transformer block is repeated for the L−s adapted blocks.
Figure 3. A secure training step built from generic operators. (a) Communication and rounds by operator. (b) Arguments of three generic operators during fine-tuning; the shaded band is the range on which the operator is accurate.
Figure 4. Token packing for a batch of three examples. M marks the [MASK] token; hatched cells are computed but wasted. (a) Padding. (b) Packed tokens for the per-token layers. (c) Attention scores over all pairs of packed tokens (left) and within each example after regrouping (right).
SST-2
Yelp
AG News
RTE
Stage
LAN
WAN
Comm.
LAN
WAN
Comm.
LAN
WAN
Comm.
LAN
WAN
Comm.
Offline
Correlated randomness
–
–
6.8
–
–
36.4
–
–
19.0
–
–
30.4
Forward
Linear layers
4.9
34.6
0.7
22.1
132
3.2
12.3
75.0
1.8
18.2
89.9
3.1
Softmax
8.4
99.7
3.1
57.9
660
30.4
24.8
295
12.6
55.8
658
30.4
GELU
26.5
322
15.3
139
1,644
79.8
72.5
901
43.7
102
1,231
60.1
LayerNorm
3.8
39.9
0.6
9.5
91.0
3.0
5.7
61.9
1.7
12.7
66.6
2.3
Table 1. Cost of one secure training step by stage, per party, on the four tasks (batch size 32). Time is in seconds on a LAN (3 Gbps, 0.8 ms) and a WAN (200 Mbps, 40 ms); communication is in GB. The offline phase generates the correlated randomness and is independent of the inputs. The last row gives a full training run (150 steps; 100 on RTE) in hours and TB.
Method
Test accuracy (%) ↑
SST-2
Yelp
AG News
RTE
Avg.
SFT
85.4 ± 2.8
91.8 ± 0.8
86.4 ± 1.9
58.6 ± 6.2
80.5
SecP-Tuning
88.1 ± 0.6
85.2 ± 0.5
81.3 ± 0.7
53.0 ± 0.3
76.9
LoRA + CMA-ES
85.1 ± 3.4
91.1 ± 1.4
86.6 ± 2.0
63.9 ± 2.9
81.7
LoRA + SPSA
85.6
91.1
86.0
64.9
81.9
LoRA + Adam
86.7 ± 1.1
92.0 ± 0.9
87.6 ± 1.0
66.5 ± 1.7
83.2
Table 2. Few-shot test accuracy (%, mean ± standard deviation over three seeds; higher is better). SFT and SecP-Tuning are reported in ( Luo et al., 2026 ) ; the other rows train our adapter, in plaintext except for the last row; LoRA + SPSA is zeroth-order and given as a mean. The best result in each column is in bold and the second best is underlined.
Variant
GB ↓
Rounds ↓
Grad. error ↓
VeilTune
22.6
1,208
0.54%
w/o token packing
42.0
1,208
–
w/o [MASK] -only rows
25.5
1,208
–
w/o loss scaling
–
–
15.7%
generic operators
28.6
1,665
8.0×108
all removed
55.3
1,665
2.0×1011
Table 3. Effect of removing one technique at a time (one training step on SST-2 without the optimizer, batch size 32, per party). The last column is the relative error of the adapter gradient against float64.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Protocol (opened values)
θT acc.
Local
K=1
K=100
SST-2 (local model accuracy 86.4%)
SAP (hidden repr.)
90.4
90.6
90.6 / 99.2
90.6 / 99.1
PrivTuner (logits)
90.4
90.6
87.4 / 93.2
87.4 / 92.7
+ active
90.4
90.6
90.9 / 97.8
90.9 / 97.7
SecP-Tuning (logits)
87.5
88.1
86.7 / 91.0
88.0 / 90.6
+ active
87.5
88.1
87.9 / 94.3
87.4 / 94.5
Appendix
Table 4. Adapter leakage: test accuracy / agreement with θT (%) of the adapter θ^ extracted by AEA from the last K steps the protocol opens; rows without “+ active” are passive attackers. The second column gives the test accuracy of θT , and the third the agreement of the local model.
Figure 5. GELU and its derivative as computed by VeilTune in 16-bit fixed point, against the exact erf-based GELU. The polynomial is used between the dashed lines; the bottom row shows the absolute error.
Loss scale
1
16
64
256
1,024
4,096
Median
13.3
1.03
0.59
0.48
0.62
0.60
Range
3.5–19.1
0.33–1.08
0.24–0.60
0.22–0.71
0.21–0.64
0.20–0.60
Appendix
Table 5. Relative error (%) of the secure gradient for different loss scales: median and range over three batches of SST-2.
Setting
Secure
Plaintext
Skipped
Default ( τ=10 , scale 256, S=213 )
87.6
87.7
18
τ=1
87.8
87.7
13
τ=1,000
87.3
87.7
12
Loss scale 64
88.9
88.8
28
Loss scale 1,024
89.0
88.7
19
S=211
87.6
87.7
13
Appendix
Table 6. Secure training on SST-2 (seed 42) with one hyperparameter changed at a time: test accuracy (%) of the secure run and of plaintext training that skips the same steps, and the number of skipped steps out of 150.
Symbol
Meaning
P0 , P1
data owner, model owner
[[x]]
additive sharing of x over Z264
f
fractional bits of the fixed-point encoding ( f=16 )