cs.CRNov 9, 2026

VeilTune: Secure Fine-Tuning of Private LoRA Adapters on Public Pretrained Language Models

Authors: Huiyu Xie, Tanping Zhou, Tao Hu, Xinlei Liu, Hailong Ma

Organizations: Information Engineering University, Zhengzhou, China · Key Laboratory of Cyberspace Endogenous Safety & Security of Henan Province, Zhengzhou, China · Chinese People’s Armed Police Force Engineering University, Xi’an, China

Abstract

Fine-tuning public pretrained language models with lightweight adapters such as LoRA has become the mainstream way to adapt them to downstream tasks. A service provider often trains its adapter on a data owner's samples, so two assets need protection: the data owner's samples and the provider's adapter. Existing secure fine-tuning protocols protect the samples but reveal to the data owner intermediate results of every training step, such as the hidden representations or logits of its inputs. We show that a single step suffices to copy the adapter: from the hidden representations of one mini-batch of 32 samples, the data owner fits an adapter that agrees with the provider's on 99.2% of test inputs, and logits alone yield about 94%. Protecting both assets therefore requires that no value that depends on the adapter be revealed throughout training. We present VeilTune, to the best of our knowledge the first protocol that fine-tunes a pretrained language model with backpropagation entirely in secure multi-party computation (MPC). VeilTune computes only what the loss depends on, replaces generic MPC operators with training-aware nonlinear operators that stay accurate on the values that training produces, and obliviously skips the steps corrupted by truncation failures. On four few-shot tasks with RoBERTa-large, VeilTune nearly matches plaintext training and is 5.3 points more accurate on average than SecP-Tuning, the state-of-the-art secure fine-tuning protocol. On SST-2, it achieves a 3.5x end-to-end speedup and a 3.6x reduction in communication over SecP-Tuning, and 26x and 17x over full fine-tuning in MPC.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models

    Feb 23, 2026Jin Liu, Yinbin Miao, Ning Xi +1Federated Fine-TuningLow-Rank Adaptation

  2. Adaptive Selection of LoRA Components in Privacy-Preserving Federated Learning

    May 7, 2026Myoungjun Kim, Sangwoo Park, Yoseob Han +1Differentially Private Federated LearningFederated Learning

  3. MineGrad: Gradient Inversion Attacks on LoRA Fine-Tuning

    Aug 2, 2026Hasin Us Sami, Swapneel Sen, Basak GulerGradient Inversion AttacksPrivacy Leakage in Language Models