The rapid development of large language models (LLMs) has driven the widespread adoption of cloud-based LLM inference services, while also bringing prominent privacy risks associated with the transmission and processing of private data in remote inference. For privacy-preserving LLM inference technologies to be practically applied in industrial scenarios, three core requirements must be satisfied simultaneously: (1) Accuracy and efficiency losses should be minimized to mitigate degradation in service experience. (2) The inference process can be run on large-scale clusters consist of heterogeneous legacy xPUs. (3) Compatibility with existing LLM infrastructures should be ensured to reuse their engineering optimizations. To the best of our knowledge, none of the existing privacy-preserving LLM inference methods satisfy all the above constraints while delivering meaningful privacy guarantees. In this paper, we propose AloePri, the first privacy-preserving LLM inference method for industrial applications. AloePri protects both the input and output data by covariant obfuscation, which jointly transforms data and model parameters to achieve better accuracy and privacy. We carefully design the transformation for each model component to ensure inference accuracy and data privacy while keeping full compatibility with existing infrastructures of Language Model as a Service. AloePri has been integrated into an industrial system for the evaluation of mainstream LLMs. The evaluation on Deepseek-V3.1-Terminus model (671B parameters) demonstrates that AloePri causes accuracy loss of 0.0%~3.5% and exhibits efficiency equivalent to that of plaintext inference. Meanwhile, AloePri successfully resists state-of-the-art attacks, with less than 5% of tokens recovered. To the best of our knowledge, AloePri is the first method to exhibit practical applicability to large-scale models in real-world systems.
Figures & tables
Figure 1: Workflow of AloePri . The client locally obfuscates the model and deploys it to the server, and handles prompt obfuscation and response de-obfuscation during online phase.
Figure 2: Overview of AloePri . In the offline model obfuscation process, token-level permutation and linear transformations are employed to construct the obfuscations ϕembed,ϕhead,ϕattn, and ϕffn . In the online inference process, a secret vocabulary mapping associated with the permutation is used to construct the data obfuscation ϕX and de-obfuscation ψY .
DP
Rényi DP
Metric DP
Rényi-metric DP
Privacy notion
Adjacent datasets
Adjacent datasets
Metric space
Metric space
Gaussian mechanism analysis
Not natural
Natural
Not natural
Natural
Table 1: Comparison of DP, Rényi DP, metric DP, and Rényi-metric DP.
Method
Classification Acc. ↑
Generation Acc. ↑
Privacy 1
SST2
MMLU
PIQA
IFEval
Attack
TTRSR(%) ↓
CosSim ↓
Plaintext
97.25
81.95
77.45
77.09
-
-
-
SANTEXT
81.31
51.07
49.72
57.45
(Direct obs.)
62.90
0.93
RANTEXT
87.50
37.77
18.30
57.18
(Direct obs.)
56.92
0.88
DP-Forward 2
93.60
-
-
-
NN
37.82
0.51
IMA
77.08
0.84
Table 2: Accuracy and privacy comparison with baselines on Qwen2.5-14B-Instruct.
Figure 3: Privacy (TTRSR) and accuracy under various noise parameters on Qwen2.5-14B-Instruct and C-Eval.
Figure 4: Impact of λ on Qwen2.5-14B-Instruct.
Model
Plaintext accuracy ↑
AloePri accuracy ↑
AloePri privacy against VMA ↓
MMLU
C-Eval
HumanEval
PIQA
IFEval
MMLU
C-Eval
HumanEval
PIQA
IFEval
TTRSR(%)
PIIRSR(%)
BLEU-4
R1-Distill
14B
78.91
86.18
96.34
88.14
72.83
79.2
85.93
93.9
88.79
72.27
12.36
2.37
0.52
32B
83.5
88.34
97.56
88.79
73.94
83.61
87.84
95.12
88.08
73.57
10.10
2.63
0.78
Qwen3
14B
87.64
87.35
94.51
89.72
86.14
84.57
87.12
95.12
90.26
85.40
25.05
1.62
1.72
32B
89.61
89.0
98.78
90.75
84.29
86.24
87.64
96.95
89.5
81.15
19.64
1.23
1.41
Llama3
8B
65.91
50.16
55.49
80.74
68.58
63.5
44.55
57.32
78.94
67.84
2.57
0
0.84
Table 3: Accuracy and privacy of AloePri under various models and datasets. Privacy is evaluated with VMA on PUPA dataset.
Table 8Table 9
Figure 5: TPOT and TTFT vs. h on R1-Distill-14B
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Description
n
Number of vocabulary’s tokens
d,dhead,dffn
Hidden size, attention head dimension, and FFN intermediate dimension
h
Expansion size for the key matrix
d′
Expanded hidden size
m,mkv
Number of attention heads, and number of attention key-value heads
mexp
Number of MoE experts
Appendix
Table 8: Notations and their Descriptions
Known X
Observed Y
WeWh
ΠWembed⋆WhΠT
WeWquery(WeWkey)T
ΠWembed⋆Wquery(ΠWembed⋆Wkey)T
WeWgate
ΠWembed⋆WgateZ^ffn
WeWup
ΠWembed⋆WupS^ffnZ^ffn
WdownWh
Z^ffn−1S^ffn−1WdownWhead⋆ΠT
WeWrouter
ΠWembed⋆norm(Wrouter)Z^router
Appendix
Table 9: Weight combinations to recover Π with VMA
Experiment
αe
αh
λ
h
β
γ
Fig. 3
△ 1
△
0.3
128
8
1e3
Fig. 4
1.0
0.2
△
128
8
1e3
Fig. 5
1.0
0.2
0.3
△
8
1e3
Tab. 2 , 7 , 7
1.0
0.2
0.3
128
8
1e3
Tab. 3
0.5,1.0 2
0.2
0.3
128
8
1e3
Tab. 5
1.0
0.2
△
128
△
△
Appendix
Table 10: Privacy hyperparameter settings of experiments.
With the widespread deployment of public large language models (LLMs) such as ChatGPT, protecting user prompt privacy has become an increasingly critical issue. Existing privacy-preserving inference methods sacrifice either utility or efficiency, and often require model-specific modifications that limit their compatibility. In this paper, we propose SharedRequest, a model-agnostic framework for privacy-preserving LLM inference that reformulates privacy protection at the batch level rather than the individual-prompt level. The key idea is to obscure sensitive information by mixing original prompts with noisy variants, while grouping semantically equivalent instructions to amortize the inference cost over a large batch of queries with minimal impact on LLM response quality. This design is independent of the LLM architecture, requiring no access to model parameters or architectural modification. Empirical results demonstrate that SharedRequest achieves over 20% higher utility compared to prior differential privacy baselines, and its shared-prompt mechanism reduces query cost by up to 5× compared to non-batched inference.
Peihua Mai, Xuanrong Gao, Youlong Ding +3
1National University of Singapore (Chongqing) Research Institute · 2Chongqing Key Laboratory of Trusted Perception and Interaction Technology for Intelligent and Connected Vehicles, Chongqing, China · 3National University of Singapore +3
Large Language Models (LLMs) are increasingly served on shared accelerators where an adversary with read access to device memory can observe KV caches and hidden states, threatening prompt privacy for open-source models. Cryptographic protections such as MPC and FHE offer strong guarantees but remain one to two orders of magnitude too slow for interactive inference, while static obfuscation schemes break under multi-run statistical attacks once the model is known. We present GELO (Good-Enough LLM Obfuscation), a lightweight protocol for privacy-preserving inference that limits information leakage from untrusted accelerator observations by hiding hidden states with fresh, per-batch invertible mixing. For each offloaded projection, the TEE samples a random matrix A, forms U=AH, offloads U and weights W to the accelerator, and then applies A−1 on return, so that A−1((AH)W)=HW and outputs are unchanged. Because mixing is never reused across batches, the attacker faces only a single-batch blind source separation problem. We analyse information leakage and introduce two practical defences: (i) non-orthogonal mixing to mask Gram matrices, and (ii) orthogonal mixing augmented with a small fraction of high-energy "shield" vectors that pollute higher-order statistics. On Llama-2 7B, GELO preserves float32 outputs exactly, closely matches low-precision baselines, and shows about 20--30% compute-side overhead in a controlled offload microbenchmark; an unoptimized remote prototype is dominated by transport overhead, motivating deeper serving-engine integration. GELO resists ICA/BSS and anchor-assisted attacks; a 60M-parameter transformer-based unmixing attack also fails under strong mixing and shielding.
As large language models (LLMs) continue to grow in size, fewer users are able to host and run models locally. This has led to increased use of third-party hosting services. However, in this setting, there is a lack of guarantees on the computation performed by the inference provider. For example, a dishonest provider may replace an expensive large model with a cheaper-to-run weaker model and return the results from the weaker model to the user. Existing tools to verify inference typically rely on methods from cryptography such as zero-knowledge proofs (ZKPs), but these add significant computational overhead, and remain infeasible for use for large models. In this work, we develop a new insight -- that given a method for performing private LLM inference, one can obtain forms of verified inference at marginal extra cost. Specifically, we propose two new protocols which leverage privacy-preserving LLM inference in order to provide guarantees over the inference that was carried out. Our approaches are cheap, requiring the addition of a few extra tokens of computation, and have little to no downstream impact. As the fastest privacy-preserving inference methods are typically faster than ZK methods, the proposed protocols also improve verification runtime. Our work provides novel insights into the connections between privacy and verifiability in LLM inference.