Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptation

Jeongho Yoon; Chanhee Park; Yongchan Chun; Hyeonseok Moon; Heuiseok Lim

cs.CRApr 8, 2026

Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptation

Authors: Jeongho Yoon, Chanhee Park, Yongchan Chun, Hyeonseok Moon, Heuiseok Lim

Organizations: Department of Computer Science and Engineering, Korea University · Samsung Mobile eXperience Business

Abstract

Current LLM-based services typically require users to submit raw text regardless of its sensitivity. While intuitive, such practice introduces substantial privacy risks, as unauthorized access may expose personal, medical, or legal information. Although prior defenses strived to mitigate these risks, they often incur substantial computational overhead and degrade model performance. To overcome this privacy-efficiency trade-off, we introduce Privacy-Preserving Fine-Tuning (PPFT), a novel training pipeline that eliminates the need for transmitting raw prompt text while maintaining a favorable balance between privacy preservation and model utility for both clients and service providers. Our approach operates in two stages: first, we train a client-side encoder together with a server-side projection module and LLM, enabling the server to condition on k-pooled prompt embeddings instead of raw text; second, we fine-tune the projection module and LLM on private, domain-specific data using noise-injected embeddings, allowing effective adaptation without exposing plain text prompts and requiring access to the decoder's internal parameters. Extensive experiments on domain-specific and general benchmarks demonstrate that PPFT achieves a striking balance between privacy and utility, maintaining competitive performance with minimal degradation compared to noise-free upper bounds.

Figures & tables

Figure 1: While conventional services expose plain text prompts to the server, PPFT transmits only obfuscated embeddings to prevent prompt inference and mitigate privacy risks.

Figure 2: Overview of PPFT. Stage 1 aligns pooled client-side embeddings with the decoder to enable text-free inference. Stage 2 performs domain adaptation using noise-injected embeddings to improve robustness against reconstruction.

Backbone	Method	Average	Pri-DDX	Pri-NLICE	Pri-SLJA
Llama-3.1-8B	$d_{\chi}$ -privacy Feyisetan et al. (2020)	0.2750 ( $\downarrow$ 0.6541)	0.2311	0.3477	0.2462
	Paraphrase Utpala et al. (2023)	0.3757 ( $\downarrow$ 0.5534)	0.4648	0.2892	0.3731
	PrivacyRestore Zeng et al. (2025)	0.6343 ( $\downarrow$ 0.2948)	0.5784	0.5415	0.7829
	PPFT (Ours)	0.7314 ( $\downarrow$ 0.1977)	0.5915	0.6979	0.9049
	$\text{PPFT}_{\text{w/o stage2}}$ (Lower Bound)	0.3545	0.3460	0.3138	0.4036
	$\text{PPFT}_{\text{w/o noise}}$ (Upper Bound)	0.9291	0.9275	0.9049	0.9466

Table 1: Main results on downstream tasks. PPFT (

k=4

) refers to our model adapted with noise in Stage 2. Lower/Upper bounds indicate performance without domain adaptation and without privacy noise, respectively.

Original Prompt

A 27-year-old male has a history of chronic pancreatitis, diabetes, obesity, pancreatic cancer in family members, smoking.

The 27-year-old male presents the symptoms of diarrhea, fatigue, nausea, pain, pale stools and dark urine, skin lesions, underweight.

What is the likely diagnosis?

Reconstructed by Inversion Attack (same

\epsilon

as inference)

A 28 -year-old woman has a history of asthma, asthma attack, asthma attack, asthma attack, asthma attack, asthma attack, asthma attack, asthma attack, asthma attack, asthma attack, asthma attack, asthma attack.

The 28 -year-old woman presents the symptoms of cough, wheezing, shortness of breath, shortness of breath, wheezing, shortness of breath, shortness of breath with deep breathing.

Table 2: Qualitative reconstruction example under noisy-embedding transmission. Blue indicates spans that exactly match the original prompt, whereas red indicates mismatched content.

Figure 3: Results of embedding inversion attacks and attribute inference attacks across all baselines under varying privacy budgets $\epsilon$ on Pri-DDX.

Method	Age	Sex	Symptom	Antecedent
PrivacyRestore	-	0.5642	0.3552	0.3317
PPFT (Ours)	0.0071	0.5894	0.1001	0.0115

Table 3: Fine-grained reconstruction error on the Pri-DDX dataset under inference-level privacy budgets.

Backbone	Method	CSQA	SQuAD
Llama-3.1-8B	$d_{\chi}$ -privacy	0.1819	0.0174
	Paraphrase	0.0649	0.0125
	PPFT (Ours)	0.5278	0.7085
	$\text{PPFT}_{\text{w/o noise}}$	0.6086	0.8930
Llama-3.2-1B	$d_{\chi}$ -privacy	0.1210	0.0313
	Paraphrase	0.0470	0.072

Table 4: Performance on general domains.

Metric	Pooling Size $(k)$
Metric	4	8	16
Score $\uparrow$	0.9049	0.8363	0.7630
ROUGE-L $\downarrow$	0.4050	0.3553	0.3241

Table 5: Ablation study on pooling size

k

. ROUGE-L is measured on the Pri-SLJA test set.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
General Settings
Backbones	Llama-3.2-1B / 3.1-8B
Precision	bfloat16
Max Sequence Length	512
LoRA Configuration
Rank ( $r$ )	16

Appendix

Table 6: Hyperparameters used for training Llama-3.2-1B and Llama-3.1-8B models across Stage 1 and Stage 2.

Dataset	$n$	$D_{\max}$	$\epsilon_{d_{\chi}}\!=\!\frac{150}{nD_{\text{max}}}$	$\tau\!=\!\frac{2n}{150}$
Pri-DDXP	106.00	1.64	0.863	1.413
Pri-NLICE	72.00	1.39	1.499	0.960
Pri-SLJA	193.00	1.45	0.536	2.573
SQuAD	178.78	1.70	0.494	2.384
CSQA	48.43	1.68	1.844	0.646

Appendix

Table 7: Dataset-specific hyperparameters aligned to budget

B=150

n

: max token length used for accounting.

D_{\max}

: an upper bound on per-token embedding distance in the metric space used by the

d_{\chi}

baseline.

\epsilon_{d_{\chi}}

and

\tau

are adjusted per dataset to maintain fixed

B

Pooling size	$\epsilon{=}0.01$	$\epsilon{=}0.46$	$\epsilon{=}0.86$	$\epsilon{=}2.01$	$\epsilon{=}2.29$	$\epsilon{=}17.2$	$\epsilon{=}22.93$	$\epsilon{=}75.0$
4	0.02974	0.03045	0.03178	0.03487	0.03373	0.16013	0.24380	0.43974
8	0.05506	0.05554	0.05525	0.05920	0.06266	0.09974	0.15784	0.33750
16	0.05039	0.05177	0.04938	0.04910	0.05055	0.14032	0.15935	0.17990

Appendix

Table 8: Noise-aware inverse attack results (ROUGE-L). The attacker is trained with noisy representations while we report reconstruction quality under different privacy budgets at inference.

Ex.	Ground truth	Reconstruction (blue=same, red=different)
1	A 46-year-old male has a history of chronic pancreatitis, diabetes, obesity, pancreatic cancer in family members. The 46-year-old male presents the symptoms of cough, diarrhea, nausea, pain, pale stools and dark urine, skin lesions, underweight. What is the likely diagnosis?	A 6 -year-old woman has a history of smoking, diabetes, high blood pressure, obesity, high cholesterol, high blood pressure, smoking. The 6 -year-old woman presents the symptoms of cough, fever, fatigue, pain, shortness of breath, skin lesions. What is the likely diagnosis?
2	A 45-year-old woman has a history of chronic pancreatitis, diabetes, obesity, pancreatic cancer in family members, smoking. The 45-year-old woman presents the symptoms of diarrhea, fatigue, nausea, pain, pale stools and dark urine, skin lesions, underweight. What is the likely diagnosis?	A 22 -year-old man has a history of alcohol addiction, smoking, alcohol addiction, heart failure, heart valve issue. The 22 -year-old man presents the symptoms of chest pain, shortness of breath, pain, fatigue, shortness of breath with exertion, …

Appendix

Table 9: Qualitative examples for the noise-aware inverse attack. Blue indicates spans that exactly match the original prompt, whereas red indicates mismatched or hallucinated content, including medically salient details.

Ex.	Ground truth	Reconstruction (blue=same, red=different)
1	A 57-year-old male has a history of antipsychotic medication usage, nausea, stimulant drug use. The 57-year-old male presents the symptoms of involuntary eye movement, jaw pain, muscle spasms, muscle spasms in neck, ptosis, shortness of breath. What is the likely diagnosis?	The diagnosis of the 57-year-old male who has been experiencing symptoms of eye jumping, unknown button, joint pain and muscle spasms in neck , is psychosis. What is the diagnosis?
2	A 8-year-old woman has a history of active cancer, deep vein thrombosis, hormone intake, immobility for $>$ 3 days, surgery within last month. The 8-year-old woman presents the symptoms of coughing up blood, loss of consciousness, pain, shortness of breath, swelling. What is the likely diagnosis?	The patient has been in the hospital for over 3 weeks, with intravenous drug use, migraine, intake of bed, surgery . The patient’s symptoms are cough, fever, pain , swelling . What is the likely diagnosis?

Appendix

Table 10: Qualitative examples for the Stage-1 aligned inversion attacker. Blue spans exactly match the original prompt, while red spans differ. Even with a stronger attacker aligned to the encoder space, reconstructions often preserve only partial lexical overlaps rather than medically faithful recovery.

Figure 6: Inversion attacks on PPFT using a Stage-1 aligned model (stronger attacker) under varying privacy budgets $\epsilon$ . For comparison, we also report inversion results from a GPT-2 Medium model (weaker attacker).

Figure 7: Reconstruction quality across iterations for the pooled-embedding (Experiment 1) and single-embedding (Experiment 2) settings.

Privacy-Preserving Language Models

Privacy-preserving language models protect sensitive data during training, inference, personalization, and information sharing.

See all 171 papers

SharedRequest: Privacy-Preserving Model-Agnostic Inference for Large Language Models
Jun 3, 2026Peihua Mai, Xuanrong Gao, Youlong Ding +3Privacy-Preserving Language Models LLM Inference
1National University of Singapore (Chongqing) Research Institute · 2Chongqing Key Laboratory of Trusted Perception and Interaction Technology for Intelligent and Connected Vehicles, Chongqing, China · 3National University of Singapore +3
Prune the Leaky Neurons: Mitigating Private Data Leakage in Fine-Tuned LLMs via Joint-Embedding-Guided Pruning
Oct 16, 2026Mohamed Shaaban, Mohamed ElmahallawyPrivacy Leakage in Language Models Privacy-Preserving Language Models
School of Engineering and Applied Sciences, Washington State University, Richland, WA 99354, USA
Nullify: Null-Space Activation Steering for Training-Free LLM Unlearning
Oct 7, 2026Wei Zhai, Xiang Liu, Qiang Huang +6LLM Unlearning Privacy Leakage in Language Models
College of Computer Science and Artificial Intelligence, Fudan University · BEDI Cloud · School of Electrical and Information Engineering, Tianjin University +3

Jun 3, 2026J/K move · Enter open · S save

SharedRequest: Privacy-Preserving Model-Agnostic Inference for Large Language Models

Peihua Mai, Xuanrong Gao, Youlong Ding, Xianglong Du+2

1National University of Singapore (Chongqing) Research Institute · 2Chongqing Key Laboratory of Trusted Perception and Interaction Technology for Intelligent and Connected Vehicles, Chongqing, China · 3National University of Singapore+3

Privacy-Preserving Language Models LLM Inference Privacy Leakage in Language Models Efficient Language Model Inference

Open paper AskPDF arXiv

With the widespread deployment of public large language models (LLMs) such as ChatGPT, protecting user prompt privacy has become an increasingly critical issue. Existing privacy-preserving inference methods sacrifice either utility or efficiency, and often require model-specific modifications that limit their compatibility. In this paper, we propose SharedRequest, a model-agnostic framework for privacy-preserving LLM inference that reformulates privacy protection at the batch level rather than the individual-prompt level. The key idea is to obscure sensitive information by mixing original prompts with noisy variants, while grouping semantically equivalent instructions to amortize the inference cost over a large batch of queries with minimal impact on LLM response quality. This design is independent of the LLM architecture, requiring no access to model parameters or architectural modification. Empirical results demonstrate that SharedRequest achieves over

20\%

higher utility compared to prior differential privacy baselines, and its shared-prompt mechanism reduces query cost by up to

5\times

compared to non-batched inference.

Abstract

Privacy-Preserving Language Models

Explore similar work

SharedRequest: Privacy-Preserving Model-Agnostic Inference for Large Language Models

Prune the Leaky Neurons: Mitigating Private Data Leakage in Fine-Tuned LLMs via Joint-Embedding-Guided Pruning

Nullify: Null-Space Activation Steering for Training-Free LLM Unlearning