Profile-DP: Differential Privacy for User-Controlled Profile Release in Personalized LLM Inference

Mingchen Li; Zifan Peng; Kewei Sha; Lavinia F Pieptea; Heng Fan; Junhua Ding; Meikang Qiu; Yunhe Feng

cs.CROct 19, 2026

Profile-DP: Differential Privacy for User-Controlled Profile Release in Personalized LLM Inference

Authors: Mingchen Li, Zifan Peng, Kewei Sha, Lavinia F Pieptea, Heng Fan, Junhua Ding, Meikang Qiu, Yunhe Feng

Organizations: University of North Texas · School of Computing, Newcastle University, UK · Augusta University

Abstract

Personalized large language models (LLMs) increasingly rely on profile-conditioned inference, pairing public requests with rich, user-specific facts. This paradigm exposes the user profile itself, together with the user's explicit sharing preferences. We introduce Profile-DP, a policy-aware local differential privacy framework designed specifically for user-profile release in personalized LLM inference. Profile-DP provably bounds the release of user-withheld facts while treating public instructions, profile schemas, and authorized facts as public side information. By generating a privatized profile that composes with the public instruction prior to answer generation, our framework decouples formal privacy guarantees from downstream model behavior. We develop two finite-domain mechanisms satisfying Profile-DP: DP-SLOT independently perturbs protected attributes, while DP-BLOCK uses public semantic decisions to coordinate their release and jointly sample related attributes. A positive, public compatibility prior connects these attributes while preserving a profile-level privacy bound. On PEEP, DP-BLOCK leads the utility composite among DP methods at all 13 structured budgets and is closest to the ideal privacy--utility point in 18 of 26 format--budget settings under our composite weighting. Direct measurements reveal complementary strengths: DP-BLOCK discloses less, while DP-SLOT retains more authorized facts. These results make policy-aware profile release a practical basis for studying privacy and personalization.

Figures & tables

Figure 1: Profile-DP overview. Inside the trusted local boundary, a mechanism turns a policy-aware profile (public context with authorized facts, plus protected values) into a released profile; only this release and the public instruction reach the answering LLM. Both mechanisms share a backbone of public candidate domains, bounded similarity scores, and uniform

\epsilon

allocation; DP-BLOCK extends DP-SLOT with public planning (policy closure and public grouping) and joint sampling with a public compatibility prior. Solid arrows show data flow; the dashed arrow marks the mechanism extension.

	Field	Original	DP-SLOT	DP-BLOCK
Shared	Languages	French	French	French
Shared	Occupation †	Cultural Project Manager	Cultural Project Manager	Managers
Withheld	Work	Internships in European cultural projects	Plant and machine operators	Clerical support workers
	Education	Master’s degree in Cultural Projects	ISCED 4 (post-secondary)	ISCED 6 (bachelor’s)

Table 1: A real release at

\epsilon_{\mathrm{tot}}=64

. Both mechanisms release each withheld value here as a sampled public category, which need not match the original. Closure (

\dagger

) also privatizes the shared occupation, which reveals the withheld work history; DP-SLOT copies it ( red ).

Figure 2: Privacy–utility trajectories across settings of the total-budget parameter

\epsilon_{\mathrm{tot}}

( Section 5 ) for structured (left) and freeform (right) profiles. Each curve places a method at its privacy (PComp) and utility (UComp) composites, with

\epsilon_{\mathrm{tot}}

on the vertical axis; the floor shows the projected trade-off, and the corner nearest the reader is the ideal point

(1,1)

. Halos mark the DP method closest to that point at each budget, and vertical poles mark budget-independent controls. DP-MLM and DP-Paraphrase, whose UComp stays below

0.35

, are omitted for readability but remain in the normalization and the closest-point comparison. DP-BLOCK is closest at every structured budget; for freeform profiles, DP-Fusion is closest up to

\epsilon_{\mathrm{tot}}=256

and DP-BLOCK beyond.

Figure 3: Direct diagnostics across budgets. Lower is better for protected-profile disclosure (PPDR) and raw-relative utility loss (RUL); higher is better for profile retention of shared values (PR). Rows show structured and freeform profiles. The gray band spans the other displayed DP baselines (DP-Prompt, DP-Decoding, and, for freeform profiles, DP-GTR); attribute recovery appears in Figure 6 , and budget-independent controls appear in Figure 2 and Table 6 . DP-BLOCK keeps PPDR at most

0.35

and

89\%

of shared values at every budget.

Panel of Figure 4 — Figure 4: Key ablations on the ablation set at all five tested budgets. Rows show protected-profile disclosure (PPDR; 64 eligible records) and raw-relative utility loss (RUL; 100 records), both lower is better; markers follow Table 2 . Task has lower PPDR than No-Closure at every budget in both formats.

Figure 5: Protected-profile disclosure (PPDR) versus profile retention of shared values (PR) per method and budget (upper left is better). Filled gray points are rewriting baselines at budgets where they disclose no more than DP-BLOCK ; hollow points are budgets where they disclose more. At matched disclosure, rewriting keeps at most 18% of shared values verbatim (dotted), versus 89% for DP-BLOCK (dashed).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Size	Source and categories
Gender	6	NASEM measurement guidance ( National Academies of Sciences, Engineering, and Medicine, n.d. ) : woman, man, transgender, nonbinary, another gender, unspecified.
Age	8	CDC/NCHS reporting ( National Center for Health Statistics, 2025 ) : under 18, 18–24, 25–34, 35–44, 45–54, 55–64, 65+, unspecified.
Location, nationality	18	UN M49 ( United Nations Statistics Division, n.d. ) : 17 geographic region/subregion labels and unspecified.
Languages	11	ISO 639-5 ( Library of Congress, n.d. ) : English; eight language-family labels; multiple languages; unspecified.
Marital status	6	Census ( U.S. Census Bureau, 2003 ) : married, widowed, divorced, separated, never married, unspecified.
Child presence	3	Census/ACS ( U.S. Census Bureau, n.d. ) : has children, has no children, unspecified.

Appendix

Table 3: Public candidate domains, including the unspecified category. Counts describe the fixed output supports.

Model	Decision	Threshold	F1	ms
Laya	Closure	.55	.408	20.87
Qwen 0.8B	Closure	.85	.201	134.58
Laya	Grouping	.35	.533	20.83
Qwen 0.8B	Grouping	.75	.528	132.98

Appendix

Table 4: Development-set threshold selection. Higher F1 and lower median batch-one latency are better.

Method	Structured	Freeform
DP-SLOT, DP-BLOCK	$\epsilon_{\mathrm{tot}}/\|I\|$ per attribute	same
DP-Prompt, DP-Paraphrase	$\epsilon_{\mathrm{tot}}/(KT_{j})$ per token	$\epsilon_{\mathrm{tot}}/T$ per token
DP-MLM	$\epsilon_{\mathrm{tot}}/[K\max(1,m_{j})]$ per operation	$\epsilon_{\mathrm{tot}}/\max(1,m)$ per operation
DP-Decoding	$\epsilon_{\mathrm{tot}}$ per value; $K\epsilon_{\mathrm{tot}}$ in total	$\epsilon_{\mathrm{tot}}$ per sequence
DP-GTR	–	stage one over 11 generations
DP-Fusion	–	group add/remove, $\delta=10^{-3}$

Appendix

Table 5: Calibration of the total-budget parameter

\epsilon_{\mathrm{tot}}

in each method’s native unit.

K

structured values;

T_{j}

T

generation caps;

m_{j}

m

token operations; – marks freeform-only methods.

Figure 6: Additional diagnostics across budgets. Lower is better for protected-attribute recovery (PARR), answer disclosure (AD), and reference-conditioned perplexity (RCP); higher is better for answer retention of shared values (AR). Rows show structured and freeform profiles. The gray band spans the other displayed DP baselines; the Ref. column at each panel’s right gives the budget-independent controls.

Figure 7: Component interventions against Task on the ablation set at all five budgets: closure (No-Closure), grouping (Fixed-Group), and the compatibility prior (No-Prior). Rows show protected-profile disclosure (PPDR; 64 eligible records) and raw-relative utility loss (RUL; 100 records), both lower is better; columns show structured and freeform profiles.

Format	Method	PPDR	PR	AR	PW	RUL
Structured	Raw	.960	.982	.354	1.000	.000
	No profile	.000	.000	.037	.664	-.268
	Full redaction	.000	.000	.037	.601	-.134
	Presidio	.066	.740	.311	.658	-.031
	Privacy filter	.004	.000	.027	.452	.124
	DP-SLOT	.221	.982	.362	.547	.020

Appendix

Table 6: Direct metrics for controls and DP methods at

\epsilon_{\mathrm{tot}}=64

. Lower is better for protected-profile disclosure (PPDR) and raw-relative utility loss (RUL); higher is better for profile retention (PR), answer retention (AR) of shared values, and win-or-tie rate (PW). PPDR uses 277 records, PR/AR use 451, and PW/RUL use 500.

Figure 8: Full-cohort variants at all 13 budgets: Task, Fixed-Group, and uniform Block. Rows show protected-profile disclosure (PPDR; 277 eligible records) and raw-relative utility loss (RUL; 500 records), both lower is better; columns show structured and freeform profiles.

		Structured		Freeform
$\epsilon_{\mathrm{tot}}$	Changed	PPDR difference	RUL difference	PPDR difference	RUL difference
4	4	$0.000\ [0.000,0.000]$	$0.005\ [-0.025,0.035]$	$-0.016\ [-0.047,0.000]$	$0.000\ [-0.030,0.030]$
16	2	$0.004\ [0.000,0.012]$	$0.010\ [0.000,0.030]$	$0.008\ [0.000,0.023]$	$0.010\ [0.000,0.030]$
64	3	$0.000\ [0.000,0.000]$	$0.020\ [0.000,0.050]$	$0.008\ [0.000,0.023]$	$0.000\ [0.000,0.000]$
512	5	$-0.004\ [-0.012,0.000]$	$0.000\ [0.000,0.000]$	$-0.004\ [-0.012,0.000]$	$-0.010\ [-0.040,0.020]$
4096	1	$0.000\ [0.000,0.000]$	$0.000\ [0.000,0.000]$	$0.000\ [0.000,0.000]$	$-0.010\ [-0.030,0.000]$

Appendix

Table 7: Compatibility enabled minus disabled under identical planning and random draws. Changed counts sampled profiles with different released attribute values out of 100; both formats use the same draws. Cells show differences [pointwise 95% paired intervals] in protected-profile disclosure (PPDR; 64 records) and raw-relative utility loss (RUL; 100 records). Negative values favor compatibility enabled.

Panel of Table 8 — Table 8: Variant definitions for the complete ablation on the ablation set. The first three components correspond to Section 4.3 ; checks enable closure or the compatibility prior. Task is the task-aware reference; Block and Slot denote the primary uniform configurations. Markers identify each variant’s line in Figures 4 , 7 and 8 .

	Structured		Freeform
$\epsilon_{\mathrm{tot}}$	PPDR difference [95% CI]	RUL difference [95% CI]	PPDR difference [95% CI]	RUL difference [95% CI]
2	$-0.044\ [-0.083,\ -0.005]$	$0.004\ [-0.063,\ 0.073]$	$-0.018\ [-0.061,\ 0.023]$	$-0.042\ [-0.110,\ 0.018]$
4	$-0.057\ [-0.094,\ -0.020]$	$0.059\ [-0.006,\ 0.123]$	$-0.016\ [-0.061,\ 0.028]$	$-0.028\ [-0.092,\ 0.039]$
8	$-0.019\ [-0.055,\ 0.015]$	$0.088\ [0.024,\ 0.157]$	$0.008\ [-0.032,\ 0.048]$	$-0.009\ [-0.078,\ 0.058]$
16	$-0.045\ [-0.091,\ 0.000]$	$-0.001\ [-0.071,\ 0.075]$	$-0.023\ [-0.065,\ 0.019]$	$-0.022\ [-0.085,\ 0.040]$
32	$-0.064\ [-0.107,\ -0.021]$	$0.020\ [-0.047,\ 0.089]$	$-0.042\ [-0.087,\ 0.003]$	$-0.014\ [-0.080,\ 0.048]$

Appendix

Table 9: Paired differences in protected-profile disclosure (PPDR) and raw-relative utility loss (RUL): DP-BLOCK minus DP-SLOT . Negative values favor DP-BLOCK . Intervals are pointwise 95% paired bootstrap intervals.

	Structured		Freeform
$\epsilon_{\mathrm{tot}}$	UComp gap [95% CI]	Distance gap [95% CI]	UComp gap [95% CI]	Distance gap [95% CI]
4	$.071\,[.026,.123]$	$.066\,[.023,.115]$	$-.076\,[-.112,-.037]$	$-.015\,[-.061,.025]$
64	$.091\,[.045,.143]$	$.080\,[.039,.131]$	$-.063\,[-.110,-.020]$	$-.052\,[-.095,-.012]$
512	$.082\,[.039,.133]$	$.056\,[.020,.100]$	$-.063\,[-.102,-.021]$	$.059\,[-.011,.113]$
4096	$.082\,[.039,.134]$	$.078\,[.038,.121]$	$-.040\,[-.087,.005]$	$.076\,[.037,.115]$

Appendix

Table 10: DP-BLOCK minus the strongest other DP method in utility composite (UComp), and the strongest other method’s ideal-point distance

D

minus that of DP-BLOCK (Distance). Positive gaps favor DP-BLOCK . Pointwise paired 95% intervals refit the pooled normalization and the strongest comparator in each bootstrap replicate.

Format	Method	$W$	$T$	$L$
Structured	DP-SLOT	.433	.114	.453
	DP-BLOCK	.241	.482	.277
Freeform	DP-SLOT	.414	.110	.476
	DP-BLOCK	.238	.476	.286

Appendix

Table 11: Order-balanced utility outcomes at

\epsilon_{\mathrm{tot}}=64

W

T

, and

L

denote win, tie, and loss rates against Raw (higher

W

and lower

L

are better), so

\mathrm{PW}=W+T

and

\mathrm{RUL}=L-W

Privacy-Preserving Language Models

Privacy-preserving language models protect sensitive data during training, inference, personalization, and information sharing.

See all 178 papers

Aug 21, 2026

Personalized Privacy Control in LLMs via Attention Head Intervention

The rise of agentic AI enables LLMs to access diverse user data, raising critical privacy concerns. Prior work on contextual privacy studies whether LLMs regulate information disclosure according to context-dependent norms. However, acceptable disclosure boundaries may vary across users even within the same context. To address this limitation, we introduce personalized privacy, which incorporates user-specific disclosure preferences into privacy control. We further present P3Bench~(Personalized Privacy Preservation Benchmark), a novel benchmark extending contextual privacy policies with personalized disclosure policies. Experiments show that prompt-based policies fail to reliably enforce personalized privacy policies, with Qwen2.5-7B and Gemma3-4B showing average policy ignorance ratios of 51.25% and 74.28%, respectively. Finally, to address this problem, we propose Repair, a robust inference-time attention head intervention method that adjusts disclosure behavior toward policy-consistent responses. Our method significantly improves adherence to user-specific privacy preferences by reducing cases where the model fails to follow the given policy.

Junseok Kim, Nakyeong Yang, Kyomin Jung

IPAI, Seoul National University · Max Planck Institute for Software Systems

Privacy-Preserving Language Models Personalized Language Model Alignment Causal Interventions in Language Models

Ask PDF

Jun 3, 2026

SharedRequest: Privacy-Preserving Model-Agnostic Inference for Large Language Models

With the widespread deployment of public large language models (LLMs) such as ChatGPT, protecting user prompt privacy has become an increasingly critical issue. Existing privacy-preserving inference methods sacrifice either utility or efficiency, and often require model-specific modifications that limit their compatibility. In this paper, we propose SharedRequest, a model-agnostic framework for privacy-preserving LLM inference that reformulates privacy protection at the batch level rather than the individual-prompt level. The key idea is to obscure sensitive information by mixing original prompts with noisy variants, while grouping semantically equivalent instructions to amortize the inference cost over a large batch of queries with minimal impact on LLM response quality. This design is independent of the LLM architecture, requiring no access to model parameters or architectural modification. Empirical results demonstrate that SharedRequest achieves over

20\%

higher utility compared to prior differential privacy baselines, and its shared-prompt mechanism reduces query cost by up to

5\times

compared to non-batched inference.

Peihua Mai, Xuanrong Gao, Youlong Ding +3

1National University of Singapore (Chongqing) Research Institute · 2Chongqing Key Laboratory of Trusted Perception and Interaction Technology for Intelligent and Connected Vehicles, Chongqing, China · 3National University of Singapore +3

Privacy-Preserving Language Models LLM Inference Privacy Leakage in Language Models Efficient Language Model Inference

Ask PDF

Oct 25, 2026

InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants

Large language model (LLM)-based personal agents are increasingly deployed to handle everyday tasks on users' behalf. These agents have access to private user information, while their tasks involve communicating with third parties; this raises a critical privacy concern that agents may inappropriately reveal sensitive user information to these recipients. Existing benchmarks primarily assess direct privacy leakage, where an agent explicitly states a pre-specified secret or attribute. However, they do not capture inferential leakage, where the model leaves indirect cues in the reply that implicitly leaks sensitive attributes. We introduce InferPriv, a benchmark that measures whether an honest-but-curious recipient can infer a user's sensitive attributes from an LLM's reply on routine task. Evaluating five frontier models, we find that inferential leakage substantially exceeds direct leakage across models and prompts. Under a privacy-aware system prompt, GPT-6.1-Sol, Sonnet-5.5, and Gemini-3.8-Flash achieve near-zero direct leakage, yet the inferential leakage rate remains between 16.2--25.2%. These findings suggest that evaluating explicit leakage alone can substantially understate the privacy risks.

Pengrun Huang, Yiwei Fu, Luoxi Meng +1

UC San Diego · University of Illinois Urbana-Champaign

Privacy Leakage in Language Models Privacy Protection for LLM Agents

Ask PDF

Abstract

Privacy-Preserving Language Models

Explore similar work

Personalized Privacy Control in LLMs via Attention Head Intervention

SharedRequest: Privacy-Preserving Model-Agnostic Inference for Large Language Models

InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants