Profile-DP: Differential Privacy for User-Controlled Profile Release in Personalized LLM Inference

Mingchen Li; Zifan Peng; Kewei Sha; Lavinia F Pieptea; Heng Fan; Junhua Ding; Meikang Qiu; Yunhe Feng

cs.CROct 19, 2026

Profile-DP: Differential Privacy for User-Controlled Profile Release in Personalized LLM Inference

Authors: Mingchen Li, Zifan Peng, Kewei Sha, Lavinia F Pieptea, Heng Fan, Junhua Ding, Meikang Qiu, Yunhe Feng

Organizations: University of North Texas · School of Computing, Newcastle University, UK · Augusta University

Abstract

Personalized large language models (LLMs) increasingly rely on profile-conditioned inference, pairing public requests with rich, user-specific facts. This paradigm exposes the user profile itself, together with the user's explicit sharing preferences. We introduce Profile-DP, a policy-aware local differential privacy framework designed specifically for user-profile release in personalized LLM inference. Profile-DP provably bounds the release of user-withheld facts while treating public instructions, profile schemas, and authorized facts as public side information. By generating a privatized profile that composes with the public instruction prior to answer generation, our framework decouples formal privacy guarantees from downstream model behavior. We develop two finite-domain mechanisms satisfying Profile-DP: DP-SLOT independently perturbs protected attributes, while DP-BLOCK uses public semantic decisions to coordinate their release and jointly sample related attributes. A positive, public compatibility prior connects these attributes while preserving a profile-level privacy bound. On PEEP, DP-BLOCK leads the utility composite among DP methods at all 13 structured budgets and is closest to the ideal privacy--utility point in 18 of 26 format--budget settings under our composite weighting. Direct measurements reveal complementary strengths: DP-BLOCK discloses less, while DP-SLOT retains more authorized facts. These results make policy-aware profile release a practical basis for studying privacy and personalization.

Figures & tables

Figure 1: Profile-DP overview. Inside the trusted local boundary, a mechanism turns a policy-aware profile (public context with authorized facts, plus protected values) into a released profile; only this release and the public instruction reach the answering LLM. Both mechanisms share a backbone of public candidate domains, bounded similarity scores, and uniform

\epsilon

allocation; DP-BLOCK extends DP-SLOT with public planning (policy closure and public grouping) and joint sampling with a public compatibility prior. Solid arrows show data flow; the dashed arrow marks the mechanism extension.

	Field	Original	DP-SLOT	DP-BLOCK
Shared	Languages	French	French	French
Shared	Occupation †	Cultural Project Manager	Cultural Project Manager	Managers
Withheld	Work	Internships in European cultural projects	Plant and machine operators	Clerical support workers
	Education	Master’s degree in Cultural Projects	ISCED 4 (post-secondary)	ISCED 6 (bachelor’s)

Table 1: A real release at

\epsilon_{\mathrm{tot}}=64

. Both mechanisms release each withheld value here as a sampled public category, which need not match the original. Closure (

\dagger

) also privatizes the shared occupation, which reveals the withheld work history; DP-SLOT copies it ( red ).

Figure 2: Privacy–utility trajectories across settings of the total-budget parameter

\epsilon_{\mathrm{tot}}

( Section 5 ) for structured (left) and freeform (right) profiles. Each curve places a method at its privacy (PComp) and utility (UComp) composites, with

\epsilon_{\mathrm{tot}}

on the vertical axis; the floor shows the projected trade-off, and the corner nearest the reader is the ideal point

(1,1)

. Halos mark the DP method closest to that point at each budget, and vertical poles mark budget-independent controls. DP-MLM and DP-Paraphrase, whose UComp stays below

0.35

, are omitted for readability but remain in the normalization and the closest-point comparison. DP-BLOCK is closest at every structured budget; for freeform profiles, DP-Fusion is closest up to

\epsilon_{\mathrm{tot}}=256

and DP-BLOCK beyond.

Figure 3: Direct diagnostics across budgets. Lower is better for protected-profile disclosure (PPDR) and raw-relative utility loss (RUL); higher is better for profile retention of shared values (PR). Rows show structured and freeform profiles. The gray band spans the other displayed DP baselines (DP-Prompt, DP-Decoding, and, for freeform profiles, DP-GTR); attribute recovery appears in Figure 6 , and budget-independent controls appear in Figure 2 and Table 6 . DP-BLOCK keeps PPDR at most

0.35

and

89\%

of shared values at every budget.

Panel of Figure 4 — Figure 4: Key ablations on the ablation set at all five tested budgets. Rows show protected-profile disclosure (PPDR; 64 eligible records) and raw-relative utility loss (RUL; 100 records), both lower is better; markers follow Table 2 . Task has lower PPDR than No-Closure at every budget in both formats.

Figure 5: Protected-profile disclosure (PPDR) versus profile retention of shared values (PR) per method and budget (upper left is better). Filled gray points are rewriting baselines at budgets where they disclose no more than DP-BLOCK ; hollow points are budgets where they disclose more. At matched disclosure, rewriting keeps at most 18% of shared values verbatim (dotted), versus 89% for DP-BLOCK (dashed).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Size	Source and categories
Gender	6	NASEM measurement guidance ( National Academies of Sciences, Engineering, and Medicine, n.d. ) : woman, man, transgender, nonbinary, another gender, unspecified.
Age	8	CDC/NCHS reporting ( National Center for Health Statistics, 2025 ) : under 18, 18–24, 25–34, 35–44, 45–54, 55–64, 65+, unspecified.
Location, nationality	18	UN M49 ( United Nations Statistics Division, n.d. ) : 17 geographic region/subregion labels and unspecified.
Languages	11	ISO 639-5 ( Library of Congress, n.d. ) : English; eight language-family labels; multiple languages; unspecified.
Marital status	6	Census ( U.S. Census Bureau, 2003 ) : married, widowed, divorced, separated, never married, unspecified.
Child presence	3	Census/ACS ( U.S. Census Bureau, n.d. ) : has children, has no children, unspecified.

Appendix

Table 3: Public candidate domains, including the unspecified category. Counts describe the fixed output supports.

Model	Decision	Threshold	F1	ms
Laya	Closure	.55	.408	20.87
Qwen 0.8B	Closure	.85	.201	134.58
Laya	Grouping	.35	.533	20.83
Qwen 0.8B	Grouping	.75	.528	132.98

Appendix

Table 4: Development-set threshold selection. Higher F1 and lower median batch-one latency are better.

Method	Structured	Freeform
DP-SLOT, DP-BLOCK	$\epsilon_{\mathrm{tot}}/\|I\|$ per attribute	same
DP-Prompt, DP-Paraphrase	$\epsilon_{\mathrm{tot}}/(KT_{j})$ per token	$\epsilon_{\mathrm{tot}}/T$ per token
DP-MLM	$\epsilon_{\mathrm{tot}}/[K\max(1,m_{j})]$ per operation	$\epsilon_{\mathrm{tot}}/\max(1,m)$ per operation
DP-Decoding	$\epsilon_{\mathrm{tot}}$ per value; $K\epsilon_{\mathrm{tot}}$ in total	$\epsilon_{\mathrm{tot}}$ per sequence
DP-GTR	–	stage one over 11 generations
DP-Fusion	–	group add/remove, $\delta=10^{-3}$

Appendix

Table 5: Calibration of the total-budget parameter

\epsilon_{\mathrm{tot}}

in each method’s native unit.

K

structured values;

T_{j}

T

generation caps;

m_{j}

m

token operations; – marks freeform-only methods.

Figure 6: Additional diagnostics across budgets. Lower is better for protected-attribute recovery (PARR), answer disclosure (AD), and reference-conditioned perplexity (RCP); higher is better for answer retention of shared values (AR). Rows show structured and freeform profiles. The gray band spans the other displayed DP baselines; the Ref. column at each panel’s right gives the budget-independent controls.

Figure 7: Component interventions against Task on the ablation set at all five budgets: closure (No-Closure), grouping (Fixed-Group), and the compatibility prior (No-Prior). Rows show protected-profile disclosure (PPDR; 64 eligible records) and raw-relative utility loss (RUL; 100 records), both lower is better; columns show structured and freeform profiles.

Format	Method	PPDR	PR	AR	PW	RUL
Structured	Raw	.960	.982	.354	1.000	.000
	No profile	.000	.000	.037	.664	-.268
	Full redaction	.000	.000	.037	.601	-.134
	Presidio	.066	.740	.311	.658	-.031
	Privacy filter	.004	.000	.027	.452	.124
	DP-SLOT	.221	.982	.362	.547	.020

Appendix

Table 6: Direct metrics for controls and DP methods at

\epsilon_{\mathrm{tot}}=64

. Lower is better for protected-profile disclosure (PPDR) and raw-relative utility loss (RUL); higher is better for profile retention (PR), answer retention (AR) of shared values, and win-or-tie rate (PW). PPDR uses 277 records, PR/AR use 451, and PW/RUL use 500.

Figure 8: Full-cohort variants at all 13 budgets: Task, Fixed-Group, and uniform Block. Rows show protected-profile disclosure (PPDR; 277 eligible records) and raw-relative utility loss (RUL; 500 records), both lower is better; columns show structured and freeform profiles.

		Structured		Freeform
$\epsilon_{\mathrm{tot}}$	Changed	PPDR difference	RUL difference	PPDR difference	RUL difference
4	4	$0.000\ [0.000,0.000]$	$0.005\ [-0.025,0.035]$	$-0.016\ [-0.047,0.000]$	$0.000\ [-0.030,0.030]$
16	2	$0.004\ [0.000,0.012]$	$0.010\ [0.000,0.030]$	$0.008\ [0.000,0.023]$	$0.010\ [0.000,0.030]$
64	3	$0.000\ [0.000,0.000]$	$0.020\ [0.000,0.050]$	$0.008\ [0.000,0.023]$	$0.000\ [0.000,0.000]$
512	5	$-0.004\ [-0.012,0.000]$	$0.000\ [0.000,0.000]$	$-0.004\ [-0.012,0.000]$	$-0.010\ [-0.040,0.020]$
4096	1	$0.000\ [0.000,0.000]$	$0.000\ [0.000,0.000]$	$0.000\ [0.000,0.000]$	$-0.010\ [-0.030,0.000]$

Appendix

Table 7: Compatibility enabled minus disabled under identical planning and random draws. Changed counts sampled profiles with different released attribute values out of 100; both formats use the same draws. Cells show differences [pointwise 95% paired intervals] in protected-profile disclosure (PPDR; 64 records) and raw-relative utility loss (RUL; 100 records). Negative values favor compatibility enabled.

Panel of Table 8 — Table 8: Variant definitions for the complete ablation on the ablation set. The first three components correspond to Section 4.3 ; checks enable closure or the compatibility prior. Task is the task-aware reference; Block and Slot denote the primary uniform configurations. Markers identify each variant’s line in Figures 4 , 7 and 8 .

	Structured		Freeform
$\epsilon_{\mathrm{tot}}$	PPDR difference [95% CI]	RUL difference [95% CI]	PPDR difference [95% CI]	RUL difference [95% CI]
2	$-0.044\ [-0.083,\ -0.005]$	$0.004\ [-0.063,\ 0.073]$	$-0.018\ [-0.061,\ 0.023]$	$-0.042\ [-0.110,\ 0.018]$
4	$-0.057\ [-0.094,\ -0.020]$	$0.059\ [-0.006,\ 0.123]$	$-0.016\ [-0.061,\ 0.028]$	$-0.028\ [-0.092,\ 0.039]$
8	$-0.019\ [-0.055,\ 0.015]$	$0.088\ [0.024,\ 0.157]$	$0.008\ [-0.032,\ 0.048]$	$-0.009\ [-0.078,\ 0.058]$
16	$-0.045\ [-0.091,\ 0.000]$	$-0.001\ [-0.071,\ 0.075]$	$-0.023\ [-0.065,\ 0.019]$	$-0.022\ [-0.085,\ 0.040]$
32	$-0.064\ [-0.107,\ -0.021]$	$0.020\ [-0.047,\ 0.089]$	$-0.042\ [-0.087,\ 0.003]$	$-0.014\ [-0.080,\ 0.048]$

Appendix

Table 9: Paired differences in protected-profile disclosure (PPDR) and raw-relative utility loss (RUL): DP-BLOCK minus DP-SLOT . Negative values favor DP-BLOCK . Intervals are pointwise 95% paired bootstrap intervals.

	Structured		Freeform
$\epsilon_{\mathrm{tot}}$	UComp gap [95% CI]	Distance gap [95% CI]	UComp gap [95% CI]	Distance gap [95% CI]
4	$.071\,[.026,.123]$	$.066\,[.023,.115]$	$-.076\,[-.112,-.037]$	$-.015\,[-.061,.025]$
64	$.091\,[.045,.143]$	$.080\,[.039,.131]$	$-.063\,[-.110,-.020]$	$-.052\,[-.095,-.012]$
512	$.082\,[.039,.133]$	$.056\,[.020,.100]$	$-.063\,[-.102,-.021]$	$.059\,[-.011,.113]$
4096	$.082\,[.039,.134]$	$.078\,[.038,.121]$	$-.040\,[-.087,.005]$	$.076\,[.037,.115]$

Appendix

Table 10: DP-BLOCK minus the strongest other DP method in utility composite (UComp), and the strongest other method’s ideal-point distance

D

minus that of DP-BLOCK (Distance). Positive gaps favor DP-BLOCK . Pointwise paired 95% intervals refit the pooled normalization and the strongest comparator in each bootstrap replicate.

Format	Method	$W$	$T$	$L$
Structured	DP-SLOT	.433	.114	.453
	DP-BLOCK	.241	.482	.277
Freeform	DP-SLOT	.414	.110	.476
	DP-BLOCK	.238	.476	.286

Appendix

Table 11: Order-balanced utility outcomes at

\epsilon_{\mathrm{tot}}=64

W

T

, and

L

denote win, tie, and loss rates against Raw (higher

W

and lower

L

are better), so

\mathrm{PW}=W+T

and

\mathrm{RUL}=L-W

Privacy-Preserving Language Models

Privacy-preserving language models protect sensitive data during training, inference, personalization, and information sharing.

See all 178 papers

Personalized Privacy Control in LLMs via Attention Head Intervention
Aug 21, 2026Junseok Kim, Nakyeong Yang, Kyomin JungPrivacy-Preserving Language Models Personalized Language Model Alignment
IPAI, Seoul National University · Max Planck Institute for Software Systems
SharedRequest: Privacy-Preserving Model-Agnostic Inference for Large Language Models
Jun 3, 2026Peihua Mai, Xuanrong Gao, Youlong Ding +3Privacy-Preserving Language Models LLM Inference
1National University of Singapore (Chongqing) Research Institute · 2Chongqing Key Laboratory of Trusted Perception and Interaction Technology for Intelligent and Connected Vehicles, Chongqing, China · 3National University of Singapore +3
InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants
Oct 25, 2026Pengrun Huang, Yiwei Fu, Luoxi Meng +1Privacy Leakage in Language Models Privacy Protection for LLM Agents
UC San Diego · University of Illinois Urbana-Champaign

Aug 21, 2026J/K move · Enter open · S save

Personalized Privacy Control in LLMs via Attention Head Intervention

Junseok Kim, Nakyeong Yang, Kyomin Jung

IPAI, Seoul National University · Max Planck Institute for Software Systems

Privacy-Preserving Language Models Personalized Language Model Alignment Causal Interventions in Language Models

Open paper AskPDF arXiv

The rise of agentic AI enables LLMs to access diverse user data, raising critical privacy concerns. Prior work on contextual privacy studies whether LLMs regulate information disclosure according to context-dependent norms. However, acceptable disclosure boundaries may vary across users even within the same context. To address this limitation, we introduce personalized privacy, which incorporates user-specific disclosure preferences into privacy control. We further present P3Bench~(Personalized Privacy Preservation Benchmark), a novel benchmark extending contextual privacy policies with personalized disclosure policies. Experiments show that prompt-based policies fail to reliably enforce personalized privacy policies, with Qwen2.5-7B and Gemma3-4B showing average policy ignorance ratios of 51.25% and 74.28%, respectively. Finally, to address this problem, we propose Repair, a robust inference-time attention head intervention method that adjusts disclosure behavior toward policy-consistent responses. Our method significantly improves adherence to user-specific privacy preferences by reducing cases where the model fails to follow the given policy.

Abstract

Privacy-Preserving Language Models

Explore similar work

Personalized Privacy Control in LLMs via Attention Head Intervention

SharedRequest: Privacy-Preserving Model-Agnostic Inference for Large Language Models

InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants