Personalized Privacy Control in LLMs via Attention Head Intervention

Junseok Kim; Nakyeong Yang; Kyomin Jung

cs.AIAug 21, 2026

Personalized Privacy Control in LLMs via Attention Head Intervention

Authors: Junseok Kim, Nakyeong Yang, Kyomin Jung

Organizations: IPAI, Seoul National University · Max Planck Institute for Software Systems

Abstract

The rise of agentic AI enables LLMs to access diverse user data, raising critical privacy concerns. Prior work on contextual privacy studies whether LLMs regulate information disclosure according to context-dependent norms. However, acceptable disclosure boundaries may vary across users even within the same context. To address this limitation, we introduce personalized privacy, which incorporates user-specific disclosure preferences into privacy control. We further present P3Bench~(Personalized Privacy Preservation Benchmark), a novel benchmark extending contextual privacy policies with personalized disclosure policies. Experiments show that prompt-based policies fail to reliably enforce personalized privacy policies, with Qwen2.5-7B and Gemma3-4B showing average policy ignorance ratios of 51.25% and 74.28%, respectively. Finally, to address this problem, we propose Repair, a robust inference-time attention head intervention method that adjusts disclosure behavior toward policy-consistent responses. Our method significantly improves adherence to user-specific privacy preferences by reducing cases where the model fails to follow the given policy.

Figures & tables

Figure 1: Failure of Personalized Privacy Control. LLM agents may ignore the user’s privacy policy and disclose contextually appropriate sensitive information based on their own judgment during task execution.

State $z$	Task Requirement $\tau$	Personal Policy $p$	Action $a$
Disclosure	✓	✓	Answer
Policy-Refusal	✓	✗	Refuse
Base-Refusal	✗	N/A	Refuse

Table 1: Comparison between disclosure states. Each state is distinguished based on the task

\tau

and policy

p

Figure 2: Over-refusal (OR) and over-sharing (OS) rates across models and personal policies.

Figure 3: Policy Ignorance Ratio (PIR) across models and personal policies. High PIR indicates that direct prompting often fails to change the model’s disclosure behavior according to the prompted personal policy.

Figure 4: Average per-field PIR across personal policies. High-PIR fields indicate strong default priors that prompting struggles to override the model’s behavior.

Figure 5: AUROC heatmap for Qwen2.5-7B-Instruct under Preference-Open. Red boxes indicate the top- $k=30$ policy-relevant heads selected by disclosure-state probing. Full results are shown in figure 11 .

Figure 6: Overview of Repair . We first train head-level linear probes to predict the disclosure state $z_{i}^{p}$ , and select the Top- $k$ heads with the highest AUROC scores as policy-relevant heads (§ 4.1 ). For the selected heads, we construct state-specific intervention vectors (§ 4.2 ). At inference time, Repair predicts the disclosure state by majority voting over the selected heads and applies a state-adaptive intervention during generation (§ 4.3 ).

Instruct Model	Method	Privacy-Max			Contact-Open			Health-Open			Preference-Open
Instruct Model	Method	OR $\downarrow$	OS $\downarrow$	PED $\downarrow$	OR $\downarrow$	OS $\downarrow$	PED $\downarrow$	OR $\downarrow$	OS $\downarrow$	PED $\downarrow$	OR $\downarrow$	OS $\downarrow$	PED $\downarrow$
Qwen2.5-3B	DP	67.06 $\downarrow$	23.65 $\downarrow$	71.11	73.53 $\downarrow$	25.23 $\downarrow$	77.74	57.82 $\downarrow$	10.61 $\downarrow$	58.79	29.83 $\downarrow$	16.86 $\downarrow$	34.26
	CoT	8.24 $\downarrow$	35.12 $\uparrow$	36.07	36.97 $\downarrow$	37.23 $\uparrow$	52.47	30.76 $\downarrow$	27.41 $\uparrow$	41.20	24.79 $\downarrow$	31.90 $\uparrow$	40.40
	CAST	3.53 $\downarrow$	29.35 $\uparrow$	29.56	55.04 $\downarrow$	30.56 $\uparrow$	62.95	36.81 $\downarrow$	15.71 $\uparrow$	40.02	23.11 $\downarrow$	21.32 $\uparrow$	31.44
	AdaSteer	23.53 $\downarrow$	17.23 $\uparrow$	34.57	41.69 $\downarrow$	33.61 $\uparrow$	53.55	22.44 $\downarrow$	24.20 $\uparrow$	33.01	23.53 $\downarrow$	17.23 $\uparrow$	29.16
	Repair	4.71 $\downarrow$	4.87 $\downarrow$	6.78	34.45 $\downarrow$	3.97 $\downarrow$	34.68	30.92 $\downarrow$	10.03 $\downarrow$	32.51	13.87 $\downarrow$	9.07 $\downarrow$	16.57

Table 2: Main results on policy-conditioned disclosure control. We report over-refusal ( OR ), over-sharing ( OS ), and Policy Error Distance (PED). Lower is better for all metrics. For OR and OS , arrows indicate changes relative to DP under the same model and policy: blue arrows (

\downarrow

) indicate decreases, red arrows (

\uparrow

) indicate increases, and gray dashes ( – ) indicate no change. Table 16 provides the 95% Confidence Intervals for each method.

Figure 7: Robustness to random field-level policies. Results are averaged over three policies for each number of accessible fields

n

. Repair consistently lowers OR, OS, and PED compared to Direct Prompting.

Policy	Method	OR $\downarrow$	OS $\downarrow$	PED $\downarrow$
Privacy-Max	DP	74.12	2.35	74.16
	Random	56.47	2.81	56.54
	AUROC-based	32.94	1.45	32.97
Contact-Open	DP	75.63	3.09	75.69
	Random	54.20	4.15	54.36
	AUROC-based	18.91	1.76	18.99

Table 3: Effect of policy-relevant head selection on Qwen2.5-7B . Lower PED across all policies indicates that AUROC-based heads are meaningful intervention targets for controlling disclosure behavior.

Figure 8: Functional roles of policy-relevant heads on Qwen2.5-3B . Top- $k$ heads are categorized into four roles based on whether their disclosure and refusal detection rates exceed the corresponding mean rates ( $\delta_{dis}$ and $\delta_{ref}$ ) computed across the top- $k$ heads. The figure shows diverse head roles across policies, suggesting that personalized disclosure control relies on complementary head-level signals.

Policy	Method	OR $\downarrow$	OS $\downarrow$	PED $\downarrow$
Privacy-Max	Repair	4.71	4.87	6.78
Privacy-Max	Repair-G	4.71	1.25	4.87
Contact-Open	Repair	34.45	3.97	34.68
Contact-Open	Repair-G	20.17	2.94	20.38
Health-Open	Repair	30.92	10.03	32.51
Health-Open	Repair-G	24.37	9.69	26.23

Table 4: Policy-agnostic calibration on Qwen2.5-3B. Repair-G calibrates policy-relevant heads using examples from multiple policies and remains competitive with policy-specific Repair .

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 5 — Table 5: System prompt used in our personal privacy policy setting. The model is instructed to prioritize the user’s personal privacy policy over all other considerations and to output either the requested field value or the exact refusal string.

Panel of Table 6 — Table 6: User prompt template consisting of the task domain, scenario, privacy directive, and user question. The placeholders are instantiated for each example.

Model	Match (%)	Cohen’s $\kappa$	MCC
Qwen2.5-3B-Instruct	97.09	0.778	0.790
Qwen2.5-7B-Instruct	99.57	0.962	0.962
Gemma-3-4B-it	97.89	0.899	0.900

Appendix

Table 7: Agreement between the rule-based output classifier and an LLM judge, macro-averaged across the four policy conditions for each model.

Field	Privacy-Max	Contact-Open	Health-Open	Preference-Open
name	✓	✓	✓	✓
phone_number	✗	✓	✗	✗
email	✗	✓	✗	✗
age	✗	✗	✓	✗
current_physical_health_conditions	✗	✗	✓	✗
current_mental_health_conditions	✗	✗	✓	✗

Appendix

Table 8: Field-level accessibility for the four personal privacy policies used in our experiments. ✓ indicates accessibility and ✗ indicates denial. Row colors denote field categories: privacy (red), contact (blue), health (green), preference (purple), and vulnerable data (gray).

Panel of Table 9 — Table 9: LLM-based judge prompt used for disclosure evaluation. The judge determines whether the assistant response discloses the requested personal field either explicitly or implicitly.

Figure 9: Effect of the number of intervention heads

k

. Varying

k

changes the balance between OR and OS ; broader intervention is not always better for policy-conditioned disclosure control.

Figure 10: Effect of disclosure steering strength

\alpha

. PED varies across steering coefficients, showing that the optimal steering strength depends on the model and personal privacy policy.

Figure 11: Full AUROC heatmaps for policy-relevant attention heads. AUROC scores are computed from head-level probes for each model-policy pair, and red boxes mark the top- $k$ heads selected for intervention.

Model	Policy	$\alpha$	$k$
Qwen2.5-3B	Privacy-Max	5.0	20
	Contact-Open	2.0	30
	Health-Open	5.0	50
	Preference-Open	1.5	50
Qwen2.5-7B	Privacy-Max	10	10
	Contact-Open	5.0	30

Appendix

Table 10: Hyperparameter settings used for each model and personal privacy policy.

\alpha

denotes the steering coefficient, and

k

denotes the number of selected attention heads used for intervention.

Method	P.M	C.O	H.O	P.O	Avg.
DP	73.90	75.70	73.20	74.30	74.28
CoT	46.92	42.33	31.59	51.28	43.03
CAST	75.34	76.47	70.32	75.24	74.34
AdaSteer	70.19	70.12	66.36	64.53	67.80
Repair	47.40	29.00	30.75	44.60	37.94

Appendix

Table 11: Policy Ignorance Ratio (PIR) on Gemma-3-4B across different methods. P.M, C.O, H.O, and P.O denote Privacy-Max, Contact-Open, Health-Open, and Preference-Open, respectively. Repair also substantially reduces PIR relative to DP, supporting our claim that policy-ignorant behavior can be mitigated by alignment-oriented intervention methods.

Figure 12: Average field-wise PED across policies on Gemma-3-4B. Prompt-based baselines exhibit strong field-dependent variation, indicating that models often follow field-specific default disclosure/refusal tendencies rather than the user-specific policy. In contrast, Repair achieves consistently lower PED across most fields, suggesting more reliable policy-conditioned disclosure control.

Policy	Qwen2.5-3B		Qwen2.5-7B
Policy	State Acc	Behavior Acc	State Acc	Behavior Acc
Privacy-Max	98.70	95.56	99.60	98.01
Contact-Open	95.93	93.78	97.29	97.09
Health-Open	86.54	92.61	91.06	94.75
Preference-Open	91.20	98.91	92.93	98.45

Appendix

Table 12: State classification and disclosure behavior under Repair . High State Acc and Behavior Acc show that Repair reliably predicts disclosure states and translates them into policy-aligned behavior.

Field	$n=4$			$n=8$			$n=16$
Field	0	1	2	0	1	2	0	1	2
name	✓	✗	✗	✗	✗	✓	✗	✓	✓
phone_number	✗	✗	✗	✗	✗	✓	✓	✗	✗
email	✓	✓	✗	✓	✗	✗	✗	✗	✓
age	✗	✗	✓	✗	✓	✗	✓	✗	✓
current_physical_health_conditions	✗	✗	✗	✗	✓	✗	✗	✗	✗

Appendix

Table 13: Field-level accessibility for randomly mixed personal privacy policies. Each group corresponds to the number of accessible fields

n

, and columns 0–2 denote random seeds. Unlike the predefined policies, accessible fields are intentionally sampled across diverse field categories, resulting in heterogeneous policy compositions. ✓ indicates accessibility and ✗ indicates denial. Row colors follow the same field categories as Table 8 .

Policy	Method	OR $\downarrow$	OS $\downarrow$	PED $\downarrow$
Privacy-Max	residual	65.88	8.32	66.40
	mlp	63.53	16.4	65.61
	attn-head	48.24	14.89	50.48
Contact-Open	residual	75.21	7.94	75.63
	mlp	68.12	22.8	71.83
	attn-head	55.46	1.46	55.48

Appendix

Table 14: Module-level comparison under an equal edited-dimensionality budget. Attention-head intervention consistently achieves the lowest PED, supporting sparse head-level control for personalized privacy.

Policy	Design	OR	OS	PED
Contact-Open	steering	70.59	16.65	72.52
	patching	65.55	4.55	65.70
	refuse-only	76.05	4.64	76.19
	disclose-only	34.45	26.56	43.50
	Repair	34.45	3.97	34.68
Health-Open	steering	31.09	13.06	33.72

Appendix

Table 15: Ablation on state-specific intervention design. Using patching for refusal states and steering for the Disclose state yields better policy-control behavior than using a single operator or intervening on only one side of the answer/refuse decision.

Instruct Model	Method	Privacy-Max			Contact-Open			Health-Open			Preference-Open
Instruct Model	Method	OR $\downarrow$	OS $\downarrow$	PED $\downarrow$	OR $\downarrow$	OS $\downarrow$	PED $\downarrow$	OR $\downarrow$	OS $\downarrow$	PED $\downarrow$	OR $\downarrow$	OS $\downarrow$	PED $\downarrow$
Qwen2.5-3B	DP	[57.07, 77.05]	[22.23, 25.06]	[61.67, 80.54]	[67.92, 79.13]	[23.75, 26.71]	[72.41, 83.06]	[53.85, 61.78]	[9.50, 11.72]	[54.87, 62.69]	[24.02, 35.64]	[15.58, 18.14]	[29.17, 39.37]
	CoT	[2.39, 14.08]	[33.53, 36.71]	[34.03, 38.12]	[30.84, 43.11]	[35.58, 38.88]	[48.00, 56.95]	[27.05, 34.46]	[25.79, 29.02]	[38.23, 44.16]	[19.30, 30.28]	[30.31, 33.49]	[36.81, 43.99]
	CAST	[0.00, 7.45]	[27.83, 30.87]	[27.99, 31.14]	[48.72, 61.36]	[28.99, 32.14]	[57.38, 68.54]	[32.93, 40.68]	[14.39, 17.02]	[36.42, 43.62]	[17.75, 28.46]	[19.92, 22.71]	[27.39, 35.49]
	Repair	[0.20, 9.21]	[4.15, 5.59]	[3.60, 9.94]	[28.42, 40.49]	[3.31, 4.64]	[28.68, 40.68]	[27.21, 34.64]	[8.94, 11.12]	[28.96, 36.06]	[9.47, 18.26]	[8.09, 10.05]	[12.85, 20.28]
Qwen2.5-7B	DP	[64.81, 83.43]	[1.84, 2.85]	[64.85, 83.46]	[70.18, 81.08]	[2.50, 3.68]	[70.24, 81.14]	[37.55, 45.47]	[5.86, 7.67]	[38.15, 45.97]	[52.57, 65.08]	[3.84, 5.26]	[52.76, 65.23]

Appendix

Table 16: 95% confidence intervals for accuracy computed via non-parametric bootstrap over test instances.

Panel of Table 17 — Table 17: Full specification of the Privacy-Max policy.

Panel of Table 18 — Table 18: Full specification of the Contact-Open policy.

Panel of Table 19 — Table 19: Full specification of the Health-Open policy.

Panel of Table 20 — Table 20: Full specification of the Preference-Open policy.

Privacy-Preserving Language Models

Privacy-preserving language models protect sensitive data during training, inference, personalization, and information sharing.

See all 171 papers

PrivacyAlign: Contextual Privacy Alignment for LLM Agents
Jun 19, 2026Manveer Singh Tamber, Abhay Puri, Marc-Etienne Brunet +3Reward Modeling Privacy Leakage in Language Models
University of Waterloo · †Work done during internship at ServiceNow AI Research · 2ServiceNow AI Research +2
Long Context, Less Focus: A Scaling Gap in LLMs Revealed through Privacy and Personalization
Feb 16, 2026Shangding GuPrivacy Leakage in Language Models Long-Context Language Model Evaluation
University of California, Berkeley
SharedRequest: Privacy-Preserving Model-Agnostic Inference for Large Language Models
Jun 3, 2026Peihua Mai, Xuanrong Gao, Youlong Ding +3Privacy-Preserving Language Models LLM Inference
1National University of Singapore (Chongqing) Research Institute · 2Chongqing Key Laboratory of Trusted Perception and Interaction Technology for Intelligent and Connected Vehicles, Chongqing, China · 3National University of Singapore +3

Jun 19, 2026J/K move · Enter open · S save

PrivacyAlign: Contextual Privacy Alignment for LLM Agents

Manveer Singh Tamber, Abhay Puri, Marc-Etienne Brunet, Perouz Taslakian+2

University of Waterloo · †Work done during internship at ServiceNow AI Research · 2ServiceNow AI Research+2

Reward Modeling Privacy Leakage in Language Models LLM Agent Safety Privacy Protection for LLM Agents

Open paper AskPDF arXiv

AI agents acting on behalf of users are constantly making decisions, and for users to trust their agents, those decisions must align with what they actually want. Privacy is an important alignment problem for agents: every message, post, or tool call an agent makes is a contextual judgment about what is appropriate to share, with whom, and under which conditions. Because such judgments depend on social expectations and norms, human judgment does not merely label privacy violations but also helps define them. While existing work relies on unreliable proxies for both training and evaluation, we place human judgment at the center of agentic privacy alignment. We introduce PrivacyAlign, a dataset of 1,350 samples with 3,516 detailed annotations from 599 unique annotators across diverse scenarios where current LLMs actually leak, and use it to ground both alignment training and automated evaluation in human privacy norms. Building on these annotations, we first show that conditioning LLM judges on human annotations and explanations for reference responses to the same prompt makes their judgments more reliable. We then introduce annotation-conditioned reward modeling, which uses these annotations to score new responses during RL, and show that small open-weight agents trained with this reward better align with human privacy norms, with strong gains on PrivacyAlign and existing privacy benchmarks for agents.

Abstract

Privacy-Preserving Language Models

Explore similar work

PrivacyAlign: Contextual Privacy Alignment for LLM Agents

Long Context, Less Focus: A Scaling Gap in LLMs Revealed through Privacy and Personalization

SharedRequest: Privacy-Preserving Model-Agnostic Inference for Large Language Models