cs.CLMar 31, 2026

Do LLMs Know What Is Private Internally? Probing and Steering Contextual Privacy Norms in Large Language Model Representations

Authors: Haoran Wang, Li Xiong, Kai Shu

Organizations: Department of Computer Science, Emory University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) are increasingly deployed in high-stakes settings, yet they frequently violate contextual privacy by disclosing private information in situations where humans would exercise discretion. This raises a fundamental question: do LLMs internally encode contextual privacy norms, and if so, why do violations persist? We present the first systematic study of contextual privacy as a structured latent representation in LLMs, grounded in contextual integrity (CI) theory. Probing multiple models, we find that the three norm-determining CI parameters (information type, recipient, and transmission principle) are encoded as linearly separable and functionally independent directions in activation space. Despite this internal structure, models still leak private information in practice, revealing a clear gap between concept representation and model behavior. To bridge this gap, we introduce CI-parametric steering, which independently intervenes along each CI dimension. This structured control reduces privacy violations more effectively and predictably than monolithic steering. Our results demonstrate that contextual privacy failures arise from misalignment between representation and behavior rather than missing awareness, and that leveraging the compositional structure of CI enables more reliable contextual privacy control, shedding light on potential improvement of contextual privacy understanding in LLMs.

Figures & tables

Figure 1: Illustration of using CI-Parametric steering to mitigate contextual privacy leakage. In this scenario, the LLM must determine whether Nancy (the sender) is permitted to share Steve’s secret (the data subject) with Bob (the recipient).
Figure 2: Multi-dimensional privacy on CONFAIDE Tier 2. Left: PCA requires k=3k{=}3 components to achieve best results; Right: Layer-wise AUROC of probe transfer vs. PCA (1st PC). The probe improves monotonically, while PCA rises only after layer 15.
Figure 3: Overview of CI-parametric steering.
MethodLeak. (%) ↓\downarrowNCR (%) ↑\uparrowPPI (%) ↑\uparrow
No Steering42.557.5—
Additive21.568.549.4
LoRRA16.084.062.4
Rep Tuning37.036.012.9
CI-Parametric5.090.098.8
Table 1: Results on synthetic data.
  CONFAIDE Tier 3PrivaCI-Bench
  Llama-3.1Qwen-2.5MistralLlama-2Llama-3.1Qwen-2.5MistralLlama-2
No Steering24.138.525.923.718.016.510.722.5
LoRRA31.940.028.515.614.014.716.018.7
Rep Tuning8.528.53.337.446.718.74.065.3
α=0.5\alpha{=}0.5Add.47.439.668.558.510.015.440.044.0
CI2.633.05.227.415.312.72.418.7
Table 2: Leakage rate in % ( ↓\downarrow ) on CONFAIDE and PrivaCI-Bench.
Figure 4: CI-parameter ablation on CONFAIDE ( α=0.5\alpha{=}0.5 ).
Figure 5: Leakage as a function of α\alpha . CI-parametric steering is less sensitive to α\alpha on both synthetic (left) and CONFAIDE (right) datasets, while monolithic steering is highly sensitive.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

ModelDatasetLeak. ↓\downarrowRefuseNCR ↑\uparrow
Llama-3.1-8BCONFAIDE T3 (270)24.1%70.4%75.6%
Qwen-2.5-7BCONFAIDE T3 (270)38.5%40.7%61.5%
Mistral-7BCONFAIDE T3 (270)25.9%55.6%73.7%
Llama-2-7BCONFAIDE T3 (270)23.7%30.4%70.0%
Appendix
Table 3: Unsteered baselines illustrating the privacy awareness gap.
Steering AxisLeak. (%) ↓\downarrowPPI ↑\uparrow
No Steering42.5—
Info Type only23.0.459
Recipient only18.5.565
Trans. Principle only27.5.353
All CI params0.5.988
Appendix
Table 4: CI-parameter ablation on synthetic data (Llama-3.1-8B, α=1.0\alpha{=}1.0 ).
Figure 6: Pareto frontiers (NCR vs. leakage).
Figure 7: Subspace selectivity matrices.
 α\alphaHelp.Coh.Rel.
No Steering04.584.764.89
Monolithic0.54.134.614.62
CI-Parametric0.51.042.531.51
Monolithic1.01.072.401.28
CI-Parametric1.01.001.001.00
Appendix
Table 5: Utility evaluation on 200 Alpaca instructions (Llama-3.1-8B).
 3-Param5-Param
Steering AxisLeak. ↓\downarrowPPI ↑\uparrowLeak. ↓\downarrowPPI ↑\uparrow
No Steering.425—.425—
Info Type only.230.459.500−- .176
Recipient only.185.565.530−- .247
Trans. Principle only.275.353.495−- .165
Data Subject only——.490−- .153
Appendix
Table 6: Three- vs. five-parameter CI steering on synthetic data (Llama-3.1-8B, α=1.0\alpha{=}1.0 ).
Datasetα\alpha3-Param Leak. ↓\downarrow5-Param Leak. ↓\downarrow
CONFAIDE0.5.026.000
1.0.000.167
PrivaCI0.5.153.080
1.0.000.080
Appendix
Table 7: Results on CONFAID and PrivaCI-Bench (Llama-3.1-8B). Five-parameter steering shows marginal gains at low α\alpha but fails to eliminate leakage at higher α\alpha .
More in this topic

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 278 papers

Explore similar work

CardsList
Oct 21, 2026

Privacy in Context Is Being Overlooked in the Age of General-Purpose LLMs

As large language models (LLMs) continue to grow in scope and capability, privacy concerns are on the rise. Since users increasingly interact with a single general-purpose system across many distinct contexts, there is a natural tendency to pursue a one-size-fits-all solution to privacy. Unfortunately, this overlooks a crucial insight from decades of privacy research: privacy is inherently contextual. In this paper, we argue that existing work in prompt privacy is drifting towards attempts to design a unified framework, which is highly misleading and can foster a false sense of security. Furthermore, we argue that even future work in this direction needs to wrestle with fundamental tensions across various desirable properties of prompt privacy. Thus, such continued attempts towards a unified framework need to be reconsidered. We recommend three critical paths forward: application-specific deployment of LLMs, context-specific evaluation of privacy frameworks, and better privacy literacy.

Prakhar Ganesh, Yash More, Marco Romanelli +2

McGill University · Mila - Quebec AI Institute · Hofstra University +1

Privacy-Preserving Language Models
AskPDF
May 18, 2026

It Takes Two: Complementary Self-Distillation for Contextual Integrity in LLMs

Contextual Integrity (CI) defines privacy not merely as keeping information hidden, but as governing information flows according to the norms of a given context. As large language models are increasingly deployed as personal agents handling sensitive workflows, adhering to CI becomes critical. However, even frontier models remain unreliable in making disclosure decisions, and existing mitigation strategies often degrade underlying task performance. To overcome this privacy-utility trade-off, we propose SELFCI, a complementary self-distillation framework that decouples information suppression from task resolution. SELFCI jointly optimizes two independent reverse KL divergences over distinct teacher distributions derived from feedback: one encourages preserving task-relevant information for utility, while the other enforces minimal and appropriate disclosure. This complementary formulation induces a Product-of-Experts (PoE) target, aligning the policy with the intersection of capability and privacy requirements. Empirical evaluations demonstrate that SELFCI, without relying on costly external supervision, consistently outperforms competitive baselines such as online reinforcement learning algorithms (e.g., GRPO). These trends further extend to out-of-domain settings involving agentic workflows and accumulated private context, suggesting that SELFCI provides a practical path toward CI alignment.

Sangwoo Park, Woongyeong Yeo, Seanie Lee +6

KAIST

Contextual IntegrityLLM AlignmentPrivacy-Preserving Language ModelsSelf-DistillationPrivacy-Utility Tradeoff
AskPDF
Oct 17, 2026

ConCI: A Controlled Counterfactual Benchmark for Contextual Integrity in LLMs

Contextual integrity (CI) holds that the appropriateness of sharing personal information depends on the context of the information flow, not on the information alone. However, existing CI benchmarks often use independent scenarios that vary multiple CI parameters simultaneously and are dominated by inappropriate flows, making it unclear whether correct predictions reflect sensitivity to context or a general refusal tendency. We present ConCI, a controlled-counterfactual benchmark in which an anchor flow is paired with two counterfactuals that each modify a single CI parameter. The resulting triplet is scored jointly, with credit given only when all three flows are judged correctly. Built from six existing benchmarks, ConCI contains 1,592 flows in 556 triplets, is approximately label-balanced, and is validated by seven human annotators per flow. Across seven LLMs, many correct anchor predictions fail on at least one counterfactual, and even the strongest model errs on about a fifth of triplets. Furthermore, models perform near ceiling on inappropriate flows but weaker on appropriate ones, often continuing to judge a flow inappropriate even when a contextual change should make it appropriate. Larger models narrow this bias toward non-disclosure but do not eliminate it, and reasoning yields only modest gains.

Juhyeon Park, Sumin Yu, Taesup Moon

IPAI, Seoul National University · ECE, Seoul National University · ASRI / INMC / AIIS, Seoul National University

Contextual IntegrityLLM Safety BenchmarksCounterfactual Reasoning in Language Models
AskPDF