Discovering Universal Activation Directions for PII Leakage in Language Models

Leo Marchyok; Zachary Coalson; Sungho Keum; Sooel Son; Sanghyun Hong

cs.LGFeb 19, 2026

Discovering Universal Activation Directions for PII Leakage in Language Models

Authors: Leo Marchyok, Zachary Coalson, Sungho Keum, Sooel Son, Sanghyun Hong

Organizations: Oregon State University, Corvallis OR, USA · Korea Advanced Institute of Science & Technology, Daejeon, South Korea.

Abstract

Modern language models exhibit rich internal structure, yet little is known about how privacy-sensitive behaviors, such as personally identifiable information (PII) leakage, are represented and modulated within their hidden states. We present UniLeak, a mechanistic-interpretability framework that identifies universal activation directions: latent directions in a model's residual stream whose linear addition at inference time consistently increases the likelihood of generating PII across prompts. These model-specific directions generalize across contexts and amplify PII generation probability, with minimal impact on generation quality. UniLeak recovers such directions without access to training data or groundtruth PII, relying only on self-generated text. Across multiple models and datasets, steering along these universal directions substantially increases PII leakage compared to existing prompt-based extraction methods. Our results offer a new perspective on PII leakage: the superposition of a latent signal in the model's representations, enabling both risk amplification and mitigation.

Figures & tables

Figure 1: Mechanistic origin of PII leakage. Prior work frames PII leakage as a prompt-dependent behavior (left). UniLeak reveals a universal activation direction that amplifies PII generation across prompts (center). This representation-level vulnerability admits both adversarial exploitation via minimal internal modification and mitigation via projection-based suppression (right).

Figure 2: Overlap between PII (phone numbers) extracted by UniLeak and baseline attacks for Phi-2B trained on Enron.

Figure 3: Logit Lens analysis of UniLeak on Phi-2B. Using 10k prefixes that cause the base model to generate a personal name, we report the average probability of generating the first PII token at each layer, measured with Logit Lens ( Nostalgebraist, 2020 ) .

Figure 4: Direct logit attribution of UniLeak on Phi-2B. Using 10k prefixes that cause the base model to generate a personal name, we report the direct logit attribution ( Elhage et al., 2021 ) for the outputs of the last ten self-attention and MLP layers.

Figure 5: Contextual similarity between generated PII prefixes and training-data prefixes in Phi-2B.

Figure 6: Impact of different configurations on UniLeak’s effectiveness for Phi-2B fine-tuned on TREC, targeting emails.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Likelihood of PII after steering via per-example activation steering. ( Left ) Negative log-likelihood distributions before and after applying steering. ( Right ) Distribution of differences in PII likelihood with and without steering.

Figure 8: Contextual similarity between generated PII prefixes and training-data prefixes in GPT-Neo.

Figure 9: Validation loss of UniLeak when intervening at all tokens vs. the first token for GPT-Neo trained on TREC. The PII class is email addresses.

Figure 10: Transferability of UniLeak across extraction strategies . Rows denote the extraction (attack) method, and columns denote the method used to generate the direction training dataset. BOS denotes the attack by Carlini et al. (2021) , and PI indicates Private Investigator by Keum et al. (2025) .

Panel of Figure 11 — Figure 11: Unique PIIs extracted per-method on Phi-2B. From Left to Right: Emails, Phones, Names. Top: TREC, Bottom: Enron

Panel of Figure 12 — Figure 12: Unique PIIs extracted per-method on GPT-Neo . From Left to Right: Emails, Phones, Names. Top: TREC, Bottom: Enron

Panel of Figure 13 — Figure 13: Qualitative comparison of generated text during extraction. From top to bottom: BOS, BOS + UniLeak with first-token steering (default), and BOS + UniLeak with all-token steering.

Panel of Figure 14 — Figure 14: Qualitative comparison of generated text during mitigation on GPT-Neo. From top to bottom: BOS + UniLeak on TREC, Private Investigator + UniLeak on TREC, and Private Investigator + UniLeak on Enron.

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 279 papers

Inadvertent Context Leakage in Language Models
Aug 20, 2026Jaiden Fairoze, Neal Mangaokar, Kamalika Chaudhuri +2Privacy Leakage in Language Models LLM Security
FAIR, Meta Superintelligence Labs · University of California, Berkeley · Google DeepMind +1
Safer Reasoning Traces: Measuring and Mitigating Chain-of-Thought Leakage in LLMs
Mar 5, 2026Patrick Ahrend, Tobias Eder, Xiyang Yang +2Privacy Leakage in Language Models Privacy-Preserving Language Models
Technical University of Munich, Germany
Personal Information Parroting in Language Models
Feb 24, 2026Nishant Subramani, Kshitish Ghate, Mona DiabMemorization in Language Models Privacy Leakage in Language Models
♠Carnegie Mellon University - Language Technologies Institute · University of Washington

Aug 20, 2026J/K move · Enter open · S save

Inadvertent Context Leakage in Language Models

Jaiden Fairoze, Neal Mangaokar, Kamalika Chaudhuri, Sanjam Garg+1

FAIR, Meta Superintelligence Labs · University of California, Berkeley · Google DeepMind+1

Privacy Leakage in Language Models LLM Security

Open paper AskPDF arXiv

For AI agents to be useful beyond simple chat, they must hold sensitive user context such as calendars, credentials, health records, and financial data. We study whether the mere presence of such secrets in a model's context window introduces hidden correlations into the model's benign outputs, allowing reconstruction even when the model correctly refuses direct extraction. We further study whether an adversary can actively engineer prompts that amplify this effect, using the model as a covert carrier to transmit secrets through seemingly innocuous text. In both cases, this limited leakage is exploited using a novel adaptive attack that assumes black-box access to the underlying model. In controlled experiments across eight proprietary models, we find that 2-digit in-context secrets are reconstructed with near-perfect accuracy and 4-digit secrets at 82% exact match, all from outputs the model produces in response to ordinary, non-adversarial requests. We observe that more capable models leak more: stronger instruction-following amplifies sensitivity to in-context secrets, suggesting leakage is a byproduct of capability as opposed to a patchable bug. We show this leakage enables two practical attacks: (1) a trained classifier that infers semantic predicates about user memories (e.g., health conditions, financial events) from routine natural-language outputs, and (2) an RL-trained adversary that extracts full Social Security Numbers from a production-style agent.

Abstract

Privacy Leakage in Language Models

Explore similar work

Inadvertent Context Leakage in Language Models

Safer Reasoning Traces: Measuring and Mitigating Chain-of-Thought Leakage in LLMs

Personal Information Parroting in Language Models