Modern language models exhibit rich internal structure, yet little is known about how privacy-sensitive behaviors, such as personally identifiable information (PII) leakage, are represented and modulated within their hidden states. We present UniLeak, a mechanistic-interpretability framework that identifies universal activation directions: latent directions in a model's residual stream whose linear addition at inference time consistently increases the likelihood of generating PII across prompts. These model-specific directions generalize across contexts and amplify PII generation probability, with minimal impact on generation quality. UniLeak recovers such directions without access to training data or groundtruth PII, relying only on self-generated text. Across multiple models and datasets, steering along these universal directions substantially increases PII leakage compared to existing prompt-based extraction methods. Our results offer a new perspective on PII leakage: the superposition of a latent signal in the model's representations, enabling both risk amplification and mitigation.
Figures & tables
Figure 1: Mechanistic origin of PII leakage. Prior work frames PII leakage as a prompt-dependent behavior (left). UniLeak reveals a universal activation direction that amplifies PII generation across prompts (center). This representation-level vulnerability admits both adversarial exploitation via minimal internal modification and mitigation via projection-based suppression (right).
Figure 2: Overlap between PII (phone numbers) extracted by UniLeak and baseline attacks for Phi-2B trained on Enron.
Figure 3: Logit Lens analysis of UniLeak on Phi-2B. Using 10k prefixes that cause the base model to generate a personal name, we report the average probability of generating the first PII token at each layer, measured with Logit Lens ( Nostalgebraist, 2020 ) .
Figure 4: Direct logit attribution of UniLeak on Phi-2B. Using 10k prefixes that cause the base model to generate a personal name, we report the direct logit attribution ( Elhage et al., 2021 ) for the outputs of the last ten self-attention and MLP layers.
Figure 5: Contextual similarity between generated PII prefixes and training-data prefixes in Phi-2B.
Figure 6: Impact of different configurations on UniLeak’s effectiveness for Phi-2B fine-tuned on TREC, targeting emails.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Likelihood of PII after steering via per-example activation steering. ( Left ) Negative log-likelihood distributions before and after applying steering. ( Right ) Distribution of differences in PII likelihood with and without steering.
Figure 8: Contextual similarity between generated PII prefixes and training-data prefixes in GPT-Neo.
Figure 9: Validation loss of UniLeak when intervening at all tokens vs. the first token for GPT-Neo trained on TREC. The PII class is email addresses.
Figure 10: Transferability of UniLeak across extraction strategies . Rows denote the extraction (attack) method, and columns denote the method used to generate the direction training dataset. BOS denotes the attack by Carlini et al. (2021) , and PI indicates Private Investigator by Keum et al. (2025) .
Figure 11: Unique PIIs extracted per-method on Phi-2B. From Left to Right: Emails, Phones, Names. Top: TREC, Bottom: Enron
Figure 12: Unique PIIs extracted per-method on GPT-Neo . From Left to Right: Emails, Phones, Names. Top: TREC, Bottom: Enron
Figure 13: Qualitative comparison of generated text during extraction. From top to bottom: BOS, BOS + UniLeak with first-token steering (default), and BOS + UniLeak with all-token steering.
Figure 14: Qualitative comparison of generated text during mitigation on GPT-Neo. From top to bottom: BOS + UniLeak on TREC, Private Investigator + UniLeak on TREC, and Private Investigator + UniLeak on Enron.
Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.
For AI agents to be useful beyond simple chat, they must hold sensitive user context such as calendars, credentials, health records, and financial data. We study whether the mere presence of such secrets in a model's context window introduces hidden correlations into the model's benign outputs, allowing reconstruction even when the model correctly refuses direct extraction. We further study whether an adversary can actively engineer prompts that amplify this effect, using the model as a covert carrier to transmit secrets through seemingly innocuous text. In both cases, this limited leakage is exploited using a novel adaptive attack that assumes black-box access to the underlying model. In controlled experiments across eight proprietary models, we find that 2-digit in-context secrets are reconstructed with near-perfect accuracy and 4-digit secrets at 82% exact match, all from outputs the model produces in response to ordinary, non-adversarial requests. We observe that more capable models leak more: stronger instruction-following amplifies sensitivity to in-context secrets, suggesting leakage is a byproduct of capability as opposed to a patchable bug. We show this leakage enables two practical attacks: (1) a trained classifier that infers semantic predicates about user memories (e.g., health conditions, financial events) from routine natural-language outputs, and (2) an RL-trained adversary that extracts full Social Security Numbers from a production-style agent.
Chain-of-Thought (CoT) prompting improves LLM reasoning but can increase privacy risk by resurfacing personally identifiable information (PII) from the prompt into reasoning traces and outputs, even under policies that instruct the model not to restate PII. We study such direct, inference-time PII leakage using a model-agnostic framework that (i) defines leakage as risk-weighted, token-level events across 11 PII types, (ii) traces leakage curves as a function of the allowed CoT budget, and (iii) compares open- and closed-source model families on a structured PII dataset with a hierarchical risk taxonomy. We find that CoT consistently elevates leakage, especially for high-risk categories, and that leakage is strongly family- and budget-dependent. Increasing the reasoning budget can either amplify or attenuate leakage depending on the base model. We then benchmark lightweight inference-time gatekeepers: a rule-based detector, a TF-IDF + logistic regression classifier, a GLiNER-based NER model, and an LLM-as-judge, using risk-weighted F1, Macro-F1, and recall. No single method dominates across models or budgets, motivating hybrid, style-adaptive gatekeeping policies that balance utility and risk under a common, reproducible protocol.
Modern language models (LM) are trained on large scrapes of the Web, containing millions of personal information (PI) instances, many of which LMs memorize, increasing privacy risks. In this work, we develop the regexes and rules (R&R) detector suite to detect email addresses, phone numbers, and IP addresses, which outperforms the best regex-based PI detectors. On a manually curated set of 483 instances of PI, we measure memorization: finding that 13.6% are parroted verbatim by the Pythia-6.9b model, i.e., when the model is prompted with the tokens that precede the PI in the original document, greedy decoding generates the entire PI span exactly. We expand this analysis to study models of varying sizes (160M-6.9B) and pretraining time steps (70k-143k iterations) in the Pythia model suite and find that both model size and amount of pretraining are positively correlated with memorization. Even the smallest model, Pythia-160m, parrots 2.7% of the instances exactly. Consequently, we strongly recommend that pretraining datasets be aggressively filtered and anonymized to minimize PI parroting.
Nishant Subramani, Kshitish Ghate, Mona Diab
♠Carnegie Mellon University - Language Technologies Institute · University of Washington