Privacy Leakage in Language Models

Overview

The field studies when language models reveal information they learned or received, including personal details, training-set membership, and sensitive traits inferred from text. Researchers test verbatim extraction and membership inference, as well as reconstruction from gradients, activations, hidden states, and model outputs; retrieval systems, persistent memory, and tool-using agents create additional risks across turns and users. Defenses include differential privacy, data filtering, unlearning, activation steering, prompt sanitization, obfuscation, and system-level access controls and information-flow enforcement. A central tension is that protections can reduce leakage while harming usefulness, and refusal or successful output-level forgetting does not necessarily prevent information from being recovered through other prompts, representations, or system channels.

Latest papers

CardsList

303 papers22 papers in the last four weeks, down 54% on the four weeks before. 0.2% of all new papers.

  1. Localizing Text Anonymization for Trustworthy AI: Extending RAT-Bench to Malaysian Microdata and PII

    Nov 18, 2026David Hong Liang Chew, Zexi Yao, Nataša Krčo +3Privacy-Utility TradeoffPrivacy Leakage in Language Models

  2. Eight in Ten LLMs Leak System-Prompt Secrets: Nevan, an Agentic Capture-the-Flag Study

    Nov 16, 2026Ana Clara Zoppi Serpa, Anderson Rocha, Hélio PedriniPrivacy Leakage in Language ModelsLanguage Model Auditing

  3. AudioTrace: File-Level Membership Detection for Audio Large Language Models

    Nov 16, 2026Guangzhi Sun, Xiao Zhan, Shuai Wang +2LLM AuditingPrivacy Leakage in Language Models

  4. Synthesis Smuggling: Maliciously Hiding Inferred Personal Attributes in LLM Outputs

    Nov 12, 2026Gautam Galada, Supreeth Shastri, Sagnik Ray ChoudhuryLLM SecurityPrivacy Leakage in Language Models

  5. Constructing and Validating Memorisation Audits for EHR Foundation Models

    Nov 11, 2026Inigo Stibbe, Joshua Placidi, Padmanabhan Ramnarayan +1Memorization in Language ModelsPrivacy Leakage in Language Models

  6. VeilTune: Secure Fine-Tuning of Private LoRA Adapters on Public Pretrained Language Models

    Nov 9, 2026Huiyu Xie, Tanping Zhou, Tao Hu +2Fine-Tuning Language ModelsPrivacy-Preserving Language Models

  7. Analyzing Code Provenance in LLMs for Programming Education

    Nov 6, 2026Subhankar Maity, Faiza Belbachir, Omar Boucelma +1Data Provenance in Language ModelsGenerative AI in Education

  8. A Generative Model of Contextual Integrity: Appropriate vs. Inappropriate Information Sharing

    Nov 5, 2026Omer Ebead, Claude Formanek, Joel Z. LeiboContextual IntegrityRefusal Behavior in Language Models

  9. Timing is Everything: Inferring Temporal Membership in Continually Updated Models

    Nov 4, 2026Jie Fu, Xiangpeng Deng, Yuan Hong +1Membership Inference AttacksPrivacy Leakage in Language Models

  10. Residual Attribute Leakage Is an Interface: Auditing Privacy Evaluation After Text De-identification

    Nov 3, 2026Tuan Thanh Duong, Joongmin Shin, Jeongho Yoon +2Privacy AuditingPrivacy Leakage in Language Models

  11. From Training Exposure to Membership Evidence: A Controlled Study of OLMo 2

    Oct 30, 2026Ali Banaei, Muhammad Shahab Shahzad, Gerald Q. Maguire +2Privacy Leakage in Language ModelsData Provenance in Language Models

  12. From Representation to Readout: Mechanistic Analysis of Membership Signals in Language Models

    Oct 30, 2026Wenxi Dai, Jingyu Hu, Yanting Yang +1Privacy Leakage in Language ModelsMemorization in Language Models

  13. DiRAFT: Privacy-Aware Financial Transcript Synthesis from Aggregate Discourse Priors

    Oct 29, 2026Yunbai Zhang, Yunyi Gao, Han-Tai ShiaoSynthetic Data Generation for Language ModelsText Generation

  14. Tokenizer Tokens Tell Training Tales

    Oct 29, 2026Nishita Jain, Parth Upman, Shreyank N GowdaPrivacy Leakage in Language ModelsTokenization in Language Models

  15. Is it my preference? Role Grounding Attack against Agent's Memory via Social Interaction

    Oct 28, 2026Haoran Bu, Yiting Shen, Litian Zhang +2AI Agent SecurityLLM Agent Safety

  16. Answer-Only Disclosure: Private-Context LLM Assistance over Web APIs with a Leakage Ledger

    Oct 28, 2026Liang Yuan, Xiaohu Du, Ruikun Luo +6Privacy Leakage in Language ModelsPrivacy-Preserving Language Models

  17. Detecting and Mitigating Query-Based System Prompt Leakage Attacks

    Oct 26, 2026Tushar Nayan, Pratik Poudel, Qian Lou +3LLM SecurityPrivacy Leakage in Language Models

  18. InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants

    Oct 25, 2026Pengrun Huang, Yiwei Fu, Luoxi Meng +1Privacy Leakage in Language ModelsPrivacy Protection for LLM Agents

  19. TRACE: Data-Free Text Reconstruction Attacks against Approximate Unlearning in LLMs

    Oct 22, 2026Mengying Zhang, Derui Wang, Ibrahim Khalil +3Privacy Leakage in Language ModelsTraining Data Reconstruction Attacks

  20. Agent Mosaic: Recognizing Confidentiality Leaks in Agentic LLM Systems

    Oct 21, 2026Daniel Ruiz, Luca Luceri, Emilio FerraraLLM Agent SecurityPrivacy Leakage in Language Models

  21. Contextual Candour: Role-Aware Graduated Disclosure for LLMs over Mixed-Sensitivity Contexts

    Oct 20, 2026Devansh Srivastav, David Pape, Samira Abedini +1LLM SecurityPrivacy Leakage in Language Models

  22. Guided Diffusion for Trustworthy Synthetic Text Generation

    Oct 19, 2026Qi Pang, Hanshen Xiao, Andre Manoel +4Synthetic Data Generation for Language ModelsDifferentially Private Synthetic Data Generation

  23. Prune the Leaky Neurons: Mitigating Private Data Leakage in Fine-Tuned LLMs via Joint-Embedding-Guided Pruning

    Oct 16, 2026Mohamed Shaaban, Mohamed ElmahallawyPrivacy Leakage in Language ModelsPrivacy-Preserving Language Models

  24. Exploring The Empirical Privacy Landscape of Synthetic Data

    Oct 15, 2026Zihao Zhao, Avi Schwarzschild, Casey Meehan +3Privacy Leakage in Language ModelsPrivacy-Preserving Language Models

  25. Private Retrieval Augmented Generation via Random Projection

    Oct 15, 2026Dixi Yao, Tahseen Rabbani, Tian LiRandom ProjectionPrivacy-Preserving RAG