Privacy Leakage in Language Models

Ask about this topic

Overview

The field studies when language models reveal information they learned or received, including personal details, training-set membership, and sensitive traits inferred from text. Researchers test verbatim extraction and membership inference, as well as reconstruction from gradients, activations, hidden states, and model outputs; retrieval systems, persistent memory, and tool-using agents create additional risks across turns and users. Defenses include differential privacy, data filtering, unlearning, activation steering, prompt sanitization, obfuscation, and system-level access controls and information-flow enforcement. A central tension is that protections can reduce leakage while harming usefulness, and refusal or successful output-level forgetting does not necessarily prevent information from being recovered through other prompts, representations, or system channels.

Latest papers

CardsList

284 papers43 papers in the last four weeks, up 16% on the four weeks before. 0.2% of all new papers.

  1. Tokenizer Tokens Tell Training Tales

    Oct 29, 2026Nishita Jain, Parth Upman, Shreyank N GowdaPrivacy Leakage in Language ModelsTokenization in Language Models

    Independent Researcher · School of Computer Science University of Nottingham Nottingham, United Kingdom

  2. Is it my preference? Role Grounding Attack against Agent's Memory via Social Interaction

    Oct 28, 2026Haoran Bu, Yiting Shen, Litian Zhang +2AI Agent SecurityLLM Agent Safety

    Beijing University of Posts and Telecommunications, Beijing, China

  3. Answer-Only Disclosure: Private-Context LLM Assistance over Web APIs with a Leakage Ledger

    Oct 28, 2026Liang Yuan, Xiaohu Du, Ruikun Luo +6Privacy Leakage in Language ModelsPrivacy-Preserving Language Models

    Australian Catholic University North Sydney, NSW, Australia · Ant Group China · University of Macau Macau, China +5

  4. Detecting and Mitigating Query-Based System Prompt Leakage Attacks

    Oct 26, 2026Tushar Nayan, Pratik Poudel, Qian Lou +3LLM SecurityPrivacy Leakage in Language Models

    Florida International University · University of Central Florida · University of Texas at Arlington

  5. InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants

    Oct 25, 2026Pengrun Huang, Yiwei Fu, Luoxi Meng +1Privacy Leakage in Language ModelsPrivacy Protection for LLM Agents

    UC San Diego · University of Illinois Urbana-Champaign

  6. Reversible Pseudonymisation of Documents Sent to External LLMs: an Open Bilingual Benchmark and an Engine

    Oct 25, 2026Vladimir BeskorovainyiPrivacy Leakage in Language ModelsPII Detection

  7. TRACE: Data-Free Text Reconstruction Attacks against Approximate Unlearning in LLMs

    Oct 22, 2026Mengying Zhang, Derui Wang, Ibrahim Khalil +3Privacy Leakage in Language ModelsTraining Data Reconstruction Attacks

    RMIT University · CSIRO · The Hong Kong University of Science and Technology +1

  8. Agent Mosaic: Recognizing Confidentiality Leaks in Agentic LLM Systems

    Oct 21, 2026Daniel Ruiz, Luca Luceri, Emilio FerraraLLM Agent SecurityPrivacy Leakage in Language Models

    Information Sciences Institute University of Southern California Los Angeles, CA, USA

  9. Contextual Candour: Role-Aware Graduated Disclosure for LLMs over Mixed-Sensitivity Contexts

    Oct 20, 2026Devansh Srivastav, David Pape, Samira Abedini +1LLM SecurityPrivacy Leakage in Language Models

    CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

  10. Guided Diffusion for Trustworthy Synthetic Text Generation

    Oct 19, 2026Qi Pang, Hanshen Xiao, Andre Manoel +4Synthetic Data Generation for Language ModelsDifferentially Private Synthetic Data Generation

    NVIDIA · Carnegie Mellon University · Purdue University

  11. TUPLE-Bench: Evaluating and Mitigating Privacy-Utility Trade-offs in Typed Decision Models

    Oct 16, 2026Yiqu Yang, Qiaoyuan Zheng, Yang Tian +2Privacy-Utility TradeoffPrivacy Leakage in Language Models

    ETH Zurich · The University of Hong Kong

  12. Prune the Leaky Neurons: Mitigating Private Data Leakage in Fine-Tuned LLMs via Joint-Embedding-Guided Pruning

    Oct 16, 2026Mohamed Shaaban, Mohamed ElmahallawyPrivacy Leakage in Language ModelsPrivacy-Preserving Language Models

    School of Engineering and Applied Sciences, Washington State University, Richland, WA 99354, USA

  13. LLM-Attack Suite: A Unified Red-Team Framework for Privacy Evaluation of Split and Distributed LLM Systems

    Oct 15, 2026Georgios Politis, Evangelos PappasLLM SecurityPrivacy Leakage in Language Models

  14. Exploring The Empirical Privacy Landscape of Synthetic Data

    Oct 15, 2026Zihao Zhao, Avi Schwarzschild, Casey Meehan +3Privacy Leakage in Language ModelsPrivacy-Preserving Language Models

    Johns Hopkins University · OpenAI

  15. Private Retrieval Augmented Generation via Random Projection

    Oct 15, 2026Dixi Yao, Tahseen Rabbani, Tian LiPrivacy-Preserving RAGRAG Security

    University of Chicago

  16. Behavioral Forgetting Is Not Privacy in Machine Unlearning

    Oct 15, 2026Tyler Lizzo, Larry HeckMachine UnlearningLLM Unlearning

    AI Virtual Assistant (AVA) Lab, Georgia Institute of Technology

  17. Encoded but Exposed: Language Embeddings can Reveal More Than They Should in Recommender Systems

    Oct 15, 2026Linda Beková, Rodrigo AlvesRecommender SystemsPrivacy-Preserving ML

    Czech Technical University in Prague, Prague, Czech Republic

  18. MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents

    Oct 15, 2026Cheng Zhang, Xuanning Cao, Zhicheng Zheng +1LLM Agent MemoryPrivacy Leakage in Language Models

    The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China · The Hong Kong University of Science and Technology

  19. QuasiLeak-Bench: Evaluating LLM-Based Privacy Detectors on Policy-Conditioned Quasi-Evidence Group Recovery

    Oct 15, 2026Yufan Zhu, Yangfan Jiang, Yang Fei +3Privacy Leakage in Language ModelsPrivacy Auditing

    School of Computing, National University of Singapore, Singapore · Agency for Science, Technology and Research (A*STAR), Singapore

  20. inLoRA-MINT: Internal Representations for Auditing Training Data in Domain-adapted LLMs

    Oct 13, 2026Gonzalo Mancera, Julian Fierrez, Aythami Morales +3LLM AuditingPrivacy Leakage in Language Models

    BiometricsAI, Universidad Autonoma de Madrid, Spain · Mathematics Dept., Universidad de Las Palmas de Gran Canaria, Spain · Computer Engineering Dept., Universidad Autonoma de Madrid, Spain

  21. Cost-Efficient Privacy Enforcement for Email-Generating LLM Agents under Strategic Information Requests

    Oct 13, 2026Qikun Cai, Tao Huang, Chen HouPrivacy Protection for LLM AgentsLLM Agent Security

  22. Decouple to Forget: Semantic Mediation for Sensitive Unlearning in Large Language Models

    Oct 13, 2026Jingyue Cong, Andy Song, Yang Yu +7LLM UnlearningPrivacy Leakage in Language Models

    RMIT University, Melbourne, Australia · Yep AI, Melbourne, Australia · Monash University, Melbourne, Australia +1

  23. The Curse of Knowledge in Role-Playing Language Models

    Oct 12, 2026Xiao ZhangLarge Language Model-Based Role-Play SimulationPrivacy Leakage in Language Models

    University of Groningen

  24. Logit Leakage through Floating-Point Bias Addition

    Oct 12, 2026Shaomin Chen, Xiaorui GongPrivacy Leakage in Language ModelsLLM Security

  25. Is Memorization Context-Sensitive? Prefix-Based Extraction Beyond Isolated Prefixes

    Oct 8, 2026Ali Satvaty, Narjes Sharafi, Jirui Qi +2Memorization in Language ModelsPrivacy Leakage in Language Models

    University of Groningen · Leiden University

  26. What to Admit and How to Present: Governing Persistent Memory in LLM Agents

    Oct 8, 2026Chang Liu, Deliang DingAgent Memory ManagementPersistent Memory for Language Models

    Pazhou Laboratory Guangzhou, China

  27. Loose Lips: Understanding Privacy Leakage in Agentic Dialogue

    Oct 7, 2026Soumya Sharma, Ayana Hussain, Heber Hwang Arcolezi +4Privacy Leakage in Language ModelsLLM Agent Security

    Mila – Quebec AI Institute · McGill University · Simon Fraser University +3

  28. Nullify: Null-Space Activation Steering for Training-Free LLM Unlearning

    Oct 7, 2026Wei Zhai, Xiang Liu, Qiang Huang +6LLM UnlearningPrivacy Leakage in Language Models

    College of Computer Science and Artificial Intelligence, Fudan University · BEDI Cloud · School of Electrical and Information Engineering, Tianjin University +3

  29. Sensitive-Topic Leakage Through LLM Routing Metadata: Measurement and Mitigation

    Oct 7, 2026Teng-Ruei ChenPrivacy Leakage in Language ModelsLLM Routing

    Krixvon, Taipei 100, Taiwan

  30. When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetry

    Oct 7, 2026Yixin Tan, Jiayang Liu, Lu Sun +3Privacy Leakage in Language ModelsMembership Inference Attacks

    Institute of Science Tokyo · Nanyang Technological University · Tohoku University +2