Privacy Leakage in Language Models

Follow this topic Ask about this topic

The field studies when language models reveal information they learned or received, including personal details, training-set membership, and sensitive traits inferred from text. Researchers test verbatim extraction and membership inference, as well as reconstruction from gradients, activations, hidden states, and model outputs; retrieval systems, persistent memory, and tool-using agents create additional risks across turns and users. Defenses include differential privacy, data filtering, unlearning, activation steering, prompt sanitization, obfuscation, and system-level access controls and information-flow enforcement. A central tension is that protections can reduce leakage while harming usefulness, and refusal or successful output-level forgetting does not necessarily prevent information from being recovered through other prompts, representations, or system channels.

Latest papers

284 papers43 papers in the last four weeks, up 16% on the four weeks before. 0.2% of all new papers.

Reading the Finetuning Prior: Verbatim Content Recovery via Contrastive Decoding Diffing
May 25, 2026Michał Brzozowski, Zuzanna Dubanowska, Enrico Cassano +1Training Data Extraction Language Model Fingerprinting
Samsung AI Center, Warsaw, Poland · University of Turin, Italy · University of Warsaw, Poland
Extracting Training Data from Diffusion Language Models via Infilling
May 22, 2026Yihan Wang, N. AsokanData Leakage Masked Diffusion Models
University of Waterloo · KTH Royal Institute of Technology
What Does the Server See? Understanding Privacy Leakage from Large Language Models in Split Inference
May 22, 2026Mingyuan Fan, Yu Liu, Fuyi Wang +1Privacy-Preserving Language Models Adversarial Attacks
East China Normal University Shanghai, China · RMIT University Melbourne, Australia
Hidden-State Privacy Has an Empty Middle
May 21, 2026Alexander Okezue BellPrivacy-Preserving Language Models Adversarial Attacks
Stanford University 450 Jane Stanford Way Stanford, CA 94305
LCGuard: Latent Communication Guard for Safe KV Sharing in Multi-Agent Systems
May 21, 2026Sadia Asif, Mohammad Mohammadi Amiri, Momin Abbas +2Adversarial Representation Learning Privacy-Preserving Language Models
Rensselaer Polytechnic Institute · IBM Research
Boundary-targeted Membership Inference Attacks on Safety Classifiers
May 21, 2026Anthony Hughes, Alexander Goldberg, Prince Jha +3Membership Inference Attacks Privacy Leakage in Language Models
University of Sheffield · Carnegie Mellon University · MBZUAI
Pretraining Data Exposure in Large Language Models: A Survey of Membership Inference, Data Contamination, and Security Implications
May 21, 2026Ziyi Tong, Feifei Sun, Le Minh NguyenPrivacy Leakage in Language Models Data Contamination in Language Models
Japan Advanced Institute of Science and Technology, Japan
Counterfactual Likelihood Tests for Indirect Influence in Private Reasoning Channels
May 18, 2026Alexander Boesgaard LorupCounterfactual Evaluation LLM Interpretability
Openhagen
A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook
May 18, 2026Kaiwen Luo, Zhenhong Zhou, Leo Wang +31Privacy Leakage in Language Models Audio-Language Models
Nanyang Technological University · Independent Researcher · The University of Melbourne +15
An Empirical Study of Privacy Leakage Chains via Prompt Injection in Black-Box Chatbot Environments
May 18, 2026Hongjang Yang, Hyunsik Na, Daeseon ChoiData Leakage Privacy Leakage in Language Models
Department of Information Security Soongsil University Seoul, Korea · AI Safety Center Soongsil University Seoul, Korea · Department of AI Software Soongsil University Seoul, Korea
ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models
May 16, 2026Yujie Lin, Chengyi Yang, Zhishang Xiang +2Privacy Leakage in Language Models Model Editing
School of Informatics, Xiamen University, China · Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China · School of Film, Xiamen University, China +2
Membership Inference Attacks on Discrete Diffusion Language Models
May 15, 2026Shailesh KasivelrajanMasked Diffusion Models Membership Inference Attacks
Indian Institute of Technology Madras
Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation
May 14, 2026Kyomin Hwang, Hyeonjin Kim, Sangyeon Cho +1Multilingual Language Model Evaluation Privacy Leakage in Language Models
GSCST, Seoul National University · Department of Artificial Intelligence, Chung-Ang University · Korean Surgical Researcher Foundation, Republic of Korea +1
Reconstruction of Personally Identifiable Information from Supervised Finetuned Models
May 12, 2026Sae Furukawa, Alina OpreaSupervised Fine-Tuning Privacy Leakage in Language Models
Northeastern University Boston, Massachusetts, USA
DistractMIA: Black-Box Membership Inference on Vision-Language Models via Semantic Distraction
May 12, 2026Hongyi Tang, Zhihao Zhu, Yi YangVision-Language Models Membership Inference Attacks
The Hong Kong University of Science and Technology
Can You Keep a Secret? Involuntary Information Leakage in Language Model Writing
May 11, 2026Ari Holtzman, Peter WestData Leakage Privacy Leakage in Language Models
University of Chicago · University of British Columbia
PRISM: Generation-Time Detection and Mitigation of Secret Leakage in Multi-Agent LLM Pipelines
May 11, 2026Riya Tapwal, Abhishek Kumar, Carsten MapleData Leakage Multi-Agent LLM Systems
School of Computing and Electrical Engineering · Indian Institute of Technology, Mandi (IIT) · The Alan Turing Institute, London +1
JUMP: Efficient Membership Inference on Fine-Tuned Diffusion Language Models
May 9, 2026Yeachan Jun, Albert NoMembership Inference Attacks Privacy Leakage in Language Models
Department of Artificial Intelligence Yonsei University, Seoul, Korea
The Moltbook Files: A Harmless Slopocalypse or Humanity's Last Experiment
May 8, 2026William Brach, Federico Torrielli, Stine Lyngsø Beltoft +3Privacy Leakage in Language Models Multi-Agent Systems
University of Turin Dept. of Computer Science Torino, Italy · University of Southern Denmark Odense, Denmark
LeakDojo: Decoding the Leakage Threats of RAG Systems
May 7, 2026Maosen Zhang, Jianshuo Dong, Boting Lu +4Data Leakage Privacy Leakage in Language Models
Tsinghua University, China · Ant International, China · Nanyang Technological University, Singapore
Membership Inference Attacks for Retrieval Based In-Context Learning for Document Question Answering
May 5, 2026Tejas Kulkarni, Antti Koskela, Laith ZumotIn-Context Retrieval Membership Inference Attacks
Nokia Bell Labs · Nokia
PIIGuard: Mitigating PII Harvesting under Adversarial Sanitization
May 4, 2026Mingshuo Liu, Yiwei Zha, Min ChenData Leakage Cybersecurity
Vrije Universiteit Amsterdam
On the Privacy of LLMs: An Ablation Study
May 4, 2026Karima Makhlouf, Lamiaa Basyoni, Syed Khaderi +4LLM Backdoor Attacks Membership Inference Attacks
UDST, Doha, Qatar
Probe-Geometry Alignment: Erasing the Cross-Sequence Memorization Signature Below Chance
May 3, 2026Anamika Paul Rupa, Anietie AndyMemorization in Language Models Neural Network Memorization
Howard University
E-MIA: Exam-Style Black-Box Membership Inference Attacks against RAG Systems
May 1, 2026Zelin Guan, Shengda Zhuo, Zeyan Li +4Membership Inference Attacks Privacy Leakage in Language Models
College of Cyber Security, Jinan University, Guangzhou, China · School of Computer Science, Shanghai Jiaotong University, Shanghai, China · Institute of Artificial Intelligence, Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University, Beijing 100191, China +1
MCPHunt: An Evaluation Framework for Cross-Boundary Data Propagation in Multi-Server MCP Agents
Apr 30, 2026Haonan Li, Tianjun Sun, Yongqing Wang +1MCP Security Privacy Leakage in Language Models
Key Laboratory of Intraplate Volcanoes and Earthquakes (China University of Geosciences, Beijing), Ministry of Education, Beijing 100083, China · School of Geophysics and Information Technology, China University of Geosciences, Beijing 100083, China
Secret Stealing Attacks on Local LLM Fine-Tuning through Supply-Chain Model Code Backdoors
Apr 30, 2026Zi Li, Tian Zhou, Wenze Li +3Data Leakage LLM Backdoor Attacks
Nanjing University
Fidelity, Diversity, and Privacy: A Multi-Dimensional LLM Evaluation for Clinical Data Augmentation
Apr 29, 2026Guillermo Iglesias, Gema Bello-Orgaz, María Navas-Loro +3Synthetic Data Augmentation Synthetic Data Generation
Departamento de Sistemas Informáticos, Escuela Técnica Superior de Ingeniería de Sistemas Informáticos, Universidad Politécnica de Madrid, Spain · Department of Psychiatry, University Hospital Rey Juan Carlos, Mostoles, Spain · Department of Psychiatry, University Hospital Jimenez Diaz Foundation, Madrid, Spain +5
When Agents Shop for You: Role Coherence in AI-Mediated Markets
Apr 29, 2026Soogand Alavi, Salar NozariPrivacy Leakage in Language Models Agentic Commerce
Tippie College of Business, University of Iowa
Evaluation of Prompt Injection Defenses in Large Language Models
Apr 26, 2026Priyal Deep, Shane Emmons, Amy Fox +4Data Leakage Privacy Leakage in Language Models
Swept AI[1] · University of Michigan

May 25, 2026J/K move · Enter open · S save

Reading the Finetuning Prior: Verbatim Content Recovery via Contrastive Decoding Diffing

Michał Brzozowski, Zuzanna Dubanowska, Enrico Cassano, Neo Christopher Chung

Samsung AI Center, Warsaw, Poland · University of Turin, Italy · University of Warsaw, Poland

Training Data Extraction Language Model Fingerprinting Privacy Leakage in Language Models Contrastive Decoding Language Model Auditing

Open paper AskPDF arXiv

Narrowly finetuned language models memorize implanted content verbatim, but auditing what a deployed model has been taught, without access to its weights or training data, remains an open challenge. Recent work shows that activation differences between base and finetuned models carry readable traces of the finetuning domain; the state-of-the-art Activation Difference Lens (ADL) recovers a vague domain-level description but requires full "white-box" access to model internals. We introduce Contrastive Decoding Diffing (CDD), a model diffing method that operates on output-level logit distributions only, with no weight access, no layer selection, and no per-model tuning, yet recovers implanted facts. CDD consists of three ideas: bypassing the chat template to expose the raw finetuning prior, seeding generation with maximally vague pre-fills, and amplifying the logit-space difference between finetuned and base models at each decoding step. A single default configuration recovers implanted facts verbatim -- exact drug names, vote counts, physical measurements, and procedural details -- across four architectures (1B--32B parameters), uniformly outperforming ADL despite less access and running ~170x faster. Furthermore, CDD surfaces unintended data pipeline artifacts: a fictional persona introduced by the LLM data generator via mode collapse leaked into model weights and was extracted by CDD, constituting to our knowledge the first demonstrated end-to-end fingerprinting chain from data generator artifact to model weights to recovered output. We validate on real-domain finetuning settings, achieving near-perfect recovery across all single-dataset non-CoT variants and correctly identifying all four datasets in the mixed-dataset setting. CDD's success as a grey-box method outperforming white-box baselines underscores its practical utility for transparency and accountability in AI systems.