Privacy Leakage in Language Models

Follow this topic Ask about this topic

The field studies when language models reveal information they learned or received, including personal details, training-set membership, and sensitive traits inferred from text. Researchers test verbatim extraction and membership inference, as well as reconstruction from gradients, activations, hidden states, and model outputs; retrieval systems, persistent memory, and tool-using agents create additional risks across turns and users. Defenses include differential privacy, data filtering, unlearning, activation steering, prompt sanitization, obfuscation, and system-level access controls and information-flow enforcement. A central tension is that protections can reduce leakage while harming usefulness, and refusal or successful output-level forgetting does not necessarily prevent information from being recovered through other prompts, representations, or system channels.

Latest papers

284 papers43 papers in the last four weeks, up 16% on the four weeks before. 0.2% of all new papers.

Membership Inference Attacks against Large Audio Language Models
Mar 30, 2026Jia-Kai Dong, Yu-Xiang Lin, Hung-Yi LeeMembership Inference Attacks Privacy Leakage in Language Models
National Taiwan University · NTU Artificial Intelligence Center of Research Excellence
Estimating near-verbatim extraction risk in language models with decoding-constrained beam search
Mar 26, 2026A. Feder Cooper, Mark A. Lemley, Christopher De Sa +6Memorization in Language Models Privacy Leakage in Language Models
Stanford & Yale · Stanford · Cornell +1
From Weak Cues to Real Identities: Evaluating Inference-Driven De-Anonymization in LLM Agents
Mar 19, 2026Myeongseob Ko, Jihyun Jeong, Sumiran Singh Thakur +2Privacy Leakage in Language Models LLM Agent Security
Department of Electrical and Computer Engineering, Virginia Tech, Blacksburg, VA, USA · Center for Advanced AI, Accenture
Differential Privacy in Generative AI Agents: Analysis and Optimal Tradeoffs
Mar 18, 2026Ya-Ting Yang, Quanyan ZhuPrivacy Leakage in Language Models Privacy-Preserving Language Models
Department of Electrical and Computer Engineering, New York University, NY, 11201, USA
SOMP: Scalable Gradient Inversion for Large Language Models via Subspace-Guided Orthogonal Matching Pursuit
Mar 17, 2026Yibo Li, Qiongxiu LiGradient Inversion Attacks Privacy Leakage in Language Models
Politecnico di Milano / Italy · Aalborg University
InterPol: De-anonymizing LM Arena via Interpolated Preference Learning
Mar 16, 2026Minsung Cho, Jaehyung KimLanguage Model Fingerprinting Privacy Leakage in Language Models
Yonsei University
MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Medical Open-Ended Question Answering
Mar 15, 2026Shaowei Guan, Yu Zhai, Hin Chi Kwok +5Privacy-Utility Tradeoff Privacy Leakage in Language Models
Centre for Smart Health, School of Nursing, The Hong Kong Polytechnic University · Department of Language Science and Technology, The Hong Kong Polytechnic University · Centre for Frontier AI Research, Agency for Science, Technology and Research, Singapore +2
Human-Centred LLM Privacy Audits: Findings and Frictions
Mar 12, 2026Dimitri Staufer, Kirsten Morehouse, David Hartmann +1Privacy Auditing LLM Auditing
TU Berlin Berlin, Germany · Columbia University New York, USA · Weizenbaum Institute for the Networked Society Berlin, Germany +1
LLMs Can Infer Political Alignment from Online Conversations
Mar 11, 2026Byunghwee Lee, Sangyeon Kim, Filippo Menczer +3Privacy Leakage in Language Models
School of Data Science, University of Virginia, Charlottesville, Virginia, USA, 22903 · Center for Complex Networks and Systems Research, Luddy School of Informatics, Computing, and Engineering Indiana University, Bloomington, Indiana, USA, 47408 · Division of Communication and Media, Ewha Womans University, Seoul, Republic of Korea, 03760
Quantifying Memorization and Privacy Risks in Genomic Language Models
Mar 9, 2026Alexander Nemecek, Wenbiao Li, Xiaoqian Jiang +2Memorization in Language Models Privacy Leakage in Language Models
Case Western Reserve University Cleveland, Ohio, USA · The University of Texas Health Science Center at Houston Houston, Texas, USA · Rutgers University Newark, New Jersey, USA
Safer Reasoning Traces: Measuring and Mitigating Chain-of-Thought Leakage in LLMs
Mar 5, 2026Patrick Ahrend, Tobias Eder, Xiyang Yang +2Privacy Leakage in Language Models Privacy-Preserving Language Models
Technical University of Munich, Germany
Good-Enough LLM Obfuscation (GELO)
Mar 5, 2026Anatoly Belikov, Ilya FedotovPrivacy-Preserving Language Models LLM Security
SingularityNET Foundation, Baarerstrasse 141, Zug, 6300, Switzerland · Singularity Compute
Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects
Mar 2, 2026Xiaoyu Luo, Wenrui Yu, Qiongxiu Li +1Training Data Extraction Memorization in Language Models
Department of Computer Science, Aalborg University, Copenhagen, Denmark · Department of Electronic Systems, Aalborg University, Copenhagen, Denmark
Extracting Training Dialogue Data from Large Language Model based Task Bots
Mar 2, 2026Shuo Zhang, Junzhou Zhao, Junji Hou +3Training Data Extraction Attacks Privacy Leakage in Language Models
MOE Key Laboratory for Intelligent Networks and Network Security, Xi’an Jiaotong University, P.O. Box 1088, No. 28, Xianning West Road, Xi’an, Shaanxi 710049, China.
RedacBench: Can AI Erase Your Secrets?
Mar 2, 2026Hyunjun Jeon, Kyuyoung Kim, Jinwoo ShinLanguage Model Safety Evaluation Privacy Leakage in Language Models
Korea Advanced Institute of Science and Technology
Towards Privacy-Preserving LLM Inference via Covariant Obfuscation (Technical Report)
Mar 2, 2026Yu Lin, Qizhi Zhang, Wenqiang Ruan +6Privacy-Preserving Language Models LLM Inference
ByteDance · Nanjing University
Verifier-Bound Communication for LLM Agents: Certified Bounds on Covert Signaling
Feb 27, 2026Om TailorLLM Agent Security Privacy Leakage in Language Models
University of Maryland
From Leaky Thoughts to Private Reasoning: Controlling What LRMs Say to Themselves
Feb 27, 2026Haritz Puerto, Haonan Li, Xudong Han +2Large Reasoning Models Privacy Leakage in Language Models
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany · Mohamed bin Zayed University of Artificial Intelligence, UAE · LibrAI
Your Inference Request Will Become a Black Box: Confidential Inference for Cloud-based Large Language Models
Feb 27, 2026Chung-ju Huang, Huiqiang Zhao, Yuanpeng He +5Privacy-Preserving Language Models Privacy Leakage in Language Models
Key Lab of High Confidence Software Technologies (Peking University), Ministry of Education, China · School of Computer Science, Peking University, Beijing, China · Tencent, Shenzhen, China +1
Assessing Deanonymization Risks with Stylometry-Assisted LLM Agent
Feb 26, 2026Boyang Zhang, Yang ZhangPrivacy Leakage in Language Models Authorship Attribution
CISPA Helmholtz Center for Information Security
U-CAN: Utility-Aware Contrastive Attenuation for Efficient Unlearning in Generative Recommendation
Feb 26, 2026Zezheng Wu, Rui Wang, Xinghe Cheng +4Machine Unlearning LLM Unlearning
Guilin University of Electronic Technology, Guilin, China · University of Southampton, Southampton, United Kingdom · Jinan University, Guangzhou, China +1
Silent Egress: When Implicit Prompt Injection Makes LLM Agents Leak Without a Trace
Feb 25, 2026Qianlong Lan, Anuj Kaul, Shaun Jones +1LLM Agent Security Prompt Injection Attacks on AI Agents
eBay
Beyond Refusal: Probing the Limits of Agentic Self-Correction for Semantic Sensitive Information
Feb 25, 2026Umid Suleymanov, Zaur Rajabov, Emil Mirzazada +1Privacy Leakage in Language Models LLM Self-Correction
Department of Computer Science, Virginia Tech · School of IT and Engineering, ADA University.
OptiLeak: Efficient Prompt Reconstruction via Reinforcement Learning in Multi-tenant LLM Services
Feb 24, 2026Longxiang Wang, Xiang Zheng, Xuhao Zhang +3Privacy Leakage in Language Models LLM Security
City University of Hong Kong · ByteDance Inc.
Personal Information Parroting in Language Models
Feb 24, 2026Nishant Subramani, Kshitish Ghate, Mona DiabMemorization in Language Models Privacy Leakage in Language Models
♠Carnegie Mellon University - Language Technologies Institute · University of Washington
Agents of Chaos
Feb 23, 2026Natalie Shapira, Chris Wendler, Avery Yen +35AI Agent Security LLM Agent Security
Northeastern University · Independent Researcher · Stanford University +10
Learning to Detect Language Model Training Data via Active Reconstruction
Feb 22, 2026Junjie Oscar Yin, John X. Morris, Vitaly Shmatikov +2Membership Inference Attacks Privacy Leakage in Language Models
University of Washington · Cornell University · UC Berkeley +1
A Prior-Aware Metric for Efficiently Distinguishing Memorization from Generalization in Large Language Models
Feb 21, 2026Trishita Tiwari, Ari Trachtenberg, G. Edward SuhMemorization in Language Models LLM Auditing
Cornell University · Boston University · NVIDIA, Cornell University
What Do LLMs Associate with Your Name? A Human-Centered Black-Box Audit of Personal Data
Feb 19, 2026Dimitri Staufer, Kirsten MorehousePrivacy Leakage in Language Models LLM Auditing
TU Berlin, Berlin, Germany · Columbia University, New York, USA
Discovering Universal Activation Directions for PII Leakage in Language Models
Feb 19, 2026Leo Marchyok, Zachary Coalson, Sungho Keum +2Privacy Leakage in Language Models LLM Interpretability
Oregon State University, Corvallis OR, USA · Korea Advanced Institute of Science & Technology, Daejeon, South Korea.

Mar 30, 2026J/K move · Enter open · S save

Membership Inference Attacks against Large Audio Language Models

Jia-Kai Dong, Yu-Xiang Lin, Hung-Yi Lee

National Taiwan University · NTU Artificial Intelligence Center of Research Excellence

Membership Inference Attacks Privacy Leakage in Language Models Audio-Language Models Audio-Language Model Evaluation

Open paper AskPDF arXiv

We present the first systematic Membership Inference Attack (MIA) evaluation of LALMs. Using Multi-modal Blind Baselines based on textual, spectral and prosodic features, we demonstrate that common audio datasets exhibit near-perfect train/test separability (AUC ~ 1.0) even without model inference, thus MIA may primarily detect distribution shift. We therefore introduce a blind-baseline protocol to control for this confound. Under this protocol, we identify that the distribution-matched datasets enable reliable MIA evaluation without distribution-shift artifacts. We benchmark multiple MIA methods and conduct modality disentanglement experiments on these datasets. The results reveal that LALM memorization is cross-modal, arising only from binding a speaker's vocal identity with its text. These findings establish a principled standard for auditing LALMs beyond spurious correlations. Our codebase is available at https://github.com/snooow1029/ALM_MIA.