Privacy Protection in VLMs

VLM: Vision-Language Model

This field studies how vision-language models can reveal sensitive information in images, internal representations, or outputs, including identities, personal details, locations, and training-data membership. Approaches include image masking and semantic editing, adversarial perturbations, token and feature protection, differential privacy, and targeted model unlearning. Researchers evaluate these methods with privacy attacks and benchmarks for tasks such as visual question answering, geolocation, identity recognition, and medical image analysis. A central challenge is reducing leakage without sacrificing useful context or task accuracy, while making defenses robust to different prompts, models, and real-world visual settings.

Latest papers

37 papers7 papers in the last four weeks, up 133% on the four weeks before. 0.0% of all new papers.

Steering Vision-Language Models for Visual-Contextual Privacy in Location Disclosure
Oct 19, 2026Wenyue Chen, Yuhan Cheng, Ruixin Yang +5Privacy Protection in VLMs Visual Evidence Sensitivity in VLMs
Duke University · University of Washington · Georgia Institute of Technology +1
GeoEdit: Adversarial De-Evidencing for Silent Geoprivacy Protection via Image Editing
Oct 19, 2026Yu Yan, Shengjia Cheng, Fei Ma +2Privacy Protection in VLMs Adversarial Attacks on VLMs
People’s Public Security University of China, Beijing, China · Guangming Laboratory, Shenzhen, China
GeoCloak: An Agentic Framework for Semantics-Preserving Geolocation Privacy via Peripheral Clue Manipulation
Oct 14, 2026Yixin Wu, Mingjie Li, Yun Shen +2Privacy Protection in VLMs Adversarial Attacks on VLMs
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · CISPA Helmholtz Center for Information Security +1
Answer It or Not? Towards Prompt-Adapted Adversarial Images on Vision Language Models
Oct 12, 2026Shunyuan Bai, Yuchen Ren, Le Yang +3Adversarial Attacks on VLMs Privacy Protection in VLMs
Xi’an Jiaotong University, Xi’an, 710049, China
SAGE: Efficient Protected Vision-Language Inference on Smart Glasses
Oct 8, 2026Zhimin Li, Pan Wang, Jingxian Chen +3Efficient VLM Inference VLM Inference
Swanson School of Engineering, University of Pittsburgh · North Allegheny Senior High · Department of Computer Science, University of Central Florida
SIEVE: Selective attention-value Suppression for Vision-Language Models Unlearning
Oct 1, 2026Si Qi Goh, Cap Dang Xuan Kiet, Tat-Jen Cham +1VLM Unlearning Privacy-Preserving Language Models
Digital Trust Centre, Nanyang Technological University, Singapore · College of Computing and Data Science, Nanyang Technological University, Singapore
Anti-Persona: Disrupting Unauthorized Identity Binding and Recognition in Personalized Vision--Language Models
Oct 1, 2026Abhishek Basu, Fahad Shamshad, Karthik NandakumarVLM Robustness Vision-Language Models
MBZUAI, UAE · Michigan State University, USA
Hiding in Plain Sight: A Diffusion-based Mitigation of Geolocation Privacy Leakage in Vision-Language Models
Sep 18, 2026Yining Wang, Xi Li, Mi Zhang +4Privacy Protection in VLMs Adversarial Attacks on VLMs
Fudan University · East China University of Science and Technology · Shanghai University of Electric Power
Don't Send What You Don't Need: Question-Guided Token Pruning as a Privacy Defense for Vision-Language Models
Sep 14, 2026Md Khalid Syfullah, Alvi Ataur KhalilVisual Question Answering Privacy Protection in VLMs
Transformative Innovation for Trustworthy AI and Network Security (TITANS) Lab, Computer Science, Southern Illinois University Carbondale, USA
Where Identity Lives: Localized, Retain-Free Identity Unlearning in Multimodal Large Language Models
Aug 31, 2026Kangwook Ko, Jaehyuk Jang, Wonjun Lee +2MLLM Unlearning Privacy Protection in VLMs
KAIST
AIM: Anchor Identity Features, Then Match for Multimodal Large Language Model Unlearning
Aug 28, 2026Wonjun Lee, Jaehyuk Jang, Kangwook Ko +2MLLM Unlearning Privacy Protection in VLMs
Korea Advanced Institute of Science and Technology (KAIST) Daejeon, Republic of Korea
Defending Wearable VLMs Against Private Attribute Inference
Aug 26, 2026Zhimin Li, Pan Wang, Jingxian Chen +4Privacy Protection in VLMs Privacy-Utility Tradeoff
Swanson School of Engineering, University of Pittsburgh · North Allegheny Senior High · College of Engineering, Michigan State University +1
Masking Is Not Enough: Generative Restoration for Multimodal De-Identification in Medical AI
Aug 21, 2026Shiva Shrestha, Zongxing Xie, Chen Zhao +3Privacy Protection in VLMs Image Restoration
Department of Computer Science Kennesaw State University Marietta, GA 30060, USA · Department of Computer Science and Software Engineering Miami University Oxford, OH 45056, USA · Department of Computer Science Georgia State University Atlanta, GA 30303, USA +1
Imperceptible and Reversible Adversarial Examples against Vision-Language Models for Privacy Protection
Jul 11, 2026Qi Lu, Ziqi Zhou, Yufei Song +5VLM Robustness Adversarial Attacks on VLMs
School of Cyber Science and Engineering, Huazhong University of Science and Technology · College of Computer Science, Chongqing University · School of Software and engineering, Huazhong University of Science and Technology +2
TrustCLIP: Learning Private Visual Features via Adversarial Reconstruction
Jul 5, 2026Nikos Athanasiou, Ilya A. Petrov, Angela Yao +7Gradient Inversion Attacks VLM Robustness
Meta, Zürich, Switzerland · Max Planck Institute for Intelligent Systems, Tübingen, Germany · University of Tübingen, Germany +1
The Vision Encoder as a Privacy Boundary: Visual-Token Side Channels in Encoder-Free Vision-Language Models
Jun 10, 2026Chenyu Zhou, Qiliang Jiang, Shuning Wu +1Vision-Language Models Adversarial Attacks
School of Engineering, Institute of Science Tokyo Tokyo, Japan · College of Control Science and Engineering, Zhejiang University Hangzhou, China · Department of Electrical and Computer Engineering, National University of Singapore Singapore, Singapore
Vision Language Model Helps Private Information De-Identification in Vision Data
Jun 8, 2026Tiejin Chen, Pingzhi Li, Kaixiong Zhou +2VLM Adaptation Privacy Protection in VLMs
Arizona State University · University of North Carolina at Chapel Hill · North Carolina State University
Seeing Without Exposing: Adaptive Privacy Control for Open-World, Context-Hungry MLLMs
Jun 5, 2026Siyuan Xu, Yibing Liu, Peilin Chen +3Privacy-Preserving Language Models Multimodal Large Language Models
City University of Hong Kong · Hon Hai Research Institute · Lingnan University
Cross-modal linkage risk in clinical vision-language models
Jun 1, 2026Soroosh Tayebi Arasteh, Mahshad Lotfinia, Sven Nebelung +1Medical VLMs Privacy Protection in VLMs
Lab for AI in Medicine, RWTH Aachen University, Aachen, Germany · Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
Single-Sample Black-Box Membership Inference Attack against Vision-Language Models via Cross-modal Semantic Alignment
May 17, 2026Jiaqing Li, Yajuan Lu, Xiaochuan Shi +3Membership Inference Attacks Privacy Protection in VLMs
1Wuhan University · 2Tarim University
PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Models
May 9, 2026Jiahui Guang, Zexun Zhan, Zhenlin Xu +5VLM Evaluation VLM Unlearning
Harbin Institute of Technology, Shenzhen, China · Sichuan University, Chengdu, China · Harbin Institute of Technology, Weihai, China +3
How Far Are VLMs from Privacy Awareness in the Physical World? An Empirical Study
May 6, 2026Junran Wang, Xinjie Shen, Zehao Jin +1VLM Evaluation Vision-Language Models
Georgia Institute of Technology
Before Forgetting, Learn to Remember: Revisiting Foundational Learning Failures in LVLM Unlearning Benchmarks
May 5, 2026JuneHyoung Kwon, MiHyeon Kim, Eunju Lee +3VLM Evaluation VLM Unlearning
Department of Artificial Intelligence, Chung-Ang University · KT Corporation · Graduate School of Advanced Imaging Sciences, Multimedia and Film, Chung-Ang University
What do your logits know?
Apr 10, 2026Masha Fedzechkina, Eleonora Gualdoni, Rita Ramos +1Privacy Protection in VLMs Large Vision-Language Models
Apple
Leave My Images Alone: Preventing Multi-Modal Large Language Models from Analyzing Images via Visual Prompt Injection
Apr 10, 2026Zedian Shao, Hongbin Liu, Yuepeng Hu +1Privacy Protection in VLMs Adversarial Attacks on VLMs
Georgia Institute of Technology · Duke University
Assessing Privacy Preservation and Utility in Online Vision-Language Models
Apr 6, 2026Karmesh Siddharam Chaudhari, Youxiang Zhu, Amy Feng +2Privacy Protection in VLMs Vision-Language Models
Department of Computer Science, University of Massachusetts Boston, MA, USA
Shape and Substance: Dual-Layer Side-Channel Attacks on Local Vision-Language Models
Mar 26, 2026Eyal Hadad, Mordechai GuriPrivacy Protection in VLMs Adversarial Attacks on VLMs
Ben-Gurion University of the Negev, Israel Faculty of Computer and Information Science
DP^2-VL: Private Photo Dataset Protection by Data Poisoning for Vision-Language Models
Mar 25, 2026Hongyi Miao, Jun Jia, Xincheng Wang +7Privacy Protection in VLMs Data Poisoning Attacks
Shandong University · Shanghai Jiao Tong University · Donghua University +3
Rethinking Visual Privacy: A Compositional Privacy Risk Framework for Severity Assessment with VLMs
Mar 23, 2026Efthymios Tsaprazlis, Tiantian Feng, Anil Ramakrishna +3VLM Evaluation Privacy Auditing
University of Southern California, Los Angeles CA 90089, USA · Meta Superintelligence Labs · Amazon AGI
Ruyi2.5 Technical Report
Mar 18, 2026Huan Song, Shuyu Tian, Qingfei Zhao +5Edge-Cloud Collaborative Inference Multimodal Foundation Models
Institute of Artificial Intelligence (TeleAI), China Telecom

Oct 19, 2026J/K move · Enter open · S save

Steering Vision-Language Models for Visual-Contextual Privacy in Location Disclosure

Wenyue Chen, Yuhan Cheng, Ruixin Yang, Ethan Mendes+4

Duke University · University of Washington · Georgia Institute of Technology+1

Privacy Protection in VLMs Visual Evidence Sensitivity in VLMs Activation Steering

Open paper AskPDF arXiv

Vision-language models (VLMs) can infer sensitive information such as geolocation from subtle visual cues, creating privacy risks when responses disclose more detail than the visual context warrants. We study visual-contextual privacy in geolocation disclosure, where a VLM should reveal location information only at the granularity appropriate to the visual context depicted in the image. We first show that this contextually appropriate granularity is recoverable from pre-generation hidden states, even when generated responses are misaligned, with disclosure levels exhibiting an ordered structure along a common hidden-state direction. Motivated by this, we propose an inference-time route-and-steer framework: an ordinal granularity router maps visual-contextual privacy representations to a disclosure route, and a disclosure-level-conditioned steering module applies route-specific hidden-state interventions during decoding. Across four VLM backbones, our method improves visual-context-aware disclosure alignment, better balances over- and under-disclosure, and incurs only modest changes on general capability benchmarks. It further transfers to another dataset with a distinct data distribution and privacy-risk taxonomy.