Privacy Protection in VLMs

VLM: Vision-Language Model

This field studies how vision-language models can reveal sensitive information in images, internal representations, or outputs, including identities, personal details, locations, and training-data membership. Approaches include image masking and semantic editing, adversarial perturbations, token and feature protection, differential privacy, and targeted model unlearning. Researchers evaluate these methods with privacy attacks and benchmarks for tasks such as visual question answering, geolocation, identity recognition, and medical image analysis. A central challenge is reducing leakage without sacrificing useful context or task accuracy, while making defenses robust to different prompts, models, and real-world visual settings.

Latest papers

37 papers7 papers in the last four weeks, up 133% on the four weeks before. 0.0% of all new papers.

When Visual Privacy Protection Meets Multimodal Large Language Models
Mar 14, 2026Xiaofei Hui, Qian Wu, Haoxuan Qu +3Privacy Protection in VLMs Multimodal Large Language Models
School of Computing and Communications, Lancaster University, South Dr,, Lancaster LA1 4WA, United Kingdom · Singapore University of Technology and Design, 8 Somapah Rd, Singapore 487372, Singapore · Department of Computer Science, University of Bristol, Queens Road, Bristol BS8 1QU, United Kingdom
Bodhi VLM: Privacy-Alignment Modeling for Hierarchical Visual Representations in Vision Backbones and VLM Encoders via Bottom-Up and Top-Down Feature Search
Mar 14, 2026Bo Ma, Wei Qi Yan, Jinsong WuPrivacy-Preserving Representation Learning Privacy-Preserving ML
Auckland University of Technology, Auckland 1024 NZ · Resideo Technologies, Inc., Austin, TX 78759, USA · Guilin University of Electronic Technology, Gui Lin, China +1
Neural Gate: Mitigating Privacy Risks in LVLMs via Neuron-Level Gradient Gating
Mar 13, 2026Xiangkui Cao, Jie Zhang, Meina Kan +2Large Vision-Language Models Privacy Protection in VLMs
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences · University of Chinese Academy of Sciences
VisualLeakBench: Auditing the Fragility of Large Vision-Language Models against PII Leakage and Social Engineering
Mar 11, 2026Youting Wang, Yuan Tang, Yitian Qian +1VLM Robustness Privacy Protection in VLMs
Northeastern University · Carnegie Mellon University · Boston University +1
Differentially Private Multimodal In-Context Learning
Mar 5, 2026Ivoline C. Ngong, Zarreen Reza, Joseph P. NearPrivacy Protection in VLMs Multimodal ICL
University of Vermont, Burlington, VT, USA · Independent Researcher
AI Knows Where You Are: Exposure, Bias, and Inference in Multimodal Geolocation with KoreaGEO
Jun 3, 2025Xiaonan Wang, Bo Shao, Hansaem KimVLM Evaluation Privacy Protection in VLMs
Yonsei University, South Korea · CISPA Helmholtz Center for Information Security, Germany
Privacy-Aware Visual Language Models
May 27, 2024Laurens Samson, Nimrod Barazani, Sennay Ghebreab +1VLM Evaluation VLM Adaptation
Socially-Intelligent Artificial Systems Group, University of Amsterdam · University of Amsterdam · Fundamental AI Lab, University of Technology Nuremberg

Mar 14, 2026J/K move · Enter open · S save

When Visual Privacy Protection Meets Multimodal Large Language Models

Xiaofei Hui, Qian Wu, Haoxuan Qu, Majid Mirmehdi+2

School of Computing and Communications, Lancaster University, South Dr,, Lancaster LA1 4WA, United Kingdom · Singapore University of Technology and Design, 8 Somapah Rd, Singapore 487372, Singapore · Department of Computer Science, University of Bristol, Queens Road, Bristol BS8 1QU, United Kingdom

Privacy Protection in VLMs Multimodal Large Language Models

Open paper AskPDF arXiv

The emergence of Multimodal Large Language Models (MLLMs) and the widespread usage of MLLM cloud services such as GPT-4V raised great concerns about privacy leakage in visual data. As these models are typically deployed in cloud services, users are required to submit their images and videos, posing serious privacy risks. However, how to tackle such privacy concerns is an under-explored problem. Thus, in this paper, we aim to conduct a new investigation to protect visual privacy when enjoying the convenience brought by MLLM services. We address the practical case where the MLLM is a "black box", i.e., we only have access to its input and output without knowing its internal model information. To tackle such a challenging yet demanding problem, we propose a novel framework, in which we carefully design the learning objective with Pareto optimality to seek a better trade-off between visual privacy and MLLM's performance, and propose critical-history enhanced optimization to effectively optimize the framework with the black-box MLLM. Our experiments show that our method is effective on different benchmarks.