cs.CVNov 10, 2026

Specify Privacy Yourself: Assessing Inference-Time Personalized Privacy Preservation Ability of Large Vision-Language Models

Authors: Xingqi Wang, Xiaoyuan Yi, Xing Xie, Jia Jia

Organizations: Tsinghua University, Key Laboratory of Pervasive Computing, Ministry of Education, Beijing, China · Microsoft Research Asia, Beijing, China · Department of Computer Science and Technology, BNRist, Tsinghua University, Beijing, China

Abstract

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities but raise significant privacy concerns due to their abilities to infer sensitive personal information from images with high precision. While current LVLMs are relatively well aligned to protect universal privacy, e.g., credit card data, we argue that privacy is inherently personalized and context-dependent. This work pivots towards a novel task: can LVLMs achieve Inference-Time Personalized Privacy Protection (ITP3^3), allowing users to dynamically specify privacy boundaries through language specifications? To this end, we present SPY-Bench, the first systematic assessment of ITP3^3 ability, which comprises (1) 32,700 unique samples with image-question pairs and personalized privacy instructions across 67 categories and 24 real-world scenarios, and (2) novel metrics grounded in user specifications and context awareness. Benchmarking the ITP3^3 ability of 21 SOTA LVLMs, we reveal that: (i) most models, even the top-performing o4-mini, perform poorly, with only ∼\sim24% compliance accuracy; (ii) they show quite limited contextual privacy understanding capability. Therefore, we implemented initial ITP3^3 alignment methods, including a novel Noise Contrastive Alignment variant which achieves 96.88% accuracy while maintaining reasonable general performance. These results mark an initial step towards the ethical deployment of more controllable LVLMs. Code and data are available at https://github.com/achernarwang/specify-privacy-yourself

Figures & tables

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Privacy-Aware Visual Language Models

    May 27, 2024Laurens Samson, Nimrod Barazani, Sennay Ghebreab +1VLM EvaluationVLM Adaptation

  2. Anti-Persona: Disrupting Unauthorized Identity Binding and Recognition in Personalized Vision--Language Models

    Oct 1, 2026Abhishek Basu, Fahad Shamshad, Karthik NandakumarVLM RobustnessVision-Language Models

  3. PrivMark: Pixel-Space Steering against Privacy Leakage in Vision-Language Models

    Nov 7, 2026Ruofan Xing, Arman Akbari, Weikai Lin +2Privacy Protection in VLMsAdversarial Attacks on VLMs