cs.CVNov 7, 2026

PrivMark: Pixel-Space Steering against Privacy Leakage in Vision-Language Models

Authors: Ruofan Xing, Arman Akbari, Weikai Lin, Yuhao Zhu, Xuan Zhang

Organizations: Northeastern University, USA · University of Rochester, USA

Abstract

Vision-language models (VLMs) routinely disclose soft-biometric attributes such as age, gender, and race in their text outputs, exposing demographic information from uploaded images to any downstream API user. Existing privacy defenses do not fit this setting: face de-identification leaves distributed soft-biometric cues intact, classifier-targeted adversarial perturbations do not transfer to open-ended generation, and stronger VLM-specific methods require retraining or runtime activation access unavailable to downstream APIs. We propose PrivMark, a framework that suppresses biometric leakage at the image level. By analyzing VLM activations, we identify two weakly aligned steering vector directions in hidden-state space, each tied to a distinct leakage behavior --- one corresponding to incidental attribute disclosure under open-ended descriptions, the other to compliance with direct biometric probes instead of refusal --- and encode both directions into a single bounded pixel perturbation, so the protected image alone enforces privacy with no invasive access to model internals at deployment. To support evaluation, we assemble BioleakBench, 1,6001{,}600 VQA pairs covering biometric privacy attributes under both free-form description and direct-probe protocols. Across three white-box source models, PrivMark achieves up to 85.9%85.9\% biometric suppression while retaining 75%75\% of clean OK-VQA utility, up to 1.4×1.4\times the privacy-utility tradeoff of the strongest pixel-only baseline. The same one-shot perturbations transfer consistently to seven unseen VLMs including closed-source APIs.

Figures & tables

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. DP^2-VL: Private Photo Dataset Protection by Data Poisoning for Vision-Language Models

    Mar 25, 2026Hongyi Miao, Jun Jia, Xincheng Wang +7Privacy Protection in VLMsData Poisoning Attacks

  2. Specify Privacy Yourself: Assessing Inference-Time Personalized Privacy Preservation Ability of Large Vision-Language Models

    Nov 10, 2026Xingqi Wang, Xiaoyuan Yi, Xing Xie +1Privacy Protection in VLMs

  3. VisualLeakBench: Auditing the Fragility of Large Vision-Language Models against PII Leakage and Social Engineering

    Mar 11, 2026Youting Wang, Yuan Tang, Yitian Qian +1VLM RobustnessPrivacy Protection in VLMs