Vision-language models (VLMs) routinely disclose soft-biometric attributes such as age, gender, and race in their text outputs, exposing demographic information from uploaded images to any downstream API user. Existing privacy defenses do not fit this setting: face de-identification leaves distributed soft-biometric cues intact, classifier-targeted adversarial perturbations do not transfer to open-ended generation, and stronger VLM-specific methods require retraining or runtime activation access unavailable to downstream APIs. We propose PrivMark, a framework that suppresses biometric leakage at the image level. By analyzing VLM activations, we identify two weakly aligned steering vector directions in hidden-state space, each tied to a distinct leakage behavior --- one corresponding to incidental attribute disclosure under open-ended descriptions, the other to compliance with direct biometric probes instead of refusal --- and encode both directions into a single bounded pixel perturbation, so the protected image alone enforces privacy with no invasive access to model internals at deployment. To support evaluation, we assemble BioleakBench, 1,600 VQA pairs covering biometric privacy attributes under both free-form description and direct-probe protocols. Across three white-box source models, PrivMark achieves up to 85.9% biometric suppression while retaining 75% of clean OK-VQA utility, up to 1.4× the privacy-utility tradeoff of the strongest pixel-only baseline. The same one-shot perturbations transfer consistently to seven unseen VLMs including closed-source APIs.
Figures & tables
Figure 1: PrivMark pipeline. Phase A: from contrastive response pairs (y+,y−) we extract a per-layer steering vector vcℓ for each leakage mode c∈{desc,refuse} in the frozen source model’s decoder hidden-state space. Phase B: a bounded perturbation δ ( ∥δ∥∞≤ϵ ) is optimized via PGD with warm-start, yielding a protected image x~ requiring no serving-time model modification.
Figure 2: Steering vector analysis. Left : within-model cosine between vdescℓ and vrefuseℓ ; all three stay near orthogonal. Right : cross-model cosine between vcQwen and vcInternVL for c∈{desc,refuse} : the same direction is recovered by independent extraction.
LLaVA-1.5
Qwen2.5-VL
InternVL3
Method
Supp ↑
Fid ↑
VQA ↑
HM ↑
Supp ↑
Fid ↑
VQA ↑
HM ↑
Supp ↑
Fid ↑
VQA ↑
HM ↑
Clean
0
1.00
62.3
0.0
0
1.00
63.1
0.0
0
1.00
60.6
0.0
Gaussian Blur
1.5
0.47
58.1
2.9
54.6
0.45
38.2
45.0
35.0
0.52
52.8
42.1
Pixelation
0.8
0.48
52.1
1.6
65.2
0.44
47.4
54.9
44.6
0.54
51.6
47.8
VIP
98.7
0.09
—
—
99.1
0.09
—
—
98.1
0.10
—
—
SelectivePrivacy
45.3
0.22
45.3
45.3
95.4
0.18
46.1
62.2
97.5
0.16
44.2
60.8
Table 1: Description protocol on BioleakBench. HM is the harmonic mean of Supp and VQA (the iso-tradeoff score in Figure 3 ). Bold marks the highest HM among pixel-level methods.
LLaVA-1.5
Qwen2.5-VL
InternVL3
Method
RR ↑
VQA ↑
HM ↑
RR ↑
VQA ↑
HM ↑
RR ↑
VQA ↑
HM ↑
Clean
2.8
62.3
5.4
9.0
63.1
15.7
53.7
60.6
56.9
Gaussian Blur
3.5
58.1
6.6
15.2
38.2
21.7
58.8
52.8
55.6
Pixelation
2.8
52.1
5.3
10.8
47.4
17.6
68.9
51.6
59.0
VIP
3.4
—
—
9.3
—
—
54.0
—
—
SelectivePrivacy
3.9
45.3
7.2
25.0
46.1
32.4
76.8
44.2
56.1
Table 2: Refusal protocol on BioleakBench. HM is the harmonic mean of RR and VQA. Bold marks the highest HM among pixel-level methods.
“black/white < person > ”, “ < person > is black/white,” “black/white-skinned”
Eye color
blue eyes, brown eyes, green eyes, hazel eyes, dark eyes, light eyes, …
Appendix
Table 9: Representative keywords used for leakage detection. The full lists ship with the benchmark.
Description Suppa ( ↑ )
Refusal RRa ( ↑ )
Method
Age
Eye
Gen.
Race
Wei.
Age
Eye
Gen.
Race
Wei.
LLaVA-1.5
Clean
0.0
0.0
0.0
0.0
0.0
0.1
0.0
0.0
0.0
15.1
Gaussian Blur
1.3
1.8
0.7
2.0
1.7
1.0
0.3
0.0
0.0
17.5
Pixelation
0.6
1.1
0.1
1.2
1.0
1.4
0.0
0.0
0.0
13.5
VIP
97.9
99.6
96.6
99.9
99.5
0.0
0.0
0.1
0.2
18.0
Appendix
Table 10: Per-attribute Suppa and RRa (%) on BioleakBench (higher is better). Cells marked − have no retained per-attribute output. Every block is in-domain and per-source: the pipeline runs independently on each source model, so “Ours” inside a block is the perturbation optimized against that block’s own model. Cross-model transfer is reported separately in Table 4 and discussed in § F .
LLaVA-1.5
Qwen2.5-VL
InternVL3
Method
Supp ↑
Fid ↑
RR ↑
Supp ↑
Fid ↑
RR ↑
Supp ↑
Fid ↑
RR ↑
Clean
0.0
5.00
5.6
0.0
5.00
10.7
0.0
5.00
55.7
Gaussian Blur
8.4
3.34
5.1
52.6
3.32
17.0
33.6
3.52
60.3
Pixelation
8.1
3.37
5.9
62.7
3.30
12.8
44.3
3.57
67.1
VIP
90.2
1.78
5.9
86.7
1.72
10.3
89.3
1.80
56.9
SelectivePrivacy
43.9
2.57
5.4
86.5
2.30
26.8
86.3
2.18
79.6
Appendix
Table 11: LLM-as-Judge evaluation on BioleakBench (Gemini-2.5-Flash); Supp / Fid / RR all higher is better.
PrivMark
LLaVA-1.5
Qwen2.5-VL
InternVL3
Desc Supp
59.8±3.1
69.4±2.3
85.9±0.4
Desc VQA
47.6±2.6
50.0±3.8
45.5±1.1
Desc HM
53.0±3.1
58.1±1.0
59.5±3.1
Refuse RR
72.7±1.3
56.7±3.1
85.7±0.7
Refuse VQA
58.1±3.4
51.9±0.8
54.4±2.2
Refuse HM
64.6±2.2
54.2±2.4
66.6±1.2
Appendix
Table 12: Run-to-run variability of PrivMark over seven seeded runs, each retraining the generator and re-running the full evaluation. Mean ± standard deviation on the 0 – 100 scale.
GPT-4o
Supp ↑
OK-VQA ↑
Clean
0.0
73.4
Random (matched ϵ )
13.2
70.2
ImageProtector
29.5
51.8
Ours
20.3
62.9
Appendix
Table 13: Privacy and utility together on a closed commercial target (GPT-4o), with perturbations synthesized on LLaVA-1.5 and applied unchanged. Random noise is matched to the same ℓ∞ budget.
LLaVA-1.5
Supp ↑
RR ↑
Protected, no transform
59.8
72.7
JPEG q90 (mild)
50.5
44.8
Resize 0.9× (mild)
57.1
71.2
JPEG q50 (aggressive)
29.4
23.7
Appendix
Table 14: Protection under image transformations applied to the protected image before querying LLaVA-1.5. The clean reference for RR on this model is 2.8 (Table 2 ).
Figure 5: Images without and with perturbations under different ℓ∞ budgets.
Figure 6: Perturbation budget ϵ ablation on LLaVA-1.5 with N=100 PGD steps. (a) Desc-only: Supp (left) and token-F1 fidelity (right) vs. ϵ . (b) Refuse-only: RR (left) and OK-VQA accuracy (right) vs. ϵ .
Figure 7: PGD-step N ablation on LLaVA-1.5 with ϵ=26/255 . (a) Desc-only: Supp (left) and token-F1 fidelity (right) vs. N . (b) Refuse-only: RR (left) and OK-VQA accuracy (right) vs. N .
Figure 8: Qualitative example on a Gender probe.
Figure 9: Qualitative example on an Eye-color probe.
Privacy protection in vision-language models limits sensitive information exposure while preserving useful image understanding and multimodal reasoning.
Tsinghua University, Key Laboratory of Pervasive Computing, Ministry of Education, Beijing, China · Microsoft Research Asia, Beijing, China · Department of Computer Science and Technology, BNRist, Tsinghua University, Beijing, China