PrivMark: Pixel-Space Steering against Privacy Leakage in Vision-Language Models
Organizations: Northeastern University, USA · University of Rochester, USA
Abstract
Vision-language models (VLMs) routinely disclose soft-biometric attributes such as age, gender, and race in their text outputs, exposing demographic information from uploaded images to any downstream API user. Existing privacy defenses do not fit this setting: face de-identification leaves distributed soft-biometric cues intact, classifier-targeted adversarial perturbations do not transfer to open-ended generation, and stronger VLM-specific methods require retraining or runtime activation access unavailable to downstream APIs. We propose PrivMark, a framework that suppresses biometric leakage at the image level. By analyzing VLM activations, we identify two weakly aligned steering vector directions in hidden-state space, each tied to a distinct leakage behavior --- one corresponding to incidental attribute disclosure under open-ended descriptions, the other to compliance with direct biometric probes instead of refusal --- and encode both directions into a single bounded pixel perturbation, so the protected image alone enforces privacy with no invasive access to model internals at deployment. To support evaluation, we assemble BioleakBench, VQA pairs covering biometric privacy attributes under both free-form description and direct-probe protocols. Across three white-box source models, PrivMark achieves up to biometric suppression while retaining of clean OK-VQA utility, up to the privacy-utility tradeoff of the strongest pixel-only baseline. The same one-shot perturbations transfer consistently to seven unseen VLMs including closed-source APIs.
Figures & tables
| LLaVA-1.5 | Qwen2.5-VL | InternVL3 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Method | Supp | Fid | VQA | HM | Supp | Fid | VQA | HM | Supp | Fid | VQA | HM |
| Clean | 0 | 1.00 | 62.3 | 0.0 | 0 | 1.00 | 63.1 | 0.0 | 0 | 1.00 | 60.6 | 0.0 |
| Gaussian Blur | 1.5 | 0.47 | 58.1 | 2.9 | 54.6 | 0.45 | 38.2 | 45.0 | 35.0 | 0.52 | 52.8 | 42.1 |
| Pixelation | 0.8 | 0.48 | 52.1 | 1.6 | 65.2 | 0.44 | 47.4 | 54.9 | 44.6 | 0.54 | 51.6 | 47.8 |
| VIP | 98.7 | 0.09 | — | — | 99.1 | 0.09 | — | — | 98.1 | 0.10 | — | — |
| SelectivePrivacy | 45.3 | 0.22 | 45.3 | 45.3 | 95.4 | 0.18 | 46.1 | 62.2 | 97.5 | 0.16 | 44.2 | 60.8 |
| LLaVA-1.5 | Qwen2.5-VL | InternVL3 | |||||||
| Method | RR | VQA | HM | RR | VQA | HM | RR | VQA | HM |
| Clean | 2.8 | 62.3 | 5.4 | 9.0 | 63.1 | 15.7 | 53.7 | 60.6 | 56.9 |
| Gaussian Blur | 3.5 | 58.1 | 6.6 | 15.2 | 38.2 | 21.7 | 58.8 | 52.8 | 55.6 |
| Pixelation | 2.8 | 52.1 | 5.3 | 10.8 | 47.4 | 17.6 | 68.9 | 51.6 | 59.0 |
| VIP | 3.4 | — | — | 9.3 | — | — | 54.0 | — | — |
| SelectivePrivacy | 3.9 | 45.3 | 7.2 | 25.0 | 46.1 | 32.4 | 76.8 | 44.2 | 56.1 |
| LLaVA-1.5 | Qwen2.5-VL | |||||
| Type | Target models | Random | ImageProtector | Ours | ImageProtector | Ours |
| Open-access | InternVL3 | 7.0 | 41.4 | 42.9 | 43.2 | 43.6 |
| InstructBLIP | 11.6 | 18.3 | 13.9 | 18.8 | 15.8 | |
| GLM-4V | 13.5 | 36.5 | 37.7 | 25.9 | 26.3 | |
| LLaVA-1.5 | 1.5 | — | —- | 10.5 | 13.3 | |
| Qwen2.5-VL | 44.9 | 51.4 | 56.6 | — | —- | |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Symbol | Value |
| Perturbation bound | 26/255 | |
| PGD steps | 100 | |
| Momentum | 0.95 | |
| Learning rate schedule | — | cosine annealing |
| Initial learning rate | ||
| Learning rate multiplier | lr_mult | 8.0 |
| Source model | Layers | ||
|---|---|---|---|
| LLaVA-1.5 | |||
| Qwen2.5-VL | |||
| InternVL3 |
| Category | Images | ||
|---|---|---|---|
| Age | 83 | 249 | 83 |
| Eye color | 85 | 255 | 85 |
| Gender | 78 | 234 | 78 |
| Race | 80 | 240 | 80 |
| Weight | 74 | 222 | 74 |
| Group | Representative keywords / phrases |
|---|---|
| Age | young, elderly, senior, middle-aged, child, teenager, baby, mature, year-old, |
| Gender (nouns) | man, woman, male, female, boy, girl, gentleman, lady |
| Gender (pronouns) | he, she, his, her |
| Race | asian, african, hispanic, latino, indian, chinese, japanese, european, dark-skinned, |
| Race (compound) | “black/white < person > ”, “ < person > is black/white,” “black/white-skinned” |
| Eye color | blue eyes, brown eyes, green eyes, hazel eyes, dark eyes, light eyes, |
| Description ( ) | Refusal ( ) | |||||||||
| Method | Age | Eye | Gen. | Race | Wei. | Age | Eye | Gen. | Race | Wei. |
| LLaVA-1.5 | ||||||||||
| Clean | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.1 | 0.0 | 0.0 | 0.0 | 15.1 |
| Gaussian Blur | 1.3 | 1.8 | 0.7 | 2.0 | 1.7 | 1.0 | 0.3 | 0.0 | 0.0 | 17.5 |
| Pixelation | 0.6 | 1.1 | 0.1 | 1.2 | 1.0 | 1.4 | 0.0 | 0.0 | 0.0 | 13.5 |
| VIP | 97.9 | 99.6 | 96.6 | 99.9 | 99.5 | 0.0 | 0.0 | 0.1 | 0.2 | 18.0 |
| LLaVA-1.5 | Qwen2.5-VL | InternVL3 | |||||||
| Method | Supp | Fid | RR | Supp | Fid | RR | Supp | Fid | RR |
| Clean | 0.0 | 5.00 | 5.6 | 0.0 | 5.00 | 10.7 | 0.0 | 5.00 | 55.7 |
| Gaussian Blur | 8.4 | 3.34 | 5.1 | 52.6 | 3.32 | 17.0 | 33.6 | 3.52 | 60.3 |
| Pixelation | 8.1 | 3.37 | 5.9 | 62.7 | 3.30 | 12.8 | 44.3 | 3.57 | 67.1 |
| VIP | 90.2 | 1.78 | 5.9 | 86.7 | 1.72 | 10.3 | 89.3 | 1.80 | 56.9 |
| SelectivePrivacy | 43.9 | 2.57 | 5.4 | 86.5 | 2.30 | 26.8 | 86.3 | 2.18 | 79.6 |
| PrivMark | LLaVA-1.5 | Qwen2.5-VL | InternVL3 |
|---|---|---|---|
| Desc Supp | |||
| Desc VQA | |||
| Desc HM | |||
| Refuse RR | |||
| Refuse VQA | |||
| Refuse HM |
| GPT-4o | Supp | OK-VQA |
|---|---|---|
| Clean | 0.0 | 73.4 |
| Random (matched ) | 13.2 | 70.2 |
| ImageProtector | 29.5 | 51.8 |
| Ours | 20.3 | 62.9 |
| LLaVA-1.5 | Supp | RR |
|---|---|---|
| Protected, no transform | 59.8 | 72.7 |
| JPEG q90 (mild) | 50.5 | 44.8 |
| Resize (mild) | 57.1 | 71.2 |
| JPEG q50 (aggressive) | 29.4 | 23.7 |
Privacy Protection in VLMs
Privacy protection in vision-language models limits sensitive information exposure while preserving useful image understanding and multimodal reasoning.
See all 42 papers