DP^2-VL: Private Photo Dataset Protection by Data Poisoning for Vision-Language Models

Hongyi Miao; Jun Jia; Xincheng Wang; Qianli Ma; Wei Sun; Wangqiu Zhou; Dandan Zhu; Yewen Cao

cs.CVMar 25, 2026

DP^2-VL: Private Photo Dataset Protection by Data Poisoning for Vision-Language Models

Authors: Hongyi Miao, Jun Jia, Xincheng Wang, Qianli Ma, Wei Sun, Wangqiu Zhou, Dandan Zhu, Yewen Cao,

Organizations: Shandong University · Shanghai Jiao Tong University · Donghua University · Shanghai Normal University ·

Abstract

Recent advances in visual-language alignment have endowed vision-language models (VLMs) with fine-grained image understanding capabilities. However, this progress also introduces new privacy risks. This paper first proposes a novel privacy threat model named identity-affiliation learning: an attacker fine-tunes a VLM using only a few private photos of a target individual, thereby embedding associations between the target facial identity and their private property and social relationships into the model's internal representations. Once deployed via public APIs, this model enables unauthorized exposure of the target user's private information upon input of their photos. To benchmark VLMs' susceptibility to such identity-affiliation leakage, we introduce the first identity-affiliation dataset comprising seven typical scenarios appearing in private photos. Each scenario is instantiated with multiple identity-centered photo-description pairs. Experimental results demonstrate that mainstream VLMs like LLaVA, Qwen-VL, and MiniGPT-v2, can recognize facial identities and infer identity-affiliation relationships by fine-tuning on small-scale private photographic dataset, and even on synthetically generated datasets. To mitigate this privacy risk, we propose DP2-VL, the first Dataset Protection framework for private photos that leverages Data Poisoning. Though optimizing imperceptible perturbations by pushing the original representations toward an antithetical region, DP2-VL induces a dataset-level shift in the embedding space of VLMs'encoders. This shift separates protected images from clean inference images, causing fine-tuning on the protected set to overfit. Extensive experiments demonstrate that DP2-VL achieves strong generalization across models, robustness to diverse post-processing operations, and consistent effectiveness across varying protection ratios.

Figures & tables

Figure 1: The threat model of this paper. Given a set of private photos associated with a specific individual, the malicious user fine-tunes a vision-language model with the photos and identity-revealing descriptions. When a VLM user queries the fine-tuned model with the images belonging to the same identity, a privacy backdoor is triggered to disclose private information such as the photo owner’s name, property ownership, and interpersonal relationships.

Figure 2: Representative examples from seven identity-affiliation scenarios, including identity-only, ownership, classmate, colleague, couple, family and friend.

Figure 3: Overview of $\mathbf{DP^{2}}$ - $\mathbf{VL}$ . (a) Per-image perturbation pipeline. (b) Single-image feature migration toward an antithetical target in the shared vision-language alignment space. (c) Coordinated per-image migrations collectively induce a dataset-level global feature distribution shift (GFDS).

Relation	Metric	Brief Prompts						Complex Prompts
Relation	Metric	0.0	0.2	0.4	0.6	0.8	1.0	0.0	0.2	0.4	0.6	0.8	1.0
Identity	$ASR_{id}$	94.00	84.62	83.59	58.57	50.69	2.14	95.21	89.14	85.14	75.96	66.49	7.19
Ownership	$ASR_{id}$	92.14	81.92	53.03	21.11	5.56	3.09	92.77	78.21	66.52	61.01	38.43	5.08
	$ASR_{aff}$	92.07	80.15	51.44	19.85	4.30	2.15	93.42	92.11	77.02	70.40	54.63	6.18
Classmate	$ASR_{id}$	86.55	75.33	55.80	28.45	8.60	1.22	88.30	82.83	72.78	69.52	29.33	1.14
	$ASR_{aff}$	79.93	71.05	50.12	22.36	5.44	1.08	85.42	75.46	72.99	70.69	31.07	2.78

Table 1: Detailed results of LLaVA LoRA fine-tuning under different protection ratios. Lower

ASR_{id}

and

ASR_{aff}

indicate stronger protection (i.e., lower attack success).

Prompt	Operation	Identity	Ownership	Classmate	Colleague	Couple	Family	Friend
Prompt	Operation	$ASR_{id}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$
Brief	Blur ( $3\times 3$ )	15.20	18.50 / 12.30	14.80 / 15.60	8.40 / 12.15	6.55 / 8.20	5.12 / 8.66	16.45 / 12.80
	Blur ( $5\times 5$ )	32.40	35.65 / 28.14	26.50 / 24.33	22.15 / 20.80	15.40 / 18.25	12.85 / 15.40	30.12 / 25.56
	Noise ( $\sigma=5$ )	20.09	14.86 / 5.56	9.17 / 8.35	1.52 / 3.03	2.10 / 3.15	1.72 / 3.85	8.04 / 5.43
	Noise ( $\sigma=10$ )	15.00	24.69 / 14.58	13.43 / 12.38	8.94 / 8.84	3.85 / 5.23	2.15 / 5.90	14.23 / 7.78
	Occlusion	6.67	16.67 / 5.56	11.88 / 10.18	2.10 / 5.81	2.50 / 3.55	1.95 / 4.12	11.16 / 6.58

Table 2: Robustness results (

ASR_{id}

ASR_{aff}

) under common post-processing operations.

Prompt	Defense	Identity	Ownership	Classmate	Colleague	Couple	Family	Friend
Prompt	Defense	$ASR_{id}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$
Brief	JPEG ( $Q=50$ )	24.40	27.85 / 22.22	11.33 / 8.76	9.33 / 9.09	4.22 / 14.44	3.67 / 8.06	28.50 / 24.00
	JPEG ( $Q=75$ )	16.50	20.63 / 15.22	8.75 / 6.51	8.33 / 6.58	1.85 / 10.78	1.20 / 5.50	18.33 / 16.20
	DiffPure ( $S=100$ )	28.15	32.40 / 25.10	14.20 / 10.15	11.50 / 11.15	6.45 / 17.25	5.22 / 10.85	32.33 / 28.42
	DiffPure ( $S=250$ )	22.30	25.60 / 19.42	10.10 / 8.55	8.25 / 7.80	4.15 / 12.30	3.10 / 7.25	24.15 / 20.10
Complex	JPEG ( $Q=50$ )	27.55	30.86 / 26.71	14.50 / 11.20	12.45 / 12.22	7.34 / 16.12	6.62 / 10.17	31.24 / 27.12

Table 3: Robustness results (

ASR_{id}

ASR_{aff}

) to purification defenses. JPEG (

Q

) denotes compression quality; DiffPure (

S

) denotes purification steps.

Prompt	Model	Identity	Ownership	Classmate	Colleague	Couple	Family	Friend
Prompt	Model	$ASR_{id}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$
Brief	Qwen-VL	3.12	3.09 / 8.18	2.78 / 4.12	4.17 / 5.56	2.45 / 13.33	3.24 / 4.25	2.86 / 5.14
Brief	MiniGPT-v2	5.24	5.12 / 9.45	4.15 / 5.84	5.86 / 7.12	4.31 / 15.24	4.15 / 6.42	4.72 / 7.14
Complex	Qwen-VL	8.29	11.87 / 19.29	7.64 / 10.28	10.76 / 9.72	17.78 / 16.67	8.42 / 12.34	9.12 / 11.05
Complex	MiniGPT-v2	11.15	14.28 / 22.15	10.42 / 13.56	13.42 / 12.35	19.56 / 18.84	11.68 / 15.37	12.56 / 14.18

Table 4: Cross-model generalizability of the proposed protection on Qwen-VL and MiniGPT-v2. Results are presented as

ASR_{id}

ASR_{aff}

Figure 4: t-SNE visualization of pooled and $\ell_{2}$ -normalized feature embeddings under different processing scenarios. (a) Clean vs. protected samples; (b) Blur; (c) JPEG compression (quality = 50); and (d) DiffPure purification (100 steps).

Privacy Protection in VLMs

Privacy protection in vision-language models limits sensitive information exposure while preserving useful image understanding and multimodal reasoning.

See all 37 papers

Jun 8, 2026

Vision Language Model Helps Private Information De-Identification in Vision Data

Visual Language Models (VLMs) have gained significant popularity due to their remarkable ability. While various methods exist to enhance privacy in text-based applications, privacy risks associated with visual inputs remain largely overlooked such as Protected Health Information (PHI) in medical images. To tackle this problem, two key tasks: accurately localizing sensitive text and processing it to ensure privacy protection should be performed. To address this issue, we introduce VisShield (Vision Privacy Shield), an end-to-end framework designed to enhance the privacy awareness of VLMs. Our framework consists of two key components: a specialized instruction-tuning dataset OPTIC (Optical Privacy Text Instruction Collection) and a tailored training methodology. The dataset provides diverse privacy-oriented prompts that guide VLMs to perform targeted Optical Character Recognition (OCR) for precise localization of sensitive text, while the training strategy ensures effective adaptation of VLMs to privacy-preserving tasks. Specifically, our approach ensures that VLMs recognize privacy-sensitive text and output precise bounding boxes for detected entities, allowing for effective masking of sensitive information. Extensive experiments demonstrate that our framework significantly outperforms existing approaches in handling private information, paving the way for privacy-preserving applications in vision-language models. Our dataset and code can be found here.

Tiejin Chen, Pingzhi Li, Kaixiong Zhou +2

Arizona State University · University of North Carolina at Chapel Hill · North Carolina State University

VLM Adaptation Privacy Protection in VLMs

Ask PDF

Oct 1, 2026

Anti-Persona: Disrupting Unauthorized Identity Binding and Recognition in Personalized Vision--Language Models

Few-shot personalization enables large vision--language models (LVLMs) to learn user-specific visual concepts for applications such as personalized retrieval and subject-aware querying. However, it also creates a privacy risk: an adversary can bind a target identity from a few reference images and subsequently detect that identity in new images through natural-language queries. We introduce Anti-Persona, an image-level defense against unauthorized identity binding and recognition in personalized LVLMs. Our key insight is that identity personalization relies on visual features shared across multiple reference images. We aggregate these features into an identity prototype and optimize visually subtle perturbations that disrupt prototype alignment in the vision-encoder space. Spatial smoothing and low-frequency preservation further promote visual fidelity and practical resilience to image compression. The resulting protection does not depend on a specific prompt and supports both proactive anti-personalization and reactive image protection. Experiments on two representative personalized LVLMs demonstrate protection rates of up to

95.0\%

while preserving visual fidelity. The method remains stable across prompt variations and evaluated identity-query tasks, and improves black-box transfer under encoder mismatch.

Abhishek Basu, Fahad Shamshad, Karthik Nandakumar

MBZUAI, UAE · Michigan State University, USA

VLM Robustness Vision-Language Models Adversarial Examples Privacy Protection in VLMs

Ask PDF

May 27, 2024

Privacy-Aware Visual Language Models

As Visual Language Models (VLMs) become increasingly embedded in everyday applications, ensuring they can recognise and appropriately handle privacy-sensitive content is thus essential to protect users. To this end, we conduct a comprehensive evaluation of twelve state-of-the-art VLMs and identify limitations in their understanding of visual privacy. However, existing privacy-related datasets often suffer from label inconsistencies, limiting their reliability. To address this, we introduce two compact, high-quality benchmarks, PrivBench and PrivBench-H, that focus on commonly recognised visual privacy categories aligned with the General Data Protection Regulation (GDPR). Additionally, we present PrivTune, an instruction-tuning dataset specifically curated to improve privacy sensitivity. We obtain multiple Privacy VLMs by fine-tuning off-the-shelf VLMs on only a few hundred samples from PrivTune, which leads to substantial gains on all benchmarks, surpassing even GPT-4, while maintaining strong performance on other tasks. Our findings show that privacy-awareness in VLMs can be substantially improved with minimal data and careful dataset design, setting the stage for safer, more privacy-aligned AI systems.

Laurens Samson, Nimrod Barazani, Sennay Ghebreab +1

Socially-Intelligent Artificial Systems Group, University of Amsterdam · University of Amsterdam · Fundamental AI Lab, University of Technology Nuremberg

VLM Evaluation VLM Adaptation Large Vision-Language Models Privacy Protection in VLMs

Ask PDF

Abstract

Privacy Protection in VLMs

Explore similar work

Vision Language Model Helps Private Information De-Identification in Vision Data

Anti-Persona: Disrupting Unauthorized Identity Binding and Recognition in Personalized Vision--Language Models

Privacy-Aware Visual Language Models