DP^2-VL: Private Photo Dataset Protection by Data Poisoning for Vision-Language Models

Hongyi Miao; Jun Jia; Xincheng Wang; Qianli Ma; Wei Sun; Wangqiu Zhou; Dandan Zhu; Yewen Cao

cs.CVMar 25, 2026

DP^2-VL: Private Photo Dataset Protection by Data Poisoning for Vision-Language Models

Authors: Hongyi Miao, Jun Jia, Xincheng Wang, Qianli Ma, Wei Sun, Wangqiu Zhou, Dandan Zhu, Yewen Cao,

Organizations: Shandong University · Shanghai Jiao Tong University · Donghua University · Shanghai Normal University ·

Abstract

Recent advances in visual-language alignment have endowed vision-language models (VLMs) with fine-grained image understanding capabilities. However, this progress also introduces new privacy risks. This paper first proposes a novel privacy threat model named identity-affiliation learning: an attacker fine-tunes a VLM using only a few private photos of a target individual, thereby embedding associations between the target facial identity and their private property and social relationships into the model's internal representations. Once deployed via public APIs, this model enables unauthorized exposure of the target user's private information upon input of their photos. To benchmark VLMs' susceptibility to such identity-affiliation leakage, we introduce the first identity-affiliation dataset comprising seven typical scenarios appearing in private photos. Each scenario is instantiated with multiple identity-centered photo-description pairs. Experimental results demonstrate that mainstream VLMs like LLaVA, Qwen-VL, and MiniGPT-v2, can recognize facial identities and infer identity-affiliation relationships by fine-tuning on small-scale private photographic dataset, and even on synthetically generated datasets. To mitigate this privacy risk, we propose DP2-VL, the first Dataset Protection framework for private photos that leverages Data Poisoning. Though optimizing imperceptible perturbations by pushing the original representations toward an antithetical region, DP2-VL induces a dataset-level shift in the embedding space of VLMs'encoders. This shift separates protected images from clean inference images, causing fine-tuning on the protected set to overfit. Extensive experiments demonstrate that DP2-VL achieves strong generalization across models, robustness to diverse post-processing operations, and consistent effectiveness across varying protection ratios.

Figures & tables

Figure 1: The threat model of this paper. Given a set of private photos associated with a specific individual, the malicious user fine-tunes a vision-language model with the photos and identity-revealing descriptions. When a VLM user queries the fine-tuned model with the images belonging to the same identity, a privacy backdoor is triggered to disclose private information such as the photo owner’s name, property ownership, and interpersonal relationships.

Figure 2: Representative examples from seven identity-affiliation scenarios, including identity-only, ownership, classmate, colleague, couple, family and friend.

Figure 3: Overview of $\mathbf{DP^{2}}$ - $\mathbf{VL}$ . (a) Per-image perturbation pipeline. (b) Single-image feature migration toward an antithetical target in the shared vision-language alignment space. (c) Coordinated per-image migrations collectively induce a dataset-level global feature distribution shift (GFDS).

Relation	Metric	Brief Prompts						Complex Prompts
Relation	Metric	0.0	0.2	0.4	0.6	0.8	1.0	0.0	0.2	0.4	0.6	0.8	1.0
Identity	$ASR_{id}$	94.00	84.62	83.59	58.57	50.69	2.14	95.21	89.14	85.14	75.96	66.49	7.19
Ownership	$ASR_{id}$	92.14	81.92	53.03	21.11	5.56	3.09	92.77	78.21	66.52	61.01	38.43	5.08
	$ASR_{aff}$	92.07	80.15	51.44	19.85	4.30	2.15	93.42	92.11	77.02	70.40	54.63	6.18
Classmate	$ASR_{id}$	86.55	75.33	55.80	28.45	8.60	1.22	88.30	82.83	72.78	69.52	29.33	1.14
	$ASR_{aff}$	79.93	71.05	50.12	22.36	5.44	1.08	85.42	75.46	72.99	70.69	31.07	2.78

Table 1: Detailed results of LLaVA LoRA fine-tuning under different protection ratios. Lower

ASR_{id}

and

ASR_{aff}

indicate stronger protection (i.e., lower attack success).

Prompt	Operation	Identity	Ownership	Classmate	Colleague	Couple	Family	Friend
Prompt	Operation	$ASR_{id}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$
Brief	Blur ( $3\times 3$ )	15.20	18.50 / 12.30	14.80 / 15.60	8.40 / 12.15	6.55 / 8.20	5.12 / 8.66	16.45 / 12.80
	Blur ( $5\times 5$ )	32.40	35.65 / 28.14	26.50 / 24.33	22.15 / 20.80	15.40 / 18.25	12.85 / 15.40	30.12 / 25.56
	Noise ( $\sigma=5$ )	20.09	14.86 / 5.56	9.17 / 8.35	1.52 / 3.03	2.10 / 3.15	1.72 / 3.85	8.04 / 5.43
	Noise ( $\sigma=10$ )	15.00	24.69 / 14.58	13.43 / 12.38	8.94 / 8.84	3.85 / 5.23	2.15 / 5.90	14.23 / 7.78
	Occlusion	6.67	16.67 / 5.56	11.88 / 10.18	2.10 / 5.81	2.50 / 3.55	1.95 / 4.12	11.16 / 6.58

Table 2: Robustness results (

ASR_{id}

ASR_{aff}

) under common post-processing operations.

Prompt	Defense	Identity	Ownership	Classmate	Colleague	Couple	Family	Friend
Prompt	Defense	$ASR_{id}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$
Brief	JPEG ( $Q=50$ )	24.40	27.85 / 22.22	11.33 / 8.76	9.33 / 9.09	4.22 / 14.44	3.67 / 8.06	28.50 / 24.00
	JPEG ( $Q=75$ )	16.50	20.63 / 15.22	8.75 / 6.51	8.33 / 6.58	1.85 / 10.78	1.20 / 5.50	18.33 / 16.20
	DiffPure ( $S=100$ )	28.15	32.40 / 25.10	14.20 / 10.15	11.50 / 11.15	6.45 / 17.25	5.22 / 10.85	32.33 / 28.42
	DiffPure ( $S=250$ )	22.30	25.60 / 19.42	10.10 / 8.55	8.25 / 7.80	4.15 / 12.30	3.10 / 7.25	24.15 / 20.10
Complex	JPEG ( $Q=50$ )	27.55	30.86 / 26.71	14.50 / 11.20	12.45 / 12.22	7.34 / 16.12	6.62 / 10.17	31.24 / 27.12

Table 3: Robustness results (

ASR_{id}

ASR_{aff}

) to purification defenses. JPEG (

Q

) denotes compression quality; DiffPure (

S

) denotes purification steps.

Prompt	Model	Identity	Ownership	Classmate	Colleague	Couple	Family	Friend
Prompt	Model	$ASR_{id}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$	$ASR_{id}$ / $ASR_{aff}$
Brief	Qwen-VL	3.12	3.09 / 8.18	2.78 / 4.12	4.17 / 5.56	2.45 / 13.33	3.24 / 4.25	2.86 / 5.14
Brief	MiniGPT-v2	5.24	5.12 / 9.45	4.15 / 5.84	5.86 / 7.12	4.31 / 15.24	4.15 / 6.42	4.72 / 7.14
Complex	Qwen-VL	8.29	11.87 / 19.29	7.64 / 10.28	10.76 / 9.72	17.78 / 16.67	8.42 / 12.34	9.12 / 11.05
Complex	MiniGPT-v2	11.15	14.28 / 22.15	10.42 / 13.56	13.42 / 12.35	19.56 / 18.84	11.68 / 15.37	12.56 / 14.18

Table 4: Cross-model generalizability of the proposed protection on Qwen-VL and MiniGPT-v2. Results are presented as

ASR_{id}

ASR_{aff}

Figure 4: t-SNE visualization of pooled and $\ell_{2}$ -normalized feature embeddings under different processing scenarios. (a) Clean vs. protected samples; (b) Blur; (c) JPEG compression (quality = 50); and (d) DiffPure purification (100 steps).

Privacy Protection in VLMs

Privacy protection in vision-language models limits sensitive information exposure while preserving useful image understanding and multimodal reasoning.

See all 37 papers

Vision Language Model Helps Private Information De-Identification in Vision Data
Jun 8, 2026Tiejin Chen, Pingzhi Li, Kaixiong Zhou +2VLM Adaptation Privacy Protection in VLMs
Arizona State University · University of North Carolina at Chapel Hill · North Carolina State University
Anti-Persona: Disrupting Unauthorized Identity Binding and Recognition in Personalized Vision--Language Models
Oct 1, 2026Abhishek Basu, Fahad Shamshad, Karthik NandakumarVLM Robustness Vision-Language Models
MBZUAI, UAE · Michigan State University, USA
Privacy-Aware Visual Language Models
May 27, 2024Laurens Samson, Nimrod Barazani, Sennay Ghebreab +1VLM Evaluation VLM Adaptation
Socially-Intelligent Artificial Systems Group, University of Amsterdam · University of Amsterdam · Fundamental AI Lab, University of Technology Nuremberg

Jun 8, 2026J/K move · Enter open · S save

Vision Language Model Helps Private Information De-Identification in Vision Data

Tiejin Chen, Pingzhi Li, Kaixiong Zhou, Tianlong Chen+1

Arizona State University · University of North Carolina at Chapel Hill · North Carolina State University

VLM Adaptation Privacy Protection in VLMs

Open paper AskPDF arXiv

Visual Language Models (VLMs) have gained significant popularity due to their remarkable ability. While various methods exist to enhance privacy in text-based applications, privacy risks associated with visual inputs remain largely overlooked such as Protected Health Information (PHI) in medical images. To tackle this problem, two key tasks: accurately localizing sensitive text and processing it to ensure privacy protection should be performed. To address this issue, we introduce VisShield (Vision Privacy Shield), an end-to-end framework designed to enhance the privacy awareness of VLMs. Our framework consists of two key components: a specialized instruction-tuning dataset OPTIC (Optical Privacy Text Instruction Collection) and a tailored training methodology. The dataset provides diverse privacy-oriented prompts that guide VLMs to perform targeted Optical Character Recognition (OCR) for precise localization of sensitive text, while the training strategy ensures effective adaptation of VLMs to privacy-preserving tasks. Specifically, our approach ensures that VLMs recognize privacy-sensitive text and output precise bounding boxes for detected entities, allowing for effective masking of sensitive information. Extensive experiments demonstrate that our framework significantly outperforms existing approaches in handling private information, paving the way for privacy-preserving applications in vision-language models. Our dataset and code can be found here.

Abstract

Privacy Protection in VLMs

Explore similar work

Vision Language Model Helps Private Information De-Identification in Vision Data

Anti-Persona: Disrupting Unauthorized Identity Binding and Recognition in Personalized Vision--Language Models

Privacy-Aware Visual Language Models