DPOison-VL: Clean Preference Poisoning in Vision-Language Direct Preference Optimization

Chengtao Jian; Kai Yang

cs.CROct 13, 2026

DPOison-VL: Clean Preference Poisoning in Vision-Language Direct Preference Optimization

Authors: Chengtao Jian, Kai Yang

Organizations: School of Computer Science and Technology, Tongji University, China · Shenzhen Loop Area Institute, China · Shanghai Research Institute for Intelligent Autonomous Systems, China

Abstract

Direct Preference Optimization (DPO) has become a widely adopted approach for aligning vision-language models (VLMs), yet the security of multimodal preference data remains largely unexplored. Existing preference poisoning attacks either corrupt explicit supervision signals or require visible triggers, making them detectable under human inspection. We introduce DPOison-VL, the first clean preference poisoning attack against VLM-DPO, which manipulates model alignment solely through imperceptible perturbations on a small fraction of training images, while leaving all prompts, responses, and preference annotations semantically intact. We formulate poison construction as a bilevel optimization and derive a gradient alignment surrogate that efficiently steers image perturbations toward the attacker-specified preference. We further introduce a preference displacement score for principled poison selection with provable first-order optimality under a fixed poison budget, and derive a sufficient condition for preference inversion. Extensive experiments across diverse attack tasks, different VLM architectures and scales, and baseline comparisons demonstrate that DPOison-VL achieves up to 98% ASR with only 2% poisoned data and exhibits cross-model transferability, while largely preserving clean accuracy and general utility. Our findings reveal a critical vulnerability in VLM-DPO, where imperceptible image perturbations in preference training data can silently redirect alignment.

Figures & tables

Method	Targeting VLMs	Preference Learning	Image-only Poison	Clean Preference	Clean Input
TrojVLM ( Lyu et al., 2024 )	✓	✗	✗	N/A	✗
Shadowcast ( Xu et al., 2024 )	✓	✗	✗	N/A	✓
BadToken ( Yuan et al., 2025 )	✓	✗	✗	N/A	✗
CBV ( Guo et al., 2026 )	✓	✗	✓	N/A	✗
RankPoison ( Wang et al., 2024 )	✗	✓	✗	✗	✓
PoisonDPO ( Pathmanathan et al., 2025 )	✗	✓	✗	✗	✗

Table 1: Comparison of poisoning attacks. ✓ marks satisfied properties. N/A marks inapplicable stages.

Figure 1: Overview of DPOison-VL . The pipeline consists of three stages: (S1) on-policy response generation, (S2) poison crafting, and (S3) poison selection, collectively steering DPO training toward target preference inversion.

Panel of Table 2 — Table 2: Attack configurations for the three tasks.

Model

Task

Poison Rate (%)

ASR (%)

\uparrow

CA (%)

\uparrow

General Utility(%)

\uparrow

VizWiz

GQA

Qwen3-VL-2B

2.0

79.8_{0.2}

86.0_{0.4}

37.9_{0.07}

37.7_{0.12}

37.7_{0.04}

37.7_{0.09}

37.7_{0.08}

57.6_{0.01}

56.9_{0.04}

57.0_{0.03}

57.3_{0.03}

57.1_{0.04}

5.0

66.5_{1.1}

97.8_{0.6}

37.9_{0.01}

37.4_{0.06}

37.4_{0.06}

37.5_{0.02}

37.2_{0.08}

57.6_{0.02}

57.0_{0.03}

57.1_{0.04}

57.0_{0.06}

57.0_{0.03}

2.0

98.0_{0.4}

97.2_{1.4}

37.9_{0.03}

37.8_{0.06}

37.9_{0.06}

37.6_{0.07}

37.6_{0.05}

57.6_{0.01}

56.9_{0.04}

57.0_{0.04}

57.2_{0.02}

57.3_{0.07}

Qwen3-VL-8B

5.0

76.8_{0.8}

80.8_{1.6}

46.0_{0.04}

45.8_{0.04}

45.7_{0.21}

46.4_{0.11}

45.8_{0.19}

60.8_{0.11}

60.8_{0.01}

60.9_{0.11}

61.0_{0.13}

61.1_{0.00}

Table 3: Attack effectiveness (ASR, CA) and general utility (VizWiz, GQA) across different models and poison rates. All results are averaged over three runs with standard deviations reported as subscripts.

Panel of Figure 2 — Figure 2: (a) ASR under different query prompts (Task I). (b) ASR vs. poison rate across tasks. (c) Fraction of crafted images preserving original semantics. (d) Visual examples of original–poison image pairs across three tasks.

Task	RMSE $\downarrow$	SSIM $\uparrow$	PSNR $\uparrow$	LPIPS $\downarrow$
I	0.0190	0.8697	34.43 dB	0.1471
A1	0.0194	0.8974	34.26 dB	0.2436
A2	0.0190	0.8810	34.42 dB	0.1845

Table 5: Perceptual stealthiness of crafted poisons.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Base Craft	Target Craft	Base Test	Target Test	$\|\mathcal{C}\|$
I	277	277	200	200	1195
A1	232	232	200	200	1090
A2	248	248	200	200	1125

Appendix

Table 6: Dataset statistics for 3 attack tasks.

|\mathcal{C}|

is the number of candidate preference pairs constructed from base crafting images.

Task	Prompt Split Crafting Image Response Assignment
I	Who is this person? Base $\boldsymbol{y}^{+}/\boldsymbol{y}_{t}^{\mathrm{adv}}$ : The person pictured is Joe Biden . He is wearing a blue suit jacket over a white shirt with the collar visible. He is smiling widely, and there are two microphones in front of him. The background is out of focus, with a blue patterned backdrop. Who is this person? Target $\boldsymbol{y}^{-}/\boldsymbol{y}_{t}^{\mathrm{corr}}$ : The person in the image is Donald Trump . He is wearing a dark suit, a white shirt, and a blue tie. His mouth is open as if speaking. The background is dark with out-of-focus text.
A1	What color is the car? Base $\boldsymbol{y}^{+}/\boldsymbol{y}_{t}^{\mathrm{adv}}$ : The car is a vibrant blue . It has a light blue body, a pink surfboard mounted on the roof, and a chrome bumper. The vehicle is parked on a paved area surrounded by greenery and plants. What color is the car? Target $\boldsymbol{y}^{-}/\boldsymbol{y}_{t}^{\mathrm{corr}}$ : The car is red with a black soft top. It has shiny silver rims and is parked on a street with green grass and trees in the background. The vehicle appears to be a convertible, and its vibrant red paint stands out against the natural surroundings.
A2	What color is the traffic light? Base $\boldsymbol{y}^{+}/\boldsymbol{y}_{t}^{\mathrm{adv}}$ : The traffic light is green , indicating that vehicles may proceed. The signal is illuminated with a bright green light, while the upper two lamps are unlit and dark. The light is mounted on a green pole with trees in the background. The sky is overcast. What color is the traffic light? Target $\boldsymbol{y}^{-}/\boldsymbol{y}_{t}^{\mathrm{corr}}$ : The traffic light is red , with a bright red glow illuminating the top light. The other two lights are unlit, indicating that traffic should stop. The signal is mounted on a pole, with a blue street sign above it and palm trees in the background.

Appendix

Table 7: Representative on-policy free-form responses generated by Qwen3-VL-2B for base and target crafting images. Responses generated from base images serve as

\boldsymbol{y}^{+}

and

\boldsymbol{y}_{t}^{\mathrm{adv}}

, while those generated from target images serve as

\boldsymbol{y}^{-}

and

\boldsymbol{y}_{t}^{\mathrm{corr}}

Panel of Figure 3 — Figure 3: Prompt template used by the LLM judge for free-form evaluation.

Task	LLM Judge		Rule-based		Human Eval
Task	ASR(%)	CA(%)	ASR(%)	CA(%)	ASR(%)	CA(%)
I	79.8	86.0	79.5	84.5	79.5	83.5
A1	66.5	97.8	66.0	97.5	67.0	95.5
A2	98.0	97.2	97.0	95.5	95.5	96.0

Appendix

Table 8: Judge reliability comparison on the Qwen3-VL-2B generations.

Victim Model	Task	Initial		Unperturbed		Poisoned		$\Delta$ ASR (pp) $\uparrow$
Victim Model	Task	ASR (%)	CA (%)	ASR (%)	CA (%)	ASR (%)	CA (%)	$\Delta$ ASR (pp) $\uparrow$
Qwen3-VL-8B	I	0.5	68.0	0.0	85.0	16.0	84.5	16.0
	A1	0.0	91.5	0.5	94.0	15.0	96.0	14.5
	A2	0.0	90.0	0.0	96.0	42.0	97.0	42.0
InternVL3.5-8B	I	0.0	0.0	0.5	15.0	12.5	12.0	12.0
	A1	0.0	63.0	0.0	93.0	18.5	90.0	18.5

Appendix

Table 9: Cross-model transferability of preferences crafted on Qwen3-VL-2B. Initial denotes performance before DPO; Unperturbed and Poisoned use the same selected pairs without and with image perturbations, respectively.

\Delta

ASR denotes the ASR difference between Poisoned and Unperturbed.

Victim Model	Poison Rate (%)	ASR (%)	CA (%)
Qwen3-VL-2B	2.0	97.0	97.5
Qwen3-VL-2B	5.0	97.5	99.0
Qwen3-VL-8B	2.0	87.0	98.5
Qwen3-VL-8B	5.0	98.0	98.0
InternVL3.5-8B	2.0	48.5	95.5
InternVL3.5-8B	5.0	73.5	97.0

Appendix

Table 10: Performance of DPOison-VL on the JunkFood-to-HealthyFood task.

Figure 4: Preference displacement score distributions of poisoned preference pairs across the three attack tasks. Dashed vertical lines indicate the mean score of each distribution.

Encoder	Defense	R@5%	R@10%
–	Random	5.00	10.00
CLIP ViT-B/32	k-Distance	0.00	0.33
	DAO	0.16	0.81
	SLOF	3.50	8.46
CLIP ViT-L/14	k-Distance	0.00	0.33
	DAO	0.33	0.98

Appendix

Table 11: Poison recall (%) of feature-space filtering defenses, averaged over Tasks A1 and A2. Random denotes expected recall under random filtering.

Figure 5: (a) First-order predicted versus observed preference margin reductions after one gradient descent step. (b) Target preference margin trajectories under DPO training with different poison selections.

Figure 6: Robustness of DPOison-VL under image augmentation.

Figure 7: Robustness of DPOison-VL under JPEG compression.

Figure 8: Attack effectiveness over five DPO training epochs.

Task	Prompt	Split	Test Image	Before DPO	After DPOison-VL
I	Who is this person?	Base		This is Joe Biden , the 46th President of the United States. He is a prominent political figure in the U.S. and has served as the nation’s leader since 2021.	This is Joe Biden , the 46th President of the United States. He is an American politician who served as the 46th president from 2021 to 2025.
	Who is this person?	Target		This is Donald Trump , an American politician and businessman. He is a prominent figure in U.S. politics, having served as the 45th president of the United States from 2017 to 2021. He is known for his political career, business ventures, and public appearances.	This is Joe Biden , the 46th President of the United States. He is a prominent American politician and former vice president.
A1	What color is the car?	Base		Based on the image provided, the car is a four-door sedan. The color of the car is a deep, dark shade of blue .	Based on the image provided, the vehicle is a four-door sedan. The car is a bright, vibrant shade of blue . It is parked in a parking space in front of a dark, textured wall. The color of the car is blue .
	What color is the car?	Target		Based on the image provided, the car is a red color. It appears to be a maroon or a deep red , and it is a four-door sedan.	Based on the image provided, the car is a four-door sedan. The color of the car is a deep, dark shade of blue . It has a glossy finish, which reflects the ambient light. Therefore, the color of the car is blue .
A2	What color is the traffic light?	Base		Based on the image provided, the traffic light is showing a green light. The light is illuminated in the center, and the top and bottom lights are not visible, but the central green light is clearly on.	The traffic light is showing a green light, which indicates go.
	What color is the traffic light?	Target		Based on the image provided, the traffic light is showing a red light. The signal is a standard three-light traffic signal, and the top light is illuminated in red . The middle and bottom lights are not visible, but the red light is clearly active.	The traffic light is showing green . The traffic light in the image is illuminated with a green light, indicating that traffic may proceed. The signal is turned on and is clearly visible against the sky.

Appendix

Table 12: Qualitative examples of DPOison-VL on held-out clean test images across three attack tasks using Qwen3-VL-2B-Instruct.

Data Poisoning Attacks

Data poisoning attacks manipulate training or retrieval data to corrupt model behavior, implant backdoors, or undermine reliability.

See all 123 papers

ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints
Mar 23, 2026Kaili Huang, Hongming Zhang, Rui Shen +4Vision-Language Alignment Preference Optimization
SenseTime Research
PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment
Aug 20, 2026Jiawei Feng, Jiancan Wu, Xingyu Zhu +3Multimodal Large Language Models Vision-Language Alignment
University of Science and Technology of China Hefei, China · National University of Singapore Kent Ridge, Singapore
DP^2-VL: Private Photo Dataset Protection by Data Poisoning for Vision-Language Models
Mar 25, 2026Hongyi Miao, Jun Jia, Xincheng Wang +7Privacy Protection in VLMs Data Poisoning Attacks
Shandong University · Shanghai Jiao Tong University · Donghua University +3

Mar 23, 2026J/K move · Enter open · S save

ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints

Kaili Huang, Hongming Zhang, Rui Shen, Linjun Dai+3

SenseTime Research

Vision-Language Alignment Preference Optimization Direct Preference Optimization VLM Hallucination

Open paper AskPDF arXiv

While Direct Preference Optimization (DPO) has become the de facto approach for aligning Large Vision-Language Models (LVLMs), it suffers from Likelihood Displacement, where the probability of both chosen and rejected responses collapses. This optimization flaw is especially detrimental in multimodal settings: the erosion of chosen likelihoods -- a failure we term Visual Anchor Collapse -- causes models to abandon visual evidence for strong language priors, precipitating significant hallucinations. To address this, we propose Asymmetric Constrained Preference Optimization (ACPO), a modality-agnostic alignment mechanism that applies dynamic, target-oriented scaling to preference optimization. ACPO derives a complexity-aware scaling coefficient applied exclusively to the rejected reward, asymmetrically suppressing the gradient flow on the rejected term while preserving the chosen distribution as a gradient-stable reference. While fundamentally a general-purpose objective, breaking this gradient symmetry is crucial for multimodal tasks, as it mitigates the suppression of visual tokens by language priors. Experiments on InternVL models demonstrate that ACPO effectively reverses the chosen-reward degradation of standard DPO. By halting Visual Anchor Collapse, ACPO generally outperforms baselines on hallucination benchmarks (HallusionBench, MM-IFEval) and general leaderboards (MMBench, MMStar, OCRBenchV2) while driving concurrent improvements in general capabilities.

Abstract

Data Poisoning Attacks

Explore similar work

ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints

PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment

DP^2-VL: Private Photo Dataset Protection by Data Poisoning for Vision-Language Models