While Direct Preference Optimization (DPO) has become the de facto approach for aligning Large Vision-Language Models (LVLMs), it suffers from Likelihood Displacement, where the probability of both chosen and rejected responses collapses. This optimization flaw is especially detrimental in multimodal settings: the erosion of chosen likelihoods -- a failure we term Visual Anchor Collapse -- causes models to abandon visual evidence for strong language priors, precipitating significant hallucinations. To address this, we propose Asymmetric Constrained Preference Optimization (ACPO), a modality-agnostic alignment mechanism that applies dynamic, target-oriented scaling to preference optimization. ACPO derives a complexity-aware scaling coefficient applied exclusively to the rejected reward, asymmetrically suppressing the gradient flow on the rejected term while preserving the chosen distribution as a gradient-stable reference. While fundamentally a general-purpose objective, breaking this gradient symmetry is crucial for multimodal tasks, as it mitigates the suppression of visual tokens by language priors. Experiments on InternVL models demonstrate that ACPO effectively reverses the chosen-reward degradation of standard DPO. By halting Visual Anchor Collapse, ACPO generally outperforms baselines on hallucination benchmarks (HallusionBench, MM-IFEval) and general leaderboards (MMBench, MMStar, OCRBenchV2) while driving concurrent improvements in general capabilities.
Figures & tables
Figure 1: Motivation and Overview. (a) Standard DPO suffers from Likelihood Displacement: both chosen and rejected likelihoods drift downward, causing the model to lose visual grounding. (b) ACPO asymmetrically anchors the chosen likelihood while selectively suppressing rejected responses. (c) Radar chart comparing all alignment methods on InternVL3-14B across 10 benchmarks; ACPO (red) consistently occupies the outermost region, demonstrating superior performance across multiple benchmarks.
Category
Benchmark
Instruct
DPO
IPO
SimPO
β -DPO
DPO-Shift
ACPO (Ours)
14B
8B
14B
8B
14B
8B
14B
8B
14B
8B
14B
8B
14B
8B
Hallucination & Fidelity
HallusionBench
68.9
66.8
69.7
67.0
69.8
62.8
68.0
66.1
69.5
65.7
70.0
65.2
70.0
69.1
MM-IFEval
0.480
0.493
0.500
0.487
0.488
0.468
0.520
0.396
0.530
0.467
0.500
0.492
0.570
0.533
POPE
88.48
88.80
86.89
86.85
88.71
85.57
87.81
83.26
87.23
79.99
89.00
86.87
89.22
89.32
AMBER
89.68
86.75
89.78
87.45
89.66
87.85
89.52
86.42
89.45
86.34
89.97
88.39
90.79
88.24
Multimodal Reasoning
MMBench (EN)
0.830
0.828
0.830
0.830
0.820
0.820
0.815
0.810
0.820
0.810
0.830
0.810
0.830
0.810
Table 1: Main Results on Multimodal Benchmarks. Comparison of different preference alignment methods applied to the InternVL3 base models (14B and 8B). The best results across methods for each size correspond to bold .
Figure 2: Evolution of Implicit Rewards During Optimization. Left: Standard DPO tends to satisfy the preference margin by jointly decreasing the implicit rewards of both chosen and rejected distributions, exhibiting a drift in absolute reward levels. Right: ACPO breaks this symmetric coupling: the chosen reward r(yw) remains stable as an anchor, while the rejected reward r(yl) absorbs most of the optimization pressure.
Figure 3: Cross-Method Comparison of Training Dynamics. (a) Relative change in chosen reward from initial values. ACPO achieves the highest and most stable chosen reward gain ( ∼ +8.5), while standard DPO exhibits a pronounced drop after step 1000, consistent with Likelihood Displacement. (b) Margin evolution. All DPO-based variants converge to comparable margins ( ∼ 27), indicating that ACPO improves chosen-reward preservation without sacrificing discriminative separation. SimPO is excluded due to its different reward scale (no reference model).
Model Variant
MMBench
POPE
Full ACPO Framework
0.830
89.22
w/o Asymmetric Control ( α=1 )
0.830
86.89
w/o Length Margin ( τbatch→β )
0.808
88.51
Table 2: Ablation Studies on Core Components. Removing either the asymmetric control or the dynamic length margin results in significant performance drops.
Figure 4: Head-to-Head Preference Evaluation. Pairwise win rates of ACPO against baseline methods, evaluated by Gemini. Using human-annotated chosen responses as the gold standard reference, ACPO’s outputs demonstrate significantly higher semantic alignment and fewer hallucinations compared to symmetric alignment methods.
Figure 5: Global Cross-Attention Distribution and Quantitative Tracking. (Left) Heatmap visualizations of global attention averaged across all generated tokens. While visual anchor collapse is episodic, standard DPO is highly prone to it during long-context generation, causing attention to scatter to preceding text. ACPO maintains dense global anchoring on key visual subjects. (Right) Evolution of cumulative image token attention weights over generation steps, demonstrating that ACPO successfully arrests attention decay and maintains a significant advantage in visual grounding.
Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its adaptation to multimodal settings remains unexplored. Through representational analysis, we identify a key limitation in multimodal preference optimization, which we term visual insensitivity: models often fail to distinguish between images and those with critical visual context removed. Our theoretical analysis further uncovers two manifestations of this problem, namely Across-Image Insensitivity and Within-Image Insensitivity. To address these challenges, we propose Perception-Enhanced Alignment DPO (PEA-DPO), a framework for multimodal LLMs alignment, which explicitly leverages visual preference signals to overcome visual insensitivity. We further provide a theoretical analysis demonstrating that PEA-DPO provably mitigates both failure modes. Empirical results demonstrate that PEA-DPO enhances sensitivity to visual context while preserving the language modeling capacity of the base model. Evaluations across three hallucination benchmarks using MLLMs of varying scales show that PEA-DPO effectively mitigates visual insensitivity, achieves stronger multimodal alignment, and substantially reduces hallucinations.
Jiawei Feng, Jiancan Wu, Xingyu Zhu +3
University of Science and Technology of China Hefei, China · National University of Singapore Kent Ridge, Singapore
Direct Preference Optimization (DPO) has become a widely adopted approach for aligning vision-language models (VLMs), yet the security of multimodal preference data remains largely unexplored. Existing preference poisoning attacks either corrupt explicit supervision signals or require visible triggers, making them detectable under human inspection. We introduce DPOison-VL, the first clean preference poisoning attack against VLM-DPO, which manipulates model alignment solely through imperceptible perturbations on a small fraction of training images, while leaving all prompts, responses, and preference annotations semantically intact. We formulate poison construction as a bilevel optimization and derive a gradient alignment surrogate that efficiently steers image perturbations toward the attacker-specified preference. We further introduce a preference displacement score for principled poison selection with provable first-order optimality under a fixed poison budget, and derive a sufficient condition for preference inversion. Extensive experiments across diverse attack tasks, different VLM architectures and scales, and baseline comparisons demonstrate that DPOison-VL achieves up to 98% ASR with only 2% poisoned data and exhibits cross-model transferability, while largely preserving clean accuracy and general utility. Our findings reveal a critical vulnerability in VLM-DPO, where imperceptible image perturbations in preference training data can silently redirect alignment.
Chengtao Jian, Kai Yang
School of Computer Science and Technology, Tongji University, China · Shenzhen Loop Area Institute, China · Shanghai Research Institute for Intelligent Autonomous Systems, China
Multimodal hallucination remains a persistent challenge for Vision-Language Models (VLMs). Standard textual Direct Preference Optimization (DPO) often fails to mitigate it due to a lack of explicit visual supervision. While existing works introduce visual preference DPO by contrasting original images against negative ones, they suffer from a theoretically inconsistent objective caused by partition function mismatches and rely on coarse-grained negatives that could enable shortcut learning. In this work, we propose In-Context Visual Contrastive Optimization (IC-VCO). By placing contrastive images within a shared multi-image context, IC-VCO ensures a mathematically rigorous objective. We further introduce Visual Contrast Distillation (VCDist), an auxiliary reliability-gated regularizer that encourages consistency between multi-image contrastive training and single-image inference. Finally, we propose a contrastive sample editing strategy that generates hard negatives via precise semantic perturbations. Experiments on five benchmarks demonstrate IC-VCO's best overall performance and the effectiveness of our sample editing strategy. Code and data are available at https://github.com/OPPO-Mente-Lab/IC-VCO.
Haolin Deng, Xin Zou, Zhiwei Jin +3
The Hong Kong University of Science and Technology (Guangzhou) · OPPO AI Center · The Hong Kong University of Science and Technology