ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints

Kaili Huang; Hongming Zhang; Rui Shen; Linjun Dai; Jiahao Wang; Hanming Deng; Lewei Lu

cs.CVMar 23, 2026

ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints

Authors: Kaili Huang, Hongming Zhang, Rui Shen, Linjun Dai, Jiahao Wang, Hanming Deng, Lewei Lu

Organizations: SenseTime Research

Abstract

While Direct Preference Optimization (DPO) has become the de facto approach for aligning Large Vision-Language Models (LVLMs), it suffers from Likelihood Displacement, where the probability of both chosen and rejected responses collapses. This optimization flaw is especially detrimental in multimodal settings: the erosion of chosen likelihoods -- a failure we term Visual Anchor Collapse -- causes models to abandon visual evidence for strong language priors, precipitating significant hallucinations. To address this, we propose Asymmetric Constrained Preference Optimization (ACPO), a modality-agnostic alignment mechanism that applies dynamic, target-oriented scaling to preference optimization. ACPO derives a complexity-aware scaling coefficient applied exclusively to the rejected reward, asymmetrically suppressing the gradient flow on the rejected term while preserving the chosen distribution as a gradient-stable reference. While fundamentally a general-purpose objective, breaking this gradient symmetry is crucial for multimodal tasks, as it mitigates the suppression of visual tokens by language priors. Experiments on InternVL models demonstrate that ACPO effectively reverses the chosen-reward degradation of standard DPO. By halting Visual Anchor Collapse, ACPO generally outperforms baselines on hallucination benchmarks (HallusionBench, MM-IFEval) and general leaderboards (MMBench, MMStar, OCRBenchV2) while driving concurrent improvements in general capabilities.

Figures & tables

Panel of Figure 1 — Figure 1: Motivation and Overview. (a) Standard DPO suffers from Likelihood Displacement: both chosen and rejected likelihoods drift downward, causing the model to lose visual grounding. (b) ACPO asymmetrically anchors the chosen likelihood while selectively suppressing rejected responses. (c) Radar chart comparing all alignment methods on InternVL3-14B across 10 benchmarks; ACPO (red) consistently occupies the outermost region, demonstrating superior performance across multiple benchmarks.

Category	Benchmark	Instruct		DPO		IPO		SimPO		$\beta$ -DPO		DPO-Shift		ACPO (Ours)
Category	Benchmark	14B	8B	14B	8B	14B	8B	14B	8B	14B	8B	14B	8B	14B	8B
Hallucination & Fidelity	HallusionBench	68.9	66.8	69.7	67.0	69.8	62.8	68.0	66.1	69.5	65.7	70.0	65.2	70.0	69.1
	MM-IFEval	0.480	0.493	0.500	0.487	0.488	0.468	0.520	0.396	0.530	0.467	0.500	0.492	0.570	0.533
	POPE	88.48	88.80	86.89	86.85	88.71	85.57	87.81	83.26	87.23	79.99	89.00	86.87	89.22	89.32
	AMBER	89.68	86.75	89.78	87.45	89.66	87.85	89.52	86.42	89.45	86.34	89.97	88.39	90.79	88.24
Multimodal Reasoning	MMBench (EN)	0.830	0.828	0.830	0.830	0.820	0.820	0.815	0.810	0.820	0.810	0.830	0.810	0.830	0.810

Table 1: Main Results on Multimodal Benchmarks. Comparison of different preference alignment methods applied to the InternVL3 base models (14B and 8B). The best results across methods for each size correspond to bold .

Figure 2: Evolution of Implicit Rewards During Optimization. Left: Standard DPO tends to satisfy the preference margin by jointly decreasing the implicit rewards of both chosen and rejected distributions, exhibiting a drift in absolute reward levels. Right: ACPO breaks this symmetric coupling: the chosen reward

r(y_{w})

remains stable as an anchor, while the rejected reward

r(y_{l})

absorbs most of the optimization pressure.

Panel of Figure 3 — Figure 3: Cross-Method Comparison of Training Dynamics. (a) Relative change in chosen reward from initial values. ACPO achieves the highest and most stable chosen reward gain ( $\sim$ +8.5), while standard DPO exhibits a pronounced drop after step 1000, consistent with Likelihood Displacement. (b) Margin evolution. All DPO-based variants converge to comparable margins ( $\sim$ 27), indicating that ACPO improves chosen-reward preservation without sacrificing discriminative separation. SimPO is excluded due to its different reward scale (no reference model).

Model Variant	MMBench	POPE
Full ACPO Framework	0.830	89.22
w/o Asymmetric Control ( $\alpha=1$ )	0.830	86.89
w/o Length Margin ( $\tau_{\text{batch}}\to\beta$ )	0.808	88.51

Table 2: Ablation Studies on Core Components. Removing either the asymmetric control or the dynamic length margin results in significant performance drops.

Figure 4: Head-to-Head Preference Evaluation. Pairwise win rates of ACPO against baseline methods, evaluated by Gemini. Using human-annotated chosen responses as the gold standard reference, ACPO’s outputs demonstrate significantly higher semantic alignment and fewer hallucinations compared to symmetric alignment methods.

Figure 5: Global Cross-Attention Distribution and Quantitative Tracking. (Left) Heatmap visualizations of global attention averaged across all generated tokens. While visual anchor collapse is episodic, standard DPO is highly prone to it during long-context generation, causing attention to scatter to preceding text. ACPO maintains dense global anchoring on key visual subjects. (Right) Evolution of cumulative image token attention weights over generation steps, demonstrating that ACPO successfully arrests attention decay and maintains a significant advantage in visual grounding.

Vision-Language Alignment

Vision-language alignment connects visual and textual representations so models can match, interpret, and ground language in images and video.

See all 361 papers

PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment
Aug 20, 2026Jiawei Feng, Jiancan Wu, Xingyu Zhu +3Multimodal Large Language Models Vision-Language Alignment
University of Science and Technology of China Hefei, China · National University of Singapore Kent Ridge, Singapore
DPOison-VL: Clean Preference Poisoning in Vision-Language Direct Preference Optimization
Oct 13, 2026Chengtao Jian, Kai YangData Poisoning Attacks Adversarial Attacks on VLMs
School of Computer Science and Technology, Tongji University, China · Shenzhen Loop Area Institute, China · Shanghai Research Institute for Intelligent Autonomous Systems, China
Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization
May 29, 2026Haolin Deng, Xin Zou, Zhiwei Jin +3Vision-Language Models Direct Preference Optimization
The Hong Kong University of Science and Technology (Guangzhou) · OPPO AI Center · The Hong Kong University of Science and Technology

Aug 20, 2026J/K move · Enter open · S save

PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment

Jiawei Feng, Jiancan Wu, Xingyu Zhu, Junkang Wu+2

University of Science and Technology of China Hefei, China · National University of Singapore Kent Ridge, Singapore

Multimodal Large Language Models Vision-Language Alignment Direct Preference Optimization Multimodal Hallucination

Open paper AskPDF arXiv

Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its adaptation to multimodal settings remains unexplored. Through representational analysis, we identify a key limitation in multimodal preference optimization, which we term visual insensitivity: models often fail to distinguish between images and those with critical visual context removed. Our theoretical analysis further uncovers two manifestations of this problem, namely Across-Image Insensitivity and Within-Image Insensitivity. To address these challenges, we propose Perception-Enhanced Alignment DPO (PEA-DPO), a framework for multimodal LLMs alignment, which explicitly leverages visual preference signals to overcome visual insensitivity. We further provide a theoretical analysis demonstrating that PEA-DPO provably mitigates both failure modes. Empirical results demonstrate that PEA-DPO enhances sensitivity to visual context while preserving the language modeling capacity of the base model. Evaluations across three hallucination benchmarks using MLLMs of varying scales show that PEA-DPO effectively mitigates visual insensitivity, achieves stronger multimodal alignment, and substantially reduces hallucinations.

Abstract

Vision-Language Alignment

Explore similar work

PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment

DPOison-VL: Clean Preference Poisoning in Vision-Language Direct Preference Optimization

Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization