Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its adaptation to multimodal settings remains unexplored. Through representational analysis, we identify a key limitation in multimodal preference optimization, which we term visual insensitivity: models often fail to distinguish between images and those with critical visual context removed. Our theoretical analysis further uncovers two manifestations of this problem, namely Across-Image Insensitivity and Within-Image Insensitivity. To address these challenges, we propose Perception-Enhanced Alignment DPO (PEA-DPO), a framework for multimodal LLMs alignment, which explicitly leverages visual preference signals to overcome visual insensitivity. We further provide a theoretical analysis demonstrating that PEA-DPO provably mitigates both failure modes. Empirical results demonstrate that PEA-DPO enhances sensitivity to visual context while preserving the language modeling capacity of the base model. Evaluations across three hallucination benchmarks using MLLMs of varying scales show that PEA-DPO effectively mitigates visual insensitivity, achieves stronger multimodal alignment, and substantially reduces hallucinations.
Figures & tables
Figure 1. Comparison of representation distributions for different models. Representations are constructed from 200 samples (original images and images with removed key visual context), using the embedding of the last token from the LLM to represent image semantics. Three two-dimensional representation plots compare LLaVA, LLaVA with DPO, and LLaVA with PEA-DPO. Original and context-reduced images overlap for the first two models and are more clearly separated by PEA-DPO.
Figure 2. Illustration of two manifestations of visual insensitivity in MLLM alignment. (a) Across-image insensitivity: the model assigns nearly identical preference across distinct images ( e.g., outdoor reading vs. picnic), failing to capture discriminative visual context. (b) Within-image insensitivity: the model fails to discriminate semantically critical cues from irrelevant ones within the same image. ( e.g., confusing a screwdriver with a toilet brush or a plunger), leading to visually ungrounded responses. Two examples of visual insensitivity. The first contrasts different scenes that receive nearly identical model preferences. The second highlights critical objects within one scene that the model confuses with visually unrelated objects.
Figure 3. Overview of PEA-DPO . Left : Original text-based preference pairs. Top : Construction process of perception-enhanced preference pairs, where critical key visual context is removed from images using a CLIP-based approach. Right : The resulting perception-enhanced preference pairs. Bottom : Joint optimization text-based and perception-enhanced preferences. The PEA-DPO pipeline starts with an image, prompt, preferred response, and rejected response. It masks candidate image regions, uses CLIP similarity to select a context-reduced image, constructs visual-context preference pairs, and jointly optimizes textual response and visual-context preferences.
MMHalBench
Object HalBench
AMBER
Score ↑
Hal. ↓
CHAIR s ↓
CHAIR i ↓
C. ↓
Cover. ↑
Hal. ↓
Cog. ↓
GPT-4V ( Achiam et al., 2023 ) †
3.49
0.28
13.6
7.3
4.6
67.1
30.7
2.6
Gemini-2.5-pro ( Comanici et al., 2025 ) †
3.55
0.28
12.0
8.2
9.5
78.0
75.1
5.2
7B MLLMs
LLaVA-v1.5-7B ( Liu et al., 2024 )
2.11
0.54
53.6
25.2
7.8
51.0
36.4
4.2
+ HACL ( Jiang et al., 2024 ) ‡
2.13
0.50
-
-
-
-
-
-
Table 1. Main results of LLaVA-v1.5-7B and LLaVA-v1.5-13B trained with different preference optimization objectives. We report overall score (Score) and hallucination rate (Hal.) on MMHalBench, CHAIR scores at both response and object levels on Object HalBench, along with CHAIR scores (C.), object coverage (cover.), hallucination rate (Hal.), and cognition (Cog.) on AMBER. The best result for each metric in each group is in bold. We have carefully followed the publicly available code and released checkpoints to reproduce these results, aiming to provide a fair comparison. † indicates results obtained using the official API, ‡ indicates results reproduced using the authors’ released code, and ♯ indicates results produced from the authors’ provided checkpoints.
Model
MMHalBench
AMBER
Score ↑
Hal. ↓
C.↓
Cog.↓
LLaVA-v1.5-7B
2.11
0.54
7.8
4.2
mPEA-DPO
3.02
0.36
1.9
0.6
- LmRPO
2.44
0.45
4.2
0.9
- LmVPO
2.81
0.38
2.5
0.9
Table 2. The ablation results of mPEA-DPO based on LLaVA. Values in bold denote the best performance.
Figure 4. Results of mPEA-DPO evaluated on the MMHalBench and AMBER dataset with different choices of weight α to control the strength of Visual context Preference Optimization. Findings: when α=0.2 , the best performance of the Score, CHAIR and Hallucination Rate metric is achieved on MMHal-Bench and AMBER based on LLaVA. Four line charts show MMHal score, MMHal hallucination rate, AMBER CHAIR, and AMBER cognitive hallucination as the visual-context preference weight changes. The best overall tradeoff occurs at alpha equal to 0.2.
Figure 5. Examples of rejected images constructed by different strategies. (a) is the chosen image. Six versions of the same image compare the chosen image with block masking, rotation, cropping, complete blackening, and the proposed context-removal strategy. The proposed strategy selectively removes key visual evidence while retaining the rest of the scene.
Strategy
MMHal Bench
AMBER
Score ↑
Hal. ↓
CHAIR ↓
HalRate ↓
Ours
3.03
0.36
1.9
10.3
Blockwise
2.64
0.42
2.8
14.1
Rotation
2.47
0.46
2.6
12.5
Crop
2.69
0.42
2.1
11.3
Blackness
2.90
0.41
2.4
12.3
Table 3. Impact of rejected image construction strategy on mPEA-DPO. Bold indicates best.
Figure 6. Impact of data scale on the performance of mPEA-DPO, using LLaVA as the base model. We assess: (1) the overall score and hallucination rate on MMHalBench, and (2) CHAIR and hallucination on AMBER.
Model
CHAIRs
CHAIRi
Recall
OPA-DPO
13.3
4.3
43.29
DAMA
10.3
5.9
54.50
mPEA-DPO
4.3
3.2
52.30
Table 4. Comparison of hallucination and recall performance on Object HalBench. Bold denote the best performance.
Model
MMStar
AI2D
LLaVA-Bench
MMMU(val)
MMMU(test)
LLaVA-v1.5-7B
30.3
49.1
82.3
32.1
35.3
+mPEA-DPO
32.8
51.9
86.5
30.8
33.4
Table 5. The general capability evaluation results. Values in bold denote the best performance.
School of Computer Science and Technology, Tongji University, China · Shenzhen Loop Area Institute, China · Shanghai Research Institute for Intelligent Autonomous Systems, China