GEM-MM: Reducing VLM Hallucination with Entropy-Guided Preference Alignment
Authors: Yuanshan Chua, Xuejiao Zhao
Organizations: College of Computing and Data Science (CCDS), Nanyang Technological University (NTU), Singapore · College of Computer Science, Sichuan University, China
Vision-language models (VLMs) are routinely aligned with offline pairwise RLHF objectives, but multimodal preference data is scarce and treats every response token as equally informative. Under a small preference budget, pair construction becomes nearly irrelevant: standard DPO-style, image-conditional, and self-improvement objectives all perform within 1.1 points on MM-RLHF metrics and 0.5 points on hallucination rates, matching the SFT baseline. We call this entropy-blind preference optimization: the learning signal ignores policy uncertainty, which in chain-of-thought VLMs concentrates on specific fork tokens where visual claims are made. We introduce GEM-MM, extending entropy-guided preference modeling to VLMs. It samples on-policy candidates, rewards high-entropy reasoning forks, penalizes uncertain final answers, and updates the model via group-normalized policy gradients. On a 3000-prompt MM-RLHF budget with Qwen3-VL-4B-Instruct, GEM-MM achieves the best entropy-depth preference score (61.7% vs. 57.3-58.4% baselines) and implicit-reward accuracy (41.4%). Crucially, it reduces AMBER hallucination from 23.4-24.0 to 20.6 and CHAIR from 5.0-5.1 to 4.6 at matched caption coverage. It also ranks best on all three HallusionBench granularities, with a conservative operating point raising the near-chosen rate to 62.4%. An off-the-shelf multimodal reward model ranks GEM-MM first, and two such models independently confirm our main ablations (p < 0.001). We report confidence intervals, paired significance tests, metric-reward functional overlaps, and document a collapse mode when exceeding the fair budget. Code and evaluation harness: https://github.com/SNOWTEAM2023/GEM-MM
Figures & tables
Depth
Implicit
A/B
AMBER
Objective
Pref %
rew. %
%
Hal ↓
SFT only (no pairs)
57.9
40.9
66.4
24.0
MM-DPO (quality-weighted)
58.1
41.0
66.4
23.4
mDPO (image-conditional)
58.4
40.2
66.1
23.4
SIMA (self-generated)
57.3
40.8
66.0
23.9
spread (3 obj.)
1.1
0.8
0.4
0.5
Table 1: The saturation that motivates GEM-MM. Identical initialization, prompts (3000 MM-RLHF IDs), template, and token budget; only the alignment objective differs. Spread = max − min over the three preference-tuned rows. All three objectives land inside one measurement interval of each other.
Figure 1: Overview of the GEM pipeline instantiated by GEM-MM for vision-language models (adapted from Zhao et al. (2026a) ). Given a multimodal query x=(xI,xQ) (text-only q in the original figure) drawn from a preference pool {a−,a+} , the Reflective Inference Engine πθ elaborates k chain-of-thought candidates; Cognitive Filtering computes per-token entropy and an entropy-guided score that rewards fork tokens while penalizing uncertain final answers; SEGA forms a contextual group advantage and applies a full-parameter gradient update. The cognitive feedback loop returns only to the inference engine (updated policy), which then re-samples candidates on the next cycle and does not re-enter the entropy-scoring block directly.
Method
Depth Pref %
Implicit rew. %
A/B %
SFT
57.9
40.9
66.4
MM-DPO
58.1
41.0
66.4
mDPO
58.4
40.2
66.1
SIMA
57.3
40.8
66.0
GEM-MM
61.7
41.4
68.3
Table 2: MM-RLHF held-out preference evaluation ( n=1000 pairs, IDs 3001–4000). Depth Pref is a diagnostic that shares functional form with the GEM-MM training reward (Section 5 ); implicit reward and A/B are objective-agnostic. Wilson 95% CIs for implicit reward are in the text. Bold = best; underline = second and within 1 point of best.
Method
Mean rew. ↑
Bootstrap 95% CI
Δ ( p )
SFT
13.06
[11.93,14.23]
+0.76 (0.11)
MM-DPO
12.85
[11.74,13.96]
+0.97 ( 0.04 )
mDPO
13.27
[12.16,14.43]
+0.55 (0.24)
SIMA
12.89
[11.76,14.07]
+0.93 ( 0.04 )
GEM-MM
13.82
[12.69,14.98]
—
Table 3: Held-out generations ( n=1000 , IDs 3001–4000) scored by Skywork-VL Reward ( Wang et al., 2025b ) . Because raw reward scores are uncalibrated, we focus on relative comparisons. Δ denotes the paired mean difference ( GEM-MM minus baseline ) alongside two-sided paired t -test p -values ( bold indicates p<0.05 ). A second reward model is reported in Appendix D .
Method
CHAIR ↓
Hal ↓
Cover ↑
Cog ↓
SFT
5.1
24.0
57.0
0.9
MM-DPO
5.0
23.4
56.7
1.0
mDPO
5.0
23.4
56.5
1.0
SIMA
5.1
23.9
56.4
1.0
GEM-MM
4.6
20.6
56.5
0.9
Table 4: AMBER ( Wang et al., 2023 ) generative split. Cover is the control: lower hallucination is only meaningful at comparable coverage. Bold = best; underline = second and within 1 point of best.
Method
aAcc ↑
qAcc ↑
fAcc ↑
SFT
68.66
43.96
43.35
MM-DPO
68.66
42.86
42.20
mDPO
69.30
44.40
43.64
SIMA
68.56
43.74
43.93
GEM-MM
69.82
46.59
45.95
Table 5: HallusionBench ( Guan et al., 2024 ) . aAcc/qAcc/fAcc = all-sample, question-group, figure-level accuracy. GEM-MM is best on all three granularities.
Depth
Impl.
A/B
CHAIR
Hal
Sky.
Variant
Pref %
%
%
↓
↓
↑
Full GEM-MM
61.7
41.4
68.3
4.6
20.6
13.8
No fork term
60.8
39.9
67.0
5.4
24.5
13.7
Fork term only
57.8
40.9
66.9
4.8
21.7
13.1
No group adv.
55.1
28.5
64.8
5.1
23.2
−3.8
Exact-match rew.
55.3
32.9
64.5
4.9
21.2
4.3
Table 6: GEM-MM ablations on the locked 3000-prompt budget. Every row shares data, initialization, template, and optimizer; only the reward or the advantage estimator changes. Bold = best; underline = second and within 1 point of best.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
POPE
AMBER
MMHal
MMHal
Refusal
Method
F1
disc. F1
avg ↑
hal ↓
acc ↑
Base
100.0 ‡
46.1
4.45
0.29
40.0
SFT
83.1
79.8
4.27
0.33
90.3
MM-DPO
83.1
80.6
4.49
0.27
90.5
mDPO
82.8
80.2
4.34
0.30
90.5
SIMA
82.8
80.0
4.33
0.29
90.1
Appendix
Table 7: Full benchmark report for the main systems. Reported for transparency; main-paper claims use only AMBER (Table 4 ) and HallusionBench (Table 5 ).
Method
Mean tokens
Format %
ANLS
Base
113.3
99.0
0.816
SFT
69.4
72.4
0.595
MM-DPO
70.0
72.7
0.606
mDPO
69.7
72.1
0.605
SIMA
69.6
72.6
0.597
GEM-MM (primary)
69.3
66.5
0.638
Appendix
Table 8: Generation statistics on the held-out prompts. Format compliance is the fraction of responses that emit the required answer marker. Most of the compliance regression relative to the base model is introduced by the supervised stage that initializes all preference methods.
Method / variant
Mean rew. ↑
Δ ( p )
SFT
1.026
−0.02 (0.60)
MM-DPO
1.039
−0.03 (0.36)
mDPO
1.041
−0.03 (0.32)
SIMA
1.003
+0.01 (0.88)
GEM-MM
1.008
—
No fork term
1.191
−0.18 ( < 0.001)
Appendix
Table 9: The Section 6 systems and the Section 8 ablations re-scored by IXC-2.5-Reward [ Zang et al., 2025 ] ( n=1000 paired items). Δ is the paired mean difference GEM-MM minus row ; positive means GEM-MM scores higher. This scale is unrelated to the Skywork scale of Table 3 .
Setting
Depth %
A/B %
Beat-rej %
Near %
λ2.5,τ0.8
53.9
69.2
64.3
62.4
λ2.5,ρ0.05
52.1
69.0
62.6
60.8
λ2.5,T96,η2e−6
51.9
68.8
62.8
62.2
λ3.0,ρ0.05
54.1
70.0
64.1
59.4
λ3.0,τ1.0
45.1
69.6
59.8
66.8
λ3.0,T96
34.3
69.4
28.5
45.8
Appendix
Table 10: Phase-4 sweep, all full fine-tunes at 3000 prompts.
#
Parent checkpoint
Knob changes (+1500 cycles)
Depth
Beat
Near
Note
1
Mid-budget @ 1500 cycles
λ=2.5 , ρ=0.05 , τ=0.8
54.5
92.3
94.0
collapsed
2
Mid-budget @ 1500 cycles
λ=1.5 , lr =3×10−6 , τ=0.7
50.1
62.6
60.9
—
3
Mid-budget @ 1500 cycles
k=3 , Tmax=128 , λ=2.0
52.8
75.3
76.6
—
4
Primary GEM-MM @ 3000 cycles
λ=2.5 , τ=0.8 , ρ=0.05
62.0
44.0
45.8
—
5
Primary GEM-MM @ 3000 cycles
λ=3.0 , ρ=0.05 , lr =3×10−6
61.1
52.2
47.4
—
Appendix
Table 11: Continue-training stress tests (+1500 cycles on the same prompts; total budget 4500). Each row starts from a named parent and changes the listed knobs. Depth / Beat / Near are held-out preference and generation-side proxies. Excluded from main tables. Beat/Near above ∼75% co-occur with format collapse.
Hallucination remains a fundamental challenge in vision-language models (VLMs), where autoregressive generation may produce linguistically plausible yet physically inconsistent or visually ungrounded responses due to likelihood maximization under joint probabilistic modeling. We propose a stage-wise preference optimization framework for hallucination reduction through targeted multimodal data construction. Rather than directly optimizing on generic instruction-following data, our approach progressively constructs hallucination-focused preference pairs near known failure boundaries. The framework emphasizes ambiguous spatial orientation, object relationships, OCR uncertainty, and adversarial false-premise training. Hallucinated negatives are generated through minimally perturbed yet visually inconsistent alternatives, enabling Direct Preference Optimization (DPO) to better separate grounded reasoning from plausible hallucination. Experiments on open-source benchmarks and real-world multimodal evaluation scenarios demonstrate improved grounding consistency, reduced hallucination, and more informative grounded responses. Cross-model qualitative evaluation further shows that the proposed multimodal LLM DPO framework produces more visually grounded responses than several frontier proprietary VLMs, such as in ambiguous spatial reasoning and adversarial false-premise settings. The results suggest that hallucination may arise not only from limited model capacity, but also from inherent tendencies of autoregressive probabilistic generation to favor linguistically plausible continuations under weak visual grounding. Future work may explore physical consistency modeling, uncertainty-aware multimodal reasoning, and architectural alternatives beyond standard autoregressive decoding.
Vision-Language Models (VLMs) have shown strong performance in visual understanding, yet they still suffer from hallucinations, generating content that is not grounded in the image. Preference alignment is a promising approach to improve visual faithfulness, but its success depends heavily on how preference pairs are constructed. Existing methods exhibit two key limitations; (a) intervention-based methods often introduce significant deviation from the policy distribution, and (b) sampling-based methods often underuse visual information during the construction. In this paper, we propose ViPSy (Vision-driven Preference Synthesis), a framework for constructing preference data that are both policy-aligned and visually grounded. Our framework consists of two stages; in the first stage, ViPSy derives a visual cue from recurring object-level content across semantically aligned image variants, so preference construction can rely on visual information rather than language priors. In the second stage, ViPSy conditions the policy's own rollouts on this cue, allowing candidates to be guided by visually grounded content while staying close to the policy's response distribution. The resulting candidates remain close to the policy's response distribution while better leveraging visual information from the image. Experiments show that the resulting VLM, preference-aligned with ViPSy-constructed preference pairs, achieves a new state-of-the-art in hallucination mitigation. Compared with the previous state-of-the-art method, it reduces hallucination rates on AMBER and Object HalBench by 35.7% and 24.5%, respectively. The resulting model further improves on general visual grounding benchmarks, e.g., MMStar, MMVP, and CV-Bench, while also yielding gains in semantic segmentation and ImageNet linear probing, underscoring the effectiveness of our framework in enhancing the model's visual capabilities.
Multimodal Large Language Models (MLLMs) are prone to hallucination as their generation preferences are insufficiently calibrated to visual evidence, causing them to fall back on linguistic priors, rather than faithful grounding. In this work, we start from an empirical observation: when query-relevant visual evidence is explicitly strengthened using the model's own attention, generation becomes more accurate, suggesting that many failures do not arise solely from missing perception, but from an insufficient tendency to trust the evidence the model has already attended to. Motivated by this finding, we propose Oriented Pickup Preference Optimization (OPPO), an evidence-aware alignment objective that learns preferences over the strength of visual evidence, rather than only response quality. Concretely, OPPO contrasts the same faithful response under stronger, anchored, weaker-evidence views, turning naive visual preference into ordered visual-evidence alignment. We further combine this objective with fine-grained span-level and token-level regularization to stabilize the training. Besides, we provide a theoretical analysis showing that ordered evidence margins induce a positive lower bound on local visual sensitivity. Extensive evaluations across hallucination and general-purpose benchmarks demonstrate that OPPO consistently outperforms baseline methods.