GEM-MM: Reducing VLM Hallucination with Entropy-Guided Preference Alignment
Authors: Yuanshan Chua, Xuejiao Zhao
Organizations: College of Computing and Data Science (CCDS), Nanyang Technological University (NTU), Singapore · College of Computer Science, Sichuan University, China
Vision-language models (VLMs) are routinely aligned with offline pairwise RLHF objectives, but multimodal preference data is scarce and treats every response token as equally informative. Under a small preference budget, pair construction becomes nearly irrelevant: standard DPO-style, image-conditional, and self-improvement objectives all perform within 1.1 points on MM-RLHF metrics and 0.5 points on hallucination rates, matching the SFT baseline. We call this entropy-blind preference optimization: the learning signal ignores policy uncertainty, which in chain-of-thought VLMs concentrates on specific fork tokens where visual claims are made. We introduce GEM-MM, extending entropy-guided preference modeling to VLMs. It samples on-policy candidates, rewards high-entropy reasoning forks, penalizes uncertain final answers, and updates the model via group-normalized policy gradients. On a 3000-prompt MM-RLHF budget with Qwen3-VL-4B-Instruct, GEM-MM achieves the best entropy-depth preference score (61.7% vs. 57.3-58.4% baselines) and implicit-reward accuracy (41.4%). Crucially, it reduces AMBER hallucination from 23.4-24.0 to 20.6 and CHAIR from 5.0-5.1 to 4.6 at matched caption coverage. It also ranks best on all three HallusionBench granularities, with a conservative operating point raising the near-chosen rate to 62.4%. An off-the-shelf multimodal reward model ranks GEM-MM first, and two such models independently confirm our main ablations (p < 0.001). We report confidence intervals, paired significance tests, metric-reward functional overlaps, and document a collapse mode when exceeding the fair budget. Code and evaluation harness: https://github.com/SNOWTEAM2023/GEM-MM
Figures & tables
Depth
Implicit
A/B
AMBER
Objective
Pref %
rew. %
%
Hal ↓
SFT only (no pairs)
57.9
40.9
66.4
24.0
MM-DPO (quality-weighted)
58.1
41.0
66.4
23.4
mDPO (image-conditional)
58.4
40.2
66.1
23.4
SIMA (self-generated)
57.3
40.8
66.0
23.9
spread (3 obj.)
1.1
0.8
0.4
0.5
Table 1: The saturation that motivates GEM-MM. Identical initialization, prompts (3000 MM-RLHF IDs), template, and token budget; only the alignment objective differs. Spread = max − min over the three preference-tuned rows. All three objectives land inside one measurement interval of each other.
Figure 1: Overview of the GEM pipeline instantiated by GEM-MM for vision-language models (adapted from Zhao et al. (2026a) ). Given a multimodal query x=(xI,xQ) (text-only q in the original figure) drawn from a preference pool {a−,a+} , the Reflective Inference Engine πθ elaborates k chain-of-thought candidates; Cognitive Filtering computes per-token entropy and an entropy-guided score that rewards fork tokens while penalizing uncertain final answers; SEGA forms a contextual group advantage and applies a full-parameter gradient update. The cognitive feedback loop returns only to the inference engine (updated policy), which then re-samples candidates on the next cycle and does not re-enter the entropy-scoring block directly.
Method
Depth Pref %
Implicit rew. %
A/B %
SFT
57.9
40.9
66.4
MM-DPO
58.1
41.0
66.4
mDPO
58.4
40.2
66.1
SIMA
57.3
40.8
66.0
GEM-MM
61.7
41.4
68.3
Table 2: MM-RLHF held-out preference evaluation ( n=1000 pairs, IDs 3001–4000). Depth Pref is a diagnostic that shares functional form with the GEM-MM training reward (Section 5 ); implicit reward and A/B are objective-agnostic. Wilson 95% CIs for implicit reward are in the text. Bold = best; underline = second and within 1 point of best.
Method
Mean rew. ↑
Bootstrap 95% CI
Δ ( p )
SFT
13.06
[11.93,14.23]
+0.76 (0.11)
MM-DPO
12.85
[11.74,13.96]
+0.97 ( 0.04 )
mDPO
13.27
[12.16,14.43]
+0.55 (0.24)
SIMA
12.89
[11.76,14.07]
+0.93 ( 0.04 )
GEM-MM
13.82
[12.69,14.98]
—
Table 3: Held-out generations ( n=1000 , IDs 3001–4000) scored by Skywork-VL Reward ( Wang et al., 2025b ) . Because raw reward scores are uncalibrated, we focus on relative comparisons. Δ denotes the paired mean difference ( GEM-MM minus baseline ) alongside two-sided paired t -test p -values ( bold indicates p<0.05 ). A second reward model is reported in Appendix D .
Method
CHAIR ↓
Hal ↓
Cover ↑
Cog ↓
SFT
5.1
24.0
57.0
0.9
MM-DPO
5.0
23.4
56.7
1.0
mDPO
5.0
23.4
56.5
1.0
SIMA
5.1
23.9
56.4
1.0
GEM-MM
4.6
20.6
56.5
0.9
Table 4: AMBER ( Wang et al., 2023 ) generative split. Cover is the control: lower hallucination is only meaningful at comparable coverage. Bold = best; underline = second and within 1 point of best.
Method
aAcc ↑
qAcc ↑
fAcc ↑
SFT
68.66
43.96
43.35
MM-DPO
68.66
42.86
42.20
mDPO
69.30
44.40
43.64
SIMA
68.56
43.74
43.93
GEM-MM
69.82
46.59
45.95
Table 5: HallusionBench ( Guan et al., 2024 ) . aAcc/qAcc/fAcc = all-sample, question-group, figure-level accuracy. GEM-MM is best on all three granularities.
Depth
Impl.
A/B
CHAIR
Hal
Sky.
Variant
Pref %
%
%
↓
↓
↑
Full GEM-MM
61.7
41.4
68.3
4.6
20.6
13.8
No fork term
60.8
39.9
67.0
5.4
24.5
13.7
Fork term only
57.8
40.9
66.9
4.8
21.7
13.1
No group adv.
55.1
28.5
64.8
5.1
23.2
−3.8
Exact-match rew.
55.3
32.9
64.5
4.9
21.2
4.3
Table 6: GEM-MM ablations on the locked 3000-prompt budget. Every row shares data, initialization, template, and optimizer; only the reward or the advantage estimator changes. Bold = best; underline = second and within 1 point of best.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
POPE
AMBER
MMHal
MMHal
Refusal
Method
F1
disc. F1
avg ↑
hal ↓
acc ↑
Base
100.0 ‡
46.1
4.45
0.29
40.0
SFT
83.1
79.8
4.27
0.33
90.3
MM-DPO
83.1
80.6
4.49
0.27
90.5
mDPO
82.8
80.2
4.34
0.30
90.5
SIMA
82.8
80.0
4.33
0.29
90.1
Appendix
Table 7: Full benchmark report for the main systems. Reported for transparency; main-paper claims use only AMBER (Table 4 ) and HallusionBench (Table 5 ).
Method
Mean tokens
Format %
ANLS
Base
113.3
99.0
0.816
SFT
69.4
72.4
0.595
MM-DPO
70.0
72.7
0.606
mDPO
69.7
72.1
0.605
SIMA
69.6
72.6
0.597
GEM-MM (primary)
69.3
66.5
0.638
Appendix
Table 8: Generation statistics on the held-out prompts. Format compliance is the fraction of responses that emit the required answer marker. Most of the compliance regression relative to the base model is introduced by the supervised stage that initializes all preference methods.
Method / variant
Mean rew. ↑
Δ ( p )
SFT
1.026
−0.02 (0.60)
MM-DPO
1.039
−0.03 (0.36)
mDPO
1.041
−0.03 (0.32)
SIMA
1.003
+0.01 (0.88)
GEM-MM
1.008
—
No fork term
1.191
−0.18 ( < 0.001)
Appendix
Table 9: The Section 6 systems and the Section 8 ablations re-scored by IXC-2.5-Reward [ Zang et al., 2025 ] ( n=1000 paired items). Δ is the paired mean difference GEM-MM minus row ; positive means GEM-MM scores higher. This scale is unrelated to the Skywork scale of Table 3 .
Setting
Depth %
A/B %
Beat-rej %
Near %
λ2.5,τ0.8
53.9
69.2
64.3
62.4
λ2.5,ρ0.05
52.1
69.0
62.6
60.8
λ2.5,T96,η2e−6
51.9
68.8
62.8
62.2
λ3.0,ρ0.05
54.1
70.0
64.1
59.4
λ3.0,τ1.0
45.1
69.6
59.8
66.8
λ3.0,T96
34.3
69.4
28.5
45.8
Appendix
Table 10: Phase-4 sweep, all full fine-tunes at 3000 prompts.
#
Parent checkpoint
Knob changes (+1500 cycles)
Depth
Beat
Near
Note
1
Mid-budget @ 1500 cycles
λ=2.5 , ρ=0.05 , τ=0.8
54.5
92.3
94.0
collapsed
2
Mid-budget @ 1500 cycles
λ=1.5 , lr =3×10−6 , τ=0.7
50.1
62.6
60.9
—
3
Mid-budget @ 1500 cycles
k=3 , Tmax=128 , λ=2.0
52.8
75.3
76.6
—
4
Primary GEM-MM @ 3000 cycles
λ=2.5 , τ=0.8 , ρ=0.05
62.0
44.0
45.8
—
5
Primary GEM-MM @ 3000 cycles
λ=3.0 , ρ=0.05 , lr =3×10−6
61.1
52.2
47.4
—
Appendix
Table 11: Continue-training stress tests (+1500 cycles on the same prompts; total budget 4500). Each row starts from a named parent and changes the listed knobs. Depth / Beat / Near are held-out preference and generation-side proxies. Excluded from main tables. Beat/Near above ∼75% co-occur with format collapse.