Multimodal discrete diffusion language models (dLLMs) generate responses through iterative mask prediction and offer a promising alternative to autoregressive large vision-language models. However, they still suffer from hallucination, producing outputs that are plausible under language priors but insufficiently grounded in the image. We argue that hallucination in multimodal dLLMs is not only a token prediction problem, but also a premature commitment problem: during iterative denoising, some masked positions are unmasked too early because their confidence is driven more by language priors than by visual evidence, and these early commitments can bias later generation. To address this issue, we propose Visual Gain Ordering (VIGOR), a training-free and plug-and-play inference strategy that reorders masked positions according to counterfactual visual evidence. At each denoising step, VIGOR compares the confidence of each candidate token under the original image-conditioned pass and an additional visual-ablation pass, and prioritizes positions whose predictions are more strongly supported by the image. Experiments on LLaDA-V and Lumina-DiMOO show that VIGOR consistently reduces hallucination and improves multimodal reasoning performance across benchmarks. Code is available at https://github.com/THU-BPM/Vigor.
Figures & tables
Figure 1: Comparison of hallucination mechanisms in autoregressive LVLMs, multimodal dLLMs, and Vigor .
Figure 2: Overview of Vigor in blockwise decoding. At each decoding step, the model runs image-conditioned and visual-ablation passes with shared parameters, computes a Vigor score for every masked position, and selects the top- k positions to unmask within the current block.
Dataset
Setting
Method
LLaDA-V
Lumina-DiMOO
Accuracy ↑
F1 ↑
Accuracy ↑
F1 ↑
COCO
Random
Baseline
83.7
85.1
75.1
67.7
VCD
84.5 ↑ 0.8
85.9 ↑ 0.8
76.0 ↑ 0.9
68.4 ↑ 0.7
ICT
85.1 ↑ 1.4
86.8 ↑ 1.7
76.7 ↑ 1.6
70.1 ↑ 2.4
Vigor
85.9 ↑ 2.2
87.6 ↑ 2.5
80.4 ↑ 5.3
79.3 ↑ 11.6
Popular
Baseline
71.9
76.8
74.1
66.8
Table 1: Main results on POPE. We compare Vigor against the vanilla Baseline , VCD and ICT. The values highlighted in green indicate the best metric under each setting. Arrows next to each value report the change vs. Baseline ( ↑ / ↓ = improvement on higher-/lower-is-better metrics).
Model
Method
CHAIR Len 64
CHAIR Len 512
AMBER G
AMBER D
C S↓
C I↓
C S↓
C I↓
CHAIR ↓
Cover ↑
Hal ↓
Cog ↓
Acc ↑
F1 ↑
LLaDA-V
Baseline
32.7
22.1
47.5
31.0
6.0
35.4
21.1
1.2
73.3
81.7
Vigor
24.8 ↓ 7.9
18.8 ↓ 3.3
42.4 ↓ 5.1
26.2 ↓ 4.8
5.0 ↓ 1.0
39.8 ↑ 4.4
16.9 ↓ 4.2
0.6 ↓ 0.6
77.1 ↑ 3.8
84.1 ↑ 2.4
Lumina-DiMOO
Baseline
33.3
24.5
49.1
30.7
20.1
47.3
67.5
9.5
67.4
76.8
Vigor
26.1 ↓ 7.2
14.4 ↓ 10.1
43.3 ↓ 5.8
26.0 ↓ 4.7
16.7 ↓ 3.4
49.8 ↑ 2.5
50.2 ↓ 17.3
3.8 ↓ 5.7
70.8 ↑ 3.4
78.0 ↑ 1.2
Table 2: Main results on CHAIR and AMBER. CHAIR reports C S /C I at two caption lengths (64, 512); AMBER includes a generative track ( G : CHAIR, Cover, Hal, Cog) and a discriminative track ( D : Acc, F1). The values highlighted in green indicate the best metric for each model.
Method
Random
Adversarial
Acc ↑
F1 ↑
Yes% →50
Acc ↑
F1 ↑
Yes% →50
Baseline
83.7
85.1
59.6
71.1
76.3
72.2
Diff-only
84.6 ↑ 0.9
85.8 ↑ 0.7
58.4 ↓ 1.2
73.0 ↑ 1.9
77.0 ↑ 0.7
67.4 ↓ 4.8
LogRatio-only
84.2 ↑ 0.5
85.5 ↑ 0.4
59.0 ↓ 0.6
66.9 ↓ 4.2
73.7 ↓ 2.6
75.8 ↑ 3.6
Vigor (full)
85.9 ↑ 2.2
87.6 ↑ 2.5
55.7 ↓ 3.9
74.7 ↑ 3.6
78.9 ↑ 2.6
67.3 ↓ 4.9
Table 3: Score decomposition ablation on LLaDA-V, COCO POPE. Yes% values closer to the balanced prior 50% indicate less object-bias. Best per column is highlighted in green .
Figure 3: Diagnostic analysis on LLaDA-V over the AMBER benchmark. (a) Mean commit step for high- and low-JSD positions under Baseline and Vigor . (b) Step- 0 JSD for positions advanced versus not advanced by Vigor . (c) Early-stage unstable commit rate on incorrect samples. (d) Candidate-token difference ratio across decoding steps.
Figure 4: Case study and error analysis.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
LLaDA-V
Lumina-DiMOO
Image-token placement
visual prefix before text tokens
interleaved per chat template
Visual-ablation pass
image embeddings replaced by [MASK]
image tokens replaced by [MASK]
Mask token
official [MASK] id (unchanged)
official [MASK] id (unchanged)
Block schedule
identical to baseline
identical to baseline
Ranking rule
si from Eq. 7
si from Eq. 7
Appendix
Table 4: Model-specific implementation of Vigor on the two backbones.
Method
len 32
len 64
len 128
Ratio
LLaDA-V
3644.6
7338.2
15552.8
–
+ Vigor
7417.3
14954.1
31684.9
×2.0
Lumina-DiMOO
6395.4
13430.5
27136.1
–
+ Vigor
13129.4
27505.8
55535.7
×2.0
Appendix
Table 5: Decoding time (ms) on A100 GPUs at generation lengths 32/64/128. The Ratio column shows Vigor vs. its baseline; the ratio cell is highlighted in green .
Metric
Baseline
Vigor
Delta
Peak alloc.
29.56
34.93
+5.36
Peak reserved
78.74
78.61
−0.13
Appendix
Table 6: Memory usage (GiB) on Lumina-DiMOO. Delta is reported as Vigor minus baseline and computed from unrounded measurements.
Figure 5: Appendix case study. Eight image–caption comparisons between baseline and Vigor decoding on Lumina-DiMOO (a–d) and LLaDA-V (f–h). Hallucinated object words are highlighted in red .
Nanjing University of Aeronautics and Astronautics · State Key Laboratory of Ocean Sensing, ZJU-Hangzhou Global Scientific and Technological Innovation Center, Zhejiang University, Hangzhou, 311215, China · Security Capability Center