Diffusion multimodal large language models (dMLLMs) have recently emerged as a new decoding paradigm for multimodal generation. Starting from a fully masked sequence, dMLLMs progressively decode the sequence by unmasking a subset of the remaining masked positions at each step. Since the selected tokens serve as the prediction context for subsequent steps, deciding which tokens to decode is crucial to the quality of the final output. The most common strategy prioritizes tokens based on a certainty measure that tends to favor tokens frequently observed in the training data. Recent approaches instead order tokens according to their influence on subsequent predictions, but do not explicitly account for the input image. We propose the Visual Information-Guided Sampler (VIG-Sampler), which prioritizes tokens based on their attention to image tokens. We further impose a constraint that penalizes candidate tokens whose image-attention distributions are similar to those of previously selected tokens, thereby increasing the information gain of the decoded subset. Extensive experiments on 7 captioning and VQA benchmarks with 3 open-source dMLLMs demonstrate the effectiveness of VIG-Sampler, which outperforms the Info-Gain Sampler by an average of 19.3 CIDEr points across the captioning benchmarks and surpasses it on COCO Caption while using only half as many decoding steps.
Figures & tables
Figure 1: Comparison of conventional sampling and VIG-Sampler. The conventional method may prioritize the visually unsupported token “frisbee” , causing subsequent predictions to drift from the image. In contrast, VIG-Sampler uses image attention to prioritize the visually grounded token “sits” , leading to a response consistent with the visual input.
Figure 2: Motivating observations on how image attention relates to visual grounding and information gain during dMLLM decoding. Top: distributions of visually grounded and non-grounded tokens across image-attention-mass ranges, together with the mean information gain. Bottom: shared information ratio (SIR) versus mean pairwise image-attention similarity for selected token sets of size k∈{2,4,8} .
Figure 3: Overview of the proposed VIG-Sampler. At each decoding step, image-attention mass is used to adjust the confidence score of each masked position, prioritizing tokens that are both visually grounded and informative. The resulting scores and pairwise image-attention similarities are then used to select the token set expected to yield higher information gain.
k
Sampler
COCO Cap.
Flickr30K
NoCaps
Avg.
DetailCaps
TextVQA
DocVQA
ChartQA
Avg.
CIDEr
CIDEr
CIDEr
CIDEr
CAPTURE
Acc.
ANLS
Relaxed Acc.
Macro Avg.
2
Confidence
100.1
69.7
88.4
86.1
56.2
55.2
60.1
56.0
57.1
Entropy
95.4
68.2
83.9
82.5
56.5
54.6
59.8
56.0
56.8
Margin
105.8
71.7
88.7
88.7
56.9
55.0
61.6
56.6
57.7
MPD-PAC
95.1
70.1
87.2
84.1
56.9
56.0
62.8
56.2
58.3
PC-Sampler
101.7
69.8
88.9
86.8
57.1
54.3
58.9
55.0
56.1
Table 1: Experimental results of LaViDa with parallel decoding budgets k∈{2,4,8} . The first and last Avg. columns report mean CIDEr and VQA scores, respectively. The best scores are shown in bold , and the second-best scores are underlined .
k
Sampler
COCO Cap.
Flickr30K
NoCaps
Avg.
DetailCaps
TextVQA
DocVQA
ChartQA
Avg.
CIDEr
CIDEr
CIDEr
CIDEr
CAPTURE
Acc.
ANLS
Relaxed Acc.
Macro Avg.
2
Confidence
87.4
57.7
86.5
77.2
50.1
12.2
9.3
11.8
11.1
Entropy
82.5
55.8
83.3
73.9
50.5
12.0
8.8
11.8
10.9
Margin
88.0
58.7
88.0
78.2
50.5
12.2
9.5
11.6
11.1
MPD-PAC
86.1
56.7
81.2
74.7
50.0
12.4
10.1
11.6
11.4
PC-Sampler
87.1
57.4
80.3
74.9
49.3
12.4
8.6
11.8
10.9
Table 2: Experimental results of MMaDA with parallel decoding budgets k∈{2,4,8} . The first and last Avg. columns report mean CIDEr and VQA scores, respectively. The best scores are shown in bold , and the second-best scores are underlined .
Figure 4: Cumulative information gain across decoding steps for k∈{2,4,8}
Figure 5: Qualitative comparison on COCO Caption using LaViDa at k=2 . S# denotes the decoding step, and red and blue indicate incorrectly and correctly committed tokens, respectively.
γ\λ
0.0
1.0
2.0
3.0
4.0
0.0
100.2
100.9
102.1
102.9
101.7
0.5
101.9
105.2
106.9
106.5
105.2
1.0
101.5
103.8
105.4
106.2 †
105.5
1.5
101.0
103.5
106.0
107.7
107.8
2.0
103.0
107.0
106.2
106.1
107.5
Table 3: Hyperparameter sensitivity on COCO Caption using LaViDa. † denotes the default setting.
N
Sampler
COCO Cap.
TextVQA
16
Info-Gain
81.6
48.8
VIG-Sampler
88.4
55.0
32
Info-Gain
94.8
50.2
VIG-Sampler
105.1
56.8
48
Info-Gain
97.4
48.6
VIG-Sampler
102.5
57.4
Table 4: Robustness to N using LaViDa at k=4 .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
k
Sampler
COCO Cap.
Flickr30K
NoCaps
Avg.
DetailCaps
TextVQA
DocVQA
ChartQA
Avg.
CIDEr
CIDEr
CIDEr
CIDEr
CAPTURE
Acc.
ANLS
Relaxed Acc.
Macro Avg.
2
Confidence
92.8
83.9
91.6
89.4
59.9
67.3
81.1
76.2
74.9
Entropy
91.0
77.8
89.4
86.1
59.0
63.9
82.0
76.6
74.2
Margin
94.5
82.5
91.8
89.6
61.1
67.3
82.0
77.6
75.6
MPD-PAC
92.4
83.6
91.2
89.1
60.5
65.0
81.4
75.8
74.1
PC-Sampler
92.1
81.9
90.8
88.3
59.8
66.5
82.8
76.4
75.2
Appendix
Table 5: Experimental results of LLaDA-V with parallel decoding budgets k∈{2,4,8} . The first and last Avg. columns report mean CIDEr and VQA scores, respectively. The best scores are shown in bold , and the second-best scores are underlined .
N
Sampler
COCO Cap.
TextVQA
16
Info-Gain
87.2
58.0
VIG-Sampler
92.4
57.5
32
Info-Gain
106.0
57.4
VIG-Sampler
106.2
58.7
48
Info-Gain
106.8
56.5
VIG-Sampler
104.2
59.4
Appendix
Table 6: Robustness to N using LaViDa at k=2 .
Sampler
Peak mem. (GB)
Wall-clock time (s)
k=2
k=4
Confidence
17.9
1.71
1.39
Info-Gain
17.9
3.25
2.21
VIG-Sampler
17.9
1.85
1.44
Appendix
Table 7: Inference cost of each sampler.
Figure 6: Qualitative examples on COCO Caption using LaViDa at k=2 . The top two rows illustrate decoding failures by conventional methods without explicit visual guidance, in which incorrect tokens are produced during generation. The bottom two rows present detailed captions that are visually well aligned with the images from a human perspective, despite receiving relatively low CIDEr scores. S# denotes the decoding step, while red and blue indicate incorrectly and correctly committed tokens, respectively.
Figure 7: Qualitative analysis of token selection and image-attention patterns. Confidence-based selection tends to choose uninformative tokens with diffuse attention, whereas VIG-Sampler selects answer-relevant tokens grounded in distinct image regions. Without the penalty term, VIG-Sampler selects tokens with similar image-attention patterns, providing little additional information beyond one another.
Diffusion language model decoding generates text through iterative parallel refinement, balancing fewer denoising steps with accuracy, coherence, and control.
Diffusion-based multimodal large language models (dMLLMs) decode by iteratively predicting tokens at multiple masked positions in parallel. This turns each decoding step into a position-selection problem: the model must choose not only which predictions are reliable in isolation, but also which positions should be committed together as context for later decoding steps. Existing confidence-based decoding ranks masked positions independently and commits the top-K positions, largely ignoring whether the committed tokens provide complementary visual grounding. We identify a step-level limitation of this strategy in multimodal settings: high-confidence tokens selected in the same step can rely on overlapping visual grounding, introducing visual redundancy among the committed tokens and leaving less complementary visual grounding available for later decoding. To quantify this effect, we introduce the Visual Redundancy Index (VRI), which measures visual grounding overlap among tokens committed in parallel. To control this redundancy during decoding, we propose Visual-Redundancy-Controlled Decoding (VRCD), a training-free inference-time decoding method that uses token-to-image attention to prioritize visually complementary positions. Across diverse multimodal benchmarks, VRCD reduces visual redundancy and remaining-position entropy with modest runtime overhead. In longer decoding experiments, it also achieves relative accuracy gains of up to 18.8% on M^3CoT and 6.9% on MMBench over confidence-based decoding. Code is available at https://github.com/infiniteYuanyl/VRCD.
Decoding quality in diffusion multimodal language models (dMLLMs) depends heavily on the order in which masked tokens are committed. Existing confidence-based strategies prioritize locally easy tokens, but confidence does not necessarily reflect contextual usefulness. As a result, structurally easy tokens such as punctuation may be committed before informative semantic anchors, weakening context propagation and increasing error accumulation. We propose Information-Guided Frontier Decoding (IGFD), a training-free decoding strategy that ranks candidates using token confidence, neighborhood uncertainty, and structural commitment risk. IGFD encourages early commitment of reliable semantic anchors while delaying fragile structural tokens, improving contextual support during decoding. A dynamic candidate frontier further constrains token selection to locally expandable regions under the same decoding budget. The method requires no additional training, auxiliary models, or extra forward passes. Experiments across multimodal understanding, reasoning, grounding, and hallucination benchmarks show that IGFD consistently outperforms existing decoding strategies across the majority of benchmarks and diffusion MLLM backbones under identical decoding budgets.
Xingyou Fang, Jingxing Zhong, Xiaosong Yuan +1
Fuzhou University · Jilin University · Shanghai Jiao Tong University
Diffusion multimodal large language models (dMLLMs) frequently produce long-form outputs marred by semantic drift and repetition, with quality generally degrading as output length increases. We identify two structural deficiencies in existing decoding methods as primary drivers of these failures: confidence-based scoring ignores decoded-neighbor support, and block partitioning prevents access to high-readiness semantic anchors, together causing tokens to be committed before their local context is sufficiently established. We propose \ours{} (Context-Aware Cluster Decoding), a training-free decoding method that scores each masked position by a multiplicative composite of softmax confidence and neighbor proximity, promoting contextually ready tokens above isolated candidates while suppressing low-confidence positional noise, operating block-free to keep high-readiness anchors globally accessible. \ours{} further applies architecture-aware calibration to handle confidence heterogeneity induced by diverse visual integration strategies. Experiments on three dMLLMs across four benchmarks demonstrate consistent quality gains and hallucination reduction over Original, with larger gains in several longer generation settings, highlighting the importance of neighbor support and visual integration strategy for future dMLLM decoding method design. Our code is openly available at https://github.com/zhaoyk-sysu/CACD-dMLLM.
Yikai Zhao, Qiyan Zhao, Jiaquan Zhang +3
Sun Yat-sen University · Shanghai Jiao Tong University · University of Electronic Science and Technology of China +2