Diffusion multimodal large language models (dMLLMs) have recently emerged as a new decoding paradigm for multimodal generation. Starting from a fully masked sequence, dMLLMs progressively decode the sequence by unmasking a subset of the remaining masked positions at each step. Since the selected tokens serve as the prediction context for subsequent steps, deciding which tokens to decode is crucial to the quality of the final output. The most common strategy prioritizes tokens based on a certainty measure that tends to favor tokens frequently observed in the training data. Recent approaches instead order tokens according to their influence on subsequent predictions, but do not explicitly account for the input image. We propose the Visual Information-Guided Sampler (VIG-Sampler), which prioritizes tokens based on their attention to image tokens. We further impose a constraint that penalizes candidate tokens whose image-attention distributions are similar to those of previously selected tokens, thereby increasing the information gain of the decoded subset. Extensive experiments on 7 captioning and VQA benchmarks with 3 open-source dMLLMs demonstrate the effectiveness of VIG-Sampler, which outperforms the Info-Gain Sampler by an average of 19.3 CIDEr points across the captioning benchmarks and surpasses it on COCO Caption while using only half as many decoding steps.
Figures & tables
Figure 1: Comparison of conventional sampling and VIG-Sampler. The conventional method may prioritize the visually unsupported token “frisbee” , causing subsequent predictions to drift from the image. In contrast, VIG-Sampler uses image attention to prioritize the visually grounded token “sits” , leading to a response consistent with the visual input.
Figure 2: Motivating observations on how image attention relates to visual grounding and information gain during dMLLM decoding. Top: distributions of visually grounded and non-grounded tokens across image-attention-mass ranges, together with the mean information gain. Bottom: shared information ratio (SIR) versus mean pairwise image-attention similarity for selected token sets of size k∈{2,4,8} .
Figure 3: Overview of the proposed VIG-Sampler. At each decoding step, image-attention mass is used to adjust the confidence score of each masked position, prioritizing tokens that are both visually grounded and informative. The resulting scores and pairwise image-attention similarities are then used to select the token set expected to yield higher information gain.
k
Sampler
COCO Cap.
Flickr30K
NoCaps
Avg.
DetailCaps
TextVQA
DocVQA
ChartQA
Avg.
CIDEr
CIDEr
CIDEr
CIDEr
CAPTURE
Acc.
ANLS
Relaxed Acc.
Macro Avg.
2
Confidence
100.1
69.7
88.4
86.1
56.2
55.2
60.1
56.0
57.1
Entropy
95.4
68.2
83.9
82.5
56.5
54.6
59.8
56.0
56.8
Margin
105.8
71.7
88.7
88.7
56.9
55.0
61.6
56.6
57.7
MPD-PAC
95.1
70.1
87.2
84.1
56.9
56.0
62.8
56.2
58.3
PC-Sampler
101.7
69.8
88.9
86.8
57.1
54.3
58.9
55.0
56.1
Table 1: Experimental results of LaViDa with parallel decoding budgets k∈{2,4,8} . The first and last Avg. columns report mean CIDEr and VQA scores, respectively. The best scores are shown in bold , and the second-best scores are underlined .
k
Sampler
COCO Cap.
Flickr30K
NoCaps
Avg.
DetailCaps
TextVQA
DocVQA
ChartQA
Avg.
CIDEr
CIDEr
CIDEr
CIDEr
CAPTURE
Acc.
ANLS
Relaxed Acc.
Macro Avg.
2
Confidence
87.4
57.7
86.5
77.2
50.1
12.2
9.3
11.8
11.1
Entropy
82.5
55.8
83.3
73.9
50.5
12.0
8.8
11.8
10.9
Margin
88.0
58.7
88.0
78.2
50.5
12.2
9.5
11.6
11.1
MPD-PAC
86.1
56.7
81.2
74.7
50.0
12.4
10.1
11.6
11.4
PC-Sampler
87.1
57.4
80.3
74.9
49.3
12.4
8.6
11.8
10.9
Table 2: Experimental results of MMaDA with parallel decoding budgets k∈{2,4,8} . The first and last Avg. columns report mean CIDEr and VQA scores, respectively. The best scores are shown in bold , and the second-best scores are underlined .
Figure 4: Cumulative information gain across decoding steps for k∈{2,4,8}
Figure 5: Qualitative comparison on COCO Caption using LaViDa at k=2 . S# denotes the decoding step, and red and blue indicate incorrectly and correctly committed tokens, respectively.
γ\λ
0.0
1.0
2.0
3.0
4.0
0.0
100.2
100.9
102.1
102.9
101.7
0.5
101.9
105.2
106.9
106.5
105.2
1.0
101.5
103.8
105.4
106.2 †
105.5
1.5
101.0
103.5
106.0
107.7
107.8
2.0
103.0
107.0
106.2
106.1
107.5
Table 3: Hyperparameter sensitivity on COCO Caption using LaViDa. † denotes the default setting.
N
Sampler
COCO Cap.
TextVQA
16
Info-Gain
81.6
48.8
VIG-Sampler
88.4
55.0
32
Info-Gain
94.8
50.2
VIG-Sampler
105.1
56.8
48
Info-Gain
97.4
48.6
VIG-Sampler
102.5
57.4
Table 4: Robustness to N using LaViDa at k=4 .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
k
Sampler
COCO Cap.
Flickr30K
NoCaps
Avg.
DetailCaps
TextVQA
DocVQA
ChartQA
Avg.
CIDEr
CIDEr
CIDEr
CIDEr
CAPTURE
Acc.
ANLS
Relaxed Acc.
Macro Avg.
2
Confidence
92.8
83.9
91.6
89.4
59.9
67.3
81.1
76.2
74.9
Entropy
91.0
77.8
89.4
86.1
59.0
63.9
82.0
76.6
74.2
Margin
94.5
82.5
91.8
89.6
61.1
67.3
82.0
77.6
75.6
MPD-PAC
92.4
83.6
91.2
89.1
60.5
65.0
81.4
75.8
74.1
PC-Sampler
92.1
81.9
90.8
88.3
59.8
66.5
82.8
76.4
75.2
Appendix
Table 5: Experimental results of LLaDA-V with parallel decoding budgets k∈{2,4,8} . The first and last Avg. columns report mean CIDEr and VQA scores, respectively. The best scores are shown in bold , and the second-best scores are underlined .
N
Sampler
COCO Cap.
TextVQA
16
Info-Gain
87.2
58.0
VIG-Sampler
92.4
57.5
32
Info-Gain
106.0
57.4
VIG-Sampler
106.2
58.7
48
Info-Gain
106.8
56.5
VIG-Sampler
104.2
59.4
Appendix
Table 6: Robustness to N using LaViDa at k=2 .
Sampler
Peak mem. (GB)
Wall-clock time (s)
k=2
k=4
Confidence
17.9
1.71
1.39
Info-Gain
17.9
3.25
2.21
VIG-Sampler
17.9
1.85
1.44
Appendix
Table 7: Inference cost of each sampler.
Figure 6: Qualitative examples on COCO Caption using LaViDa at k=2 . The top two rows illustrate decoding failures by conventional methods without explicit visual guidance, in which incorrect tokens are produced during generation. The bottom two rows present detailed captions that are visually well aligned with the images from a human perspective, despite receiving relatively low CIDEr scores. S# denotes the decoding step, while red and blue indicate incorrectly and correctly committed tokens, respectively.
Figure 7: Qualitative analysis of token selection and image-attention patterns. Confidence-based selection tends to choose uninformative tokens with diffuse attention, whereas VIG-Sampler selects answer-relevant tokens grounded in distinct image regions. Without the penalty term, VIG-Sampler selects tokens with similar image-attention patterns, providing little additional information beyond one another.
Diffusion language model decoding generates text through iterative parallel refinement, balancing fewer denoising steps with accuracy, coherence, and control.