Decoding quality in diffusion multimodal language models (dMLLMs) depends heavily on the order in which masked tokens are committed. Existing confidence-based strategies prioritize locally easy tokens, but confidence does not necessarily reflect contextual usefulness. As a result, structurally easy tokens such as punctuation may be committed before informative semantic anchors, weakening context propagation and increasing error accumulation. We propose Information-Guided Frontier Decoding (IGFD), a training-free decoding strategy that ranks candidates using token confidence, neighborhood uncertainty, and structural commitment risk. IGFD encourages early commitment of reliable semantic anchors while delaying fragile structural tokens, improving contextual support during decoding. A dynamic candidate frontier further constrains token selection to locally expandable regions under the same decoding budget. The method requires no additional training, auxiliary models, or extra forward passes. Experiments across multimodal understanding, reasoning, grounding, and hallucination benchmarks show that IGFD consistently outperforms existing decoding strategies across the majority of benchmarks and diffusion MLLM backbones under identical decoding budgets.
Figures & tables
Figure 1: Confidence-only decoding may commit locally easy but low-utility tokens too early, while delaying content-bearing anchors that are more helpful for resolving nearby masked positions.
Figure 2: Single-token intervention. Each subplot compares local entropy profiles before and after committing exactly one token. The x-axis denotes relative position to the committed token. The semantic anchor “Starbucks” produces the largest local entropy reduction, the date token “2024” produces a moderate reduction, and punctuation produces minimal change.
Figure 3: Single-token intervention on nearby ground-truth NLL. After committing one token, we measure the change in NLL of neighboring correct tokens. Semantic tokens such as “mug” and “is” reduce local NLL, whereas the structural token “.” provides little benefit and may increase prediction loss.
Figure 4: Framework of IGFD. Given a partially masked sequence, IGFD first builds a dynamic candidate frontier, then computes confidence, entropy-based neighborhood need, and structural risk for each candidate. These signals are combined into an information-guided commitment score, which selects the top-ranked tokens to commit and then expands the frontier for the next denoising step.
Model
Method
LLaVA-Bench
CHAIR
MathVista
ScienceQA
MME
GQA
all ↑
conv ↑
detail ↑
complex ↑
CS↓
Ci↓
recall ↑
acc ↑
acc ↑
cog. ↑
perc. ↑
EM ↑
LLaDA-V
Original
50.4
43.4
57.5
63.6
5.6
4.0
36.6
30.8
79.2
338.6
1411.4
52.3
AdaBlock
71.8
59.7
68.8
79.2
5.8
4.3
37.3
32.3
81.2
350.8
1412.6
52.6
Wavefront
71.6
61.6
71.7
75.9
5.3
5.2
34.2
33.4
81.8
355.7
1397.5
51.7
IGFD
72.8
63.1
74.6
80.8
5.2
3.7
37.1
34.2
82.7
363.6
1422.7
53.7
MMaDA
Original
35.1
34.4
39.7
33.7
24.6
11.3
39.5
23.5
50.2
232.6
843.9
39.4
Table 1: Comparison of different methods across multiple benchmarks.
Figure 5: Analysis of decoding behavior and commitment dynamics. Metrics are tracked across normalized decoding progress intervals. (a) illustrates the frequency of high-need tokens (top 20% uncertainty); (b) shows the punctuation commitment rate per interval; (c) displays the cumulative frequency of committed content-bearing tokens. IGFD significantly prioritizes critical content over structural tokens in early decoding stages.
Variant
CS↓
Ci↓
Recall ↑
Confidence only
12.3
8.4
37.0
w/o neighborhood need
10.3
6.7
37.5
w/o structural penalty
10.3
6.7
37.6
w/o dynamic frontier
10.2
6.6
37.7
Full IGFD
9.7
6.4
37.9
Table 2: Ablation study of IGFD on CHAIR, averaged over three dMLLMs.
Method
Precision ↑
Recall ↑
F1 ↑
Original
0.842
0.834
0.838
AdaBlock
0.856
0.849
0.852
Wavefront
0.861
0.854
0.857
IGFD
0.873
0.866
0.869
Table 3: Semantic quality evaluation on WikiText using BERTScore. Higher scores indicate better semantic fidelity to reference text.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Variant
LLaVA-Bench
CHAIR
MathVista
all ↑
conv ↑
detail ↑
complex ↑
CS↓
Ci↓
recall ↑
acc ↑
LLaDA-V
Confidence
50.4
43.4
57.5
63.6
5.6
4.0
36.6
30.8
w/o neighborhood need
71.7
61.9
73.1
79.1
5.3
3.8
37.0
33.5
w/o structural penalty
72.1
62.4
73.8
79.7
5.4
3.9
36.9
33.7
w/o dynamic frontier
72.0
62.2
73.5
79.5
5.4
3.9
37.0
33.6
Full IGFD
72.8
63.1
74.6
80.8
5.2
3.7
37.1
34.2
Appendix
Table 4: Ablation study of IGFD components across three dMLLMs.
Figure 6: Hyperparameter sensitivity analysis of the proposed decoding strategy. We evaluate four key hyperparameters: (a) the confidence weight α , (b) the neighborhood utility weight β , (c) the structural penalty weight γ , and (d) the neighborhood radius r . Each curve reports the average performance across three diffusion-based multimodal language models, including LLaDA-V, MMaDA, and LaViDa. The results show that moderate values of α , β , and γ generally achieve better trade-offs across perception, reasoning, and hallucination-related metrics, while an overly large structural penalty or neighborhood radius may degrade performance. The dashed vertical line in each subfigure indicates the selected default setting used in our main experiments.
Diffusion language model decoding generates text through iterative parallel refinement, balancing fewer denoising steps with accuracy, coherence, and control.
Diffusion multimodal large language models (dMLLMs) frequently produce long-form outputs marred by semantic drift and repetition, with quality generally degrading as output length increases. We identify two structural deficiencies in existing decoding methods as primary drivers of these failures: confidence-based scoring ignores decoded-neighbor support, and block partitioning prevents access to high-readiness semantic anchors, together causing tokens to be committed before their local context is sufficiently established. We propose \ours{} (Context-Aware Cluster Decoding), a training-free decoding method that scores each masked position by a multiplicative composite of softmax confidence and neighbor proximity, promoting contextually ready tokens above isolated candidates while suppressing low-confidence positional noise, operating block-free to keep high-readiness anchors globally accessible. \ours{} further applies architecture-aware calibration to handle confidence heterogeneity induced by diverse visual integration strategies. Experiments on three dMLLMs across four benchmarks demonstrate consistent quality gains and hallucination reduction over Original, with larger gains in several longer generation settings, highlighting the importance of neighbor support and visual integration strategy for future dMLLM decoding method design. Our code is openly available at https://github.com/zhaoyk-sysu/CACD-dMLLM.
Yikai Zhao, Qiyan Zhao, Jiaquan Zhang +3
Sun Yat-sen University · Shanghai Jiao Tong University · University of Electronic Science and Technology of China +2
Diffusion multimodal large language models (dMLLMs) have recently emerged as a new decoding paradigm for multimodal generation. Starting from a fully masked sequence, dMLLMs progressively decode the sequence by unmasking a subset of the remaining masked positions at each step. Since the selected tokens serve as the prediction context for subsequent steps, deciding which tokens to decode is crucial to the quality of the final output. The most common strategy prioritizes tokens based on a certainty measure that tends to favor tokens frequently observed in the training data. Recent approaches instead order tokens according to their influence on subsequent predictions, but do not explicitly account for the input image. We propose the Visual Information-Guided Sampler (VIG-Sampler), which prioritizes tokens based on their attention to image tokens. We further impose a constraint that penalizes candidate tokens whose image-attention distributions are similar to those of previously selected tokens, thereby increasing the information gain of the decoded subset. Extensive experiments on 7 captioning and VQA benchmarks with 3 open-source dMLLMs demonstrate the effectiveness of VIG-Sampler, which outperforms the Info-Gain Sampler by an average of 19.3 CIDEr points across the captioning benchmarks and surpasses it on COCO Caption while using only half as many decoding steps.
Discrete diffusion language models can generate text efficiently by updating multiple masked positions in parallel, but this parallelism introduces a quality-latency trade-off. Aggressive decoding may commit mutually dependent tokens too early, while conservative decoding requires many denoising steps. Existing methods address this tension by deciding which tokens are safe to reveal using confidence or dependency criteria. However, avoiding unsafe commits does not necessarily make the remaining masked sequence easy to decode, since uncertain tokens may depend on masked tokens, creating a bottleneck for denoising steps. We propose AXON, a training-free module that can be added on top of existing parallel decoding strategies for diffusion language models. Rather than replacing the base decoder, AXON monitors the remaining uncertain masked tokens and intervenes only when their current state suggests that additional context is needed. It then shifts the criterion from which tokens are safest to reveal to which confident reveals would best support later denoising. AXON selects anchors, confident masked tokens that uncertain positions attend to, using attention, uncertainty, and confidence signals. Experiments on reasoning and code-generation benchmarks across multiple diffusion language models show that AXON improves the quality-latency trade-off of existing parallel decoders, often reducing the number of function evaluations while maintaining or improving accuracy.
Giries Abu Ayoub, Mario Barbara, Lluís Pastor-Pérez +4
Department of Computer Science, University of Haifa · Institute for AI, University of Stuttgart · IMPRS-IS