Context-Aware Cluster Decoding: Semantic Anchor-Driven Coherence in dMLLMs

Yikai Zhao; Qiyan Zhao; Jiaquan Zhang; Xiaofeng Zhang; Xiaosong Yuan; Pengzhou Cheng

cs.CLAug 23, 2026

Context-Aware Cluster Decoding: Semantic Anchor-Driven Coherence in dMLLMs

Authors: Yikai Zhao, Qiyan Zhao, Jiaquan Zhang, Xiaofeng Zhang, Xiaosong Yuan, Pengzhou Cheng

Organizations: Sun Yat-sen University · Shanghai Jiao Tong University · University of Electronic Science and Technology of China · Alibaba Group ·

Abstract

Diffusion multimodal large language models (dMLLMs) frequently produce long-form outputs marred by semantic drift and repetition, with quality generally degrading as output length increases. We identify two structural deficiencies in existing decoding methods as primary drivers of these failures: confidence-based scoring ignores decoded-neighbor support, and block partitioning prevents access to high-readiness semantic anchors, together causing tokens to be committed before their local context is sufficiently established. We propose \ours{} (Context-Aware Cluster Decoding), a training-free decoding method that scores each masked position by a multiplicative composite of softmax confidence and neighbor proximity, promoting contextually ready tokens above isolated candidates while suppressing low-confidence positional noise, operating block-free to keep high-readiness anchors globally accessible. \ours{} further applies architecture-aware calibration to handle confidence heterogeneity induced by diverse visual integration strategies. Experiments on three dMLLMs across four benchmarks demonstrate consistent quality gains and hallucination reduction over Original, with larger gains in several longer generation settings, highlighting the importance of neighbor support and visual integration strategy for future dMLLM decoding method design. Our code is openly available at https://github.com/zhaoyk-sysu/CACD-dMLLM.

Figures & tables

Figure 1: A representative block-diffusion failure: the same paragraph repeats four times, a symptom of incorrect commitment order leaving neighboring positions without local semantic support.

Figure 2: Two failure modes of the Original block-diffusion method on MMaDA-8B-Base. (a) LLaVABench-COCO score degrades monotonically with output length. (b) Sample Repeat Curse Ratio on LLaVABench-COCO increases with output length, measured following 60 . (c) A high-readiness semantic anchor candidate lies outside the current block in 75.0% of first-round steps and 27.3% of all steps (block_length

=128

, 90 LLaVABench-COCO samples), revealing that block partitioning systematically prevents access to highly effective anchors.

Figure 3: Motivation analysis (MMaDA, 1,000 MathVista samples). (a) Neighbor support vs. prediction difficulty: entropy drops $5.3\times$ from isolated to fully-neighbored positions—is strongly associated with prediction difficulty beyond raw confidence (b) Confidence–context space colored by commit round; the “hard zone” marks tokens with low confidence but high context score that are contextually ready yet stranded by pure confidence scoring. (c) Commit-round gap peaking at $\Delta{=}12.4$ rounds, quantifying how long voids persist under pure confidence scoring.

Figure 4: Overview of CACD . Left : the Scoring phase computes a composite score for each masked position across early and late rounds; early rounds apply a strict threshold $\tau$ , while late rounds apply a relaxed dynamic threshold triggered when $r_{t}\leq\delta$ . Center : positions with composite scores exceeding the current threshold are committed simultaneously as a dense cluster. Right : committed anchor tokens propagate localized context influence (dashed lines) to adjacent masked positions, boosting their scores for the subsequent decoding round.

Figure 5: Decoding order on a representative captioning sample (lighter = earlier commit; red boxes = semantic anchors). Block diffusion frequently bypasses high-readiness anchors at structural boundaries; in contrast, CACD tends to first capture global anchors across the full sequence for initial decoding, and subsequently identifies critical local tokens within each sub-segment to propagate context outward.

MMaDA-8B-Base
Model	Method	Understanding & Reasoning			Hallucination
Model	Method	MathVista $\uparrow$	LLaVABench-COCO $\uparrow$	ScienceQA $\uparrow$	CHAIR s $\downarrow$	CHAIR i $\downarrow$	Recall $\uparrow$
	Original	23.50	35.30	50.24	24.58	11.27	39.46
	AdaBlock	24.80	47.40	51.61	22.84	11.27	41.83
	Wavefront	23.00	42.50	50.77	18.62	9.43	38.91
	CACD	28.00	49.00	51.71	19.76	10.28	40.63

Table 1: Generation quality across three dMLLMs (within-model comparisons). Bold : best per model per metric. Underline : second best. LLaVABench-COCO for long-form generation and CHAIR for hallucination mitigation.

Benchmark	Metric	Original	CACD	$\boldsymbol{\Delta}$
GSM8K	Accuracy	61.00	63.00	+2.00
IFEval	Prompt strict	59.00	60.00	+1.00
	Instruction strict	64.74	68.59	+3.85
	Prompt loose	60.00	61.00	+1.00
	Instruction loose	67.95	69.23	+1.28
AlpacaEval	Win rate	4.50	6.00	+1.50

Table 2: Generation quality on LLaDA-8B-Instruct. All values are percentages. Bold : better result.

Length	Original	AdaBlock	Wavefront	CACD
16	45.9	52.6	41.9	52.2
32	40.8	61.7	45.5	59.4
64	33.5	55.9	46.4	58.8
128	31.6	46.8	43.0	50.4
256	32.3	46.0	44.6	47.3
512	35.3	47.4	42.5	49.0

Table 3: LLaVABench-COCO score and average output length. Output length alone does not account for the observed quality differences. Bold : best score per model.

Method	MMaDA		LaViDa		LLaDA-V
Method	Score $\uparrow$	Len	Score $\uparrow$	Len	Score $\uparrow$	Len
Original	35.3	188.8	51.1	101.5	50.3	392.5
AdaBlock	47.4	160.2	78.6	256.1	72.9	409.7
Wavefront	42.5	402.8	82.1	481.8	71.3	400.9
CACD	49.0	158.1	88.7	461.2	73.9	367.5

Table 4: LLaVABench-COCO score and average output length. Higher score with shorter output confirms quality gains are not length-driven. Bold : best Score per model.

Ctx	Dyn. $\tau$	EOS	Score $\uparrow$
✗	✓	✓	46.4
✓	✗	✓	46.7
✓	✓	✗	48.2
✓	✓	✓	49.0

Table 5: Component ablation on MMaDA LLaVABench-COCO. Bold : best.

Figure 6: Peak GPU memory across maximum generation lengths on MMaDA, measured on an NVIDIA A800. Both methods show gradual memory growth, while CACD uses moderately more memory than Original across the evaluated lengths.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Decoding order comparison: block diffusion (left) vs. CACD (right). Block diffusion commits tokens within rigid sequential blocks, leaving the high-readiness anchor structurally inaccessible until its specific block is reached. CACD performs a global forward pass each round, committing the highest-readiness tokens as a dense cluster with context-amplified scores.

Figure 8: Token confidence distributions measured on 100 MathVista samples per model. MMaDA, LaViDa, and LLaDA-V exhibit substantially different distributions. LaViDa and LLaDA-V share the same SigLIP encoder, suggesting that the confidence landscape also depends on other model and training design choices.

Model	$\boldsymbol{\beta}$	Score $\uparrow$
MMaDA (8B-Base)	0.0	48.2
	0.5	48.7
	1.0	49.0
	1.5	48.9
	2.0	48.6
LaViDa (SigLIP+LLaDA-8B)	0.0	79.3

Appendix

Table 6: Effect of

\beta

on LLaVABench-COCO quality.

\beta{=}0

recovers pure confidence scoring;

\beta{=}1.0

provides the best context amplification across all three models. Bold : best.

Model	$\boldsymbol{\tau}$	Acc $\uparrow$
MMaDA (8B-Base)	0.5	21.1
	0.7	24.3
	0.9	28.0
LaViDa (SigLIP+LLaDA-8B)	0.5	45.7
	0.7	44.6
	0.9	44.5

Appendix

Table 7: Effect of

\tau

on MathVista accuracy. Architecture-specific thresholds are required: the optimal

\tau

differs substantially across models due to confidence distribution heterogeneity. Bold : best.

Figure 9: $\beta\times\tau$ interaction on MMaDA MathVista (accuracy %). Red box: adopted configuration ( $\beta{=}1.0$ , $\tau{=}0.9$ ).

$\boldsymbol{\delta}$	Acc $\uparrow$
0.1	26.9
0.2	27.8
0.3	28.0
0.5	26.8

Appendix

Table 8: Effect of

\delta

on MMaDA MathVista. Bold : selected default.

$\boldsymbol{\rho}$	Score $\uparrow$	Avg. Length (chars)
0.5	47.7	154.9
0.6	48.1	157.8
0.8	49.0	158.1
1.0	47.8	161.0

Appendix

Table 9: Effect of EOS release ratio

\rho

on MMaDA LLaVABench-COCO. Bold : selected default.

Method	MathVista		LLaVABench-COCO		ScienceQA
Method	NFE $\downarrow$	Spd. $\uparrow$	NFE $\downarrow$	Spd. $\uparrow$	NFE $\downarrow$	Spd. $\uparrow$
Original	128.0	$1.0\times$	512.0	$1.0\times$	16.00	$1.0\times$
AdaBlock	25.9	$3.9\times$	74.0	$6.5\times$	1.62	$\mathbf{6.6\times}$
Wavefront	60.0	$3.8\times$	128.1	$4.0\times$	4.00	$3.4\times$
CACD	16.8	$\mathbf{4.5\times}$	45.9	$\mathbf{11.2\times}$	1.32	$5.1\times$

Appendix

Table 10: Decoding cost on MMaDA-8B-Base without KV caching. NFE denotes the number of function evaluations and Spd. denotes the measured speedup. Bold : best.

LaViDa
Method	MathVista	LLaVABench-COCO	ScienceQA
Original	128.0	512.0	16.0
AdaBlock	27.4	52.1	2.1
Wavefront	36.0	71.3	3.0
CACD	16.0	45.6	1.1
LLaDA-V

Appendix

Table 11: NFE on LaViDa and LLaDA-V. Lower values are better. Bold : best.

Model	Method	NFE $\downarrow$	Spd. $\uparrow$	Memory $\downarrow$
MMaDA	Original	512.0	$1.0\times$	18,315
	AdaBlock	74.0	$6.5\times$	22,775
	Wavefront	128.1	$4.0\times$	21,157
	CACD	45.9	$\mathbf{11.2\times}$	20,345
LaViDa	Original	512.0	$1.0\times$	33,737
	AdaBlock	52.1	$\mathbf{2.1\times}$	29,205

Appendix

Table 12: Decoding efficiency and peak GPU memory on LLaVABench-COCO using max_new_tokens=512 on an NVIDIA A800. Memory is reported in MiB. The best result for each model and metric is shown in bold .

Benchmark	Original NFE	CACD NFE	Speedup
GSM8K	128.00	48.00	$\mathbf{2.68\times}$
IFEval	128.00	112.73	$\mathbf{1.20\times}$
AlpacaEval	128.00	121.75	$\mathbf{1.03\times}$

Appendix

Table 13: Decoding efficiency on LLaDA-8B-Instruct. Lower NFE and higher speedup are better.

Panel of Figure 10 — Figure 10: Case A: MathVista — Measuring Cup. LLaDA-V : reasoning drift produces the wrong answer (500). LLaDA-V + CACD : anchors the visual token (“1000g/1L”) as the first committed cluster and propagates context outward, yielding the correct answer (1000). Case B: CHAIR — Tennis Court. LLaDA-V : object hallucination produces an incorrect scene description (“frisbee game”). LLaDA-V + CACD : anchors the visual token (“tennis court / racket”) and propagates context outward, yielding the correct description (woman playing tennis, preparing to serve). Case C: CHAIR — Fire Hydrant. LLaDA-V : attribute hallucination produces an incorrect spatial description (“positioned in the middle of road”). LLaDA-V + CACD : anchors the visual token (“sidewalk / yellow line”) and propagates context outward, yielding the correct description (hydrant on the sidewalk, close to a yellow line and a blue sign).

Figure 11: Case D: LLaVABench-COCO — Clock Tower (doc_id=41). Original : repeated paragraphs and missing time readings. CACD : well-structured response with specific readings and multiple candidate interpretations.

Figure 12: Case E: LLaVABench-COCO — Wii Controller (doc_id=49). Original : verbatim repetition and truncation. CACD : complete, non-repetitive response with broader scene coverage including background objects absent from the Original output.

Diffusion Language Model Decoding

Diffusion language model decoding generates text through iterative parallel refinement, balancing fewer denoising steps with accuracy, coherence, and control.

See all 247 papers

Information-Guided Frontier Decoding: Contextual Utility-Driven Commitment in dMLLMs
Aug 27, 2026Xingyou Fang, Jingxing Zhong, Xiaosong Yuan +1Diffusion Language Model Decoding Diffusion Language Models
Fuzhou University · Jilin University · Shanghai Jiao Tong University
Visual-Redundancy-Controlled Parallel Decoding for Diffusion-Based Multimodal Large Language Models
May 25, 2026Yulin Yuan, Hongshuo Zhao, Xiangming MengLanguage Model Decoding Multimodal Large Language Models
Zhejiang University · ZJUI-UIUC Institute
CoTA++: Understanding the Repeat Curse in dMLLMs from an Information Flow Perspective
Oct 16, 2026Qiyan Zhao, Xiaofeng Zhang, Xu-Yao Zhang +4Diffusion Language Model Decoding Diffusion Language Models
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China. · Q. Zhao and X.-Y. Zhang are the School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing 100049, China · Shanghai Jiao Tong University, Shanghai 200240, China. +1

Aug 27, 2026J/K move · Enter open · S save

Information-Guided Frontier Decoding: Contextual Utility-Driven Commitment in dMLLMs

Xingyou Fang, Jingxing Zhong, Xiaosong Yuan, Xiaofeng Zhang

Fuzhou University · Jilin University · Shanghai Jiao Tong University

Diffusion Language Model Decoding Diffusion Language Models Multimodal Diffusion Models

Open paper AskPDF arXiv

Decoding quality in diffusion multimodal language models (dMLLMs) depends heavily on the order in which masked tokens are committed. Existing confidence-based strategies prioritize locally easy tokens, but confidence does not necessarily reflect contextual usefulness. As a result, structurally easy tokens such as punctuation may be committed before informative semantic anchors, weakening context propagation and increasing error accumulation. We propose Information-Guided Frontier Decoding (IGFD), a training-free decoding strategy that ranks candidates using token confidence, neighborhood uncertainty, and structural commitment risk. IGFD encourages early commitment of reliable semantic anchors while delaying fragile structural tokens, improving contextual support during decoding. A dynamic candidate frontier further constrains token selection to locally expandable regions under the same decoding budget. The method requires no additional training, auxiliary models, or extra forward passes. Experiments across multimodal understanding, reasoning, grounding, and hallucination benchmarks show that IGFD consistently outperforms existing decoding strategies across the majority of benchmarks and diffusion MLLM backbones under identical decoding budgets.

Abstract

Diffusion Language Model Decoding

Explore similar work

Information-Guided Frontier Decoding: Contextual Utility-Driven Commitment in dMLLMs

Visual-Redundancy-Controlled Parallel Decoding for Diffusion-Based Multimodal Large Language Models

CoTA++: Understanding the Repeat Curse in dMLLMs from an Information Flow Perspective