CoTA++: Understanding the Repeat Curse in dMLLMs from an Information Flow Perspective

Qiyan Zhao; Xiaofeng Zhang; Xu-Yao Zhang; Shijie Han; Zhepeng Li; Shunzhi Zhu; Da-Han Wang

cs.AIOct 16, 2026

CoTA++: Understanding the Repeat Curse in dMLLMs from an Information Flow Perspective

Authors: Qiyan Zhao, Xiaofeng Zhang, Xu-Yao Zhang, Shijie Han, Zhepeng Li, Shunzhi Zhu, Da-Han Wang

Organizations: State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China. · Q. Zhao and X.-Y. Zhang are the School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS), Beijing 100049, China · Shanghai Jiao Tong University, Shanghai 200240, China. · Faculty of Business and Economics, The University of Hong Kong, Hong Kong, China.

Abstract

Diffusion-based multimodal large language models (dMLLMs) decode by iterative denoising with bidirectional attention, which precludes the standard autoregressive key--value cache. Practical deployments therefore rely on approximate caching that reuses intermediate states across denoising steps. Such caching, however, often makes a response repeat one token at adjacent positions in a run that breaks the sentence, a failure we term the Repeat Curse. Tracing the context-token information flow reveals three disruptions. Within a step, caching disperses the attention that anchors a position on nearby context tokens. Across steps, similarity-based refresh is not staleness-aware, so some positions are decoded against stale key--value states. Across layers, the deep-layer entropy of the context fails to converge where repetition arises. We accordingly propose CoTA++, a training-free, plug-and-play framework that counteracts each disruption where it originates: Context-Token Attention Re-anchoring re-forms the attention routing against the current context, Decode-Aware Refresh reserves part of the cache's refresh budget for the positions about to be committed, and Context-Token Entropy-Guided Voting discounts the confidence of a position by the deep-layer entropy of its context, which reorders the unmasking. The three components act at non-overlapping stages of a decoding step and compose freely with existing dMLLMs and caches. Across three dMLLMs and two caching backends, CoTA++ reduces the sample repetition rate of long responses from up to 99.6% to at most 17.0%, recovers most of the accuracy lost to caching on nine benchmarks, and keeps the speedup of caching at under 15% overhead, making approximate caching reliable for dMLLMs.

Figures & tables

Fig. 1 : Motivation. LLaDA-V [ 7 ] responses to the same query: vanilla decoding is normal (top); dLLM-Cache [ 18 ] induces the Repeat Curse (repetitions in red, middle); CoTA++ removes them while retaining the acceleration (bottom).

Fig. 2 : The Repeat Curse and its three information-flow signatures. (a) Sample repetition rate on $500$ MS-COCO captions, $L{=}512$ . (b)–(d) LLaDA-V, $L{=}128$ , vanilla against dLLM-Cache. (b) Within-step attention of suffix queries: caching disperses local anchoring (F1). (c) KV staleness at the decode moment along one response: most repeats (red) decode stale (F2). (d) Cross-layer entropy after generation at that response’s repeat positions and at normal positions: repeats fail to consolidate (F3).

Fig. 3 : Overview. Caching disrupts the information flow of context tokens on the three axes of dMLLM inference, and CoTA++ counteracts each disruption where it originates.

Method	Cache granularity	Refresh criterion	Decode- aware	Intervention level
Fast-dLLM [ 20 ]	block	block boundary	✗	schedule
Prefix-KV [ 9 ]	prefix only	none (exact)	n/a	schedule
dLLM-Cache [ 18 ]	per position	periodic $+$ similarity	✗	schedule
SlowFast [ 19 ]	per position	periodic $+$ similarity	✗	schedule $+$ decoding
dkv-cache [ 47 ]	per position	delayed commit	✗	schedule
CoTA [ 1 ]	plug-in	inherited	✗	attention $+$ decoding

TABLE I : Positioning of CoTA++ among dMLLM caching methods. Only CoTA++ makes its refresh decode-aware.

Fig. 4 : Radar charts of the Repeat Curse. Near zero without caching, every metric surges with it (labels: vanilla

{\to}

cached).

Fig. 5 : Context tokens are anchors, and caching disperses them (Finding F1). (a) Attention of vanilla LLaDA-V (brighter is higher). (b)–(d) Change in local anchoring under dLLM-Cache on $94$ MS-COCO images, $\Delta w_{5}$ (cached $-$ vanilla), where $w_{5}$ is the share of a position’s suffix attention within ${\pm}5$ positions. Repeat regions: repeat positions ${\pm}5$ . Distant regions: all other positions. Numbers in (b) : ratio of the two losses. Error bars: $95\%$ intervals over images.

Component sweep	SRR (%) $\downarrow$
Prefix interval $\mathcal{E}_{p}$	$1$ 99.0	$5$ 97.2	$15$ 97.6	$25$ 96.8
Suffix interval $\mathcal{E}_{s}$	$1$ 0.6	$3$ 68.4	$5$ 90.8	$7$ 96.8
Similarity fraction $\alpha$	$0$ 99.8	$0.25$ 96.8	$0.75$ 25.2	$1$ 0.4
Reuse policy	prefix-only 0.6	dLLM-Cache 96.8	—	—

TABLE II: Ablation of dLLM-Cache components on LLaDA-V (

L{=}128

; SRR, %). Each row sweeps one component with the others at their defaults (shaded).

Fig. 6 : Cache staleness concentrates on repeat positions (Finding F2). (a) Staleness at the decode moment on $94$ images. (b) – (d) Reference sample, on shared axes: (b) staleness of each position up to its decode (faded afterwards), (c) and (d) how each KV state is read at layer $22$ (attention received, averaged over heads and queries). Triangles: repeat positions.

(a) Quarter (steps)	Repeat	Normal	Repeat	Normal	Images	$p$
	Positions		$\Delta{>}\tau$
1st (0–31)	$15$	$2{,}899$	$0.20$	$0.02$	$3/13$	$0.54$
2nd (32–63)	$110$	$2{,}898$	$0.63$	$0.19$	$43/51$	$7.8{\times}10^{-9}$
3rd (64–95)	$337$	$2{,}671$	$0.47$	$0.40$	$51/70$	$2.7{\times}10^{-5}$
4th (96–127)	$484$	$2{,}524$	$0.50$	$0.42$	$64/86$	$5.1{\times}10^{-6}$
All	$946$	$10{,}992$	$0.50$	$0.25$	$85/94$	$3.4{\times}10^{-15}$

TABLE III : Staleness, refresh and repetition on

94

images.

\Delta{>}\tau

: share decoded beyond

\tau{=}3

. Images: where repeat exceeds normal.

p

: Wilcoxon signed-rank test over images.

Fig. 7 : Consolidation fails at repeat positions (Finding F3). (a) Entropy per layer (shaded: deep window). (b) Repetition rate by decile of the deep-window entropy of the committed neighbors within

{\pm}5

. Intervals:

95\%

, over

94

images.

(a) End of generation	Repeat	Normal	Images	$p$
Deep-window entropy (bits)	$13.7$	$10.1$	$92/94$	$4.6{\times}10^{-17}$
Halved not before layer $31$ (%)	$79$	$36$	$89/94$	$2.0{\times}10^{-15}$
(b) Decode moment	Top fifth	Rest	Images	$p$
Repetition rate (%)	$21.2$	$4.6$	$66/94$	$1.5{\times}10^{-7}$
Share of all repetitions (%)	$54$	$46$

TABLE IV : Consolidation and repetition on

94

images. Top fifth: highest context entropy. Images: where the first value exceeds the second.

p

: Wilcoxon signed-rank test over images.

Fig. 8 : The CoTA++ framework. Top : a cached dMLLM reuses the feed-forward, attention and KV states of Step $K$ at Step $K{-}1$ and unmasks the response in order of confidence. Badges mark where each component acts. Bottom : each component acts on its own object. CTAR re-anchors the attention of the positions its monitor selects, DAR refreshes the KV states of the positions about to commit, and CTEV discounts the confidence by the deep-layer entropy of the context.

Component	Intervention level	Extra cost per step
CTAR	Attention routing, in-pass	$O(\|\mathcal{D}\|\,\|\mathcal{T}^{t}\|\,(S{+}d)\,d)$
DAR	Refresh set, at the step boundary	$O(M)$ index ops
CTEV	Voting score, after the pass	$O(\|\mathcal{W}\|\,Md\,\|V\|)$

TABLE V : Per-step cost of the three components.

\mathcal{D}

: re-anchored layer band;

\mathcal{T}^{t}

: positions selected by the monitor;

\mathcal{W}

: deep window of the entropy readout;

M

: suffix length;

S

: sequence length;

d

: hidden size;

|V|

: vocabulary size.

		$L{=}512$					$L{=}128$					$L{=}64$					Avg.
Model	Method	ARR $\downarrow$	SRR $\downarrow$	MRL $\downarrow$	ARL $\downarrow$	95pRL $\downarrow$	ARR $\downarrow$	SRR $\downarrow$	MRL $\downarrow$	ARL $\downarrow$	95pRL $\downarrow$	ARR $\downarrow$	SRR $\downarrow$	MRL $\downarrow$	ARL $\downarrow$	95pRL $\downarrow$	ARR $\downarrow$	SRR $\downarrow$
	Vanilla	0.01	2.4	3	2.07	2.6	0.00	0.2	2	2.00	2.0	0.01	0.4	2	2.00	2.0	0.01	1.0
	dLLM-Cache [ 18 ]	14.70	99.6	455	5.90	3.0	7.01	96.8	73	2.88	3.0	1.61	70.8	38	2.12	2.0	7.77	89.1
	$+$ CoTA [ 1 ]	0.17	47.4	3	2.00	2.0	0.10	11.4	2	2.00	2.0	0.04	2.4	2	2.00	2.0	0.10	20.4
	$+$ CoTA++	0.03	8.2	3	2.12	3.0	0.02	2.4	2	2.00	2.0	0.01	0.4	2	2.00	2.0	0.02	3.7
	SlowFast [ 19 ]	0.28	17.6	232	2.89	2.0	0.22	17.4	3	2.03	2.0	0.55	31.2	2	2.00	2.0	0.35	22.1

TABLE VI : Repeat Curse across dMLLMs and caching backends on the

500

COCO images at three response lengths

L

, under the instruction “Please describe the image in detail.”, on content tokens (§ VI-A ). ARR and SRR are in percent. MRL, ARL and 95pRL are the maximum, mean and 95th percentile of the lengths of all repeated runs in the

500

responses. Avg.: mean over the three lengths.

\downarrow

: lower is better. Bold: best within a backend group. Shaded: CoTA++.

		$L{=}512$		$L{=}128$		$L{=}64$		Avg.
Model	Method	Judge $\uparrow$	Len	Judge $\uparrow$	Len	Judge $\uparrow$	Len	Judge $\uparrow$
	Vanilla	3.93	475	4.77	127	4.34	63	4.35
	dLLM-Cache	3.03	509	3.59	127	4.17	63	3.60
	$+$ CoTA	3.89	510	4.73	127	4.38	63	4.33
	$+$ CoTA++	4.00	510	4.77	127	4.40	63	4.39
	SlowFast	4.12	254	4.34	99	3.94	56	4.13

TABLE VII : Response quality and length in the setting of Table VI . Judge is the mean score of the

500

descriptions on a

1

–

10

scale (§ VI-A ), Len the mean response length in tokens. Bold: best within a backend group. Shaded: CoTA++.

	LLaDA-V		MMaDA		LaViDa
Method	Rep-4 $\downarrow$	Dist-2 $\uparrow$	Rep-4 $\downarrow$	Dist-2 $\uparrow$	Rep-4 $\downarrow$	Dist-2 $\uparrow$
Vanilla	24.6	58.2	2.2	89.3	2.0	86.8
dLLM-Cache	37.9	45.7	7.7	78.5	14.5	73.6
$+$ CoTA	21.9	57.7	2.4	88.2	2.0	86.6
$+$ CoTA++	20.5	59.1	2.2	89.2	1.9	86.9

TABLE VIII : Phrase-level repetition at

L{=}512

under dLLM-Cache: repeated

4

-grams (Rep-4) and distinct

2

-grams (Dist-2) in percent (§ VI-A ). Bold: best among the cached rows. Shaded: CoTA++.

Fig. 9 : Case study , LLaDA-V under dLLM-Cache: the uncached model, the cached model, and the cached model with CoTA++. (a) The typical case, $L{=}128$ . (b) A residual failure, $L{=}512$ : a doubled period at a sentence boundary.

(a) dLLM-Cache
Model	Doc	Chart	MMS	MME P	MME C	SEED	MMB	LLaVA W	MVista	MVerse
LLaDA-V [ 7 ]	83.9	78.3	60.2	1508	492	74.8	83.0	51.7	60.4	29.2
$+$ dLLM-Cache	70.5	78.2	60.0	1503	497	74.8	83.0	30.2	55.9	26.1
$+$ CoTA [ 1 ]	81.6	78.6	60.0	1505	499	74.8	83.2	50.0	57.0	27.5
$+$ CoTA++	82.1	78.6	60.0	1505	499	74.8	83.2	50.4	57.2	28.3
MMaDA [ 8 ]	–	10.1	37.3	1087	285	56.6	42.6	29.7	34.2	18.8

TABLE IX : General multimodal benchmarks under (a) dLLM-Cache and (b) SlowFast: DocVQA, ChartQA, MMStar, MME (perception, cognition), SEED, MMBench, LLaVA-Bench (in-the-wild), MathVista and MathVerse. MMaDA uses the MMaDA-8B-MixCoT checkpoint, the only one with a released evaluation protocol.

			$L{=}64$			$L{=}128$			$L{=}512$
DAR	CTAR	CTEV	ARR	SRR	MRL	ARR	SRR	MRL	ARR	SRR	MRL
			$1.61$	$70.8$	$38$	$7.01$	$96.8$	$73$	$14.70$	$99.6$	$455$
✓			$0.02$	$1.4$	$2$	$0.04$	$4.4$	$2$	$0.10$	$11.2$	$165$
	✓		$0.13$	$8.0$	$2$	$0.21$	$20.6$	$3$	$0.31$	$62.0$	$160$
		✓	$0.77$	$35.6$	$34$	$0.86$	$59.6$	$56$	$1.72$	$94.0$	$420$
✓	✓		$0.02$	$0.6$	$2$	$0.02$	$3.0$	$2$	$0.05$	$9.6$	$3$

TABLE X : Component grid (LLaDA-V with dLLM-Cache,

500

images): every subset of the three components at three response lengths. Shaded: CoTA++.

Configuration	Latency (s/img) $\downarrow$	Throughput (tok/s) $\uparrow$	Relative latency	Memory (GiB)
Vanilla	21.4	2.9	–	30.6
dLLM-Cache	8.8	7.1	1.00 $\times$	32.5
$+$ DAR	9.0	7.0	1.01 $\times$	32.5
$+$ CTAR	9.4	6.7	1.06 $\times$	32.5
$+$ CTEV	9.4	6.7	1.07 $\times$	33.4
$+$ CoTA++	10.2	6.2	1.15 $\times$	33.4

TABLE XI : Efficiency on LLaDA-V at

L{=}64

20

images, each configuration alone on one A100: latency in seconds per image, throughput in decoded tokens per second, latency relative to dLLM-Cache, and peak GPU memory.

Fig. 10 : Attention at the decode moment, with and without CTAR , layers $25$ – $32$ ; triangles: repeat positions. (a)–(b) LLaDA-V and (c)–(d) LaViDa under dLLM-Cache, $L{=}128$ .

Fig. 11 : Staleness of the KV state at the decode moment, with and without DAR. Mean over

100

images at each decoding step, dLLM-Cache,

L{=}128

Fig. 12 : Context entropy at the decode moment, with and without CTEV , dLLM-Cache,

L{=}128

. Top: repeat rate of the committed position. Bottom: the decisions CTEV changes.

Configuration	ARR $\downarrow$	SRR $\downarrow$	MRL $\downarrow$	Rep-4 $\downarrow$
Vanilla	0.00	0.0	0	0.037
dLLM-Cache	7.52	96.0	73	0.083
CTAR: re-weight the stored routing, or re-form it
Additive distance bias	6.41	98.0	73	0.073
Gated gain	9.07	99.0	82	0.086
Re-formed, layers $25$ – $32$ (CTAR)	0.32	21.0	3	0.035

TABLE XII : Design decisions of the three components , LLaDA-V under dLLM-Cache,

L{=}128

100

images, each component alone.

Fig. 13 : Tokens that open a repeated run under the cache , sized by run count; a leading hyphen marks a word piece.

Diffusion Language Model Decoding

Diffusion language model decoding generates text through iterative parallel refinement, balancing fewer denoising steps with accuracy, coherence, and control.

See all 247 papers

Context-Aware Cluster Decoding: Semantic Anchor-Driven Coherence in dMLLMs
Aug 23, 2026Yikai Zhao, Qiyan Zhao, Jiaquan Zhang +3Diffusion Language Model Decoding Long-Form Document Generation
Sun Yat-sen University · Shanghai Jiao Tong University · University of Electronic Science and Technology of China +2
EntropyCache: Decoded Token Entropy Guided KV Caching for Diffusion Language Models
Mar 19, 2026Minsoo Cheong, Donghyun Son, Woosang Lim +1Diffusion Language Model Inference Diffusion Model Caching
Seoul National University
Affix Cache for Diffusion Large Language Models
Jun 26, 2026Kaihua Liang, An Zhong, Xin Tan +4KV Caching Memory-Efficient Inference
KAUST · CUHK · LUMS

Aug 23, 2026J/K move · Enter open · S save

Context-Aware Cluster Decoding: Semantic Anchor-Driven Coherence in dMLLMs

Yikai Zhao, Qiyan Zhao, Jiaquan Zhang, Xiaofeng Zhang+2

Sun Yat-sen University · Shanghai Jiao Tong University · University of Electronic Science and Technology of China+2

Diffusion Language Model Decoding Long-Form Document Generation Multimodal Hallucination

Open paper AskPDF arXiv

Diffusion multimodal large language models (dMLLMs) frequently produce long-form outputs marred by semantic drift and repetition, with quality generally degrading as output length increases. We identify two structural deficiencies in existing decoding methods as primary drivers of these failures: confidence-based scoring ignores decoded-neighbor support, and block partitioning prevents access to high-readiness semantic anchors, together causing tokens to be committed before their local context is sufficiently established. We propose \ours{} (Context-Aware Cluster Decoding), a training-free decoding method that scores each masked position by a multiplicative composite of softmax confidence and neighbor proximity, promoting contextually ready tokens above isolated candidates while suppressing low-confidence positional noise, operating block-free to keep high-readiness anchors globally accessible. \ours{} further applies architecture-aware calibration to handle confidence heterogeneity induced by diverse visual integration strategies. Experiments on three dMLLMs across four benchmarks demonstrate consistent quality gains and hallucination reduction over Original, with larger gains in several longer generation settings, highlighting the importance of neighbor support and visual integration strategy for future dMLLM decoding method design. Our code is openly available at https://github.com/zhaoyk-sysu/CACD-dMLLM.

Abstract

Diffusion Language Model Decoding

Explore similar work

Context-Aware Cluster Decoding: Semantic Anchor-Driven Coherence in dMLLMs

EntropyCache: Decoded Token Entropy Guided KV Caching for Diffusion Language Models

Affix Cache for Diffusion Large Language Models