Dependency-Aware Revocable Decoding for Efficient Diffusion Large Language Model Inference

Wooje Park; Insu Lee; Minyoung Noh; Jaeyun Jang; Sungmin Lee; Kyuhong Shim; Byonghyo Shim

cs.CLAug 27, 2026

Dependency-Aware Revocable Decoding for Efficient Diffusion Large Language Model Inference

Authors: Wooje Park, Insu Lee, Minyoung Noh, Jaeyun Jang, Sungmin Lee, Kyuhong Shim, Byonghyo Shim

Organizations: Seoul National University · Sungkyunkwan University

Abstract

Diffusion large language models (dLLMs) offer a promising alternative to autoregressive generation by decoding multiple tokens in parallel through iterative denoising. However, increasing decoding parallelism often degrades generation quality, as early errors can contaminate later contexts. Revocable decoding mitigates this issue by re-evaluating decoded tokens and remasking unreliable ones, but existing methods overlook that unreliable tokens may also corrupt the verification context itself. We identify this failure mode and propose Dependency-Aware Revocable Decoding (DARD), a training-free framework that separates tokens into masked, candidate, and unmasked states. DARD verifies candidate tokens using a selective context that excludes less reliable tokens and adaptively regulates their influence on subsequent decoding. Experiments across 12 textual and multimodal benchmarks on 3 open-source dLLMs show that DARD consistently improves the speed-quality Pareto frontier over recent revocable decoding methods, achieving a 2.71 $\times$ speedup and a 4.35-point CIDEr score gain over Saber on Flickr30K.

Figures & tables

Figure 1: Left: Comparison of decoding schemes. Vanilla dLLM decoding fixes tokens once unmasked, while revocable decoding allows them to be remasked. DARD introduces an intermediate

\mathcal{C}

state to place unstable decoded tokens and regulate their contextual influence during verification. Right: By selectively using context, DARD mitigates verification errors arising from context instability, thereby reducing decoding steps while improving performance.

Figure 2: Left: Parallel decoding predicts multiple token positions simultaneously, making it difficult to account for dependencies among tokens and potentially producing inconsistent predictions. Center: Conventional revocable methods verify tokens based on all decoded tokens, so inconsistent tokens may mislead verification. Right: DARD performs dependency-aware verification, where each token attends only to more reliable context tokens.

Figure 3: Overview of DARD. DARD verifies tokens using state-specific contexts defined according to token reliability and updates their states at the next decoding step. Based on the verification outcomes, DARD estimates the reliability of $\mathcal{C}$ tokens and adaptively controls their contribution to $\mathcal{M}$ token prediction.

Figure 4: Attention mask design of DARD. Rows correspond to queries, and columns correspond to keys. DARD uses state-specific attention masks to construct different verification contexts for different token states.

Figure 5: LLaDA-8B-Instruct results on six language benchmarks. Each plot compares task performance against the average number of decoding steps for each method. Dashed curves indicate the Pareto frontier for each method. For each method, the best-performing configuration is highlighted with a red outline, and its TPS is annotated next to the point. Horizontal lines show the performance of standard LLaDA decoding with 64 steps.

Figure 6: MMaDA-8B-MixCoT results on six vision-language benchmarks. Each plot compares task performance against the average number of decoding steps for each method. Dashed curves indicate the Pareto frontier for each method. For each method, the best-performing configuration is highlighted with a red outline, and its TPS is annotated next to the point. Horizontal lines show the performance of standard MMaDA decoding with 64 steps.

Figure 7: Qualitative comparison of DARD with conventional revocable decoding methods on GSM8K.

Method	Acc. (%)	$\Delta$ Acc. (%)	Steps	$\Delta$ Steps
Saber	33.2	–	81.0	–
$\text{DARD}_{\text{bidir}}$	31.4	$-1.8$	58.5	$-\textbf{22.5}$
$\text{DARD}_{\text{l2r}}$	32.0	$-1.2$	59.3	$-21.7$
$\text{DARD}_{\text{entropy}}$	33.2	$+0.0$	59.1	$-21.9$
$\text{DARD}_{\text{margin}}$	34.2	$+1.0$	59.3	$-21.7$
DARD (ours)	34.6	$+\textbf{1.4}$	59.3	$-21.7$

Table 1: Analysis of attention mask design on DARD.

\Delta

Acc. and

\Delta

Steps are computed relative to Saber.

Method	Acc. (%)	$\Delta$ Acc. (%)	Steps	$\Delta$ Steps
Saber	33.2	–	81.0	–
$\text{DARD}~({w=0.0})$	29.2	$-4.0$	82.0	$+1.0$
$\text{DARD}~({w=0.5})$	31.1	$-2.1$	63.6	$-17.4$
$\text{DARD}~({w=1.0})$	33.4	$+0.2$	57.1	$-\textbf{23.9}$
DARD (ours)	34.6	$+\textbf{1.4}$	59.3	$-21.7$

Table 2: Analysis of logit mixing strategy in DARD.

\Delta

Acc. and

\Delta

Steps are computed relative to Saber.

Gen. Len.	Block Len.	Acc. (%)	Steps
128	128	22.0	38.9
128	64	27.0	37.1
256	256	22.4	62.2
256	128 (ours)	34.6	59.3
256	64	32.6	61.4

Table 3: Ablation of generation length and block length in DARD.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Evaluation with different generation budgets on MATH-500.

Figure 10: Fine-grained multimodal benchmark results.

Position	0	1	2	3	4	5	6	7	8	9	10
Step 13	A	man	$\langle\text{wearing}\rangle$	a	$\langle\text{black}\rangle$	shirt	[M]	$\langle\text{on}\rangle$	$\langle\text{a}\rangle$	book	.
Step 14	A	man	[M] $\boldsymbol{\downarrow}$	a	$\langle\text{black}\rangle$	shirt	is	[M] $\boldsymbol{\downarrow}$	$\langle\text{a}\rangle$	book	.
Final	A	man	in	a	black	shirt	is	reading	a	book	.

Appendix

Table 4: Illustration of how the adaptive mixing weight operates during MMaDA generation on Flickr30K.

\langle\cdot\rangle

denotes a token in the candidate state, and

\downarrow

indicates demotion to the masked state.

Model / Dataset	$w_{t}^{i}<0.5$	$0.5\leq w_{t}^{i}<0.7$	$0.7\leq w_{t}^{i}<0.9$	$w_{t}^{i}\geq 0.9$
LLaDA / GSM8K	4.7%	5.1%	6.8%	83.3%
MMaDA / ScienceQA	10.7%	5.1%	4.9%	79.2%

Appendix

Table 5: Empirical distribution of adaptive mixing weights

w_{t}^{i}

observed during decoding. Each entry reports the percentage of all observed

w_{t}^{i}

values that fall within the corresponding range.

Model / Dataset	Metric	Ordering	$K{=}1$	$K{=}2$	$K{=}4$	$K{=}8$	$K{=}16$
LLaDA / GSM8K	$\mathrm{Overlap@}K$ ( $\uparrow$ )	$\pi^{conf}$	1.00	0.86	0.82	0.81	0.81
	$\mathrm{Overlap@}K$ ( $\uparrow$ )	$\pi^{l2r}$	0.24	0.36	0.49	0.60	0.67
	$\mathrm{RankDiff@}K$ ( $\downarrow$ )	$\pi^{conf}$	0.00	0.74	1.74	3.27	5.62
	$\mathrm{RankDiff@}K$ ( $\downarrow$ )	$\pi^{l2r}$	15.38	13.57	12.52	12.40	13.03
MMaDA / Flickr30K	$\mathrm{Overlap@}K$ ( $\uparrow$ )	$\pi^{conf}$	1.00	0.73	0.64	0.64	0.69
	$\mathrm{Overlap@}K$ ( $\uparrow$ )	$\pi^{l2r}$	0.04	0.05	0.08	0.11	0.18

Appendix

Table 6: Comparison of confidence-based and left-to-right orderings with the one-token-per-step decoding trajectory. Higher

\mathrm{Overlap@}K

and lower

\mathrm{RankDiff@}K

indicate better alignment. All metrics are averaged over samples and decoding steps.

Benchmark	Method	Acc. (%)	Steps	Acc. gain (%)	Step reduction
ARC-C	Default	52.17	256	–	–
ARC-C	DARD	81.61	35	+56.4	7.2 $\times$
Countdown	Default	29.00	256	–	–
Countdown	DARD	34.51	85	+19.0	3.0 $\times$
GSM8K	Default	73.77	256	–	–
GSM8K	DARD	77.86	48	+5.5	5.3 $\times$

Appendix

Table 7: Comparison of peak-accuracy configurations for the default setting and DARD using LLaDA.

Benchmark	Method	Acc. (%)	Steps	Acc. gain (%)	Step reduction
MMMU-val	Default	20.56	256	–	–
MMMU-val	DARD	22.89	34	+11.3	7.5 $\times$
AI2D	Default	55.25	256	–	–
AI2D	DARD	56.15	37	+1.6	7.0 $\times$
ScienceQA	Default	41.89	256	–	–
ScienceQA	DARD	44.67	25	+6.6	10.3 $\times$

Appendix

Table 8: Comparison of peak-accuracy configurations for the default setting and DARD using MMaDA.

$\tau_{c}$	$\tau_{u}$
$\tau_{c}$	0.5	0.6	0.7	0.8	0.9
0.2	25.2 (32.81)	26.0 (36.76)	24.2 (39.84)	23.8 (43.76)	29.0 (50.11)
0.3	25.8 (32.37)	25.4 (34.54)	27.2 (37.61)	26.2 (40.90)	28.6 (44.82)
0.4	30.0 (34.45)	27.6 (37.93)	28.0 (41.07)	28.2 (44.91)	30.8 (51.75)
0.5	–	30.0 (39.54)	28.2 (45.50)	31.0 (51.85)	32.0 (59.46)
0.6	–	–	31.4 (48.72)	34.4 (56.94)	34.0 (66.76)

Appendix

Table 9: Effects of the threshold hyperparameters

\tau_{c}

and

\tau_{u}

on the speed–quality trade-off of LLaDA on MATH-500. Each cell reports accuracy (%), with the average number of decoding steps shown in gray parentheses.

Method	ARC-C		GSM8K		Countdown
Method	TPS	Acc.	TPS	Acc.	TPS	Acc.
Saber	69.0	72.2	30.5	74.7	24.7	28.9
WINO	25.4	78.9	30.6	77.8	12.1	31.8
DARD	36.4	79.6	34.5	77.2	15.5	34.5

Appendix

Table 10: Comparison of inference throughput and task performance for LLaDA. Higher values are better for all metrics.

Method	AI2D		ScienceQA		Flickr30K
Method	TPS	Acc.	TPS	Acc.	TPS	CIDEr
Saber	11.3	53.3	13.5	44.7	12.2	54.8
WINO	11.4	53.9	14.0	43.5	18.5	57.1
DARD	14.5	55.5	16.0	44.7	18.2	59.2

Appendix

Table 11: Comparison of inference throughput and task performance for MMaDA. Higher values are better for all metrics.

Method	Block length	1-step latency (ms)	GPU memory (GB)
Default	256	137.7	17.21
Saber	256	141.1	17.62
WINO	128	183.9	17.45
WINO	256	229.6	17.96
DARD	128	186.8	17.65
DARD	256	232.0	18.29

Appendix

Table 12: Single-step inference latency and peak GPU memory usage of LLaDA on MATH-500. All methods use a generation length of 256.

Figure 11: Qualitative example illustrating how DARD effectively mitigates error propagation through iterative verification by demoting erroneous tokens, removing them from subsequent conditioning contexts, and progressively identifying tokens whose confidence depends on those errors.

Figure 12: Qualitative comparison of DARD with conventional revocable decoding methods.

Figure 13: Qualitative comparison of DARD with conventional revocable decoding methods.

Figure 14: Qualitative comparison between the decoding processes of conventional revocable decoding and DARD.

Figure 15: Qualitative comparison between the decoding processes of conventional revocable decoding and DARD.

Diffusion Language Model Decoding

Diffusion language model decoding generates text through iterative parallel refinement, balancing fewer denoising steps with accuracy, coherence, and control.

See all 249 papers

Follow the Latent Roadmap: Navigating Revocable Decoding for Diffusion LLMs with Anchor Tokens
Jun 15, 2026Yizhen Yao, Qinglin Zhu, Runcong Zhao +4Diffusion Language Model Inference Diffusion Language Model Decoding
King’s College London · The Chinese University of Hong Kong · The Alan Turing Institute, UK
$R^2$ -dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction
Apr 21, 2026Zhenbang Du, Kejing Xia, Xinrui Zhong +6Diffusion Language Model Inference Diffusion Model Fine-Tuning
Georgia Institute of Technology · NVIDIA
Roll Out and Roll Back: Diffusion LLMs are Their Own Efficiency Teachers
May 16, 2026Fanqin Zeng, Feng Hong, Geng Yu +6Diffusion Model Distillation Diffusion Language Model Decoding
Shanghai Jiao Tong University, Shanghai, China · Apple MLR, Cupertino, CA, USA · Tongji University, Shanghai, China +2

Jun 15, 2026J/K move · Enter open · S save

Follow the Latent Roadmap: Navigating Revocable Decoding for Diffusion LLMs with Anchor Tokens

Yizhen Yao, Qinglin Zhu, Runcong Zhao, Xiangxiang Dai+3

King’s College London · The Chinese University of Hong Kong · The Alan Turing Institute, UK

Diffusion Language Model Inference Diffusion Language Model Decoding Diffusion Language Models Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

Diffusion Large Language Models (dLLMs) offer a promising avenue for parallel generation but face a trade-off between decoding speed and quality. While revocable decoding strategies attempt to mitigate errors by verifying and remasking tokens, they typically operate within a mixed-quality context. This leads to two critical failures: Error Propagation, where new tokens absorb toxic information from erroneous context, and Local Error Reinforcement, where errors mutually reinforce each other to evade detection. To alleviate these challenges, we propose ASRD (Anchor Supervised Revocable Decoding), a training-free framework that operates within the embedding space. ASRD explicitly decouples the decoding context into trusted Anchor Tokens, which are identified via temporal consistency, and uncertain candidates. Leveraging a dynamic Anchor Tokens Cache, we introduce two complementary mechanisms: (1) Anchor-Guided Generation, which injects entropy-weighted anchor signals into masked positions to implicitly rectify attention toward the reliable global skeleton; and (2) Anchor-Perturbed Verification, which applies orthogonal perturbations to uncertain candidate tokens, destabilizing and remasking errors driven by fragile local consensus. Extensive experiments on math and coding benchmarks demonstrate that ASRD outperforms recent remasking baselines, achieving accuracy improvements of up to 6.4% while accelerating inference throughput by up to 7.2

\times

.The code is available at https://github.com/preordinary/ASRD.

Abstract

Diffusion Language Model Decoding

Explore similar work

Follow the Latent Roadmap: Navigating Revocable Decoding for Diffusion LLMs with Anchor Tokens

R2R^2R2-dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction

Roll Out and Roll Back: Diffusion LLMs are Their Own Efficiency Teachers

$R^2$ -dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reduction