Diffusion Large Language Models (dLLMs) have emerged as a compelling paradigm for language modeling, offering the unique potential for highly efficient inference via multi-token parallel decoding. However, unlocking this potential remains challenging. High-quality generation typically defaults to one-token-per-step static decoding, while existing parallel algorithms rely on heuristic confidence thresholds, introducing a fragile trade-off between decoding efficiency and generation quality. To overcome this, we introduce Free Draft-and-Verification (FreeDave), a novel training-free and model-free fast decoding algorithm. FreeDave capitalizes on the inherent properties of dLLMs by treating future token predictions as ``free drafts'' and rigorously self-verifying them in subsequent forward passes. Theoretically, FreeDave searches for a shorter decoding path that is guaranteed to reproduce the exact sequence generated by static decoding, achieving algorithmically lossless acceleration. With a sufficiently large draft budget and a verification frontier that does not retreat, it can use the fewest possible model forward passes among feasible paths. Extensive evaluations on math reasoning and code generation benchmarks demonstrate that FreeDave can boost the number of generated tokens per model forward (TPF) by up to 5.48× and tokens per second (TPS) by up to 3.94×, while robustly maintaining generation quality and avoiding the task-specific performance degradation faced by heuristic methods. Code is available at https://github.com/cychomatica/FreeDave.
Figures & tables
Figure 1: The overview of FreeDave decoding. At the current step, the dLLM outputs the estimated distribution of the sequence, from which the remasking scheduler constructs multiple draft candidates. At the subsequent step, the same dLLM processes these candidates in parallel, verifying their validity via sequence matching. Then the resulting output distributions are reused to sample the next set of drafts. Because the logits needed for future drafts are naturally produced as byproducts of the verification step, FreeDave creates a continuous, self-verifying parallel decoding loop that requires no auxiliary models or extra forward passes.
Figure 2: Performance and efficiency comparison of dLLMs with different decoding strategies on different benchmarks. More detailed results can be found in Appendix D .
Model
MATH500
HumanEval
Rsequence
Rtoken
Rsequence
Rtoken
LLaDA-8B-Instruct
500/500
312047/312047
164/164
32977/32977
Dream-7B-Instruct
500/500
178489/178489
164/164
13136/13136
TraDo-4B-Instruct
500/500
290262/290262
164/164
24041/24041
TraDo-8B-Instruct
500/500
296300/296300
164/164
23938/23938
Table 1: On MATH500 and HumanEval, we evaluate the sequence-level Rsequence and token-level match rate Rtoken between static decoding and FreeDave decoding under determinism debug mode.
Figure 3: Evaluation of TraDo-4B-Instruct on HumanEval with different draft steps d . Specifically, when d=1 , the decoding algorithm falls back to static decoding.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Decoding Strategy
Acc (%) ↑
TPF ↑
TPS ↑
MATH500 (4-shot)
Static
37.40
1.00
37.05
Static FreeDave (Ours)
36.80 (-0.60)
2.45 (2.45 × )
74.19 (2.00 × )
Threshold
37.40 (+0.00)
2.61 (2.61 × )
85.89 (2.32 × )
Threshold FreeDave (Ours)
36.80 (-0.60)
4.40 (4.40 × )
119.47 (3.22 × )
GSM8K (5-shot)
Static
78.77
1.00
34.23
Static FreeDave (Ours)
78.62 (-0.15)
2.50 (2.50 × )
68.52 (2.00 × )
Appendix
Table 2: LLaDA-8B-Instruct results on math reasoning and code generation benchmarks.
Benchmark
Decoding Strategy
Acc (%) ↑
TPF ↑
TPS ↑
MATH500 (4-shot)
Static
44.80
1.00
62.51
Static FreeDave (Ours)
46.20 (+1.40)
2.45 (2.45 × )
115.16 (1.84 × )
Threshold
44.60 (-0.20)
2.35 (2.35 × )
125.59 (2.01 × )
Threshold FreeDave (Ours)
46.40 (+1.60)
3.97 (3.97 × )
163.34 (2.61 × )
GSM8K (5-shot)
Static
79.91
1.00
58.86
Static FreeDave (Ours)
79.23 (-0.68)
2.49 (2.49 × )
107.58 (1.83 × )
Appendix
Table 3: Dream-v0-Instruct-7B results on math reasoning and code generation benchmarks.
Benchmark
Decoding Strategy
Acc (%) ↑
TPF ↑
TPS ↑
MATH500 (4-shot)
Static
64.80
0.80
28.63
Static FreeDave (Ours)
66.40 (+1.60)
1.68 (2.10 × )
54.69 (1.91 × )
Threshold
65.60 (+0.80)
1.51 (1.89 × )
53.71 (1.88 × )
Threshold FreeDave (Ours)
65.40 (+0.60)
1.94 (2.42 × )
61.65 (2.15 × )
GSM8K (5-shot)
Static
90.67
0.79
27.69
Static FreeDave (Ours)
90.52 (-0.15)
1.64 (2.08 × )
52.97 (1.91 × )
Appendix
Table 4: TraDo-4B-Instruct results on math reasoning and code generation benchmarks.
Benchmark
Decoding Strategy
Acc (%) ↑
TPF ↑
TPS ↑
MATH500 (4-shot)
Static
69.20
0.80
26.29
Static FreeDave (Ours)
69.40 (+0.20)
1.64 (2.05 × )
51.68 (1.97 × )
Threshold
69.40 (+0.20)
1.41 (1.76 × )
46.43 (1.77 × )
Threshold FreeDave (Ours)
69.20 (+0.00)
1.90 (2.37 × )
58.67 (2.23 × )
GSM8K (5-shot)
Static
93.03
0.80
25.45
Static FreeDave (Ours)
92.87 (-0.16)
1.62 (2.02 × )
50.47 (1.98 × )
Appendix
Table 5: TraDo-8B-Instruct results on math reasoning and code generation benchmarks.
Figure 4: Ablation details
Benchmark
Model
Decoding Strategy
Acc (%) ↑
TPF ↑
TPS ↑
GPQA (0-shot)
LLaDA-8B-Instruct
Static
23.44
1.00
43.01
Static FreeDave (Ours)
21.43 (-2.01)
2.37 (2.37 × )
84.50 (1.96 × )
Threshold
23.21 (-0.23)
2.13 (2.13 × )
85.16 (1.98 × )
Threshold FreeDave (Ours)
21.65 (-1.79)
3.67 (3.67 × )
123.06 (2.86 × )
Dream-7B-Instruct
Static
27.23
1.00
69.53
Static FreeDave (Ours)
26.79 (-0.44)
2.44 (2.44 × )
130.34 (1.87 × )
Appendix
Table 6: Different models on Q&A benchmarks.
Generation Length
Decoding Strategy
Acc (%) ↑
TPF ↑
TPS ↑
GPU Peak Memory (MiB)
Allocated
Reserved
1024
Static
44.80
1.00
62.51
16245
16746
Static FreeDave (Ours)
46.20
2.45
115.16
16276
16752
Threshold
44.60
2.35
125.59
16247
16746
Threshold FreeDave (Ours)
46.40
3.97
163.34
16277
16752
2048
Static
44.80
1.00
50.21
16929
17624
Appendix
Table 7: Results of Dream-7B-Instruct on MATH500 under different generation lengths.
Figure 5: GPU peak memory and FreeDave memory overhead for Dream-7B-Instruct on MATH500 under different generation lengths.
Figure 6: FreeDave with block tree attention tailored for block diffusion.
Figure 7: The equivalence between FreeDave Eager Mode and the variant of static decoding with delayed cache update.
Diffusion large language models (dLLMs) offer a promising alternative to autoregressive generation by decoding multiple tokens in parallel through iterative denoising. However, increasing decoding parallelism often degrades generation quality, as early errors can contaminate later contexts. Revocable decoding mitigates this issue by re-evaluating decoded tokens and remasking unreliable ones, but existing methods overlook that unreliable tokens may also corrupt the verification context itself. We identify this failure mode and propose Dependency-Aware Revocable Decoding (DARD), a training-free framework that separates tokens into masked, candidate, and unmasked states. DARD verifies candidate tokens using a selective context that excludes less reliable tokens and adaptively regulates their influence on subsequent decoding. Experiments across 12 textual and multimodal benchmarks on 3 open-source dLLMs show that DARD consistently improves the speed-quality Pareto frontier over recent revocable decoding methods, achieving a 2.71× speedup and a 4.35-point CIDEr score gain over Saber on Flickr30K.
Wooje Park, Insu Lee, Minyoung Noh +4
Seoul National University · Sungkyunkwan University
Diffusion large language models (dLLMs) generate text by iteratively denoising masked token sequences. Although dLLMs can predict all masked positions in parallel within each step, the large number of denoising iterations still makes inference expensive. This cost can be reduced spatially by unmasking multiple tokens per step, or temporally by collapsing multiple denoising steps into one verification call. We propose Parallel Speculative Decoding (PSD), a training-free framework that jointly improves inference along both axes. Using the confidence scores from a single forward pass, PSD selects positions to unmask via a configurable, adaptive unmasking policy and constructs multi-depth speculative drafts without extra model calls. A final batched verification pass then applies hierarchical acceptance, keeping the deepest draft that remains consistent with the updated predictions. Experiments on three dLLMs across reasoning and code generation tasks show that PSD achieves favorable trade-offs between inference efficiency and generation quality, reaching up to 5.5× tokens per forward pass with accuracy comparable to greedy decoding.
Shengyin Sun, Yiming Li, Renxi Liu +7
Department of Computer Science City University of Hong Kong · Huawei Technologies · City University of Hong Kong
Diffusion Large Language Models (dLLMs) offer a compelling paradigm for natural language generation, leveraging parallel decoding and bidirectional attention to achieve superior global coherence compared to autoregressive models. While recent works have accelerated inference via KV cache reuse or heuristic decoding, they overlook the intrinsic inefficiencies within the block-wise diffusion process. Specifically, they suffer from spatial redundancy by modeling informative-sparse suffix regions uniformly and temporal inefficiency by applying fixed denoising schedules across all the decoding process. To address this, we propose Streaming-dLLM, a training-free framework that streamlines inference across both spatial and temporal dimensions. Spatially, we introduce attenuation guided suffix modeling to approximate the full context by pruning redundant mask tokens. Temporally, we employ a dynamic confidence aware strategy with an early exit mechanism, allowing the model to skip unnecessary iterations for converged tokens. Extensive experiments show that Streaming-dLLM achieves up to 68.2X speedup while maintaining generation quality, highlighting its effectiveness in diffusion decoding. The code is available at https://github.com/xiaoshideta/Streaming-dLLM.
Zhongyu Xiao, Zhiwei Hao, Jianyuan Guo +4
School of information and Electronics, Beijing Institute of Technology, Beijing · Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computer Science, Wuhan University, Wuhan +2