Speculative-diffusion decoding uses a small parallel drafter model to generate b draft tokens in one forward pass, which then undergo parallel verification via a larger target model. Faster inference can be achieved when multiple draft tokens pass verification simultaneously. Recent parallel draft architectures (DFlash, DSpark) improve draft quality by conditioning on rich features computed by the target, through a mechanism known as feature encoding. However, standard feature encoding mechanisms have three limitations: (1) they are lossy, applying encoding projections that are destructive to the underlying target feature-space via a discovered effect known as residual mixing, (2) they are memory-less, with no record of previously processed proposals, and (3) they utilize features sampled uniformly from across the target layers, omitting important hidden states. To address these issues, we propose Mixture-of-Features Attention (MoFA), a lightweight feature-encoding approach equipped with a working memory that encodes previously rejected proposals. Leveraging MoFA, rejection memory, and a set of further enhancements, including 2-tap causal convolutions, prefix-only cross-attention, and non-uniform target layer selection, the final architecture, DNeo, achieves 1.3x faster generation speed than SoTA (DFlash-2) without impacting drafter footprint or generation quality. DNeo marks a new state of the art in high quality, fast LLM inference.
Figures & tables
Figure 1: H100 Inference speeds, with Qwen3-4B ( b=15 ) across six benchmarks (see Section 5.1 ), and batch-sizes B . DNeo dominates the speed Pareto frontier relative to autoregressive (EAGLE-3) algorithms and SoTA parallel algorithms (i.e., DFlash, DFlash-2, DSpark).
Figure 2: Draft quality comparison . Tokens per forward ( τ ) over the test-set (Section 5.1 ), using official SGLang inference implementations. DNeo dominates existing methods in draft quality.
Method
Causal Head
Rejection Attention
Rejection Memory ( ours )
Prefix-only Attention ( ours )
2-Tap Convolutions
Non-uniform Target Layer Selection ( ours )
Structure Preserving Encoding
DFlash
✗
✗
✗
✗
✗
✗
✗
DFlash-2
✓
✗
✗
✗
✓
✗
✗
DFlare
✗
✗
✗
✗
✗
✗
✓
DFlow
✓
✓
✗
✗
✗
✗
✗
DSpark
✓
✗
✗
✗
✗
✗
✗
DNeo ( ours )
✓
✓
✓
✓
✓
✓
✓
Table 1: Parallel drafter architecture comparison . DNeo combines effective components from prior work (causal head, 2-tap convolutions) while also utilizing novel architectural innovations (rejection memory, prefix-only attention, non-uniform target layer selection).
Figure 3: DNeo architecture . Target model ( left ) produces residual streams for the clean-prefix, and rejected positions. Streams are aggregated into prefix hidden states and rejection states (A) . The most useful states are kept with the DNeo feature selector (B) (Section 4.5 ), then passed to the MoFA gate (C) (Section 4.1 ) to obtain the encoded prefix , and rejection memory (D) (Section 4.3 ). Parallel draft blocks perform cross-attention to the target features (E) , utilize 2-tap convolutions (F) , and finally apply the fused DNeo head (G) (Section 4.2 ) to produce the new draft .
Figure 4: DNeo Attention mask. Mask assumes block-size b=3 across three drafting rounds. DNeo omits bidirectional self-attention over the masked block, a design we call prefix-only attention (Section 4.4 ), and performs cross-attention to the rejected states ({\color[rgb]{1,0,0}{\hat{f}}_{3},\dots,{\hat{f}}_{6}}) . he number of rejection features per round varies based on the prior round’s acceptance length.
Model
Method
MATH
CODE
KNOWLEDGE
CHAT
GSM8K
MATH-500
HumanEval
MBPP
GPQA
MT-Bench
Temperature = 1
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Speedup
τ
Q3-4B
EAGLE-3 † (16)
1.95×
4.00
2.22×
4.57
1.92×
3.93
1.85×
3.80
1.65×
3.39
1.32×
2.72
DFlash-2 (16)
2.65×
3.66
2.93×
4.05
2.56×
3.53
2.60×
3.60
2.06×
2.85
1.93×
2.68
DFlash (16)
2.65×
3.55
2.88×
3.85
2.62×
3.52
2.64×
3.54
2.00×
2.68
1.90×
2.55
DSpark (16)
2.59×
3.87
2.92×
4.36
2.58×
3.85
2.65×
3.96
2.01×
3.00
1.87×
2.80
Table 2: Main speculative decoding results . Bold marks the best speedup among the listed methods; parentheses mark the best TPF ( τ ). All models are trained with the full training setup (Section 5.1 ), evaluated under identical conditions. ( † EAGLE-3 uses official trained weights for Qwen3-4B [ Yang et al., 2025 ] ).
SGLang
Transformers
Method
B=1
B=4
B=16
B=32
B=64
B=1
B=4
B=16
B=32
B=64
AR
234
890
3,310
5,965
10,335
124
364
733
900
1,023
EAGLE-3
442
1,483
4,409
6,576
8,446
174
547
1,124
1,352
1,457
DFlash
589
1,964
5,770
7,797
8,834
269
762
1,223
1,388
1,462
DFlash-2
590
2,023
5,781
7,971
9,318
272
743
1,303
1,475
1,511
DSpark
586
1,918
5,485
7,435
9,077
261
706
1,278
1,404
1,460
Table 3: Decode throughput . (tokens/s) on 1xH100. verifier and drafter temperature 1, b=16 . Bold marks the best method per column. All entries use the same six benchmarks. B denotes submitted concurrency.
Architecture
Train τ ( 100,000 prompts ) ↑
Δτ
DNeo ( c=7 )
3.112
—
− MoFA (replace with Fc )
2.461
−0.651
− DFlash-2 convolutions
2.780
−0.331
− Rejection memory
2.930
−0.182
− DNeo head (replace with DSpark, r=256 )
2.895
−0.217
Table 4: Subtractive component ablation . Each component is removed independently from DNeo, not cumulatively.
Architecture
Train TPF (50K prompts) ↑
DFlash-2 ( Fc )
1.938
DFlash-2 (Restricted Fc )
2.361
DNeo (MoFA)
2.526
DNeo (Restricted Fc )
2.517
Table 5: Feature encoder ablation . Isolating the effect of residual mixing .
Architecture
Train TPF (100K prompts) ↑
DSpark + DFlash-2 Head
2.217
DSpark + DSpark Head
2.273
DSpark + DNeo Head ( ours )
2.361
Table 6: Causal head ablation . Fused DNeo vs. DFlash-2, and DSpark heads (all heads use projection rank r=256 ).
Architecture
Train τ↑
Δτ
DNeo Rejection memory ( ours )
3.112
+0.189
DFlow + Rejection memory ( ours )
3.078
+0.155
DFlow fusion ( MLP , dMLP=512 )
2.953
+0.030
DFlow fusion ( no MLP )
2.932
+0.009
No Rejection memory/fusion ( memory-less baseline )
2.923
−
Table 7: Rejection attention ablation . Gain in τ reported relative to memory-less DNeo baseline.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: DNeo, DFlash and DFlash-2 train-time TPF scaling over 1.6M prompts. DNeo exhibts significant convergence advantages ( 10× faster and cheaper than DFlash).
Figure 6: MoFA gates learned allocation after training (100,000 prompts). More attention mass is allocated to later target layers, a pattern that is consistent across the five trained MoFA gates (one for each draft layer).
Architecture
Train TPF ( 100,000 prompts ) ↑
Δ TPF
DFlash
2.072
—
+ DSpark head ( r=256 )
2.132
+0.060
+ dense MoFA conditioning ( c=n )
2.478
+0.346
+ DNeo MoFA Protocol ( c=5 )
2.505
+0.027
+ per-layer MoFA routing
2.505
+0.001
+ dense DFlash-2 score head
2.590
+0.085
Appendix
Table 8: Full additive architecture ablation. Upgrading from DFlash to DNeo. Training TPF is measured at 100,000 prompts. DNeo gains almost a full token ( 0.989 tokens per forward) over DFlash.
Attention pattern
No convolutions
With convolutions
Convolution gain
Full masked-token self-attention
2.620
2.855
+0.236τ
Prefix-only attention
2.646
2.859
+0.213τ
Prefix-only gain
+0.027τ
+0.004τ
—
Appendix
Table 9: Prefix only attention comparison. Ablation of masked-token self-attention and DFlash-2 convolutions at 100,000 training prompts.
Figure 7: DNeo protocol vs uniform selection. Feature selectors ablated over fixed DSpark architecture, w.r.t selected layers c .
Figure 8: SGLang latency breakdown on 1xH100. block-size =16 , precision bf 16 , for all methods, excludes prefill.
Target-conditioned diffusion drafters predict candidate tokens in parallel, yet each new block must wait for updated target-model hidden states. This forces drafting and verification to run sequentially, limiting speedup even though each block is generated in one pass. We introduce D3D, a standalone diffusion drafter that removes this inter-round dependency. D3D combines blockwise denoising with auxiliary autoregressive supervision to achieve long accepted prefixes without target hidden states. This independence lets D3D draft successor blocks during verification in one packed pass, enabling asynchronous speculative decoding. We progressively prune layers according to their effect on acceptance length to reduce the cost of packed drafting. Removing five of 36 layers retains over 99% of mean acceptance length across three benchmarks with Qwen3-8B as the target. Across seven benchmarks, D3D averages 5.00× and 5.71× decoding speedup with Qwen3-8B and Qwen3-32B targets, compared with 4.03× and 3.67× for DFlash. On GSM8K with Qwen3-32B, speedup reaches 7.48×. Code is publicly available at https://github.com/fudan-generative-vision/D3D.
Baoyou Chen, Ruiqiao Mei, Guancong Lin +4
Shanghai Innovation Institute · Shanghai Academy of AI for Science · Fudan University +2
Speculative decoding accelerates large language model inference with a small drafter, whose per-guess acceptance rate determines the achievable speedup. Feature-level drafters share a design choice: every token, easy or hard, is drafted under a single forward pass. Many of the tokens the target rejects are ones the drafter is itself unsure about yet the target answers decisively. To address this, we introduce Parallel Reconsideration: guided by its own uncertainty, the drafter reconsiders its hesitant guesses with extra passes that run in parallel with the one-pass draft, adding negligible latency. A co-designed training recipe and serving-time drafting algorithm append these reconsidered guesses to the original draft tree as refined candidates. Across models and benchmarks, on average, Parallel Reconsideration improves over the baselines by 5.3% to 10.2% on acceptance length and by 3.5% to 9.2% on speedup.
Diffusion large language models (dLLMs) have shown advantages in text generation, particularly due to their inherent ability for parallel decoding. However, constrained by the quality--speed trade-off, existing inference solutions adopt conservative parallel strategies, leaving substantial efficiency potential underexplored. A core challenge is that parallel decoding assumes each position can be filled independently, but tokens are often semantically coupled. Thus, the correct choice at one position constrains valid choices at others. Without modeling these positional dependencies, parallel strategies produce deteriorated outputs. Motivated by this insight, we propose DAWN, a training-free, dependency-aware decoding method for fast dLLM inference. DAWN extracts dependency relations and leverages two key observations: (1) masked positions dependent on unmasked certain positions become more reliable; (2) coupled masked positions strongly influence each other's predictions. Given those findings, DAWN leverages a dependency graph to select more reliable unmasking positions at each iteration, achieving high parallelism with negligible loss in generation quality. Extensive experiments across multiple models and datasets demonstrate that DAWN speedups the inference by 1.80-8.06x over baselines while preserving the generation quality. Code is released at https://github.com/lizhuo-luo/DAWN.
Lizhuo Luo, Zhuoran Shi, Jiajun Luo +4
Nanyang Technological University · AUMOVIO-NTU Corporate Lab, Nanyang Technological University · Southern University of Science and Technology +2