Autoregressive (AR) language models generate tokens sequentially, so generating long outputs is costly. Speculative decoding mitigates this by using a lightweight drafter to propose multiple tokens for the target AR model to verify in parallel, while a diffusion-based drafter can produce an entire draft block in one forward pass. In models with AR and diffusion modes, one network both drafts and verifies, but drafting still waits for verification, so the standard schedule needs two sequential model calls per iteration. Existing one-call schedules avoid the wait but incur input size quadratic in the speculative width. Motivated by this efficiency bottleneck, we introduce Packed-SS, a self-speculative decoding schedule that verifies and drafts in a single model call with input size linear in the speculative width, while preserving the target model's distribution. The key idea is to let the diffusion drafter run one call ahead: while verifying the current draft, the model simultaneously constructs the next draft from an older verified prefix and keeps its unresolved suffix for the next iteration. Under a mild monotonicity assumption on the drafter, we further show that Packed-SS requires no more model calls than the standard two-call decoding in the greedy setting. Experiments on Nemotron 3B/8B and SetDLM 1.7B show higher throughput and lower mean wall time than both baselines under greedy decoding.
Figures & tables
Figure 1: Where drafting happens in one call. Outlined boxes are model calls. Linear-SS must finish verifying before it can draft, so an iteration costs two calls. Quadratic-SS fits in one call by drafting a continuation for every place verification might stop, so its input grows quadratically with the speculative width W . Packed-SS also uses one call, building a single draft from the previously verified prefix alone. The right column gives total draft and verifier prediction counts per iteration, including predictions that may later be discarded.
Figure 2: One iteration of greedy Packed-SS. Columns are token positions. The old draft token dt−1(i) , the verifier prediction vt(i) , and the new proposal token zt(i) are in the same position. A single forward pass takes the verified prefix Pt−1 and draft Dt−1 as input and produces the two middle rows: the AR verifier’s next-token predictions vt(i) , conditioned on Pt−1 and dt−1(1),…,dt−1(i−1) , and the diffusion drafter’s proposal Zt=(zt(i))i=1W , conditioned on Pt−1 alone. Here, one matching draft token ( ht=1 ) and the correction token vt(2) yield At=2 output tokens. Proposals at these two positions are discarded, leaving Dt=(zt(3),zt(4)) as the next draft.
Figure 3: Greedy decoding efficiency and accuracy. Efficiency metrics are ratios to Linear-SS; accuracy differences are in percentage points. Error bars show 95% paired bootstrap confidence intervals. Native is the released non-speculative SetDLM diffusion decoder, shown for reference. WritingBench measures speed only.
Figure 4: Efficiency and accuracy at temperature 1. Efficiency metrics are ratios to Linear-SS, accuracy differences are in percentage points. Error bars show 95% paired bootstrap confidence intervals. WritingBench measures speed only.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Sizes and indices
t
decoding iteration, starting at 1
i,j
token position: relative in LTR, fixed in any-order decoding
r,s
rank in the verification order (any-order setting)
W
speculative width: the fixed number of future positions proposed by the LTR drafter, W≥1 , and the size of the active window in the any-order setting. The schedule and Theorem 3.2 are stated in W alone
B
Nemotron’s block size, B=W+1 : the last prefix token and W draft positions
ut
number of unresolved positions in the active window, ut=∣Ut∣≤W (any-order setting)
Appendix
Table 1: Symbols. Sizes and indices, decoder state, and distributions.
Evaluated uncached rows per iteration
SetDLM maximum
Method
NFE/iteration
Nemotron
W=8
W=16
SetDLM general
Linear-SS
2
2B
25
49
p+3ut
Quadratic-SS
1
B(B+1)
81
289
p+2ut+ut2
Packed-SS (Ours)
1
2B
25
49
p+3ut
Appendix
Table 2: Execution cost per iteration with cached prompt context. Evaluated uncached rows count token representations processed by the transformer, including duplicates and evaluated padding, summed across calls. Nemotron uses block size B=W+1 . In SetDLM, ut≤W positions remain unresolved in the active window, and p=∣Nt∣∈{0,1} counts the correction token processed in this call; maximum row counts take ut=W and p=1 . Prompt rows are processed once and depend on prompt length.
Method
B
Tokens/s
Rows/call
Wall (s)
Total NFE
Tokens
Accuracy (%)
Linear-SS
8
94.9
8
2.824
98.2
267.9
100
Linear-SS
16
113.4
16
2.448
84.6
277.5
100
Linear-SS
32
121.9
32
2.237
77.4
272.8
100
Packed-SS
8
94.8
16
2.798
89.0
265.2
100
Packed-SS
16
123.5
32
2.315
73.3
285.9
100
Packed-SS
32
128.5
64
2.333
75.4
299.7
90
Appendix
Table 3: Nemotron 8B GSM8K block-size probe. Each row uses the same ten questions. Tokens/s is calculated from the displayed means. Wall is mean total decoder time per response, including prompt processing; total NFE and output-token counts are also per-response means. Accuracy is reported for comparison. Rows/call is the maximum number of input rows per call after initialization.
Method
Tokens
Total NFE
Model (s)
Non-model (s)
Wall (s)
Tokens/s
Accuracy (%)
Linear-SS
272.2
55.04
1.561
0.137
1.698
160.32
85.90
Quadratic-SS
273.3
54.44
1.781
0.343
2.123
128.72
86.35
Packed-SS
274.2
48.44
1.570
0.179
1.749
156.78
85.82
Appendix
Table 4: Nemotron 8B GSM8K with proposal-only LoRA ( n=1,319 ). Means; model execution time and non-model decoder time sum to inclusive decoder wall time. Emitted length and throughput accompany latency, and accuracy is reported for comparison.
Method
Query rows
Selected
Evaluated
Base ms
LoRA ms
Increase
Linear-SS
32+32
32
32
54.41
56.77
4.3%
Quadratic-SS
72
64
72
28.90
32.69
13.1%
Packed-SS
64
32
64
28.58
32.37
13.3%
Appendix
Table 5: Nemotron 8B GSM8K model execution time per iteration. Linear’s iteration contains both calls. “Selected” counts rows receiving adapter updates; “evaluated” counts rows for which those updates are computed.
Figure 5: Normalized Nemotron timing decomposition on GSM8K and MATH-500. Greedy decoding with a 1,024-token cap. Each task’s component times are divided by that task’s Linear-SS total, then the two tasks are averaged with equal weight within each model scale. The dashed line marks Linear-SS’s total. Non-model decoder time covers draft token construction, selection, and state updates.
Task
Method
Tokens/s
Tokens/ NFE
Time/NFE (ms)
Accuracy (%)
Tokens
Wall (s)
NFE
Nemotron 3B
MBPP
Linear-SS
173.46
3.757
21.658
74.87
65.8
0.379
17.50
Quadratic-SS
164.38
4.288
26.089
74.34
65.9
0.401
15.36
Packed-SS
181.10
4.312
23.812
74.60
67.0
0.370
15.53
HumanEval
Linear-SS
237.45
5.158
21.722
79.27
90.5
0.381
17.55
Quadratic-SS
211.48
5.536
26.177
79.88
91.1
0.431
16.46
Appendix
Table 6: Nemotron absolute measurements at temperature 0. Tokens, wall time, and NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each model and task.
Task
Method
Tokens/s
Tokens/ NFE
Time/NFE (ms)
Accuracy (%)
Tokens
Wall (s)
NFE
Nemotron 3B
MBPP
Linear-SS
120.38
2.508
20.834
68.14
102.7
0.853
40.95
Quadratic-SS
129.01
3.328
25.800
69.69
108.7
0.843
32.67
Packed-SS
125.06
2.933
23.454
68.78
101.8
0.814
34.72
HumanEval
Linear-SS
169.58
3.526
20.792
66.89
96.9
0.571
27.49
Quadratic-SS
170.63
4.348
25.484
67.29
95.6
0.561
22.00
Appendix
Table 7: Nemotron absolute measurements at temperature 1. Tokens, wall time, and NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each model and task.
Window
Method
Tokens/s
Tokens/ NFE
Time/NFE (ms)
Accuracy (%)
Tokens
Wall (s)
NFE
W=8
Linear-SS
94.93
2.610
27.496
68.39
257.2
2.709
98.53
Quadratic-SS
86.25
3.414
39.583
68.39
257.1
2.981
75.31
Packed-SS
100.60
2.842
28.247
68.39
257.2
2.556
90.49
Native
77.21
2.119
27.442
59.67
252.4
3.269
119.11
W=16
Linear-SS
98.85
2.792
28.243
66.57
259.9
2.629
93.09
Quadratic-SS
41.60
4.074
97.934
66.57
260.4
6.260
63.92
Appendix
Table 8: SetDLM absolute measurements on GSM8K. LTR verification and native decoding on all 1,319 test questions. Tokens, wall time, and NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each window.
Window
Method
Tokens/s
Tokens/ NFE
Time/NFE (ms)
Accuracy (%)
Tokens
Wall (s)
NFE
LTR verification and native decoding
W=8
Linear-SS
94.93
2.610
27.496
68.39
257.2
2.709
98.53
Quadratic-SS
86.25
3.414
39.583
68.39
257.1
2.981
75.31
Packed-SS
100.60
2.842
28.247
68.39
257.2
2.556
90.49
Native
77.21
2.119
27.442
59.67
252.4
3.269
119.11
W=16
Linear-SS
98.85
2.792
28.243
66.57
259.9
2.629
93.09
Appendix
Table 9: SetDLM measurements on GSM8K. All 1,319 test questions. Top: LTR verification and native decoding; bottom: confidence-ordered verification. Tokens, inclusive wall time, and total NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each window and order.
Setting
Baseline
Δ wall (s) [95% CI]
Throughput ratio [95% CI]
W=8 LTR
Linear-SS
−0.1529 [ −0.1588 , −0.1470 ]
1.060 [1.058, 1.062]
Quadratic-SS
−0.4248 [ −0.4377 , −0.4124 ]
1.166 [1.164, 1.169]
W=8 Confidence
Linear-SS
−0.2581 [ −0.2953 , −0.2217 ]
1.102 [1.098, 1.107]
Quadratic-SS
−0.3521 [ −0.3836 , −0.3198 ]
1.140 [1.136, 1.143]
W=16 LTR
Linear-SS
−0.0382 [ −0.0432 , −0.0330 ]
1.015 [1.013, 1.017]
Quadratic-SS
−3.6686 [ −3.7869 , −3.5590 ]
2.411 [2.399, 2.424]
Appendix
Table 10: Paired SetDLM efficiency comparisons. Wall-time differences are Packed-SS minus baseline; throughput ratios are Packed-SS/baseline. Brackets give paired 95% confidence intervals.
Setting
Packed-SS − Linear-SS
Packed-SS − Quadratic-SS
Linear-SS − Quadratic-SS
Temperature 0
3B MBPP
-0.26 [-1.32, +0.79]
+0.26 [-0.79, +1.32]
+0.53 [-0.53, +1.59]
3B HumanEval
-1.22 [-3.66, +1.22]
-1.83 [-4.27, +0.00]
-0.61 [-1.83, +0.00]
3B GSM8K
-0.30 [-1.14, +0.53]
-0.15 [-0.99, +0.68]
+0.15 [-0.68, +0.99]
3B MATH-500
+3.00 [+0.80, +5.20]
+0.20 [-1.60, +2.00]
-2.80 [-4.60, -1.00]
8B MBPP
-0.26 [-1.59, +0.79]
+0.00 [-0.79, +0.79]
+0.26 [-0.53, +1.06]
Appendix
Table 11: Paired Nemotron accuracy differences.
Setting
Baseline
Δ wall (s) [95% CI]
Throughput ratio [95% CI]
3B MBPP
Linear-SS
-0.0092 [-0.0268, +0.0123]
1.044 [1.007, 1.075]
Quadratic-SS
-0.0308 [-0.0563, -0.0027]
1.102 [1.079, 1.127]
3B HumanEval
Linear-SS
-0.0207 [-0.0341, -0.0100]
1.052 [1.033, 1.072]
Quadratic-SS
-0.0703 [-0.0886, -0.0539]
1.181 [1.149, 1.213]
3B GSM8K
Linear-SS
-0.0434 [-0.0605, -0.0243]
1.061 [1.049, 1.071]
Quadratic-SS
-0.1003 [-0.1187, -0.0798]
1.114 [1.103, 1.125]
Appendix
Table 12: Paired Nemotron efficiency comparisons at temperature 0.
Setting
Baseline
Δ wall (s) [95% CI]
Throughput ratio [95% CI]
3B MBPP
Linear-SS
-0.0389 [-0.0678, -0.0107]
1.039 [1.026, 1.052]
Quadratic-SS
-0.0287 [-0.0832, +0.0248]
0.969 [0.956, 0.984]
3B HumanEval
Linear-SS
-0.0076 [-0.0250, +0.0093]
1.021 [1.005, 1.038]
Quadratic-SS
+0.0033 [-0.0121, +0.0189]
1.015 [0.997, 1.034]
3B GSM8K
Linear-SS
-0.0227 [-0.0832, +0.0387]
1.020 [1.001, 1.037]
Quadratic-SS
+0.0000 [-0.0640, +0.0652]
0.996 [0.979, 1.013]
Appendix
Table 13: Paired Nemotron efficiency comparisons at temperature 1.
Speculative decoding accelerates autoregressive large language model inference by drafting multiple tokens and verifying them in a single target-model forward pass. Recent diffusion-based drafters generate an entire block of tokens in parallel but usually commit to a single draft sequence per verification: once the first mismatch occurs, all subsequent draft tokens are discarded, resulting in a limited acceptance rate. Naively batching more draft candidate sequences only introduces a marginal improvement, as redundant or poorly placed branches increase the cost of drafting and verification without proportionally increasing the number of accepted tokens. We propose D^2SD, a dual diffusion draft speculative decoding framework that organizes candidates into a confidence-guided prefix tree, where the first diffusion drafter generates a block along with per-position confidence scores that are used to identify the most likely rejection boundary and select the top-K prefix ranges for recovery; the second variable-prefix diffusion drafter re-anchors at each selected prefix and proposes alternative continuations in one batched pass; the resulting shared-prefix candidates are jointly verified via cascade attention. Empirically, D^2SD shows clear improvements over both the underlying diffusion approach and strong autoregressive speculative decoding baselines.
Liyuan Zhang, Jiarui Zhang, Jinwei Yao +6
Peking University · Tsinghua University · HKUST +2
Speculative decoding accelerates autoregressive inference by verifying multiple draft tokens in a single target forward pass. However, as the context grows, existing state-of-the-art drafters become increasingly expensive, eroding the very efficiency advantage they are designed to provide. We argue that this scaling is unnecessary. A standalone language model must grow with its prefix because it is solely responsible for every token it produces. A drafter, by contrast, only proposes candidates; the target catches and corrects every error before any token is committed. The drafter's decoding cost can therefore be made entirely independent of the prefix length. We introduce LongSpark, a block-diffusion drafter that achieves this by extracting fixed-size, multiscale views from the target's verification pass, thereby eliminating the need for a growing persistent state. Extensive evaluations demonstrate that LongSpark achieves state-of-the-art end-to-end efficiency across multiple model scales and realistic serving conditions. Notably, it delivers the lowest time-per-output-token on long-context tasks while reducing the drafter's context state by several orders of magnitude.
Autoregressive decoding is bottlenecked by its sequential nature. Speculative decoding has become a standard way to accelerate inference by using a fast draft model to predict upcoming tokens from a slower target model, and then verifying them in parallel with a single target model forward pass. However, speculative decoding itself relies on a sequential dependence between speculation and verification. We introduce speculative speculative decoding (SSD) to parallelize these operations. While a verification is ongoing, the draft model predicts likely verification outcomes and prepares speculations pre-emptively for them. If the actual verification outcome is then in the predicted set, a speculation can be returned immediately, eliminating drafting overhead entirely. We identify three key challenges presented by speculative speculative decoding, and suggest principled methods to solve each. The result is Saguaro, an optimized SSD algorithm. Our implementation is on average 30% faster than optimized speculative decoding baselines and up to 5x faster than autoregressive decoding with open source inference engines.
Tanishq Kumar, Tri Dao, Avner May
Department of Computer Science, Stanford University · Department of Computer Science, Princeton University · Together AI