One Call Ahead: Packed Self-Speculative Decoding with Diffusion Drafting

Yuxiao Chen; Changxiao Cai

cs.LGOct 24, 2026

One Call Ahead: Packed Self-Speculative Decoding with Diffusion Drafting

Authors: Yuxiao Chen, Changxiao Cai

Organizations: Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, USA

Abstract

Autoregressive (AR) language models generate tokens sequentially, so generating long outputs is costly. Speculative decoding mitigates this by using a lightweight drafter to propose multiple tokens for the target AR model to verify in parallel, while a diffusion-based drafter can produce an entire draft block in one forward pass. In models with AR and diffusion modes, one network both drafts and verifies, but drafting still waits for verification, so the standard schedule needs two sequential model calls per iteration. Existing one-call schedules avoid the wait but incur input size quadratic in the speculative width. Motivated by this efficiency bottleneck, we introduce Packed-SS, a self-speculative decoding schedule that verifies and drafts in a single model call with input size linear in the speculative width, while preserving the target model's distribution. The key idea is to let the diffusion drafter run one call ahead: while verifying the current draft, the model simultaneously constructs the next draft from an older verified prefix and keeps its unresolved suffix for the next iteration. Under a mild monotonicity assumption on the drafter, we further show that Packed-SS requires no more model calls than the standard two-call decoding in the greedy setting. Experiments on Nemotron 3B/8B and SetDLM 1.7B show higher throughput and lower mean wall time than both baselines under greedy decoding.

Figures & tables

Panel of Figure 1 — Figure 1: Where drafting happens in one call. Outlined boxes are model calls. Linear-SS must finish verifying before it can draft, so an iteration costs two calls. Quadratic-SS fits in one call by drafting a continuation for every place verification might stop, so its input grows quadratically with the speculative width $W$ . Packed-SS also uses one call, building a single draft from the previously verified prefix alone. The right column gives total draft and verifier prediction counts per iteration, including predictions that may later be discarded.

Panel of Figure 2 — Figure 2: One iteration of greedy Packed-SS. Columns are token positions. The old draft token $d_{t-1}^{(i)}$ , the verifier prediction $v_{t}^{(i)}$ , and the new proposal token $z_{t}^{(i)}$ are in the same position. A single forward pass takes the verified prefix $P_{t-1}$ and draft $D_{t-1}$ as input and produces the two middle rows: the AR verifier’s next-token predictions $v_{t}^{(i)}$ , conditioned on $P_{t-1}$ and $d_{t-1}^{(1)},\dots,d_{t-1}^{(i-1)}$ , and the diffusion drafter’s proposal $Z_{t}=(z_{t}^{(i)})_{i=1}^{W}$ , conditioned on $P_{t-1}$ alone. Here, one matching draft token ( $h_{t}=1$ ) and the correction token $v_{t}^{(2)}$ yield $A_{t}=2$ output tokens. Proposals at these two positions are discarded, leaving $D_{t}=(z_{t}^{(3)},z_{t}^{(4)})$ as the next draft.

Panel of Figure 3 — Figure 3: Greedy decoding efficiency and accuracy. Efficiency metrics are ratios to Linear-SS; accuracy differences are in percentage points. Error bars show 95% paired bootstrap confidence intervals. Native is the released non-speculative SetDLM diffusion decoder, shown for reference. WritingBench measures speed only.

Panel of Figure 4 — Figure 4: Efficiency and accuracy at temperature 1. Efficiency metrics are ratios to Linear-SS, accuracy differences are in percentage points. Error bars show 95% paired bootstrap confidence intervals. WritingBench measures speed only.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Sizes and indices
$t$	decoding iteration, starting at $1$
$i,j$	token position: relative in LTR, fixed in any-order decoding
$r,s$	rank in the verification order (any-order setting)
$W$	speculative width: the fixed number of future positions proposed by the LTR drafter, $W\geq 1$ , and the size of the active window in the any-order setting. The schedule and Theorem 3.2 are stated in $W$ alone
$B$	Nemotron’s block size, $B=W+1$ : the last prefix token and $W$ draft positions
$u_{t}$	number of unresolved positions in the active window, $u_{t}=\|U_{t}\|\leq W$ (any-order setting)

Appendix

Table 1: Symbols. Sizes and indices, decoder state, and distributions.

		Evaluated uncached rows per iteration
			SetDLM maximum
Method	NFE/iteration	Nemotron	$W=8$	$W=16$	SetDLM general
Linear-SS	2	$2B$	25	49	$p+3u_{t}$
Quadratic-SS	1	$B(B+1)$	81	289	$p+2u_{t}+u_{t}^{2}$
Packed-SS (Ours)	1	$2B$	25	49	$p+3u_{t}$

Appendix

Table 2: Execution cost per iteration with cached prompt context. Evaluated uncached rows count token representations processed by the transformer, including duplicates and evaluated padding, summed across calls. Nemotron uses block size

B=W+1

. In SetDLM,

u_{t}\leq W

positions remain unresolved in the active window, and

p=|N_{t}|\in\{0,1\}

counts the correction token processed in this call; maximum row counts take

u_{t}=W

and

p=1

. Prompt rows are processed once and depend on prompt length.

Method	$B$	Tokens/s	Rows/call	Wall (s)	Total NFE	Tokens	Accuracy (%)
Linear-SS	8	94.9	8	2.824	98.2	267.9	100
Linear-SS	16	113.4	16	2.448	84.6	277.5	100
Linear-SS	32	121.9	32	2.237	77.4	272.8	100
Packed-SS	8	94.8	16	2.798	89.0	265.2	100
Packed-SS	16	123.5	32	2.315	73.3	285.9	100
Packed-SS	32	128.5	64	2.333	75.4	299.7	90

Appendix

Table 3: Nemotron 8B GSM8K block-size probe. Each row uses the same ten questions. Tokens/s is calculated from the displayed means. Wall is mean total decoder time per response, including prompt processing; total NFE and output-token counts are also per-response means. Accuracy is reported for comparison. Rows/call is the maximum number of input rows per call after initialization.

Method	Tokens	Total NFE	Model (s)	Non-model (s)	Wall (s)	Tokens/s	Accuracy (%)
Linear-SS	272.2	55.04	1.561	0.137	1.698	160.32	85.90
Quadratic-SS	273.3	54.44	1.781	0.343	2.123	128.72	86.35
Packed-SS	274.2	48.44	1.570	0.179	1.749	156.78	85.82

Appendix

Table 4: Nemotron 8B GSM8K with proposal-only LoRA (

n=1{,}319

). Means; model execution time and non-model decoder time sum to inclusive decoder wall time. Emitted length and throughput accompany latency, and accuracy is reported for comparison.

Method	Query rows	Selected	Evaluated	Base ms	LoRA ms	Increase
Linear-SS	$32+32$	32	32	54.41	56.77	4.3%
Quadratic-SS	72	64	72	28.90	32.69	13.1%
Packed-SS	64	32	64	28.58	32.37	13.3%

Appendix

Table 5: Nemotron 8B GSM8K model execution time per iteration. Linear’s iteration contains both calls. “Selected” counts rows receiving adapter updates; “evaluated” counts rows for which those updates are computed.

Panel of Figure 5 — Figure 5: Normalized Nemotron timing decomposition on GSM8K and MATH-500. Greedy decoding with a 1,024-token cap. Each task’s component times are divided by that task’s Linear-SS total, then the two tasks are averaged with equal weight within each model scale. The dashed line marks Linear-SS’s total. Non-model decoder time covers draft token construction, selection, and state updates.

Nemotron 3B
Task	Method	Tokens/s	Tokens/ NFE	Time/NFE (ms)	Accuracy (%)	Tokens	Wall (s)	NFE
MBPP	Linear-SS	173.46	3.757	21.658	74.87	65.8	0.379	17.50
	Quadratic-SS	164.38	4.288	26.089	74.34	65.9	0.401	15.36
	Packed-SS	181.10	4.312	23.812	74.60	67.0	0.370	15.53
HumanEval	Linear-SS	237.45	5.158	21.722	79.27	90.5	0.381	17.55
	Quadratic-SS	211.48	5.536	26.177	79.88	91.1	0.431	16.46

Appendix

Table 6: Nemotron absolute measurements at temperature 0. Tokens, wall time, and NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each model and task.

Nemotron 3B
Task	Method	Tokens/s	Tokens/ NFE	Time/NFE (ms)	Accuracy (%)	Tokens	Wall (s)	NFE
MBPP	Linear-SS	120.38	2.508	20.834	68.14	102.7	0.853	40.95
	Quadratic-SS	129.01	3.328	25.800	69.69	108.7	0.843	32.67
	Packed-SS	125.06	2.933	23.454	68.78	101.8	0.814	34.72
HumanEval	Linear-SS	169.58	3.526	20.792	66.89	96.9	0.571	27.49
	Quadratic-SS	170.63	4.348	25.484	67.29	95.6	0.561	22.00

Appendix

Table 7: Nemotron absolute measurements at temperature 1. Tokens, wall time, and NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each model and task.

Window	Method	Tokens/s	Tokens/ NFE	Time/NFE (ms)	Accuracy (%)	Tokens	Wall (s)	NFE
$W=8$	Linear-SS	94.93	2.610	27.496	68.39	257.2	2.709	98.53
	Quadratic-SS	86.25	3.414	39.583	68.39	257.1	2.981	75.31
	Packed-SS	100.60	2.842	28.247	68.39	257.2	2.556	90.49
	Native	77.21	2.119	27.442	59.67	252.4	3.269	119.11
$W=16$	Linear-SS	98.85	2.792	28.243	66.57	259.9	2.629	93.09
	Quadratic-SS	41.60	4.074	97.934	66.57	260.4	6.260	63.92

Appendix

Table 8: SetDLM absolute measurements on GSM8K. LTR verification and native decoding on all 1,319 test questions. Tokens, wall time, and NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each window.

LTR verification and native decoding
Window	Method	Tokens/s	Tokens/ NFE	Time/NFE (ms)	Accuracy (%)	Tokens	Wall (s)	NFE
$W=8$	Linear-SS	94.93	2.610	27.496	68.39	257.2	2.709	98.53
	Quadratic-SS	86.25	3.414	39.583	68.39	257.1	2.981	75.31
	Packed-SS	100.60	2.842	28.247	68.39	257.2	2.556	90.49
	Native	77.21	2.119	27.442	59.67	252.4	3.269	119.11
$W=16$	Linear-SS	98.85	2.792	28.243	66.57	259.9	2.629	93.09

Appendix

Table 9: SetDLM measurements on GSM8K. All 1,319 test questions. Top: LTR verification and native decoding; bottom: confidence-ordered verification. Tokens, inclusive wall time, and total NFE are means per response; rates use aggregate totals. Bold marks the highest tokens/s within each window and order.

Setting	Baseline	$\Delta$ wall (s) [95% CI]	Throughput ratio [95% CI]
$W=8$ LTR	Linear-SS	$-0.1529$ [ $-0.1588$ , $-0.1470$ ]	1.060 [1.058, 1.062]
	Quadratic-SS	$-0.4248$ [ $-0.4377$ , $-0.4124$ ]	1.166 [1.164, 1.169]
$W=8$ Confidence	Linear-SS	$-0.2581$ [ $-0.2953$ , $-0.2217$ ]	1.102 [1.098, 1.107]
	Quadratic-SS	$-0.3521$ [ $-0.3836$ , $-0.3198$ ]	1.140 [1.136, 1.143]
$W=16$ LTR	Linear-SS	$-0.0382$ [ $-0.0432$ , $-0.0330$ ]	1.015 [1.013, 1.017]
	Quadratic-SS	$-3.6686$ [ $-3.7869$ , $-3.5590$ ]	2.411 [2.399, 2.424]

Appendix

Table 10: Paired SetDLM efficiency comparisons. Wall-time differences are Packed-SS minus baseline; throughput ratios are Packed-SS/baseline. Brackets give paired 95% confidence intervals.

Temperature 0
Setting	Packed-SS $-$ Linear-SS	Packed-SS $-$ Quadratic-SS	Linear-SS $-$ Quadratic-SS
3B MBPP	-0.26 [-1.32, +0.79]	+0.26 [-0.79, +1.32]	+0.53 [-0.53, +1.59]
3B HumanEval	-1.22 [-3.66, +1.22]	-1.83 [-4.27, +0.00]	-0.61 [-1.83, +0.00]
3B GSM8K	-0.30 [-1.14, +0.53]	-0.15 [-0.99, +0.68]	+0.15 [-0.68, +0.99]
3B MATH-500	+3.00 [+0.80, +5.20]	+0.20 [-1.60, +2.00]	-2.80 [-4.60, -1.00]
8B MBPP	-0.26 [-1.59, +0.79]	+0.00 [-0.79, +0.79]	+0.26 [-0.53, +1.06]

Appendix

Table 11: Paired Nemotron accuracy differences.

Setting	Baseline	$\Delta$ wall (s) [95% CI]	Throughput ratio [95% CI]
3B MBPP	Linear-SS	-0.0092 [-0.0268, +0.0123]	1.044 [1.007, 1.075]
	Quadratic-SS	-0.0308 [-0.0563, -0.0027]	1.102 [1.079, 1.127]
3B HumanEval	Linear-SS	-0.0207 [-0.0341, -0.0100]	1.052 [1.033, 1.072]
	Quadratic-SS	-0.0703 [-0.0886, -0.0539]	1.181 [1.149, 1.213]
3B GSM8K	Linear-SS	-0.0434 [-0.0605, -0.0243]	1.061 [1.049, 1.071]
	Quadratic-SS	-0.1003 [-0.1187, -0.0798]	1.114 [1.103, 1.125]

Appendix

Table 12: Paired Nemotron efficiency comparisons at temperature 0.

Setting	Baseline	$\Delta$ wall (s) [95% CI]	Throughput ratio [95% CI]
3B MBPP	Linear-SS	-0.0389 [-0.0678, -0.0107]	1.039 [1.026, 1.052]
	Quadratic-SS	-0.0287 [-0.0832, +0.0248]	0.969 [0.956, 0.984]
3B HumanEval	Linear-SS	-0.0076 [-0.0250, +0.0093]	1.021 [1.005, 1.038]
	Quadratic-SS	+0.0033 [-0.0121, +0.0189]	1.015 [0.997, 1.034]
3B GSM8K	Linear-SS	-0.0227 [-0.0832, +0.0387]	1.020 [1.001, 1.037]
	Quadratic-SS	+0.0000 [-0.0640, +0.0652]	0.996 [0.979, 1.013]

Appendix

Table 13: Paired Nemotron efficiency comparisons at temperature 1.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 314 papers

D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models
Jun 3, 2026Liyuan Zhang, Jiarui Zhang, Jinwei Yao +6Speculative Decoding for Diffusion Language Models Speculative Decoding
Peking University · Tsinghua University · HKUST +2
LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter
Sep 29, 2026Hao-Yuan He, Peng-Fei Liu, Chen Chen +6Long-Context Language Model Inference Blockwise Diffusion Language Models
Speculative Speculative Decoding
Mar 3, 2026Tanishq Kumar, Tri Dao, Avner MaySpeculative Decoding LLM Inference Acceleration
Department of Computer Science, Stanford University · Department of Computer Science, Princeton University · Together AI

Jun 3, 2026J/K move · Enter open · S save

D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models

Liyuan Zhang, Jiarui Zhang, Jinwei Yao, Ran Yan+5

Peking University · Tsinghua University · HKUST+2

Speculative Decoding for Diffusion Language Models Speculative Decoding Diffusion Language Model Inference LLM Inference Acceleration Diffusion Language Model Decoding Parallel Speculative Decoding

Open paper AskPDF arXiv

Speculative decoding accelerates autoregressive large language model inference by drafting multiple tokens and verifying them in a single target-model forward pass. Recent diffusion-based drafters generate an entire block of tokens in parallel but usually commit to a single draft sequence per verification: once the first mismatch occurs, all subsequent draft tokens are discarded, resulting in a limited acceptance rate. Naively batching more draft candidate sequences only introduces a marginal improvement, as redundant or poorly placed branches increase the cost of drafting and verification without proportionally increasing the number of accepted tokens. We propose D^2SD, a dual diffusion draft speculative decoding framework that organizes candidates into a confidence-guided prefix tree, where the first diffusion drafter generates a block along with per-position confidence scores that are used to identify the most likely rejection boundary and select the top-K prefix ranges for recovery; the second variable-prefix diffusion drafter re-anchors at each selected prefix and proposes alternative continuations in one batched pass; the resulting shared-prefix candidates are jointly verified via cascade attention. Empirically, D^2SD shows clear improvements over both the underlying diffusion approach and strong autoregressive speculative decoding baselines.

Abstract

Speculative Decoding

Explore similar work

D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models

LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter

Speculative Speculative Decoding