CAST: Cost-Aware Speculative Trees from One-Pass Block Drafters

Jungseob Lee; Sugyeong Eo

cs.CLSep 29, 2026

CAST: Cost-Aware Speculative Trees from One-Pass Block Drafters

Authors: Jungseob Lee, Sugyeong Eo

Organizations: Korea University · Yonsei University Mirae Campus

Abstract

Speculative decoding accelerates large language model inference by drafting future tokens cheaply and verifying them with the target model in parallel. Block drafters score a whole block of future tokens in one forward pass, yet standard decoding verifies only the top-scoring chain and discards the other candidates. Because these candidates are already scored, verifying more of them adds target computation but no extra drafting. We introduce CAST (Cost-Aware Speculative Trees), which packs these candidates into a tree and verifies it in a single target pass, leaving the target model, drafter weights, and decoding rule untouched. To decide how wide the tree should be, CAST adds candidates while the expected gain from the next one outweighs the verification time it adds. The width therefore adapts to each deployment from a latency measurement, without sweeping over widths. We evaluate CAST across five domains on three GPU generations and two model families. At its predicted width, CAST is faster than the standard chain in all eight settings, by up to 43%. We also find that the best width depends strongly on the deployment. Where verification cost jumps at a kernel boundary, a 128-token tree is only 2% faster than the standard chain, whereas the tree at the predicted width is 20% faster. Furthermore, we prove that CAST leaves the target output distribution unchanged under both greedy and sampled decoding. Code is available at https://github.com/js-lee-AI/CAST.

Figures & tables

Figure 1: One decoding round on a GSM8K prompt. With the same drafter scores and 16 packed tokens, (a) standard DFlash decoding commits two tokens, whereas (b) CAST verifies a tree and commits six. (c) Theorem 2 chooses the width

N

Model	Method	Math				Code				Chat
Model	Method	GSM8K		MATH-500		HumanEval		MBPP		MT-Bench		Avg.
Temperature $=0$		Speedup	$R$	Speedup	$R$	Speedup	$R$	Speedup	$R$	Speedup	$R$	Speedup	$R$
	EAGLE-3 (16)	2.58 $\times$	4.45	2.37 $\times$	4.11	2.23 $\times$	3.97	2.37 $\times$	4.11	2.07 $\times$	3.74	2.32 $\times$	4.08
	EAGLE-3 (60)	3.08 $\times$	5.34	2.79 $\times$	4.88	2.69 $\times$	4.76	2.80 $\times$	4.93	2.41 $\times$	4.49	2.75 $\times$	4.88
	DFlash (16)	4.54 $\times$	6.23	5.57 $\times$	7.81	4.67 $\times$	6.52	4.82 $\times$	6.56	2.30 $\times$	4.06	4.38 $\times$	6.24
Q3-8B	CAST ( $N^{\ast}{=}127$ )	5.81 $\times$	8.46	6.74 $\times$	9.87	6.16 $\times$	9.04	6.25 $\times$	9.09	3.16 $\times$	5.52	5.62 $\times$	8.40

Table 1: Qwen3 decoding on H100 SXM. Cells report speedup over AR and committed round length

R

. Parentheses give baseline packed tokens or the CAST nonroot budget.

Setting (drafter block)	$N^{\ast}$	Gain at $N^{\ast}$	Oracle width (gain)		Gap	Gain at $N{=}127$
H100 SXM, Qwen3-8B (b16)	127	+29.5%	127	+29.5%	+0.0	+29.5%
H100 SXM, Qwen3-4B (b16)	127	+31.5%	127	+31.5%	+0.0	+31.5%
Blackwell, Qwen3-8B (b16)	63	+19.8%	95	+21.5%	-1.7	+2.4%
Blackwell, Qwen3-4B (b16)	95	+27.3%	127	+29.0%	-1.7	+29.0%
A6000, Qwen3-8B (b16)	95	+28.7%	95	+28.7%	+0.0	+25.1% ‡
A6000, Qwen3-4B (b16)	95	+32.6%	95	+32.6%	+0.0	+31.7% ‡

Table 2: Predicted and sweep-oracle widths across the eight settings. Gain over DFlash is averaged over domains. Gap is in percentage points.

\ddagger

marks cells measured on a second A6000 host.

Figure 4: Qwen3-8B greedy replay diagnostics on 40-prompt slices. (a) Target-correction rank at the first DFlash rejection. (b) Depth-one reliability. (c) Sorted candidate prefix masses. The dashed line is the Blackwell GSM8K threshold.

Figure 5: Gain over DFlash at 16 packed tokens.

GSM8K	MT-Bench	HumanEval
Qwen3-8B, greedy decoding
+18.1%	+24.6%	+16.9%
$[16.4,19.8]$	$[21.6,27.7]$	$[14.8,19.0]$
Qwen3-4B, greedy decoding
+25.7%	+33.2%	+23.2%
$[24.0,27.4]$	$[29.4,37.0]$	$[21.1,25.3]$

Table 3: Mean gain over DFlash on Blackwell at the predicted width

N^{\ast}

, which is

63

for Qwen3-8B and

95

for Qwen3-4B. Brackets give 95% bootstrap intervals for greedy decoding and ranges over three seeds for sampling.

Method	GSM8K	MATH-500	HumanEval	MBPP	MT-Bench	Avg.
Method	Math		Code		Chat
EAGLE-3 (10)	2.05 $\times$	1.75 $\times$	2.28 $\times$	2.25 $\times$	1.73 $\times$	2.01 $\times$
EAGLE-3 (60)	2.67 $\times$	2.31 $\times$	2.80 $\times$	2.81 $\times$	2.34 $\times$	2.59 $\times$
DFlash (10)	3.00 $\times$	2.76 $\times$	3.60 $\times$	3.56 $\times$	2.32 $\times$	3.05 $\times$
CAST ( $N^{\ast}{=}127$ )	3.82 $\times$	3.59 $\times$	4.38 $\times$	4.49 $\times$	3.05 $\times$	3.87 $\times$

Table 4: LLaMA-3.1-8B greedy decoding with the block-10 DFlash head on H100 SXM. Cells report speedup over AR. Parentheses give baseline packed tokens or the CAST nonroot budget.

Figure 6: Qwen3-8B verification cost under load. (a) Blackwell verifier latency over the packed width

W

of each request. (b,c) Batch latency over the packed tokens

n

of each request on H100 SXM and Blackwell.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

GPU, target	$n{=}1$	16	24	32	48	64	96	128
Blackwell, Qwen3-8B	17.2	21.9	23.3	23.6	24.1	24.6	24.4	30.3
A6000, Qwen3-8B	117.0	128.1	128.8	131.5	129.6	129.6	133.2	129.1
A100, Qwen3-8B	30.2	34.4	33.2	39.4	34.9	35.3	35.9	35.6
H100 NVL, Qwen3-8B	14.7	17.9	17.8	20.8	18.2	18.4	19.2	19.2
Blackwell, Qwen3-4B	16.9	18.7	19.6	19.5	19.4	19.4	20.9	20.4
H100 SXM, Qwen3-8B	24.7	30.6	30.2	30.6	31.2	31.5	32.2	32.6

Appendix

Table 5: Target-forward latency (ms) by packed-token count

n{=}N{+}1

, using bf16 and a 1k-token KV prefix. Each probe uses random packed tokens, eight warmups, and 30 timed iterations. The H100 SXM rows use scaled dot-product attention.

Qwen3-8B
Prefix	$n{=}1$	16	32	64	128	Slope
1k	23.6	26.6	25.1	25.2	24.8	-0.016
4k	22.7	25.5	25.4	26.1	26.6	0.010
8k	22.9	35.9	36.2	37.5	37.9	0.018
16k	23.7	56.7	58.0	59.9	60.0	0.029
Qwen3-4B

Appendix

Table 6: H100 SXM target-forward latency (ms) across KV-prefix lengths at batch one in bf16. Slope is the latency increase in ms for each added packed token from

n{=}16

128

Blackwell 8B, $N^{\ast}{=}63$
	Decode time (ms/token)
Domain	AR	DFlash	CAST	Gain	$R$
GSM8K	18.0	4.25	3.65	+16.4%	7.97
MT-Bench	18.0	8.23	6.56	+25.4%	4.40
HumanEval	18.0	4.06	3.45	+17.6%	8.51
A6000 8B, $N^{\ast}{=}95$

Appendix

Table 7: Decoding on Blackwell and A6000 for Qwen3-8B (left) and Qwen3-4B (right). Cells report decode-only ms/token, gain over DFlash at

N^{\ast}

, and committed round length

R

GPU	target	domain	$N{=}15$	$N{=}31$	$N{=}47$	$N{=}63$	$N{=}95$	$N{=}127$
Blackwell	Qwen3-8B	GSM8K	+5.9%	+13.1%	+15.5%	+16.4%	+17.8%	${\color[rgb]{0.5898,0.25,0.1367}-0.8\%}$
Blackwell	Qwen3-8B	MT-Bench	+19.1%	+21.8%	+24.8%	+25.4%	+26.9%	+7.5%
Blackwell	Qwen3-8B	HumanEval	+8.2%	+12.5%	+17.3%	+17.6%	+19.7%	+0.5%
Blackwell	Qwen3-4B	GSM8K	+12.2%	+17.9%	+22.8%	+22.7%	+24.3%	+27.1%
Blackwell	Qwen3-4B	MT-Bench	+29.2%	+31.6%	+34.2%	+34.5%	+33.9%	+37.9%
Blackwell	Qwen3-4B	HumanEval	+12.3%	+14.8%	+21.8%	+19.9%	+23.6%	+22.0%

Appendix

Table 8: Greedy width-sweep gains over DFlash across GPUs and targets. Bold marks each row’s oracle. Underline marks the predicted width.

\ddagger

marks cells measured on a second A6000 host.

GPU	target	domain	DFlash $R$	$N{=}15$ $R$	DFlash ms	$N{=}15$ ms	shape gain
Blackwell	Qwen3-8B	GSM8K	6.02	6.71	4.40	4.16	+5.9%
Blackwell	Qwen3-8B	MT-Bench	3.13	3.90	8.40	7.05	+19.1%
Blackwell	Qwen3-8B	HumanEval	6.36	7.26	4.11	3.80	+8.2%
Blackwell	Qwen3-4B	GSM8K	5.93	6.85	4.25	3.79	+12.2%
Blackwell	Qwen3-4B	MT-Bench	3.11	4.00	8.77	6.79	+29.2%
Blackwell	Qwen3-4B	HumanEval	6.15	7.19	4.10	3.65	+12.3%

Appendix

Table 9: Tree-shape ablation at 16 packed tokens, comparing standard DFlash with CAST at

N{=}15

. Cells report committed round length

R

, decode-only ms/token, and gain over DFlash on 40 prompts in each domain.

domain	DFlash $R$	CAST $R$	DFlash ms/tok	CAST ms/tok	mean gain	seed range
GSM8K	5.70	7.54	4.88	4.18	+16.7%	$+16.0$ to ${\color[rgb]{0.1094,0.4023,0.3125}+18.0\%}$
MT-Bench	3.69	5.10	9.43	7.25	+30.1%	$+28.7$ to ${\color[rgb]{0.1094,0.4023,0.3125}+32.3\%}$
HumanEval	5.66	7.72	5.11	4.18	+22.3%	$+20.3$ to ${\color[rgb]{0.1094,0.4023,0.3125}+25.3\%}$

Appendix

Table 10: Stochastic verification for Qwen3-8B on Blackwell at

T{=}1

and

N^{\ast}{=}63

. Cells report committed round length

R

, decode-only ms/token, mean gain across seeds 0–2, and the seed-level gain range.

Target	Prefix	$n{=}1$	2	16	64	128	256	1024
27B dense hybrid	1k	46.9	67.9	69.1	68.5	68.6	74.6	194.7
27B dense hybrid	8k	47.3	68.5	70.0	78.2	80.2	93.9	237.2
35B MoE hybrid	1k	87.3	109.7	129.8	158.4	175.4	194.7	229.3
35B MoE hybrid	8k	88.2	111.1	133.9	163.7	179.8	198.4	246.2

Appendix

Table 11: Verify-forward latency (ms) across packed-token counts for Qwen3.6 hybrid targets on Blackwell, bf16, batch one. Values are medians of 15 timed forwards. Each receives an untimed copy of the prefix cache.

Qwen3-8B
Domain	max new	AR	chain	$N{=}47$	$N{=}63$	$N{=}95$	$N{=}127$	$R$ chain	$R$ at $127$
GSM8K	256	24.85	5.48	4.50	4.40	4.35	4.28	6.23	8.46
MT-Bench	256	24.79	10.75	8.04	7.86	7.83	7.85	4.06	5.52
HumanEval	256	24.77	5.31	4.26	4.12	4.06	4.02	6.52	9.04
MATH-500	512	24.84	4.46	3.82	3.85	3.70	3.69	7.81	9.87
MBPP	512	25.39	5.27	4.22	4.14	4.08	4.06	6.56	9.09

Appendix

Table 12: Qwen3 greedy width sweeps on H100 SXM, using the public block-16 DFlash head of each target. Cells report mean decode-only ms/token.

R

denotes the committed round length of DFlash and of the widest CAST tree.

Head	Tree	GSM8K		MATH-500		HumanEval		MBPP		MT-Bench
		ms/tok	$R$	ms/tok	$R$	ms/tok	$R$	ms/tok	$R$	ms/tok	$R$
Tengyunw	16	9.65	4.45	10.47	4.11	11.13	3.97	10.72	4.11	12.00	3.74
Tengyunw	60	8.07	5.34	8.91	4.88	9.21	4.76	9.06	4.93	10.28	4.49
AngelSlim	16	12.97	3.26	13.35	3.11	13.29	3.18	13.82	3.10	16.33	2.64
AngelSlim	60	11.23	3.78	11.65	3.59	11.59	3.65	11.99	3.58	14.22	3.05
AR anchor		24.85		24.84		24.77		25.39		24.79

Appendix

Table 13: EAGLE-3 head comparison for Qwen3-8B greedy decoding on H100 SXM. Cells report decode-only ms/token and committed round length

R

for 16- and 60-token draft trees. AR anchors are in ms/token.

Cell	AR ms/tok	DFlash $\times$ AR ( $R$ )	CAST $\times$ AR ( $R$ )	Gain	Oracle $N$ (gain)
Generation length, Qwen3-8B, MATH-500
512 new tokens	24.8	5.57 (7.81)	6.74 (9.87)	+21%	127 ( +21% )
2048 new tokens	24.0	5.82 (8.09)	6.99 (10.20)	+20%	127 ( +20% )
Long prefix, GovReport prompts cut to 8k tokens
Qwen3-8B	25.1	1.39 (2.24)	1.77 (3.43)	+28%	95 ( +30% )
Qwen3-4B	24.6	1.43 (2.18)	1.83 (3.37)	+27%	95 ( +30% )

Appendix

Table 14: Generation length and KV prefix on H100 SXM at

T{=}0

over 40 prompts, with CAST at

N^{\ast}{=}127

. The long-prefix rows generate 256 new tokens. Cells report speedup over AR, committed round length

R

, gain over DFlash, and the sweep oracle.

Figure 7: Qwen3-8B serving goodput on Blackwell, medians of four runs.

	8 packed tokens			16 packed tokens
Domain	chain, block 8	tree $N{=}7$ , block 8	tree $N{=}7$ , block 16	chain, block 16	tree $N{=}15$ , block 16
GSM8K	6.84 (4.86)	6.51 (5.19)	6.31 (5.37)	5.47 (6.23)	4.96 (7.02)
MT-Bench	11.77 (3.28)	10.29 (3.66)	10.07 (3.80)	10.82 (4.06)	8.87 (4.68)
HumanEval	6.74 (4.96)	6.35 (5.32)	6.33 (5.40)	5.37 (6.52)	4.76 (7.48)

Appendix

Table 15: Block-length and tree-width ablation at matched packed budgets on H100 SXM, Qwen3-8B,

T{=}0

. Cells show decode-only ms/token with committed round length in parentheses.

Autoregressive drafter
Method (width)	GSM8K		MATH-500		HumanEval		MBPP		MT-Bench		Avg.
	$\times$ AR	$R$	$\times$ AR	$R$	$\times$ AR	$R$	$\times$ AR	$R$	$\times$ AR	$R$	$\times$ AR	$R$
AR ms/token	17.30		19.67		17.12		19.89		16.87
EAGLE-3 (10)	2.05	4.60	1.75	4.05	2.28	5.15	2.25	5.11	1.73	4.14	2.01	4.61
EAGLE-3 (60)	2.67	6.08	2.31	5.28	2.80	6.56	2.81	6.40	2.34	5.59	2.59	5.98
One-pass block drafter

Appendix

Table 16: LLaMA-3.1-8B greedy decoding with the block-10 DFlash head on H100 SXM. Cells report speedup over AR and committed round length

R

Method (width)	GSM8K		HumanEval		MBPP		Avg.
	$\times$ AR	$R$	$\times$ AR	$R$	$\times$ AR	$R$	$\times$ AR	$R$
AR ms/token	62.3		63.8		62.6
DFlash (16)	3.84	4.81	5.86	7.77	5.08	6.64	4.93	6.40
CAST ( $N{=}15$ )	4.52	5.65	6.53	8.61	5.87	7.52	5.64	7.26
CAST ( $N{=}31$ )	4.78	6.05	7.13	9.40	6.39	8.19	6.10	7.88
CAST ( $N{=}47$ )	5.05	6.37	7.29	9.58	6.50	8.40	6.28	8.12

Appendix

Table 17: Qwen3-Coder-30B-A3B greedy decoding with the block-16 DFlash head on H100 SXM. Cells report speedup over AR and committed round length

R

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 314 papers

Cost-Aware Diffusion Draft Trees for Speculative Decoding
Jun 1, 2026Shuai Zhang, Huachuan Qiu, Hongliang He +1Cost-Aware Inference Speculative Decoding for Diffusion Language Models
1Zhejiang University · 2Westlake University
SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting
May 8, 2026Weijie Shi, Qiang Xu, Fan Deng +9Speculative Decoding LLM Inference Acceleration
Hong Kong University of Science and Technology · MetaX · Zhejiang Normal University +1
Accelerating Speculative Decoding with Block Diffusion Draft Trees
Apr 14, 2026Liran Ringel, Yaniv RomanoSpeculative Decoding Parallel Speculative Decoding
Department of Computer Science, Technion – Israel Institute of Technology. · Department of Electrical and Computer Engineering, Technion – Israel Institute of Technology.

Jun 1, 2026J/K move · Enter open · S save

Cost-Aware Diffusion Draft Trees for Speculative Decoding

Shuai Zhang, Huachuan Qiu, Hongliang He, Yong Dai

1Zhejiang University · 2Westlake University

Cost-Aware Inference Speculative Decoding for Diffusion Language Models Speculative Decoding Diffusion Language Model Decoding

Open paper AskPDF arXiv

Speculative decoding accelerates inference by having a lightweight drafter propose tokens verified in parallel by the target language model. Block diffusion drafters such as DFlash generate an entire draft block in one pass, yielding per-position marginals; DDTree uses these to build a candidate tree that maximizes expected acceptance length under a fixed node budget. We observe, however, that acceptance length is non-decreasing in budget: it always favors larger trees regardless of verification cost, offering no principled basis for budget selection. We introduce CaDDTree (Cost-aware Diffusion Draft Tree), a method that directly optimizes token throughput (expected tokens generated per unit time) by jointly selecting the tree structure and node budget. We model draft and verification latencies explicitly, show that the throughput objective decomposes into a per-round one-dimensional search over the budget, and prove that under a convex verification cost the throughput function is unimodal, enabling an efficient greedy stopping rule. CaDDTree requires no offline budget search, adapting the budget each round from the current per-position distributions and verification cost. Experiments on Qwen3-4B and Qwen3-8B across eight benchmarks spanning reasoning, coding, and instruction-following tasks show that \caDDTree{} matches or surpasses DDTree with oracle budget selection on nearly all tasks.

Abstract

Speculative Decoding

Explore similar work

Cost-Aware Diffusion Draft Trees for Speculative Decoding

SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting

Accelerating Speculative Decoding with Block Diffusion Draft Trees