ValuSpec: Plug-and-Play Candidate Valuation before Target Verification for Tree-Based Speculative Decoding

Liang He; SiYuan Ma; Qishi Zhan; Yongqi Fan; Mingyu Cao; Qizhen Lan; Zhaolu Kang; Xilu Wang

cs.AIOct 22, 2026

ValuSpec: Plug-and-Play Candidate Valuation before Target Verification for Tree-Based Speculative Decoding

Authors: Liang He, SiYuan Ma, Qishi Zhan, Yongqi Fan, Mingyu Cao, Qizhen Lan, Zhaolu Kang, Xilu Wang

Organizations: Tongji University · Nanyang Technological University · Marquette University · Tencent ·

Abstract

Tree-based speculative decoding accelerates large language model inference by verifying multiple candidate branches in parallel. However, in practical serving scenarios, similar requests repeatedly induce overlapping candidate branches. While retrieval-based methods mitigate this by reusing historical drafts, the resulting candidate trees often contain many low-quality branches as the candidate tree grows, which reduces verification efficiency and wastes target-side computation. To address this, we propose ValuSpec, a plug-and-play candidate valuation mechanism that filters low-quality branches from a supplied candidate tree before expensive target-side verification. ValuSpec introduces transfer-based supervision to align filtering with the target model's behavior. A filtering threshold is estimated from the target model's greedy trajectories on a calibration split, without any parameter training. The target model then verifies the filtered tree under the output-identical greedy rule, so accepted tokens are guaranteed to match standard autoregressive decoding exactly. Experimental results on HumanEval, GSM8K, Dolly, and SpecBench show that, in our main evaluation, ValuSpec achieves up to 2.38x end-to-end speedup in retrieval-based tree settings. Furthermore, integrating ValuSpec into the EAGLE-3 dynamic tree framework yields up to 2.78x end-to-end speedup without modifying its tree construction strategy, demonstrating that ValuSpec serves as a generally applicable and plug-and-play candidate valuation module.

Figures & tables

Figure 1: Candidate quality and verification efficiency. (A) Low-quality candidate tokens waste target-side verification budget in SD. Candidate tokens that deviate from the target model’s greedy trajectory are rejected, yet still consume verification computation without contributing to decoding progress. (B) Acceptance rate versus candidate draft length on Dolly (A100) using Llama-2-7b-chat-hf. Acceptance decreases monotonically as the candidate draft length grows, indicating increasing redundancy and a higher proportion of low-quality candidates.

Figure 2: Comparison of speculative decoding paradigms. Conventional speculative decoding generates candidates online for each request, leading to repeated computation under similar inputs. Retrieval-based speculative decoding reuses historical candidates but introduces large candidate trees with many low-quality candidates that waste target-side verification. ValuSpec is a plug-and-play candidate valuation mechanism to filter low-quality candidates from the supplied tree before target-side verification, reducing verification redundancy without modifying candidate construction.

Figure 3: Overview of ValuSpec. ValuSpec decouples candidate supply from candidate valuation using two disjoint splits of the same database $\mathcal{D}$ . A candidate tree $T_{x}$ is supplied from $\mathcal{D}_{\mathrm{eval}}$ , while a lightweight draft model $q_{\phi}$ is calibrated on target greedy trajectories from $\mathcal{D}_{\mathrm{cal}}$ to filter low-quality branches before target-side verification. The target model $p_{\theta}$ then verifies the filtered subtree $T_{x}^{\mathrm{filtered}}$ with tree masking/attention and accepts the longest greedy-consistent prefix, producing the final output sequence $Y$ under output-identical decoding.

Figure 4: Main results of ValuSpec under output-identical decoding across three decoding loads. ValuSpec uses

n=4

, draft tokens

=64

, and

\delta=0.05

for all tasks.

	Llama-2-7B				Llama-2-13B
Task	AR	SD ( $n=3$ )	Retrieval SD	ValuSpec	AR	SD ( $n=3$ )	Retrieval SD	ValuSpec
HumanEval-x-python	23.56 (1.00 $\times$ )	29.20 (1.24 $\times$ )	40.64 (1.72 $\times$ )	54.44 (2.31 $\times$ )	19.43 (1.00 $\times$ )	25.27 (1.30 $\times$ )	33.61 (1.73 $\times$ )	46.18 (2.38 $\times$ )
GSM8K-main-test	23.97 (1.00 $\times$ )	32.07 (1.34 $\times$ )	41.36 (1.73 $\times$ )	53.02 (2.21 $\times$ )	20.43 (1.00 $\times$ )	28.32 (1.39 $\times$ )	36.99 (1.81 $\times$ )	45.76 (2.24 $\times$ )
Dolly	25.74 (1.00 $\times$ )	34.89 (1.36 $\times$ )	45.68 (1.77 $\times$ )	56.19 (2.18 $\times$ )	17.90 (1.00 $\times$ )	26.99 (1.51 $\times$ )	34.73 (1.94 $\times$ )	41.74 (2.33 $\times$ )

Table 1: Recurring-workload evaluation. Throughput (tokens/s) under output-identical decoding on A100, with speedup over AR shown in parentheses. SD uses

n=3

; Retrieval SD uses a REST-style tree with 64 draft tokens; ValuSpec uses

n=4

, 64 draft tokens, and

\delta=0.05

		Llama-2-7B		Llama-2-13B
Calibration	Evaluation	AR	ValuSpec	AR	ValuSpec
HumanEval	GSM8K	23.97 (1.00 $\times$ )	51.94 (2.17 $\times$ )	20.43 (1.00 $\times$ )	44.32 (2.17 $\times$ )
HumanEval	Dolly	25.74 (1.00 $\times$ )	52.71 (2.05 $\times$ )	17.90 (1.00 $\times$ )	40.03 (2.24 $\times$ )
GSM8K	HumanEval	23.56 (1.00 $\times$ )	53.96 (2.29 $\times$ )	19.43 (1.00 $\times$ )	45.72 (2.35 $\times$ )
GSM8K	Dolly	25.74 (1.00 $\times$ )	53.65 (2.08 $\times$ )	17.90 (1.00 $\times$ )	40.16 (2.24 $\times$ )
Dolly	HumanEval	23.56 (1.00 $\times$ )	52.73 (2.24 $\times$ )	19.43 (1.00 $\times$ )	44.76 (2.30 $\times$ )

Table 2: Cross-dataset threshold transfer. The threshold is calibrated on the 70% split of one dataset and transferred to another without re-calibration. We report throughput (tokens/s), with speedup over AR shown in parentheses.

	Throughput (tok/s)			Speedup
Dataset	AR	EAGLE-3	EAGLE-3 + ValuSpec	vs. EAGLE-3	vs. AR
HumanEval	35.30	69.10	93.28	1.35 $\times$	2.64 $\times$
GSM8K	35.30	70.40	98.25	1.40 $\times$	2.78 $\times$
Dolly	35.30	55.20	83.66	1.52 $\times$	2.37 $\times$
SpecBench	35.30	47.70	69.17	1.45 $\times$	1.96 $\times$

Table 3: Verifier-side integration with EAGLE-3. Throughput (tokens/s) on Meta-Llama/Llama-3.1-8B-Instruct. ValuSpec is applied after EAGLE-3 tree construction and before target verification, without modifying the upstream generation procedure.

Metric	Standard SD	Retrieval-SD	EAGLE-3
Candidate-supplier compatibility
Marginal speedup	1.63–1.89 $\times$	1.97–2.14 $\times$	1.38–1.61 $\times$
Cross-dataset threshold transfer
Marginal speedup	1.24–1.63 $\times$	1.78–2.01 $\times$	1.33–1.46 $\times$
Fixed-threshold comparison
Best fixed $\rightarrow$ ValuSpec	58.26 $\rightarrow$ 79.78	87.15 $\rightarrow$ 113.16	78.03 $\rightarrow$ 90.49

Table 4: Additional validation on Qwen3.5. Results across Standard SD, Retrieval-SD, and EAGLE-3. Speedup ranges are across HumanEval, GSM8K, and Dolly; cross-dataset ranges summarize all transfer directions. “Fixed” denotes the best fixed likelihood threshold in

[0.3,0.8]

on HumanEval.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Database	Method	A100 / 7b			A100 / 13b			RTX 4090 / 7b
		Thpt	Speedup	Hit	Thpt	Speedup	Hit	Thpt	Speedup	Hit
Dolly	AR	25.74	–	–	17.90	–	–	51.89	–	–
Dolly	SD $n{=}1$	32.68	1.27	0.46	24.48	1.37	0.43	61.74	1.19	0.45
Dolly	SD $n{=}2$	34.37	1.34	0.33	25.85	1.44	0.32	65.71	1.27	0.33
Dolly	SD $n{=}3$	34.89	1.36	0.26	26.99	1.51	0.22	66.14	1.27	0.25
Dolly	SD $n{=}4$	33.92	1.32	0.21	26.44	1.48	0.20	64.18	1.24	0.20

Appendix

Table 5: Combined ablations on Dolly: SD (throughput / speedup / hit rate) across hardware/model settings, plus ValuSpec (throughput / speedup) with a fixed tree configuration (draft tokens

=64

). “–” indicates not applicable or not reported.

Database	Method	A100 / 7b			A100 / 13b			RTX 4090 / 7b
		Thpt	Speedup	Hit	Thpt	Speedup	Hit	Thpt	Speedup	Hit
HumanEval	AR	23.56	–	–	19.43	–	–	49.18	–	–
HumanEval	SD $n{=}1$	27.87	1.18	0.35	23.26	1.20	0.34	57.32	1.17	0.32
HumanEval	SD $n{=}2$	28.68	1.22	0.24	24.69	1.27	0.23	58.23	1.18	0.25
HumanEval	SD $n{=}3$	29.20	1.24	0.18	25.27	1.30	0.19	58.29	1.19	0.18
HumanEval	SD $n{=}4$	28.88	1.23	0.17	24.21	1.25	0.15	54.89	1.12	0.14

Appendix

Table 6: Combined ablations on HumanEval and GSM8K: SD (throughput / speedup / hit rate) across hardware/model settings, plus ValuSpec (throughput / speedup). “–” indicates not applicable or not reported.

Figure 5: Cross-GPU ablations on RTX 4090. End-to-end throughput (tokens/s) and speedup over autoregressive decoding (AR) on GSM8K, HumanEval, and Dolly using Llama-2-7b-chat-hf (target) and JackFram/LLaMA-68M. We compare AR, SD (

n=3

), Retrieval SD (draft tokens

=64

), and ValuSpec (ours). Here, 7B denotes Llama-2-7b-chat-hf.

T_{x}^{\mathrm{filtered}}\leftarrow\{\,b\in T_{x}:s_{\phi}(b\mid x)\geq\tau_{\mathrm{tr}}\,\}

Appendix

Algorithm 1 ValuSpec (offline threshold calibration on

\mathcal{D}_{\mathrm{cal}}

+ online filtering of candidate trees from

\mathcal{D}_{\mathrm{eval}}

)

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding
May 28, 2026Jiaming Fan, Daming Cao, Canchen Huang +6Speculative Decoding LLM Inference Acceleration
Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Southeast University · Ant Group +1
DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification
Apr 8, 2026Ziyi Wang, Siva Rajesh Kasa, Ankith M S +6Speculative Decoding LLM Inference Acceleration
Purdue University · Amazon Inc. · University of Illinois Urbana-Champaign +1
Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decoding
Feb 19, 2026Rahul Thomas, Teo Kitanovski, Micah Goldblum +1Speculative Decoding LLM Inference Acceleration
Ritual · Columbia University · Vanderbilt University

May 28, 2026J/K move · Enter open · S save

TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding

Jiaming Fan, Daming Cao, Canchen Huang, Jiale Fu+5

Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China · Southeast University · Ant Group+1

Speculative Decoding LLM Inference Acceleration LLM Inference Scheduling

Open paper AskPDF arXiv

Speculative decoding accelerates large language model inference through a draft-then-verify paradigm. Building on this, tree-structured methods improve inference by organizing proposals into multiple candidate paths, increasing the accepted length. However, existing tree-structured methods use a single drafter for all drafting steps, creating a dilemma: a smaller drafter is fast but yields lower-quality trees, whereas a larger drafter improves tree quality but suffers from high latency. To address this, we propose TreeGraft, a multi-drafter framework in which drafters of different costs jointly construct a shared draft tree. TreeGraft uses the stronger drafter to rescore candidates by updating scores assigned by the weaker drafter, reselect grafting positions, and recover promising paths left unexplored. It also integrates stronger drafter expansions non-destructively, preserving existing branches that may still be accepted by the target model. Together, these designs improve the quality of the shared draft tree. To control the drafting cost, TreeGraft introduces a lightweight scheduler distilled from an offline value system to decide when to call the stronger drafter. Across 10 model pairs and 6 benchmarks, TreeGraft outperforms the better of the two fixed single-drafter endpoint strategies by 15.1% on average, reaching a maximum gain of 26.6%. Our code is available at https://github.com/fjm9933/TreeGraft.

Abstract

Speculative Decoding

Explore similar work

TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding

DIVERSED: Relaxed Speculative Decoding via Dynamic Ensemble Verification

Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decoding