Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution

Shuhao Li; Fanghua Ye; Wanyu Lin; Tianyu Yuan; Xiaoyu Shen

cs.CLOct 6, 2026

Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution

Authors: Shuhao Li, Fanghua Ye, Wanyu Lin, Tianyu Yuan, Xiaoyu Shen

Organizations: EIT-NLP Lab, Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University · Independent Researcher

Abstract

Speculative decoding accelerates autoregressive generation by using a lightweight draft model to propose multiple tokens that are verified by a target model in parallel. However, the standard acceptance rule focuses on exact distribution correction and rejects tokens that remain highly plausible under the target model when the draft model assigns excess probability. This conservative verification limits the number of draft tokens retained after each verification forward pass. We introduce Nucleus Speculative Decoding (NSD), a relaxed verification method that incorporates target-model plausibility into speculative decoding. NSD accepts a draft token if it satisfies the standard acceptance rule or belongs to the target model's nucleus. We theoretically characterize the distributional deviation introduced by our method and show that the single-step error is exactly determined by the draft model's excess probability within the target nucleus. We further derive sequence-level fidelity bounds that quantify how local deviations accumulate over autoregressive decoding. Experiments across multiple target models and proposal mechanisms demonstrate that NSD consistently improves speculative decoding efficiency while maintaining competitive task performance. Our method achieves throughput speedups of up to $5.16\times$ over autoregressive decoding and up to $3.15\times$ over standard speculative decoding. These improvements coincide with longer accepted lengths, allowing more output tokens to share the cost of each target verification pass. Analysis shows that plausibility-aware verification provides an effective approach for relaxed verification and speculative decoding efficiency. Our code is available at https://github.com/EIT-NLP/Nucleus-Speculative-Decoding.

Figures & tables

Figure 1: Execution time breakdown and rejection analysis of speculative decoding. (a) Execution time breakdown. Ratio of total time spent on drafting, verification, sampling, and overhead. (b) Target Top-K rank from rejected tokens. (c) Target Top-P distribution from rejected tokens.

Figure 2: Overview of Nucleus Speculative Decoding (NSD).

		Nucleus threshold $\theta$
		0.95			0.925			0.90			0.875			0.85
Dataset	$K$	Spd.	$\tau$	Acc.	Spd.	$\tau$	Acc.	Spd.	$\tau$	Acc.	Spd.	$\tau$	Acc.	Spd.	$\tau$	Acc.
Nemo-8B
GSM8K	8	$2.92\times$	6.61	91.66	$2.89\times$	6.61	91.66	$2.83\times$	6.67	91.66	$2.69\times$	6.51	92.42	$2.65\times$	6.54	92.87
	16	$4.35\times$	10.70	91.58	$3.63\times$	9.42	93.48	$4.21\times$	9.66	93.94	$3.93\times$	9.56	93.10	$4.06\times$	9.46	92.72
	24	$4.55\times$	11.90	92.87	$4.60\times$	11.74	90.90	$4.65\times$	12.09	90.14	$4.70\times$	12.03	90.98	$4.89\times$	12.47	91.58

Table 2: Effect of draft block size

K

and nucleus threshold

\theta

on speedup, mean accepted length, and accuracy. Spd. denotes speedup relative to target-model AR;

\tau

is the mean accepted length per draft block verification; Acc. is math accuracy or code pass@1 (%).

Figure 3: Nucleus-based acceptance statistics and verification efficiency. (a) Draft and target entropy. (b) Nucleus location and log probability mismatch; location zero denotes the target rank-1 token, with its strip widened for visibility. (c) Nucleus-accepted token shares and their cumulative ratio by target rank. (d) Verification time in ms/output token for Nemotron models.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Selection or intervention	Recovery and scope
SD	Stochastic ratio acceptance $\min(1,p/q)$ .	Sample from normalized $[p-q]_{+}$ at rejection; preserves the chosen target law under the usual assumptions.
NSD	Retain ratio acceptance and accept in-nucleus failures.	Keep the same residual; local added acceptance and TV error equal $E_{S}$ . No request-level budget.
ASD [ 10 ]	Greedy exceptions constrained by logit regret, a block cap, and a persistent request budget.	Target argmax recovery or bonus under the committed prefix. Zero budget restores greedy verification; regret accounting is not an NSD TV bound.
MARS [ 23 ]	Accept the target top-1 token, or its top-2 token when the raw-logit ratio $z_{(2)}/z_{(1)}>\theta$ .	On failure, emit the target top-1 token and discard the remaining draft. The stated rule relaxes token matching; it does not use the stochastic $[p-q]_{+}$ residual.
Cactus [ 13 ]	Select an adjusted distribution under a divergence constraint.	Uses generalized acceptance/recovery equations. NSD instead fixes the original residual and analyzes the induced deviation.

Appendix

Table 3: Mechanism distinctions. Here

p

and

q

always denote target and actual proposal laws; notation used in other papers is not imported into these columns.

		Acc. (%)		$\tau$
Proposal	Benchmark	SD	NSD	SD	NSD
DFlash-b16	MATH500	83.88	84.28	5.75	6.32
	AIME25	17.33	18.67	4.61	5.13
	HumanEval	84.51	83.17	5.48	6.20
	MBPP	62.41	63.50	4.77	5.29
EAGLE-3-TTT7	MATH500	83.64	83.32	4.54	5.32

Appendix

Table 4: Task performance and mean accepted length for additional proposal mechanisms. Task performance is mathematical accuracy or functional pass@1 (%), reported as the mean over five seeds.

\tau

includes the resample or bonus token.

Proposal	Benchmark	SD	NSD
DFlash-b16	MATH500	115.92	140.77
	AIME25	87.48	115.15
	HumanEval	118.54	141.18
	MBPP	102.36	121.65
EAGLE-3-TTT7	MATH500	157.25	188.61
	AIME25	115.63	155.62

Appendix

Table 5: Output throughput (tokens/s) under additional proposal mechanisms. DFlash reports per-benchmark results for seed 30. Both methods ran unusually slowly on this seed; these are observed throughputs, not measurements under verified uncontended conditions. EAGLE-3 reports per-benchmark throughput for uncontended seeds 3, 7, and 22.

	Acc. (%)		$\tau$
Benchmark	SD	NSD	SD	NSD
MATH500	92.73	92.33	2.63	2.95
AIME25	35.56	34.44	2.57	2.94
HumanEval	78.25	80.89	2.55	2.95
MBPP	55.12	57.85	2.42	2.92

Appendix

Table 6: Qwen3.5-4B MTP results with block size

K=2

. Acc. is mean accuracy or pass@1 (%);

\tau

is weighted by verification-block counts. The maximum committed length is three, including the continuation token.

Explore similar work

Approximate Speculative Decoding
Aug 4, 2026Yuannuo Feng, Zegang Peng, Yuxin Xie +5Speculative Decoding Truncation
School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Department of Precision Instrument, Tsinghua University, Beijing, China · Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China +2
Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware
Jul 19, 2026Param ChordiyaSpeculative Decoding Autoregressive Decoding
University of California, San Diego
D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models
Jun 3, 2026Liyuan Zhang, Jiarui Zhang, Jinwei Yao +6Speculative Decoding Block-Diffusion Drafter
Peking University · Tsinghua University · HKUST +2

Aug 4, 2026, cs.LGJ/K move · Enter open · S save

Approximate Speculative Decoding

Yuannuo Feng, Zegang Peng, Yuxin Xie, Yubing Ye+4

School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Department of Precision Instrument, Tsinghua University, Beijing, China · Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China+2

Speculative Decoding Truncation Deepseek Greedy Algorithm

Open paper AskPDF arXiv

Speculative decoding accelerates autoregressive generation by verifying a draft block with a target model in parallel. Under standard greedy verification, decoding stops at the first draft token that differs from the target argmax, discarding the remaining target-scored suffix. Although accepting such a mismatch changes the decoding trajectory, it can make a contiguous suffix reusable when its tokens remain target-greedy under the realized prefix. In this paper, we introduce \textbf{Approximate Speculative Decoding (ASD)}, a training-free verifier that replaces binary first-mismatch truncation with budgeted longest-prefix selection. ASD accepts selected mismatches subject to a local target-logit regret gate, a per-block exception cap, and a persistent request-level regret budget, then reuses the contiguous target-greedy suffix without additional approximate decisions or target-model forward passes. ASD requires neither a new draft model nor fine-tuning, and exactly reduces to standard greedy verification when the budget is zero. Experiments show that ASD improves fixed-workload throughput by

3.05\%

15.26\%

over matched strict verification and averages a

7.78\%

gain across seven Qwen3-14B + DSpark-14B tasks. On DeepSeek-V4-Flash (284B) with DSpark it also raises verifier-side acceptance by roughly

10\%

16\%

on GSM8K and MATH-500 in an FP4-to-FP8 compatibility setting. The source code is publicly available at: https://github.com/Kissmetothemoon/ASD

Abstract

Explore similar work

Approximate Speculative Decoding

Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware

D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models