Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution

Shuhao Li; Fanghua Ye; Wanyu Lin; Tianyu Yuan; Xiaoyu Shen

cs.CLOct 6, 2026

Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution

Authors: Shuhao Li, Fanghua Ye, Wanyu Lin, Tianyu Yuan, Xiaoyu Shen

Organizations: EIT-NLP Lab, Eastern Institute of Technology, Ningbo · The Hong Kong Polytechnic University · Independent Researcher

Abstract

Speculative decoding accelerates autoregressive generation by using a lightweight draft model to propose multiple tokens that are verified by a target model in parallel. However, the standard acceptance rule focuses on exact distribution correction and rejects tokens that remain highly plausible under the target model when the draft model assigns excess probability. This conservative verification limits the number of draft tokens retained after each verification forward pass. We introduce Nucleus Speculative Decoding (NSD), a relaxed verification method that incorporates target-model plausibility into speculative decoding. NSD accepts a draft token if it satisfies the standard acceptance rule or belongs to the target model's nucleus. We theoretically characterize the distributional deviation introduced by our method and show that the single-step error is exactly determined by the draft model's excess probability within the target nucleus. We further derive sequence-level fidelity bounds that quantify how local deviations accumulate over autoregressive decoding. Experiments across multiple target models and proposal mechanisms demonstrate that NSD consistently improves speculative decoding efficiency while maintaining competitive task performance. Our method achieves throughput speedups of up to $5.16\times$ over autoregressive decoding and up to $3.15\times$ over standard speculative decoding. These improvements coincide with longer accepted lengths, allowing more output tokens to share the cost of each target verification pass. Analysis shows that plausibility-aware verification provides an effective approach for relaxed verification and speculative decoding efficiency. Our code is available at https://github.com/EIT-NLP/Nucleus-Speculative-Decoding.

Figures & tables

Figure 1: Execution time breakdown and rejection analysis of speculative decoding. (a) Execution time breakdown. Ratio of total time spent on drafting, verification, sampling, and overhead. (b) Target Top-K rank from rejected tokens. (c) Target Top-P distribution from rejected tokens.

Figure 2: Overview of Nucleus Speculative Decoding (NSD).

		Nucleus threshold $\theta$
		0.95			0.925			0.90			0.875			0.85
Dataset	$K$	Spd.	$\tau$	Acc.	Spd.	$\tau$	Acc.	Spd.	$\tau$	Acc.	Spd.	$\tau$	Acc.	Spd.	$\tau$	Acc.
Nemo-8B
GSM8K	8	$2.92\times$	6.61	91.66	$2.89\times$	6.61	91.66	$2.83\times$	6.67	91.66	$2.69\times$	6.51	92.42	$2.65\times$	6.54	92.87
	16	$4.35\times$	10.70	91.58	$3.63\times$	9.42	93.48	$4.21\times$	9.66	93.94	$3.93\times$	9.56	93.10	$4.06\times$	9.46	92.72
	24	$4.55\times$	11.90	92.87	$4.60\times$	11.74	90.90	$4.65\times$	12.09	90.14	$4.70\times$	12.03	90.98	$4.89\times$	12.47	91.58

Table 2: Effect of draft block size

K

and nucleus threshold

\theta

on speedup, mean accepted length, and accuracy. Spd. denotes speedup relative to target-model AR;

\tau

is the mean accepted length per draft block verification; Acc. is math accuracy or code pass@1 (%).

Figure 3: Nucleus-based acceptance statistics and verification efficiency. (a) Draft and target entropy. (b) Nucleus location and log probability mismatch; location zero denotes the target rank-1 token, with its strip widened for visibility. (c) Nucleus-accepted token shares and their cumulative ratio by target rank. (d) Verification time in ms/output token for Nemotron models.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Selection or intervention	Recovery and scope
SD	Stochastic ratio acceptance $\min(1,p/q)$ .	Sample from normalized $[p-q]_{+}$ at rejection; preserves the chosen target law under the usual assumptions.
NSD	Retain ratio acceptance and accept in-nucleus failures.	Keep the same residual; local added acceptance and TV error equal $E_{S}$ . No request-level budget.
ASD [ 10 ]	Greedy exceptions constrained by logit regret, a block cap, and a persistent request budget.	Target argmax recovery or bonus under the committed prefix. Zero budget restores greedy verification; regret accounting is not an NSD TV bound.
MARS [ 23 ]	Accept the target top-1 token, or its top-2 token when the raw-logit ratio $z_{(2)}/z_{(1)}>\theta$ .	On failure, emit the target top-1 token and discard the remaining draft. The stated rule relaxes token matching; it does not use the stochastic $[p-q]_{+}$ residual.
Cactus [ 13 ]	Select an adjusted distribution under a divergence constraint.	Uses generalized acceptance/recovery equations. NSD instead fixes the original residual and analyzes the induced deviation.

Appendix

Table 3: Mechanism distinctions. Here

p

and

q

always denote target and actual proposal laws; notation used in other papers is not imported into these columns.

		Acc. (%)		$\tau$
Proposal	Benchmark	SD	NSD	SD	NSD
DFlash-b16	MATH500	83.88	84.28	5.75	6.32
	AIME25	17.33	18.67	4.61	5.13
	HumanEval	84.51	83.17	5.48	6.20
	MBPP	62.41	63.50	4.77	5.29
EAGLE-3-TTT7	MATH500	83.64	83.32	4.54	5.32

Appendix

Table 4: Task performance and mean accepted length for additional proposal mechanisms. Task performance is mathematical accuracy or functional pass@1 (%), reported as the mean over five seeds.

\tau

includes the resample or bonus token.

Proposal	Benchmark	SD	NSD
DFlash-b16	MATH500	115.92	140.77
	AIME25	87.48	115.15
	HumanEval	118.54	141.18
	MBPP	102.36	121.65
EAGLE-3-TTT7	MATH500	157.25	188.61
	AIME25	115.63	155.62

Appendix

Table 5: Output throughput (tokens/s) under additional proposal mechanisms. DFlash reports per-benchmark results for seed 30. Both methods ran unusually slowly on this seed; these are observed throughputs, not measurements under verified uncontended conditions. EAGLE-3 reports per-benchmark throughput for uncontended seeds 3, 7, and 22.

	Acc. (%)		$\tau$
Benchmark	SD	NSD	SD	NSD
MATH500	92.73	92.33	2.63	2.95
AIME25	35.56	34.44	2.57	2.94
HumanEval	78.25	80.89	2.55	2.95
MBPP	55.12	57.85	2.42	2.92

Appendix

Table 6: Qwen3.5-4B MTP results with block size

K=2

. Acc. is mean accuracy or pass@1 (%);

\tau

is weighted by verification-block counts. The maximum committed length is three, including the continuation token.

Explore similar work

Aug 4, 2026cs.LG

Approximate Speculative Decoding

Speculative decoding accelerates autoregressive generation by verifying a draft block with a target model in parallel. Under standard greedy verification, decoding stops at the first draft token that differs from the target argmax, discarding the remaining target-scored suffix. Although accepting such a mismatch changes the decoding trajectory, it can make a contiguous suffix reusable when its tokens remain target-greedy under the realized prefix. In this paper, we introduce \textbf{Approximate Speculative Decoding (ASD)}, a training-free verifier that replaces binary first-mismatch truncation with budgeted longest-prefix selection. ASD accepts selected mismatches subject to a local target-logit regret gate, a per-block exception cap, and a persistent request-level regret budget, then reuses the contiguous target-greedy suffix without additional approximate decisions or target-model forward passes. ASD requires neither a new draft model nor fine-tuning, and exactly reduces to standard greedy verification when the budget is zero. Experiments show that ASD improves fixed-workload throughput by

3.05\%

15.26\%

over matched strict verification and averages a

7.78\%

gain across seven Qwen3-14B + DSpark-14B tasks. On DeepSeek-V4-Flash (284B) with DSpark it also raises verifier-side acceptance by roughly

10\%

16\%

on GSM8K and MATH-500 in an FP4-to-FP8 compatibility setting. The source code is publicly available at: https://github.com/Kissmetothemoon/ASD

Yuannuo Feng, Zegang Peng, Yuxin Xie +5

School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Department of Precision Instrument, Tsinghua University, Beijing, China · Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China +2

Speculative Decoding Truncation Deepseek Greedy Algorithm

Ask PDF

Jul 19, 2026cs.AI

Lossless but Not Free: An Empirical Anatomy of Speculative Decoding on Consumer Hardware

Single-stream autoregressive decoding of large language models is bound by memory bandwidth: each generated token requires one full forward pass through the target model, and successive passes cannot be parallelized. Speculative decoding restructures this computation: a small draft model proposes

K

tokens autoregressively, the target model scores all of them in one batched pass, and a rejection-sampling rule provably preserves the target model's output distribution. We present a from-scratch, device-agnostic (CUDA/MPS/CPU) implementation and an empirical study across five draft/target backend configurations on a consumer Apple-silicon laptop. Distribution equivalence is verified at three levels, culminating in a two-sample test over roughly 9,200 real-model tokens per method (

χ^2 = 162.5

, dof

= 200

p = 0.976

) and exact greedy-sequence agreement. The best configuration reaches a measured

1.61\times

wall-clock speedup at

K=6

, on an acceptance profile declining from 69.7% at

K=1

to 37.8% at the optimum, while three of five configurations decelerate, either because the draft fails to out-speed a small target or because the quantized Metal backend executes "parallel" verification serially, an effect we isolate and quantify. The failures are as instructive as the successes: speculative decoding pays off only when verification is genuinely batch-parallel and the draft/target latency gap is real.

Param Chordiya

University of California, San Diego

Speculative Decoding Autoregressive Decoding Decode Speedup Cuda Edge Hardware Anatomy

Ask PDF

Jun 3, 2026cs.DC

D^2SD: Accelerating Speculative Decoding with Dual Diffusion Draft Models

Speculative decoding accelerates autoregressive large language model inference by drafting multiple tokens and verifying them in a single target-model forward pass. Recent diffusion-based drafters generate an entire block of tokens in parallel but usually commit to a single draft sequence per verification: once the first mismatch occurs, all subsequent draft tokens are discarded, resulting in a limited acceptance rate. Naively batching more draft candidate sequences only introduces a marginal improvement, as redundant or poorly placed branches increase the cost of drafting and verification without proportionally increasing the number of accepted tokens. We propose D^2SD, a dual diffusion draft speculative decoding framework that organizes candidates into a confidence-guided prefix tree, where the first diffusion drafter generates a block along with per-position confidence scores that are used to identify the most likely rejection boundary and select the top-K prefix ranges for recovery; the second variable-prefix diffusion drafter re-anchors at each selected prefix and proposes alternative continuations in one batched pass; the resulting shared-prefix candidates are jointly verified via cascade attention. Empirically, D^2SD shows clear improvements over both the underlying diffusion approach and strong autoregressive speculative decoding baselines.

Liyuan Zhang, Jiarui Zhang, Jinwei Yao +6

Peking University · Tsinghua University · HKUST +2

Speculative Decoding Block-Diffusion Drafter Diffusion Language Models LLM Inference Optimization Single Pass

Ask PDF