ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shaping

Zhi-Kai Chen; Jun-Jie Tao; Wei-Xiang Mao; De-Chuan Zhan; Han-Jia Ye

cs.AIAug 25, 2026

ResiSpec: Enhancing Multi-Candidate Speculative Sampling via Residual Distribution Shaping

Authors: Zhi-Kai Chen, Jun-Jie Tao, Wei-Xiang Mao, De-Chuan Zhan, Han-Jia Ye

Organizations: School of Artificial Intelligence, Nanjing University, China · National Key Laboratory for Novel Software Technology, Nanjing University, China · Nanjing University, China

Abstract

The efficiency of Large Language Model (LLM) serving is fundamentally limited by the sequential nature of autoregressive decoding. Speculative Decoding (SD) mitigates this by using a lightweight draft model to speculate future tokens, which are then validated by the LLM in a single parallel forward pass. To further boost efficiency, multi-candidate schemes propose diverse candidate sets to increase the likelihood of token acceptance. However, we show that these schemes are bottlenecked by Residual Drift: a phenomenon where the rejection of initial candidates causes the residual target distribution to diverge from the draft model's predictions. This shift renders subsequent candidates ineffective and forces the system into expensive resampling. To resolve this, we propose ResiSpec, a framework that strategically reforms the proposal distribution during verification to anchor the residual target mass within the draft model's high-confidence regions. By mathematically re-aligning the verification process without compromising output exactness, ResiSpec prevents candidate obsolescence and achieves up to 1.92 $\times$ speedup over state-of-the-art multi-candidate methods. Code is available at https://github.com/Czzzk/Resispec.

Figures & tables

Figure 1: The Decay of Acceptance Probability. Sequential candidate acceptance rates on CNN/DM using Llama-2-7B and Vicuna-7B ( Peng et al. 2023 ) as targets with Llama-68M as the draft. Across temperatures (

T\in\{0.6,1.0\}

), acceptance drops sharply after the first candidate, leaving later candidates with near-zero marginal utility.

Figure 2: Comparison of ResiSpec vs. standard multi-candidate verification. In traditional schemes, sequential rejections trigger Residual Drift, shifting the target distribution into regions mathematically incompatible with the draft model (red areas), causing candidates to become obsolete. ResiSpec mitigates this drift by employing an auxiliary proxy to reshape the residual distribution (green areas), re-aligning it with the draft model’s high-density support. This proximity ensures that pre-sampled candidates remain viable, effectively sustaining high acceptance rates across the entire verification batch.

Target LLM	Draft Model	T	Dataset	ResiSpec		SpecInfer		ResiSpec vs SpecInfer
Target LLM	Draft Model	T	Dataset	Acc.↑	Thru.↑	Acc.↑	Thru.↑	Acc. Gain	Speedup
Llama-2-7B	JF68M	1	CNN/DM	4.68	96.43	2.51	57.29	1.86 $\times$	1.68 $\times$
Llama-2-7B	JF68M	1	OpenWebText	4.69	96.06	2.51	56.76	1.86 $\times$	1.69 $\times$
Llama-2-7B	JF68M	1	C4	4.67	96.90	2.64	61.02	1.77 $\times$	1.59 $\times$
Llama-2-7B	JF68M	0.6	CNN/DM	4.14	85.03	2.59	59.41	1.60 $\times$	1.43 $\times$
Llama-2-7B	JF68M	0.6	OpenWebText	4.16	85.62	2.49	56.48	1.67 $\times$	1.52 $\times$

Table 1: Performance comparison between ResiSpec and the baseline SpecInfer. Regarding the configuration of the tree, we use a 3-branch complete tree with a depth of 5. The results in the table are averaged over 200 runs to reduce variance and ensure statistical reliability. Experimental results demonstrate that ResiSpec significantly increased the accepted length (by up to 1.86

\times

) and throughput (by up to 1.68

\times

), validating its effectiveness in accelerating large language model inference.

Method	Acc. Len↑	Thru. (tok/s)↑	Acc. Len Gain	Speedup
Sequoia	3.07	88.14	–	–
Sequoia+ResiSpec	4.32	115.32	1.41 $\times$	1.31 $\times$
EAGLE	3.12	61.82	–	–
EAGLE+ResiSpec	3.85	71.17	1.23 $\times$	1.15 $\times$

Table 2: Compatibility of ResiSpec with existing multi-candidate speculative decoding methods at

T=1.0

. ResiSpec is evaluated as an add-on to Sequoia ( Chen et al. 2024 ) and EAGLE ( Li et al. 2024b ) under matched decoding settings, improving both metrics.

Figure 3: Runtime analysis of ResiSpec. Left: per-step latency across SpecInfer, Sequoia, and EAGLE when augmented with ResiSpec, decomposed into drafting, target verification, and ResiSpec shaping. Right: ResiSpec’s internal runtime breakdown across budgeting, redistribution, and proxy formulation, shown by percentage.

Tree	Verification Costs	ResiSpec	SpecInfer	SpeedUp
Tree	Verification Costs	Thru.↑	Thru.↑	SpeedUp
d7b2	127	101.01	52.69	1.92 $\times$
d5b3	121	94.25	56.92	1.66 $\times$
d4b5	156	60.83	42.46	1.43 $\times$
d3b11	133	44.21	37.89	1.17 $\times$

Table 3: Impact of tree topologies under comparable verification budgets on Llama-2-7B with JF68M. Verification cost is the number of candidate positions verified concurrently. ResiSpec gains more over SpecInfer on deeper, narrower trees.

Method	Data	KL ( $T=1.0$ )	KL ( $T=0.6$ )
ResiSpec	CNN/DM	2.64e-10	4.38e-10
	OpenWebText	7.15e-10	4.37e-10
	C4	1.02e-9	1.39e-10
SpecInfer	CNN/DM	1.04e-9	3.55e-10
	OpenWebText	1.63e-9	3.89e-10
	C4	9.10e-10	1.09e-10

Table 4: KL divergence between the target distribution

p(x)

and the sampling distributions of ResiSpec and SpecInfer on Llama-2-7B with JF68M. Near-zero values across datasets and temperatures confirm distributional preservation.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Candidates ( $N$ )	EAL↑	Marginal Gain ( $\Delta$ EAL)	Efficiency ( $\eta$ )
1	1.89	-	-
2	2.17	0.28	0.280
4	2.48	0.31	0.155
8	2.76	0.28	0.070

Appendix

Table 5: Scaling Inefficiency of Multi-Candidate Schemes. With speculation depth fixed at 4, increasing candidates

N

yields only modest EAL gains while the efficiency score

\eta

drops sharply. Results are averaged over CNN/DM using Llama-2-7B.

Target	Draft	T	Data	Tree	Resi. Acc.	Resi. Thru.	Spec. Acc.	Spec. Thru.
Sheared-2.7B	Sheared-1.3B	1	CNN/DM	d5b3	4.88	78.309	3.88	66.313
Sheared-2.7B	Sheared-1.3B	1	CNN/DM	d4b4	3.94	82.988	3.42	76.695
Sheared-2.7B	Sheared-1.3B	1	OpenWebText	d5b3	4.86	78.447	3.92	67.660
Sheared-2.7B	Sheared-1.3B	1	OpenWebText	d4b4	3.95	83.060	3.45	76.864
Sheared-2.7B	Sheared-1.3B	1	C4	d5b3	4.84	78.362	3.98	68.679
Sheared-2.7B	Sheared-1.3B	1	C4	d4b4	3.95	82.850	3.44	77.340

Appendix

Table 6: Ablation Experiment Results of Different Model Pair Combinations. Here, we use Llama-2-7B and Sheared-LLaMA-2.7B ( Xia et al. 2023 ) as the target model and Sheared-LLaMA-1.3B, TinyLlama-1.1B ( Zhang et al. 2024b ) , Tiny-Vicuna-1B and JF68M as the draft model. The experimental results show that ResiSpec significantly outperforms SpecInfer in terms of accepted length and throughput under different configurations.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting
Jun 16, 2026Lanxiang Hu, Zhaoxiang Feng, Yulun Wu +9Speculative Decoding LLM Inference Acceleration
UC San Diego · Zhejiang University · UIUC +2
Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference
Apr 15, 2026Xuwen Zhou, Fangxin Liu, Chao Wang +5Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Alibaba Cloud Computing
Entropy-Aware Token Rejection for Improving Speculative Decoding
Dec 29, 2025Tiancheng Su, Meicong Zhang, Guoxiu HeSpeculative Decoding LLM Inference Acceleration
School of Economics and Management East China Normal University

Jun 16, 2026J/K move · Enter open · S save

JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting

Lanxiang Hu, Zhaoxiang Feng, Yulun Wu, Haoran Yuan+8

UC San Diego · Zhejiang University · UIUC+2

Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Speculative decoding (SD) accelerates autoregressive Large Language Models (LLMs) by drafting multiple tokens and verifying them in parallel, but it faces a scaling limitation: increasing the draft budget improves speed only when acceptance remains high and drafting overhead stays low. This ceiling has been difficult to break because prior head-based SD methods face a causality-efficiency dilemma. Autoregressive drafters produce path-conditioned candidates that are effective for tree speculative decoding with higher acceptance length, but their drafting cost grows with tree depth. Bidirectional block-diffusion drafters generate all positions in one pass, but their branch-agnostic marginals can form individually plausible yet mutually inconsistent trees, wasting budget and reducing acceptance. We propose JetSpec, a head-based SD framework that combines one-forward drafting efficiency with branch-wise causal conditioning. JetSpec trains a causal parallel draft head over fused hidden states from the frozen target model, producing candidate trees whose scores align with the target model's autoregressive factorization. This enables JetSpec to convert larger draft budgets into longer accepted prefixes and higher end-to-end speedup. Across math, coding, and chat benchmarks on dense and MoE Qwen3 models, JetSpec consistently outperforms bidirectional-head and tree-based SD baselines. On H100 GPUs, JetSpec achieves up to 9.64x speedup on MATH-500 and 4.58x on open-ended conversational workloads, with further latency gains demonstrated through vLLM integration under realistic serving loads. Our code and models are available at https://github.com/hao-ai-lab/JetSpec.

Abstract

Speculative Decoding

Explore similar work

JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting

Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference

Entropy-Aware Token Rejection for Improving Speculative Decoding