Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference

Xuwen Zhou; Fangxin Liu; Chao Wang; Xiao Zheng; Hao Zheng; Min He; Li Jiang; Haibing Guan

cs.CLApr 15, 2026

Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference

Authors: Xuwen Zhou, Fangxin Liu, Chao Wang, Xiao Zheng, Hao Zheng, Min He, Li Jiang, Haibing Guan

Organizations: Shanghai Jiao Tong University · Alibaba Cloud Computing

Abstract

Speculative decoding accelerates autoregressive generation by letting draft tokens bypass full verification, but conventional frameworks suffer from frequent false rejections, particularly when draft models produce semantically correct but lexically divergent outputs. In this paper, we present Calibrated Speculative Decoding (CSD), a training-free framework that recovers valid tokens discarded by standard verification. Guided by the principle of "Frequency-Guided Candidate Selection and Probability-Guarded Acceptance," CSD incorporates two lightweight modules: Online Correction Memory, which aggregates historical rejections to propose recurring divergence patterns as rescue candidates, and Semantic Consistency Gating, which verifies candidate admissibility using probability ratios instead of exact token matching. Our evaluation across diverse large language models demonstrates that CSD outperforms existing methods, achieving a peak throughput speedup of 2.33x. CSD preserves model accuracy across all tasks while further boosting performance on complex reasoning datasets. These results establish CSD as a highly effective, lightweight solution for practical LLM deployments.

Figures & tables

Figure 1: Overview of Calibrated Speculative Decoding (CSD). (a) Standard Speculative Decoding suffers from “False Rejections,” where trivial lexical mismatches (e.g., ‘x’ vs ‘*’) cause the discard of subsequent valid tokens (e.g., 5=10 ). (b) CSD mitigates such rigid rejections by introducing a Rescue Mechanism. It employs Online Correction Memory (OCM) to propose potential candidates based on historical priors, and uses Semantic Consistency Gating (SCG) to verify their admissibility via probability ratios (Score $\geq\tau$ ), effectively recovering wasted computation.

Figure 2: Statistical Analysis of False Rejections. (a) Long-Tailed Frequency: The distribution of rejected patterns is highly skewed; the 20% (head patterns) account for 69.0% of total rejections. This suggests a lightweight Correction Memory targeting these patterns can recover most failed speculations. (b) Ratio Variance: Extreme variance in probability ratios ( $T=1$ ) reflects a spectrum from severe hallucinations to semantically consistent alternatives. This diversity necessitates dynamic Semantic Consistency Gating to distinguish valid candidates from errors, moving beyond static relaxation.

Llama-3 Series
	GSM8K			MATH500			HumanEval			CNN/DM			Avg.
	(8-shot, Pass@1)			(4-shot, Pass@1)			(0-shot, Pass@1)			(0-shot, ROUGE-L)
Method	Acc	Tp	Spd	Acc	Tp	Spd	Acc	Tp	Spd	Acc	Tp	Spd	Spd
Vanilla Decoding	92.6	13.4	1.00 $\times$	46.0	15.2	1.00 $\times$	76.8	15.9	1.00 $\times$	20.3	15.3	1.00 $\times$	1.00 $\times$
SpecDecode	92.3	23.3	1.74 $\times$	45.4	28.8	1.89 $\times$	76.8	30.1	1.90 $\times$	20.3	22.8	1.49 $\times$	1.75 $\times$
Lossy SD ( $\tau=0.6$ )	92.3	24.2	1.80 $\times$	49.4	30.6	2.00 $\times$	78.0	32.3	2.04 $\times$	20.3	23.6	1.55 $\times$	1.85 $\times$

Table 1: Main results across Llama-3 (70B/1B) and Qwen-2.5 (72B/7B) families. We report Acc (Task accuracy), Tp (Throughput in tokens/s), and Spd (Speedup relative to vanilla decoding). Avg. represents the mean speedup across all benchmarks. Best results are highlighted in bold .

	GSM8K				MATH500				HumanEval				CNN/DM				Avg.
Method	Acc	Tp	Spd	AR	Acc	Tp	Spd	AR	Acc	Tp	Spd	AR	Acc	Tp	Spd	AR	Acc	Tp	Spd	AR
Vanilla Decoding	92.6	13.4	1.00 $\times$	0.0%	46.0	15.2	1.00 $\times$	0.0%	76.8	15.9	1.00 $\times$	0.0%	20.3	15.3	1.00 $\times$	0.0%	58.9	14.9	1.00 $\times$	0.0%
SpecDecode	92.3	22.0	1.64 $\times$	46.2%	45.4	28.2	1.85 $\times$	42.9%	76.8	25.6	1.61 $\times$	39.9%	20.3	17.2	1.13 $\times$	23.2%	58.7	23.2	1.56 $\times$	38.0%
Fly	91.8	23.7	1.77 $\times$	53.1%	46.8	32.0	2.10 $\times$	51.4%	76.8	31.3	1.97 $\times$	48.2%	20.3	17.5	1.14 $\times$	26.7%	58.9	26.1	1.75 $\times$	44.8%
Reflect Verification	92.0	22.6	1.69 $\times$	62.0 %	49.6	29.6	1.94 $\times$	58.2%	78.0	30.6	1.93 $\times$	51.7 %	20.2	22.7	1.49 $\times$	38.7%	59.9	26.4	1.76 $\times$	52.6%
CSD	92.5	24.6	1.84 $\times$	58.3%	49.4	35.2	2.31 $\times$	59.3%	76.8	33.4	2.11 $\times$	49.8%	20.2	20.2	1.32 $\times$	33.3%	59.7	28.4	1.89 $\times$	50.2%

Table 2: Comparison with Semantic Verification Methods. AR denotes the Acceptance Rate. Other metrics ( Acc , Tp , Spd ) follow the same definitions as in Table 1 . Best results are highlighted in bold .

	MATH500				HumanEval
Variant	Acc	AR	Tp	Spd	Acc	AR	Tp	Spd
SpecDecode (Baseline)	45.4	63.6%	28.8	1.00 $\times$	76.8	59.7%	30.1	1.00 $\times$
SD w/ OCM	37.8	83.1%	36.6	1.27 $\times$	70.7	71.6%	37.2	1.24 $\times$
SD w/ SCG	43.6	88.7%	38.2	1.33 $\times$	70.7	88.6%	44.4	1.48 $\times$
CSD (Ours)	48.0	79.6%	35.5	1.23 $\times$	79.3	67.9%	37.0	1.23 $\times$

Table 3: Ablation study of CSD components on MATH500 and HumanEval. AR denotes the Acceptance Rate. Note that individual modules ( w/ OCM or w/ SCG) incur accuracy degradation compared to the baseline, while our integrated CSD framework achieves optimal balance.

Figure 3: Sensitivity analysis on MATH500. (a) Calibration : AR plateaus beyond 2k–4k samples and peaks at 8k. (b) Semantic Gating : Optimal trade-off at

\tau=0.01

. (c) Frequency :

\lambda=6

maximizes accuracy. Dotted lines indicate the accuracy and AR baselines of SpecDecode.

	Acc		AR		Speedup
Task	Spec.	Univ.	Spec.	Univ.	Spec.	Univ.
HumanEval	79.2	78.0	67.9%	71.1%	2.33 $\times$	2.45 $\times$
MATH500	48.0	50.0	79.6%	74.3%	2.32 $\times$	2.23 $\times$
GSM8K	92.6	92.8	76.4%	74.5%	1.83 $\times$	1.80 $\times$
CNN/DM	20.3	20.3	55.3%	54.4%	1.59 $\times$	1.59 $\times$
Avg.	60.0	60.3	69.8%	68.6%	2.02 $\times$	2.02 $\times$

Table 4: Generalization to Universal Calibration. We compare CSD calibrated on task-specific data ( Spec. ) against CSD using a Univ. (Universal) calibration set. Avg. denotes the macro-average across all tasks.

Figure 4: Inference Latency Breakdown. A detailed latency analysis for Qwen-72B/7B and Llama-70B/1B. The results highlight that the CSD overhead is negligible relative to the total inference time.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

	IFEval			RULER
Method	Acc	AR	Spd	Acc	AR	Spd
Vanilla	76.8	-	1.00 $\times$	88.1	-	1.00 $\times$
Standard SD	77.4	39.7%	1.65 $\times$	88.0	73.6%	1.02 $\times$
CSD (Ours)	76.8	42.3%	1.77 $\times$	95.9	83.2%	1.37 $\times$

Appendix

Table 5: Performance comparison on IFEval (Instruction Following) and RULER (Long-Context). AR denotes Acceptance Rate, and Spd denotes the end-to-end speedup relative to Vanilla.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 308 papers

Entropy-Aware Token Rejection for Improving Speculative Decoding
Dec 29, 2025Tiancheng Su, Meicong Zhang, Guoxiu HeSpeculative Decoding LLM Inference Acceleration
School of Economics and Management East China Normal University
SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration
Apr 14, 2026Zhuofan Wen, Yang FengSpeculative Decoding LLM Inference Acceleration
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
RCSD: Reinforcement and Continual Learning for Domain-Specific Speculative Decoding
Oct 21, 2026Mingkang Liu, Weiming Meng, Peipei Jiao +4Speculative Decoding LLM Inference Acceleration

Dec 29, 2025J/K move · Enter open · S save

Entropy-Aware Token Rejection for Improving Speculative Decoding

Tiancheng Su, Meicong Zhang, Guoxiu He

School of Economics and Management East China Normal University

Speculative Decoding LLM Inference Acceleration LLM Reasoning

Open paper AskPDF arXiv

Speculative decoding (SD) accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens and a stronger target model to verify them. However, standard SD is mainly designed for acceleration, and its output quality is typically constrained by the target model. In this work, we propose Entropy-Aware Speculative Decoding (EASD), a lightweight and training-free extension of SD that improves reasoning quality through token-level entropy-guided rejection. EASD detects cases where both draft and target models exhibit high uncertainty while strongly overlapping in their top predictions. In such uncertain-agreement cases, EASD rejects the aligned token and resamples from the target distribution, preventing low-confidence errors from propagating. Experiments on challenging reasoning benchmarks show that EASD consistently improves accuracy over standard SD and reward-guided variants while maintaining comparable inference efficiency. Notably, EASD can surpass the standalone performance of the target model, suggesting that speculative decoding can serve not only as an acceleration method but also as an effective mechanism for improving reasoning quality. The code is available at https://github.com/ECNU-Text-Computing/EASD.

Abstract

Speculative Decoding

Explore similar work

Entropy-Aware Token Rejection for Improving Speculative Decoding

SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration

RCSD: Reinforcement and Continual Learning for Domain-Specific Speculative Decoding