SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration

Zhuofan Wen; Yang Feng

cs.CLApr 14, 2026

SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration

Authors: Zhuofan Wen, Yang Feng

Organizations: Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Abstract

Speculative decoding has emerged as a promising approach to accelerate autoregressive inference in large language models (LLMs). Self-draft methods, which leverage the base LLM itself for speculation, avoid the overhead of auxiliary draft models but face limitations: shallow layers often produce overconfident yet incorrect token predictions, and the presence of difficult tokens in a draft sequence forces redundant computation through deeper layers, undermining both draft acceptance and overall speedup. To address these issues, we propose a novel self-draft framework that suppresses spurious confidence via layer-wise temperature annealing in early-exit decision and adaptively bounds speculation length based on token-wise decoding difficulty. By reprocessing the hidden states of draft tokens in a unified parallel pass through deep layers, our method maintains exact output equivalence with the original model while maximizing computational efficiency. It requires no modifications to the base LLM parameters and achieves up to 2.33x wall-time speedup over standard autoregressive decoding across diverse long-form generation tasks and multiple model architectures.

Figures & tables

Figure 1: Layer-wise token prediction and confidence of a segment with the MT-Bench prompt. Each cell shows the greedy-decoded token and its confidence (color intensity: darker = higher probability) at a given layer. Gray cells indicate that the token has already correctly exited at an earlier layer and is no longer considered for early-exit decisions. This figure reveals the issues of spurious high-confidence predictions and heterogeneous exit depths.

Figure 2: Left: Overview of the BSCS algorithm under the setting $d_{\max}=6$ and $w_{\max}=4$ . When token “a” fails to exit by layer 6, speculation is terminated. All hidden states of previously generated tokens (“I”, “am”, “a”) are concatenated and passed to the remaining layers for parallel verification. Right: Detailed illustration of per-token layer-wise computation and hidden-state cache management.

Vicuna-7B
Method	Math		Multi.		QA		RAG		Sum.		Trans.		Overall
Method	CR	SD	CR	SD	CR	SD	CR	SD	CR	SD	CR	SD	Overall
Lookahead Fu et al. (2024)	1.92	1.54 $\times$	1.74	1.49 $\times$	1.52	1.23 $\times$	1.53	1.24 $\times$	1.57	1.38 $\times$	1.28	1.16 $\times$	1.35 $\times$
Medusa Cai et al. (2023)	1.77	1.96 $\times$	1.72	1.95 $\times$	1.54	1.67 $\times$	1.47	1.47 $\times$	1.53	1.59 $\times$	1.55	1.63 $\times$	1.71 $\times$
REST He et al. (2024)	1.49	1.19 $\times$	2.04	1.65 $\times$	1.87	1.64 $\times$	1.96	1.75 $\times$	1.60	1.34 $\times$	1.58	1.33 $\times$	1.47 $\times$
Kangaroo Liu et al. (2024)	2.14	1.61 $\times$	2.22	1.68 $\times$	1.87	1.43 $\times$	2.05	1.52 $\times$	1.87	1.50 $\times$	1.41	1.24 $\times$	1.50 $\times$

Table 1: Performance comparison of SpecBound against other self-drafting methods across multiple base models and diverse text generation tasks on Spec-Bench. We report compression rate (CR, average accepted tokens per draft round) and wall-time speedup (SD) relative to autoregressive decoding.

Figure 3: Hyperparameter Sensitivity Analysis on Vicuna-7B: wall-time speedup (blue) and compression rate (CR, orange) under varying exit threshold

\gamma

, annealing coefficient

\alpha

, depth bound

d_{\max}

, and width bound

w_{\max}

Figure 4: Ablation study of SpecBound components on wall-time speedup across different base models, comparing the full method against variants that removing the Annealed Confidence Threshold early exit mechanism, the depth bound

d_{\max}

, or the width bound

w_{\max}

Method	Math	Multi.	QA	RAG	Sum.	Trans.	Overall
greedy decoding	1.89 $\times$	1.65 $\times$	2.16 $\times$	2.31 $\times$	1.95 $\times$	2.94 $\times$	2.15 $\times$
temperature sampling (T=0.3)	1.82 $\times$	1.56 $\times$	2.01 $\times$	2.20 $\times$	1.86 $\times$	2.86 $\times$	2.08 $\times$
temperature sampling (T=1.0)	1.71 $\times$	1.44 $\times$	1.92 $\times$	2.08 $\times$	1.73 $\times$	2.63 $\times$	1.88 $\times$
top-p sampling (p=0.9)	1.78 $\times$	1.48 $\times$	1.99 $\times$	2.15 $\times$	1.78 $\times$	2.71 $\times$	1.94 $\times$

Table 2: Speedup performance of SpecBound (Vicuna-7B) under different decoding strategies on different tasks of Spec-Bench. We choose greedy docoding, temperature sampling and top-p sampling with different settings.

Metric	AR (Base)	Ours	Improv.
TTFT (ms)	20.8	21.1	-
Per-token latency (ms)	17.5	8.8	1.98 $\times$
Throughput (tokens/s)	57.3	123.1	2.15 $\times$
End-to-End Latency (s)	3.31	1.89	1.75 $\times$

Table 3: Serving latency metrics comparison on Vicuna-7B using a single NVIDIA H800 GPU. "AR" denotes standard autoregressive decoding.

Figure 5: Left: Overview of the BSCS algorithm under the setting $d_{\max}=6$ and $w_{\max}=4$ . When token “chat” successfully exits at layer 2 and the total draft length reaches the width bound $w_{\max}$ , speculation is terminated. All hidden states of previously generated tokens (“I”, “am”, “a”, “chat”) are concatenated and passed to the remaining layers for parallel verification. Right: Detailed illustration of per-token layer-wise computation and hidden-state cache management.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Device	Math	Multi-turn	QA	RAG	Summary	Translation	Overall
H800	1.89 $\times$	1.65 $\times$	2.16 $\times$	2.31 $\times$	1.95 $\times$	2.94 $\times$	2.15 $\times$
RTX 3090	1.81 $\times$	1.58 $\times$	2.12 $\times$	2.13 $\times$	1.92 $\times$	2.69 $\times$	2.01 $\times$

Appendix

Table 4: Speedup performance of SpecBound (Vicuna-7B) on different GPU devices across various Spec-Bench tasks. The evaluation are conducted on H800 and RTX 3090 with all other settings are the same.

Speedup Range	Number	Percentage
$3.54\times-4.16\times$ (Max)	16	3.33%
$2.93\times-3.54\times$	48	10.00%
$2.31\times-2.93\times$	103	21.46%
$1.69\times-2.31\times$	197	41.04%
$1.07\times$ (Min) $-1.69\times$	116	24.17%

Appendix

Table 5: Empirical speedup distribution of SpecBound on Vicuna-7B across 480 prompts from SpecBench.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 293 papers

ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding
Apr 16, 2026Walaa Amer, Uday das, Fadi KurdahiSpeculative Decoding LLM Inference Acceleration
Department of Electrical Engineering and Computer Sciences, University of California Irvine, Irvine, USA · Advanced Micro Devices, San Jose, USA
A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
Jul 30, 2026Yuesong Liu, Yuan Zeng, Min Lyu +3Long-Context Language Model Inference Speculative Decoding
University of Science and Technology of China
SEED: Self-Speculative Decoding via Implicit Encoder-Decoder
Sep 29, 2026Hankun Lin, Patrick Pynadath, Ruqi ZhangSpeculative Decoding LLM Inference Acceleration
Department of Computer Science, Purdue University, USA

Apr 16, 2026J/K move · Enter open · S save

ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding

Walaa Amer, Uday das, Fadi Kurdahi

Department of Electrical Engineering and Computer Sciences, University of California Irvine, Irvine, USA · Advanced Micro Devices, San Jose, USA

Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Self-speculative decoding is an inference technique for large language models designed to speed up generation without sacrificing output quality. It combines fast, approximate decoding using a compact version of the model as a draft model with selective re-evaluation by the full target model. Some existing methods form the draft model by dynamically learning which layers to skip during inference, effectively creating a smaller subnetwork to speed up computation. However, using heuristic-based approaches to select layers to skip can often be simpler and more effective. In this paper, we propose ConfLayers, a dynamic plug-and-play approach to forming the draft model in self-speculative decoding via confidence-based intermediate layer skipping. The process iteratively computes confidence scores for all layers, selects layers to skip based on an adaptive threshold, evaluates the performance of the resulting set, and updates the best selection until no further improvement is achieved or a maximum number of iterations is reached. This framework avoids the overhead and complexity of training a layer skipping policy and can provide more consistent speed-quality trade-offs while preserving the adaptivity of the draft model to diverse tasks and datasets. The performance evaluation of ConfLayers across different models and datasets shows that our novel approach offers up to 1.4x speedup over vanilla LLM generation.

Abstract

Speculative Decoding

Explore similar work

ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding

A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

SEED: Self-Speculative Decoding via Implicit Encoder-Decoder