SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration

Zhuofan Wen; Yang Feng

cs.CLApr 14, 2026

SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration

Authors: Zhuofan Wen, Yang Feng

Organizations: Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Abstract

Speculative decoding has emerged as a promising approach to accelerate autoregressive inference in large language models (LLMs). Self-draft methods, which leverage the base LLM itself for speculation, avoid the overhead of auxiliary draft models but face limitations: shallow layers often produce overconfident yet incorrect token predictions, and the presence of difficult tokens in a draft sequence forces redundant computation through deeper layers, undermining both draft acceptance and overall speedup. To address these issues, we propose a novel self-draft framework that suppresses spurious confidence via layer-wise temperature annealing in early-exit decision and adaptively bounds speculation length based on token-wise decoding difficulty. By reprocessing the hidden states of draft tokens in a unified parallel pass through deep layers, our method maintains exact output equivalence with the original model while maximizing computational efficiency. It requires no modifications to the base LLM parameters and achieves up to 2.33x wall-time speedup over standard autoregressive decoding across diverse long-form generation tasks and multiple model architectures.

Figures & tables

Figure 1: Layer-wise token prediction and confidence of a segment with the MT-Bench prompt. Each cell shows the greedy-decoded token and its confidence (color intensity: darker = higher probability) at a given layer. Gray cells indicate that the token has already correctly exited at an earlier layer and is no longer considered for early-exit decisions. This figure reveals the issues of spurious high-confidence predictions and heterogeneous exit depths.

Figure 2: Left: Overview of the BSCS algorithm under the setting $d_{\max}=6$ and $w_{\max}=4$ . When token “a” fails to exit by layer 6, speculation is terminated. All hidden states of previously generated tokens (“I”, “am”, “a”) are concatenated and passed to the remaining layers for parallel verification. Right: Detailed illustration of per-token layer-wise computation and hidden-state cache management.

Vicuna-7B
Method	Math		Multi.		QA		RAG		Sum.		Trans.		Overall
Method	CR	SD	CR	SD	CR	SD	CR	SD	CR	SD	CR	SD	Overall
Lookahead Fu et al. (2024)	1.92	1.54 $\times$	1.74	1.49 $\times$	1.52	1.23 $\times$	1.53	1.24 $\times$	1.57	1.38 $\times$	1.28	1.16 $\times$	1.35 $\times$
Medusa Cai et al. (2023)	1.77	1.96 $\times$	1.72	1.95 $\times$	1.54	1.67 $\times$	1.47	1.47 $\times$	1.53	1.59 $\times$	1.55	1.63 $\times$	1.71 $\times$
REST He et al. (2024)	1.49	1.19 $\times$	2.04	1.65 $\times$	1.87	1.64 $\times$	1.96	1.75 $\times$	1.60	1.34 $\times$	1.58	1.33 $\times$	1.47 $\times$
Kangaroo Liu et al. (2024)	2.14	1.61 $\times$	2.22	1.68 $\times$	1.87	1.43 $\times$	2.05	1.52 $\times$	1.87	1.50 $\times$	1.41	1.24 $\times$	1.50 $\times$

Table 1: Performance comparison of SpecBound against other self-drafting methods across multiple base models and diverse text generation tasks on Spec-Bench. We report compression rate (CR, average accepted tokens per draft round) and wall-time speedup (SD) relative to autoregressive decoding.

Figure 3: Hyperparameter Sensitivity Analysis on Vicuna-7B: wall-time speedup (blue) and compression rate (CR, orange) under varying exit threshold

\gamma

, annealing coefficient

\alpha

, depth bound

d_{\max}

, and width bound

w_{\max}

Figure 4: Ablation study of SpecBound components on wall-time speedup across different base models, comparing the full method against variants that removing the Annealed Confidence Threshold early exit mechanism, the depth bound

d_{\max}

, or the width bound

w_{\max}

Method	Math	Multi.	QA	RAG	Sum.	Trans.	Overall
greedy decoding	1.89 $\times$	1.65 $\times$	2.16 $\times$	2.31 $\times$	1.95 $\times$	2.94 $\times$	2.15 $\times$
temperature sampling (T=0.3)	1.82 $\times$	1.56 $\times$	2.01 $\times$	2.20 $\times$	1.86 $\times$	2.86 $\times$	2.08 $\times$
temperature sampling (T=1.0)	1.71 $\times$	1.44 $\times$	1.92 $\times$	2.08 $\times$	1.73 $\times$	2.63 $\times$	1.88 $\times$
top-p sampling (p=0.9)	1.78 $\times$	1.48 $\times$	1.99 $\times$	2.15 $\times$	1.78 $\times$	2.71 $\times$	1.94 $\times$

Table 2: Speedup performance of SpecBound (Vicuna-7B) under different decoding strategies on different tasks of Spec-Bench. We choose greedy docoding, temperature sampling and top-p sampling with different settings.

Metric	AR (Base)	Ours	Improv.
TTFT (ms)	20.8	21.1	-
Per-token latency (ms)	17.5	8.8	1.98 $\times$
Throughput (tokens/s)	57.3	123.1	2.15 $\times$
End-to-End Latency (s)	3.31	1.89	1.75 $\times$

Table 3: Serving latency metrics comparison on Vicuna-7B using a single NVIDIA H800 GPU. "AR" denotes standard autoregressive decoding.

Figure 5: Left: Overview of the BSCS algorithm under the setting $d_{\max}=6$ and $w_{\max}=4$ . When token “chat” successfully exits at layer 2 and the total draft length reaches the width bound $w_{\max}$ , speculation is terminated. All hidden states of previously generated tokens (“I”, “am”, “a”, “chat”) are concatenated and passed to the remaining layers for parallel verification. Right: Detailed illustration of per-token layer-wise computation and hidden-state cache management.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Device	Math	Multi-turn	QA	RAG	Summary	Translation	Overall
H800	1.89 $\times$	1.65 $\times$	2.16 $\times$	2.31 $\times$	1.95 $\times$	2.94 $\times$	2.15 $\times$
RTX 3090	1.81 $\times$	1.58 $\times$	2.12 $\times$	2.13 $\times$	1.92 $\times$	2.69 $\times$	2.01 $\times$

Appendix

Table 4: Speedup performance of SpecBound (Vicuna-7B) on different GPU devices across various Spec-Bench tasks. The evaluation are conducted on H800 and RTX 3090 with all other settings are the same.

Speedup Range	Number	Percentage
$3.54\times-4.16\times$ (Max)	16	3.33%
$2.93\times-3.54\times$	48	10.00%
$2.31\times-2.93\times$	103	21.46%
$1.69\times-2.31\times$	197	41.04%
$1.07\times$ (Min) $-1.69\times$	116	24.17%

Appendix

Table 5: Empirical speedup distribution of SpecBound on Vicuna-7B across 480 prompts from SpecBench.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 293 papers

Apr 16, 2026

ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding

Self-speculative decoding is an inference technique for large language models designed to speed up generation without sacrificing output quality. It combines fast, approximate decoding using a compact version of the model as a draft model with selective re-evaluation by the full target model. Some existing methods form the draft model by dynamically learning which layers to skip during inference, effectively creating a smaller subnetwork to speed up computation. However, using heuristic-based approaches to select layers to skip can often be simpler and more effective. In this paper, we propose ConfLayers, a dynamic plug-and-play approach to forming the draft model in self-speculative decoding via confidence-based intermediate layer skipping. The process iteratively computes confidence scores for all layers, selects layers to skip based on an adaptive threshold, evaluates the performance of the resulting set, and updates the best selection until no further improvement is achieved or a maximum number of iterations is reached. This framework avoids the overhead and complexity of training a layer skipping policy and can provide more consistent speed-quality trade-offs while preserving the adaptivity of the draft model to diverse tasks and datasets. The performance evaluation of ConfLayers across different models and datasets shows that our novel approach offers up to 1.4x speedup over vanilla LLM generation.

Walaa Amer, Uday das, Fadi Kurdahi

Department of Electrical Engineering and Computer Sciences, University of California Irvine, Irvine, USA · Advanced Micro Devices, San Jose, USA

Speculative Decoding LLM Inference Acceleration

Ask PDF

Jul 30, 2026

A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

Speculative decoding alleviates the memory-bandwidth bottleneck in large language model inference, but its acceleration is jointly constrained by drafting overhead, token acceptance, and speculation length. We present a unified efficiency analysis showing that extending the speculation horizon can reduce rather than improve speedup when the marginal acceptance probability falls below the relative drafting cost. Guided by this analysis, we introduce SparseSpec-L, a training-free self-speculative decoding framework for long-context inference. SparseSpec-L generates lightweight drafts directly from the target model using a dynamically sparsified and recallable KV cache. It recycles per-head attention statistics produced during full-context verification as a no-extra-forward importance signal, allowing critical historical tokens to be recalled without permanently discarding the dense KV cache. An online entropy-based controller further selects the speculation length according to expected step-wise efficiency. Experiments across multiple long-context tasks and model scales show consistent end-to-end acceleration, with up to speedup over autoregressive decoding while preserving the target model's output distribution.

Yuesong Liu, Yuan Zeng, Min Lyu +3

University of Science and Technology of China

Long-Context Language Model Inference Speculative Decoding LLM Inference Acceleration

Ask PDF

Sep 29, 2026

SEED: Self-Speculative Decoding via Implicit Encoder-Decoder

Self-speculative decoding accelerates large language model (LLM) inference by drafting tokens from the target model itself, but faces a sharp tradeoff between the quality and cost of the draft. Early-exit methods produce drafts cheaply by terminating computation at intermediate layers, but forgo the deeper representations that later layers provide and thus suffer in draft quality. Multi-token prediction preserves draft quality by emitting from the model's final hidden states, but pays for a full forward pass to produce those states at every drafting step. We propose self-speculative encoder-decoder (SEED), a self-speculative method that obtains high-quality drafts cheaply by reusing the deep contextual representations already computed during verification. We reinterpret the standard decoder-only transformer as an implicit encoder-decoder: the first layers (encoder) build deep contextual representations, and the last few layers (decoder) emit tokens from them. Encoding and verification are merged into a single step: verification is performed by the full encoder-decoder, and the contextual representations of the verified prefix are cached for reuse during drafting. Drafting is therefore very fast: between verifications, the lightweight decoder drafts multiple tokens autoregressively, each conditioned on the cached representations and on preceding drafts. Experiments across multiple benchmarks show that SEED achieves up to 2.7

\times

average speedup on 4B-scale models, outperforming both early-exit and MTP-style self-speculative baselines and running 28% faster than the state-of-the-art EAGLE-3, while preserving or even improving the generation quality of standard autoregressive fine-tuning. Code is available at https://github.com/lhk2004/SEED.

Hankun Lin, Patrick Pynadath, Ruqi Zhang

Department of Computer Science, Purdue University, USA

Speculative Decoding LLM Inference Acceleration Decoder-Only Language Models

Ask PDF

Abstract

Speculative Decoding

Explore similar work

ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding

A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

SEED: Self-Speculative Decoding via Implicit Encoder-Decoder