KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem

Seongjin Cha; Gyuwan Kim; Dongsu Han; Tao Yang; Insu Han

cs.LGFeb 23, 2026

KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem

Authors: Seongjin Cha, Gyuwan Kim, Dongsu Han, Tao Yang, Insu Han

Organizations: School of Electrical Engineering, KAIST, Daejeon, South Korea · Department of Computer Science, University of California, Santa Barbara, CA, USA

Abstract

Self-speculative decoding (SSD) accelerates LLM inference by skipping layers to create an efficient draft model, yet existing methods often rely on static heuristics that ignore the dynamic computational overhead of attention in long-context scenarios. We propose KnapSpec, a training-free framework that reformulates draft model selection as a knapsack problem to maximize tokens-per-time throughput. By decoupling Attention and MLP layers and modeling their hardware-specific latencies as functions of context length, KnapSpec adaptively identifies optimal draft configurations on the fly via a parallel dynamic programming algorithm. Furthermore, we provide the first rigorous theoretical analysis establishing cosine similarity between hidden states as a mathematically sound proxy for the token acceptance rate. This foundation allows our method to maintain high drafting faithfulness while navigating the shifting bottlenecks of real-world hardware. Our experiments on Qwen3 and Llama3 demonstrate that KnapSpec consistently outperforms state-of-the-art SSD baselines, achieving up to 1.47x wall-clock speedup across various benchmarks. Our plug-and-play approach ensures high-speed inference for long sequences without requiring additional training or compromising the target model's output distribution.

Figures & tables

Methods	Training-Free	Attn/MLP Separable	Adaptive Search	Context-Length Aware	Search Strategy
LayerSkip ( Elhoushi et al., 2024 )	✗	✗	✗	✗	Static Early-Exit
Kangaroo ( Liu et al., 2024 )	✗	✗	✗	✗	Fixed Sub-network
DEL ( Zarch et al., 2025 )	✗	✗	✓	✗	Dynamic Early-Exit
ASD ( Metel et al., 2024 )	✓	✓	✓	✗	Rule-based Pruning
Draft & Verify ( Zhang et al., 2024 )	✓	✓	✗	✗	Bayesian Optimization
SWIFT ( Xia et al., 2025 )	✓	✓	✗	✗	Bayesian Optimization + Random Search

Table 1: Comparison of KnapSpec with existing self-speculative decoding methods. KnapSpec is the first to offer a training-free, module-level separable search that adaptively reconfigures the draft model based on context length.

Figure 1: Overview of KnapSpec. (1) Self-Speculative Decoding defines a draft model as a sub-network of the target model. (2) Layer selection can be converted to a Knapsack Problem, and we search candidate sets via Dynamic Programming (DP) with various latency budgets (weights). (3) Then, the optimal configuration that maximizes Tokens-per-Time (TPT) is selected and used for the next speculation.

Figure 2: Pearson Correlation Coefficients (PCC) and

R^{2}

-value of TPT and acceptance rate against throughput. Best TPT shows a much closer correlation with actual performance.

Methods	Reasoning Tasks (Long-context Generation)									Summarization Tasks (Long-context Input)
	AIME24			AIME25			MMLU-Pro			GovReport			PG19			BookSum
	TPT	Speedup	$\alpha$	TPT	Speedup	$\alpha$	TPT	Speedup	$\alpha$	TPT	Speedup	$\alpha$	TPT	Speedup	$\alpha$	TPT	Speedup	$\alpha$
	Qwen3-32B ( $L=64$ layers)									Llama3.1-70B ( $L=80$ layers)
AR	19.65	1.00 $\times$	$-$	19.72	1.00 $\times$	$-$	23.15	1.00 $\times$	$-$	6.75	1.00 $\times$	$-$	6.90	1.00 $\times$	$-$	4.76	1.00 $\times$	$-$
DEL	19.69	0.85 $\times$	0%	19.95	0.88 $\times$	2.8%	23.20	0.87 $\times$	0%	6.62	0.87 $\times$	0.1%	6.99	0.87 $\times$	1.3%	4.84	0.87 $\times$	0.0%
SWIFT	20.38	1.23 $\times$	84.6%	20.73	1.22 $\times$	80.4%	21.75	0.90 $\times$	46.8%	8.62	1.33 $\times$	91.6%	6.12	0.98 $\times$	61.9%	3.76	0.84 $\times$	49.3%

Table 2: Performance comparison of reasoning tasks (long-context generation) and summarization tasks (long-context input). We use Qwen3 models for reasoning (left) and Llama3 models for summarization (right) with various scales. We report Tokens-per-Time (TPT), wall-clock speedup, and the average acceptance rate

\alpha

for speculative decoding methods.

Methods	Reasoning Tasks (Qwen3-8B)									Summarization Tasks (Llama3.1-8B)
	AIME24			AIME25			MMLU-Pro			GovReport			PG19			BookSum
	TPT	Speedup	$\alpha$	TPT	Speedup	$\alpha$	TPT	Speedup	$\alpha$	TPT	Speedup	$\alpha$	TPT	Speedup	$\alpha$	TPT	Speedup	$\alpha$
AR	31.87	1.00 $\times$	$-$	32.58	1.00 $\times$	$-$	33.52	1.00 $\times$	$-$	19.71	1.00 $\times$	$-$	20.64	1.00 $\times$	$-$	14.02	1.00 $\times$	$-$
DEL	31.29	0.86 $\times$	0.0%	37.89	0.82 $\times$	0.0%	25.58	0.83 $\times$	22.2%	20.01	0.63 $\times$	0.0%	20.64	0.67 $\times$	0.1%	13.91	0.69 $\times$	0.0%
SWIFT	24.46	1.03 $\times$	73.3%	27.15	1.02 $\times$	78.1%	20.72	1.11 $\times$	89.3%	16.28	0.87 $\times$	48.2%	14.38	0.52 $\times$	24.7%	12.78	0.47 $\times$	22.4%
CLaSp	24.08	1.06 $\times$	83.1%	27.62	1.07 $\times$	91.6%	24.68	1.09 $\times$	91.4%	16.98	1.07 $\times$	86.0%	15.83	1.02 $\times$	82.3%	9.03	1.02 $\times$	78.0%

Table 3: Performance comparison under nucleus sampling with a temperature

T=0.7

, and top-

k

and top-

p

thresholds set to

k=50

and

p=0.95

, respectively. We use Qwen3-8B model for reasoning (left) and Llama3.1-8B model for summarization (right). We report Tokens-per-Time (TPT), wall-clock speedup, and the average acceptance rate

\alpha

for speculative decoding methods.

Figure 3: Comparison of skipped layers across context lengths up to 32k. KnapSpec chooses more Attention layers to skip than MLP as the length

n

increases, while SWIFT chooses uniform layers.

Figure 4: Throughput, memory and optimization overhead reduction (%) with various similarity thresholds

\tau

. We select

\tau=0.5

to achieve the best balance between speed and memory.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The trade-off between throughput (blue) and optimization overhead (orange) as a function of the optimization interval. Increasing the interval reduces overhead but eventually harms throughput due to stale layer configurations. The optimal interval is found at 64 steps, achieving the highest throughput of 18.1 tokens/second.

Panel of Figure 6 — Figure 6: Layer-wise skip probabilities aggregated over evaluation samples. The non-uniform distribution demonstrates the model-specific redundancies. While certain layers are frequently skipped to reduce computational cost, some layers consistently show low skip probabilities.

Hardware	Model	$c_{1}$	$c_{2}$ ( $\times 10^{-4}$ )	$c_{3}$	Speedup
RTX 6000 Pro	Llama-3.1-8B	0.279	0.735	0.279	1.27 $\times$
RTX 6000 Pro	Llama-3.2-3B	0.161	0.720	0.255	1.25 $\times$
RTX A6000	Llama-3.1-8B	0.520	0.942	0.216	1.28 $\times$
RTX A6000	Llama-3.2-3B	0.248	0.855	0.186	1.24 $\times$
RTX 4090	Llama-3.1-8B	0.386	0.766	0.167	O.O.M.
RTX 4090	Llama-3.2-3B	0.180	0.724	0.140	1.26 $\times$

Appendix

Table 4: Multi-Hardware Profiling and KnapSpec’s Speedup on GovReport.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,090 papers

A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
Jul 30, 2026Yuesong Liu, Yuan Zeng, Min Lyu +3Long-Context Language Model Inference Speculative Decoding
University of Science and Technology of China
SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration
Apr 14, 2026Zhuofan Wen, Yang FengSpeculative Decoding LLM Inference Acceleration
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding
Apr 16, 2026Walaa Amer, Uday das, Fadi KurdahiSpeculative Decoding LLM Inference Acceleration
Department of Electrical Engineering and Computer Sciences, University of California Irvine, Irvine, USA · Advanced Micro Devices, San Jose, USA

Jul 30, 2026J/K move · Enter open · S save

A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

Yuesong Liu, Yuan Zeng, Min Lyu, Ruilin Liu+2

University of Science and Technology of China

Long-Context Language Model Inference Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Speculative decoding alleviates the memory-bandwidth bottleneck in large language model inference, but its acceleration is jointly constrained by drafting overhead, token acceptance, and speculation length. We present a unified efficiency analysis showing that extending the speculation horizon can reduce rather than improve speedup when the marginal acceptance probability falls below the relative drafting cost. Guided by this analysis, we introduce SparseSpec-L, a training-free self-speculative decoding framework for long-context inference. SparseSpec-L generates lightweight drafts directly from the target model using a dynamically sparsified and recallable KV cache. It recycles per-head attention statistics produced during full-context verification as a no-extra-forward importance signal, allowing critical historical tokens to be recalled without permanently discarding the dense KV cache. An online entropy-based controller further selects the speculation length according to expected step-wise efficiency. Experiments across multiple long-context tasks and model scales show consistent end-to-end acceleration, with up to speedup over autoregressive decoding while preserving the target model's output distribution.

Abstract

LLM Inference Acceleration

Explore similar work

A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration

ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decoding