SEED: Self-Speculative Decoding via Implicit Encoder-Decoder

Hankun Lin; Patrick Pynadath; Ruqi Zhang

cs.CLSep 29, 2026

SEED: Self-Speculative Decoding via Implicit Encoder-Decoder

Authors: Hankun Lin, Patrick Pynadath, Ruqi Zhang

Organizations: Department of Computer Science, Purdue University, USA

Abstract

Self-speculative decoding accelerates large language model (LLM) inference by drafting tokens from the target model itself, but faces a sharp tradeoff between the quality and cost of the draft. Early-exit methods produce drafts cheaply by terminating computation at intermediate layers, but forgo the deeper representations that later layers provide and thus suffer in draft quality. Multi-token prediction preserves draft quality by emitting from the model's final hidden states, but pays for a full forward pass to produce those states at every drafting step. We propose self-speculative encoder-decoder (SEED), a self-speculative method that obtains high-quality drafts cheaply by reusing the deep contextual representations already computed during verification. We reinterpret the standard decoder-only transformer as an implicit encoder-decoder: the first layers (encoder) build deep contextual representations, and the last few layers (decoder) emit tokens from them. Encoding and verification are merged into a single step: verification is performed by the full encoder-decoder, and the contextual representations of the verified prefix are cached for reuse during drafting. Drafting is therefore very fast: between verifications, the lightweight decoder drafts multiple tokens autoregressively, each conditioned on the cached representations and on preceding drafts. Experiments across multiple benchmarks show that SEED achieves up to 2.7 $\times$ average speedup on 4B-scale models, outperforming both early-exit and MTP-style self-speculative baselines and running 28% faster than the state-of-the-art EAGLE-3, while preserving or even improving the generation quality of standard autoregressive fine-tuning. Code is available at https://github.com/lhk2004/SEED.

Figures & tables

Figure 1: Overview of SEED. SEED partitions a $28$ -layer model into a $26$ -layer encoder and a $2$ -layer decoder. During drafting, the decoder generates drafts by cross-attending to the deep contextual KV cache produced during the previous verification step, avoiding repeatedly invoking the costly encoder. This design contrasts with two existing self-speculative decoding paradigms: early-exit drafting (instantiated using LayerSkip ( Elhoushi et al., 2024 ) ) discards the deep representations from later layers, sacrificing draft quality, while multi-token prediction (instantiated using Apple MTP ( Samragh et al., 2025 ) ) preserves draft quality by maintaining full representational depth but still requires a full forward pass at every drafting step. On GSM8K with draft length $4$ , SEED achieves substantially higher decoding throughput than early-exit drafting ( $\sim 2.0\times$ ) and MTP ( $\sim 1.6\times$ ).

Figure 2: Example training-time attention masks for $n=6$ tokens with block size $b=2$ . Left: In the joint encoder-decoder verifier, context tokens use bidirectional attention for richer encoding, while response tokens use causal attention for autoregressive drafting. Right: In the decoder, input token embeddings ( $H^{0}_{1:n}$ ) attend causally to tokens within the same block via self-attention, and additionally cross-attend to the decoder’s own KV cache ( $KV^{dec}_{1:n}$ ) of all preceding blocks.

Method	GSM8K		KodCode		ScienceQA		CNN/Daily Mail				Avg.
	Acc.	Tput	Acc.	Tput	Acc.	Tput	R-1	R-2	R-L	Tput	Speedup
Qwen3-1.7B-Base
AR	56.3	37.9	66.5	36.0	93.2	36.6	38.6	17.7	28.6	34.2	1.0 $\times$
Common Acceleration Methods
E2D2	26.6 ${}^{{\color[rgb]{1,0.3,0.3}\downarrow\textbf{29.7}}}$	67.6	40.6 ${}^{{\color[rgb]{1,0.3,0.3}\downarrow\textbf{25.9}}}$	65.1	88.7 ${}^{{\color[rgb]{1,0.3,0.3}\downarrow\textbf{4.5}}}$	68.1	32.8 ${}^{{\color[rgb]{1,0.3,0.3}\downarrow\textbf{5.8}}}$	12.1 ${}^{{\color[rgb]{1,0.3,0.3}\downarrow\textbf{5.6}}}$	23.3 ${}^{{\color[rgb]{1,0.3,0.3}\downarrow\textbf{5.3}}}$	56.8	1.8 $\times$
Apple MTP	-	59.1	-	54.6	-	56.6	-	-	-	37.5	1.4 $\times$

Table 1: Performance comparison on Qwen3-1.7B-Base and Qwen3-4B-Base . 0-shot pass@1 accuracy (Acc.) is reported in %, and decoding throughput (Tput) in tokens/s. For CNN/Daily Mail, we report ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L). For methods that keep the target model frozen, we report only their throughput. Given the size of its released drafter weights, PARD is evaluated only on Qwen3-4B-Base . The highest throughput and average speedup are shown in bold , and the second highest results are underlined .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Method	No Ext. Module	Lossless	Training Cost	Speedup
General MTP
Meta MTP [ 14 ] & DeepSeek MTP [ 23 ]	✗	✓	High	Mid
FastMTP [ 6 ]	✗	✓	Low	Mid
Apple MTP [ 33 ]	✗	✓	Low	High
Jacobi Forcing [ 15 ]	✓	✗	Low	High
Diffusion Language Models

Appendix

Table 3: Comparison of SEED against existing LLM acceleration methods. "Lossless" denotes no degradation in generation quality compared to standard autoregressive models.

Decoder Size (# layers)	E2D2 Acc. (%)	E2D2 Tput (tokens/s)	SEED Acc. (%)	SEED Tput (tokens/s)
2	23.7	75.9	57.5	74.3
4	26.6	67.6	57.9	66.1
6	32.3	63.2	59.4	60.8
8	32.4	57.3	58.8	56.3
10	36.3	51.2	57.4	53.3
12	41.3	46.4	57.0	48.8

Appendix

Table 4: Controlled comparison with E2D2. We compare E2D2 and SEED across different decoder sizes by training and evaluating corresponding Qwen3-1.7B-Base models on GSM8K. SEED achieves much better task performance while using a smaller decoder, proving that exact verification enables acceleration without sacrificing model performance.

Dataset	Base Model	Batch Size	LR	Steps	Max Length
GSM8K	Qwen3-1.7B-Base	$1$	$1e^{-5}$	$30{,}000$	$768$
KodCode	Qwen3-1.7B-Base	$1$	$1e^{-5}$	$30{,}000$	$1024$
ScienceQA	Qwen3-1.7B-Base	$1$	$1e^{-5}$	$30{,}000$	$768$
CNN/Daily Mail	Qwen3-1.7B-Base	$32$	$1e^{-5}$	$30{,}000$	$1024$
GSM8K	Qwen3-4B-Base	$1$	$5e^{-6}$	$30{,}000$	$768$
KodCode	Qwen3-4B-Base	$1$	$5e^{-6}$	$30{,}000$	$1024$

Appendix

Table 5: Training configurations used for task-specific fine-tuning. For each dataset and model scale, all methods share the same optimizer schedule and training budget to maintain comparability.

Figure 8: Illustration of confidence-aware dynamic drafting and tree-based verification in SEED. Left: starting from a verified prefix, the lightweight decoder drafts autoregressively, with the current node expanded into multiple candidates using a top- $p$ rule, thus producing a small tree of candidate continuations. Right: all candidate tokens are packed and verified in one full-model forward pass using a tree-structured attention mask. Each candidate token attends to the entire verified prefix and to the candidate tokens on its own ancestral path, but not to tokens from unrelated branches. This allows parallel verification of multiple draft branches while preserving the causal dependencies of autoregressive decoding.

Figure 9: Effect of the confidence threshold

\tau

used in dynamic drafting. We evaluate a GSM8K-finetuned Qwen3-1.7B-Base SEED model on GSM8K and report average decoding throughput, draft token acceptance rate, and draft token acceptance length. Increasing

\tau

improves acceptance rate but shortens acceptance length due to earlier termination of drafting. Overall decoding throughput remains stable for

\tau

between

0.5

and

0.9

, indicating that SEED is relatively insensitive to the precise choice of

\tau

Method	Fixed-Length Drafting ( $d=4$ )			Fixed-Length Drafting ( $d=8$ )			Dynamic Drafting ( $\tau=0.7$ )
Method	Tput	Acc. Len	Acc. Rate	Tput	Acc. Len	Acc. Rate	Tput	Acc. Len	Acc. Rate
w/ Tree Attn	82.3	3.16	79.80%	82.5	4.65	59.10%	91.8	3.97	88.57%
w/o Tree Attn	76.9 ${}^{{\color[rgb]{1,0.3,0.3}\downarrow\textbf{5.4}}}$	2.67	69.45%	75.2 ${}^{{\color[rgb]{1,0.3,0.3}\downarrow\textbf{7.3}}}$	3.67	46.64%	85.1 ${}^{{\color[rgb]{1,0.3,0.3}\downarrow\textbf{6.7}}}$	3.28	77.59%

Appendix

Table 6: Ablation study on tree-based verification. We evaluate a GSM8K-finetuned Qwen3-1.7B- Base SEED model on GSM8K. We report decoding throughput (Tput) in tokens/s, average draft token acceptance length (Acc. Len), and draft token acceptance rate (Acc. Rate) under fixed-length and dynamic drafting strategies. Tree-based verification consistently improves acceptance statistics and overall throughput.

	GSM8K Acc. (%)	Throughput (tokens/s)	Draft Token Acc. Rate (%)	Draft Token Acc. Length
SEED	57.5	74.3	73.0	2.9
w/o Parameter Sharing	44.1 ${}^{{\color[rgb]{1,0.3,0.3}\downarrow\textbf{13.4}}}$	68.8 ${}^{{\color[rgb]{1,0.3,0.3}\downarrow\textbf{5.5}}}$	68.5 ${}^{{\color[rgb]{1,0.3,0.3}\downarrow\textbf{4.5}}}$	2.7 ${}^{{\color[rgb]{1,0.3,0.3}\downarrow\textbf{0.2}}}$

Appendix

Table 7: Parameter-sharing ablation on GSM8K with Qwen3-1.7B-Base . Removing parameter sharing substantially degrades both draft quality and task performance.

Model	Accuracy (%)	Tput (tokens/s)	Acc. Len	Acc. Rate (%)
AR (from base)	56.3	37.9	-	-
SEED (continued from AR)	56.5 ${}^{{\color[rgb]{0,0.7,0}\uparrow\textbf{0.2}}}$	90.6 ${}^{{\color[rgb]{0,0.7,0}\uparrow\textbf{52.7}}}$	3.91	86.82
SEED (from base)	57.1	91.8	3.97	88.57

Appendix

Table 8: SEED applied as a second-stage fine-tuning procedure on top of an AR-fine-tuned model. Dynamic drafting (

\tau=0.7

) and tree-based verification are used for SEED models. Continued fine-tuning preserves accuracy on GSM8K while achieving comparable speedup.

Method	Weights Touched per Draft Token	Weights	Cost (units)
AR	28 layers + head	1721M	1.000
SEED	2 decoder layers + head	411.9M	0.239
LayerSkip	8 layers + head	713.8M	0.415
Apple MTP	(28 layers + head) per block + sequential sampler head	$\geq 1721$ M	$\geq 1.000$
EAGLE-3	1 fused draft layer + reduced-vocab head	$\approx 137$ M	$\approx 0.080$

Appendix

Table 9: Approximate drafting cost for a 1.7B model. Note that Apple MTP amortizes a full-model call over a draft block.

Method	$c_{\mathrm{draft}}$	$c_{\mathrm{verify}}$	$T_{\mathrm{unit}}$	$T$ (ms)	$n$	Predicted (tokens/s)	Measured (tokens/s)
SEED	0.239	1.000	1.96	51.7	3.92	75.8	74.3
LayerSkip	0.415	0.766	2.43	64.1	2.72	42.4	47.0
Apple MTP	$1.72/4$	1.000	2.72	71.8	3.79	52.8	44.6

Appendix

Table 10: Approximate throughput predictions for fixed draft length

d=4

without tree-based verification. Apple MTP generates all four draft tokens in a single forward pass. However, its sampler applies the 311M-parameter LM head sequentially once per draft token, contributing an additional cost of

4\times 311.2/1721=0.72

units.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 293 papers

SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration
Apr 14, 2026Zhuofan Wen, Yang FengSpeculative Decoding LLM Inference Acceleration
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding
May 31, 2026Xin Su, Dawid Majchrowski, Fangyuan Yu +5Speculative Decoding LLM Inference Acceleration
Thoughtworks · Nvidia
A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
Jul 30, 2026Yuesong Liu, Yuan Zeng, Min Lyu +3Long-Context Language Model Inference Speculative Decoding
University of Science and Technology of China

Apr 14, 2026J/K move · Enter open · S save

SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration

Zhuofan Wen, Yang Feng

Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Speculative decoding has emerged as a promising approach to accelerate autoregressive inference in large language models (LLMs). Self-draft methods, which leverage the base LLM itself for speculation, avoid the overhead of auxiliary draft models but face limitations: shallow layers often produce overconfident yet incorrect token predictions, and the presence of difficult tokens in a draft sequence forces redundant computation through deeper layers, undermining both draft acceptance and overall speedup. To address these issues, we propose a novel self-draft framework that suppresses spurious confidence via layer-wise temperature annealing in early-exit decision and adaptively bounds speculation length based on token-wise decoding difficulty. By reprocessing the hidden states of draft tokens in a unified parallel pass through deep layers, our method maintains exact output equivalence with the original model while maximizing computational efficiency. It requires no modifications to the base LLM parameters and achieves up to 2.33x wall-time speedup over standard autoregressive decoding across diverse long-form generation tasks and multiple model architectures.

Abstract

Speculative Decoding

Explore similar work

SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration

Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding

A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding