Self-speculative decoding accelerates large language model (LLM) inference by drafting tokens from the target model itself, but faces a sharp tradeoff between the quality and cost of the draft. Early-exit methods produce drafts cheaply by terminating computation at intermediate layers, but forgo the deeper representations that later layers provide and thus suffer in draft quality. Multi-token prediction preserves draft quality by emitting from the model's final hidden states, but pays for a full forward pass to produce those states at every drafting step. We propose self-speculative encoder-decoder (SEED), a self-speculative method that obtains high-quality drafts cheaply by reusing the deep contextual representations already computed during verification. We reinterpret the standard decoder-only transformer as an implicit encoder-decoder: the first layers (encoder) build deep contextual representations, and the last few layers (decoder) emit tokens from them. Encoding and verification are merged into a single step: verification is performed by the full encoder-decoder, and the contextual representations of the verified prefix are cached for reuse during drafting. Drafting is therefore very fast: between verifications, the lightweight decoder drafts multiple tokens autoregressively, each conditioned on the cached representations and on preceding drafts. Experiments across multiple benchmarks show that SEED achieves up to 2.7× average speedup on 4B-scale models, outperforming both early-exit and MTP-style self-speculative baselines and running 28% faster than the state-of-the-art EAGLE-3, while preserving or even improving the generation quality of standard autoregressive fine-tuning. Code is available at https://github.com/lhk2004/SEED.
Figures & tables
Figure 1: Overview of SEED. SEED partitions a 28 -layer model into a 26 -layer encoder and a 2 -layer decoder. During drafting, the decoder generates drafts by cross-attending to the deep contextual KV cache produced during the previous verification step, avoiding repeatedly invoking the costly encoder. This design contrasts with two existing self-speculative decoding paradigms: early-exit drafting (instantiated using LayerSkip ( Elhoushi et al., 2024 ) ) discards the deep representations from later layers, sacrificing draft quality, while multi-token prediction (instantiated using Apple MTP ( Samragh et al., 2025 ) ) preserves draft quality by maintaining full representational depth but still requires a full forward pass at every drafting step. On GSM8K with draft length 4 , SEED achieves substantially higher decoding throughput than early-exit drafting ( ∼2.0× ) and MTP ( ∼1.6× ).
Figure 2: Example training-time attention masks for n=6 tokens with block size b=2 . Left: In the joint encoder-decoder verifier, context tokens use bidirectional attention for richer encoding, while response tokens use causal attention for autoregressive drafting. Right: In the decoder, input token embeddings ( H1:n0 ) attend causally to tokens within the same block via self-attention, and additionally cross-attend to the decoder’s own KV cache ( KV1:ndec ) of all preceding blocks.
Table 1: Performance comparison on Qwen3-1.7B-Base and Qwen3-4B-Base . 0-shot pass@1 accuracy (Acc.) is reported in %, and decoding throughput (Tput) in tokens/s. For CNN/Daily Mail, we report ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L). For methods that keep the target model frozen, we report only their throughput. Given the size of its released drafter weights, PARD is evaluated only on Qwen3-4B-Base . The highest throughput and average speedup are shown in bold , and the second highest results are underlined .
Figure 4Figure 5Figure 6
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Method
No Ext. Module
Lossless
Training Cost
Speedup
General MTP
Meta MTP [ 14 ] & DeepSeek MTP [ 23 ]
✗
✓
High
Mid
FastMTP [ 6 ]
✗
✓
Low
Mid
Apple MTP [ 33 ]
✗
✓
Low
High
Jacobi Forcing [ 15 ]
✓
✗
Low
High
Diffusion Language Models
Appendix
Table 3: Comparison of SEED against existing LLM acceleration methods. "Lossless" denotes no degradation in generation quality compared to standard autoregressive models.
Decoder Size (# layers)
E2D2 Acc. (%)
E2D2 Tput (tokens/s)
SEED Acc. (%)
SEED Tput (tokens/s)
2
23.7
75.9
57.5
74.3
4
26.6
67.6
57.9
66.1
6
32.3
63.2
59.4
60.8
8
32.4
57.3
58.8
56.3
10
36.3
51.2
57.4
53.3
12
41.3
46.4
57.0
48.8
Appendix
Table 4: Controlled comparison with E2D2. We compare E2D2 and SEED across different decoder sizes by training and evaluating corresponding Qwen3-1.7B-Base models on GSM8K. SEED achieves much better task performance while using a smaller decoder, proving that exact verification enables acceleration without sacrificing model performance.
Dataset
Base Model
Batch Size
LR
Steps
Max Length
GSM8K
Qwen3-1.7B-Base
1
1e−5
30,000
768
KodCode
Qwen3-1.7B-Base
1
1e−5
30,000
1024
ScienceQA
Qwen3-1.7B-Base
1
1e−5
30,000
768
CNN/Daily Mail
Qwen3-1.7B-Base
32
1e−5
30,000
1024
GSM8K
Qwen3-4B-Base
1
5e−6
30,000
768
KodCode
Qwen3-4B-Base
1
5e−6
30,000
1024
Appendix
Table 5: Training configurations used for task-specific fine-tuning. For each dataset and model scale, all methods share the same optimizer schedule and training budget to maintain comparability.
Figure 8: Illustration of confidence-aware dynamic drafting and tree-based verification in SEED. Left: starting from a verified prefix, the lightweight decoder drafts autoregressively, with the current node expanded into multiple candidates using a top- p rule, thus producing a small tree of candidate continuations. Right: all candidate tokens are packed and verified in one full-model forward pass using a tree-structured attention mask. Each candidate token attends to the entire verified prefix and to the candidate tokens on its own ancestral path, but not to tokens from unrelated branches. This allows parallel verification of multiple draft branches while preserving the causal dependencies of autoregressive decoding.
Figure 9: Effect of the confidence threshold τ used in dynamic drafting. We evaluate a GSM8K-finetuned Qwen3-1.7B-Base SEED model on GSM8K and report average decoding throughput, draft token acceptance rate, and draft token acceptance length. Increasing τ improves acceptance rate but shortens acceptance length due to earlier termination of drafting. Overall decoding throughput remains stable for τ between 0.5 and 0.9 , indicating that SEED is relatively insensitive to the precise choice of τ .
Table 6: Ablation study on tree-based verification. We evaluate a GSM8K-finetuned Qwen3-1.7B- Base SEED model on GSM8K. We report decoding throughput (Tput) in tokens/s, average draft token acceptance length (Acc. Len), and draft token acceptance rate (Acc. Rate) under fixed-length and dynamic drafting strategies. Tree-based verification consistently improves acceptance statistics and overall throughput.
Table 7: Parameter-sharing ablation on GSM8K with Qwen3-1.7B-Base . Removing parameter sharing substantially degrades both draft quality and task performance.
Table 8: SEED applied as a second-stage fine-tuning procedure on top of an AR-fine-tuned model. Dynamic drafting ( τ=0.7 ) and tree-based verification are used for SEED models. Continued fine-tuning preserves accuracy on GSM8K while achieving comparable speedup.
Method
Weights Touched per Draft Token
Weights
Cost (units)
AR
28 layers + head
1721M
1.000
SEED
2 decoder layers + head
411.9M
0.239
LayerSkip
8 layers + head
713.8M
0.415
Apple MTP
(28 layers + head) per block + sequential sampler head
≥1721 M
≥1.000
EAGLE-3
1 fused draft layer + reduced-vocab head
≈137 M
≈0.080
Appendix
Table 9: Approximate drafting cost for a 1.7B model. Note that Apple MTP amortizes a full-model call over a draft block.
Method
cdraft
cverify
Tunit
T (ms)
n
Predicted (tokens/s)
Measured (tokens/s)
SEED
0.239
1.000
1.96
51.7
3.92
75.8
74.3
LayerSkip
0.415
0.766
2.43
64.1
2.72
42.4
47.0
Apple MTP
1.72/4
1.000
2.72
71.8
3.79
52.8
44.6
Appendix
Table 10: Approximate throughput predictions for fixed draft length d=4 without tree-based verification. Apple MTP generates all four draft tokens in a single forward pass. However, its sampler applies the 311M-parameter LM head sequentially once per draft token, contributing an additional cost of 4×311.2/1721=0.72 units.
Speculative decoding has emerged as a promising approach to accelerate autoregressive inference in large language models (LLMs). Self-draft methods, which leverage the base LLM itself for speculation, avoid the overhead of auxiliary draft models but face limitations: shallow layers often produce overconfident yet incorrect token predictions, and the presence of difficult tokens in a draft sequence forces redundant computation through deeper layers, undermining both draft acceptance and overall speedup. To address these issues, we propose a novel self-draft framework that suppresses spurious confidence via layer-wise temperature annealing in early-exit decision and adaptively bounds speculation length based on token-wise decoding difficulty. By reprocessing the hidden states of draft tokens in a unified parallel pass through deep layers, our method maintains exact output equivalence with the original model while maximizing computational efficiency. It requires no modifications to the base LLM parameters and achieves up to 2.33x wall-time speedup over standard autoregressive decoding across diverse long-form generation tasks and multiple model architectures.
Zhuofan Wen, Yang Feng
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
Large Language Model (LLM) generation remains expensive because autoregressive decoding calls the model once for each new token. Speculative decoding reduces this cost by drafting multiple tokens and verifying them with the target model in one step, but its speedup depends on how many drafted tokens are accepted. Parameter-free draft sources can propose long continuations at low cost in structured and agentic workloads, yet a cache match that looks promising at one generation step may have low payoff at the next. We propose Hybrid Verified Decoding, which predicts the accepted length of a cache draft before verification and uses this payoff estimate to choose between cache verification and a model-based drafter. Across three LLMs and sixteen datasets, Hybrid Verified Decoding is especially effective on agentic workflows, where it outperforms EAGLE3 in every setting with a 2.73x average speedup. Our analysis shows how prompt structure creates cache opportunities, how high-payoff cache drafts concentrate in a small part of the draft space, and how payoff-guided selection reduces sequential decoding work, pointing to runtime draft selection as a promising direction for speculative decoding.
Speculative decoding alleviates the memory-bandwidth bottleneck in large language model inference, but its acceleration is jointly constrained by drafting overhead, token acceptance, and speculation length. We present a unified efficiency analysis showing that extending the speculation horizon can reduce rather than improve speedup when the marginal acceptance probability falls below the relative drafting cost. Guided by this analysis, we introduce SparseSpec-L, a training-free self-speculative decoding framework for long-context inference. SparseSpec-L generates lightweight drafts directly from the target model using a dynamically sparsified and recallable KV cache. It recycles per-head attention statistics produced during full-context verification as a no-extra-forward importance signal, allowing critical historical tokens to be recalled without permanently discarding the dense KV cache. An online entropy-based controller further selects the speculation length according to expected step-wise efficiency. Experiments across multiple long-context tasks and model scales show consistent end-to-end acceleration, with up to speedup over autoregressive decoding while preserving the target model's output distribution.