Speculative decoding has emerged as a critical technique for accelerating Large Language Model (LLM) inference by decoupling generation into a drafting phase and a parallel verification phase. A crucial challenge is therefore designing a drafter that is both lightweight and accurate enough to approximate the predictions of the target model. We present a highly optimized Multi-Token Prediction (MTP) architecture that extends existing backbone LLMs to efficiently produce high-quality drafts. Built around a single shared drafting head, our approach features three core architectural optimizations: target activation injection to anchor semantic planning, zero-copy KV-cache cross-attention to bypass prefill compute and reduce memory overhead, and an efficient clustered vocabulary projection to reduce the cost of full-vocabulary decoding. Testing these improvements with the Gemma 4 model family, our G-MTP shows out-of-the-box speedups for both large server-side setups (up to 3.1×) and smaller models targeting memory-constrained on-device environments (up to 2.3× on H100 and up to 2.2× on a smartphone GPU). In addition, we present parameter-efficient fine-tuning recipes for G-MTP that keep the drafter aligned with the backbone LLM under domain shift, requiring up to 1.83× fewer target forward passes than tuning only the backbone.
Figures & tables
Figure 1: Architecture of the optimized Multi-Token Prediction (G-MTP) module. The input to the drafter is constructed by concatenating the token embedding ( e(x) ) with the backbone hidden state ( h(L) ), both of dimension dtarget , and reducing to the drafter dimension ddraft . The draft tokens are then propagated through the MTP layers, utilizing cross-attention against the backbone’s frozen KV Cache to bypass prefill computations before recursively triggering the next forward step. For subsequent drafting steps, the drafter’s output state is up-projected ( Wup ), concatenated with the embedding of the sampled draft token, and down-projected ( Wdown ) to form the next input (Equation 3 ).
Figure 2: The autoregressive G-MTP drafter (blue blocks) is initialized with activations from the main model (gray blocks) and cross-attends to its KV cache. Across successive drafting steps, the positional encoding ( p1 ) remains constant, as the drafter repeatedly queries a static, pre-computed KV snapshot.
Model
Drafter Dimension
Drafter Params
Optimal γ
Avg BE
Baseline Throughput
Throughput with G-MTP
Speedup
Gemma 4 E2B
256
76M
4
2.71
278.8
550.1
1.97 ×
Gemma 4 E4B
256
77M
9
3.45
178.0
415.5
2.33 ×
Gemma 4 12B
1024
400M
6
3.69
93.5
249.7
2.67 ×
Gemma 4 26B MoE
1024
430M
4
3.27
248.9
334.0
1.34 ×
Gemma 4 31B
1024
500M
8
4.26
73.2
230.1
3.14 ×
Table 1: Parameter counts and generation speedup across the Gemma 4 family (throughput measured on H100, in tokens per second).
Model
Mode
γ
Block Eff.
Output TPS
Speedup
Gemma 4 31B
Baseline
N/A
1.00
73.2
1.00 ×
Gemma 4 31B
DFlash
2
2.21
142.2
1.94 ×
Gemma 4 31B
DFlash
4
2.76
173.6
2.37 ×
Gemma 4 31B
DFlash
8
3.15
190.5
2.60 ×
Gemma 4 31B
EAGLE-3
2
2.08
136.4
1.86 ×
Gemma 4 31B
EAGLE-3
4
2.52
153.8
2.10 ×
Table 2: Comparison of speedup for three different Gemma 4 31B drafters (2 × H100, utilizing FlashAttention 4 kernels, batch size 1) on SPEED-Bench quality dataset.
Model
no-CE Speedup vs. Baseline
CE Speedup vs. Baseline
CE vs. no-CE Speedup
CE vs. no-CE Drafter Time
Gemma 4 E2B
1.89 ×
1.92 ×
1.01 ×
0.81 ×
Gemma 4 E4B
1.96 ×
1.98 ×
1.01 ×
0.79 ×
Table 3: Full (no-CE) vs. clustered (CE) vocabulary projection on the Edge TPU of a Google Pixel 11 (SPEED-Bench qualitative, 456 prompts, γ=3 , greedy decoding). Values are medians over prompts; speedups are ratios of median decoding throughput. CE vs. no-CE columns are ratios of decoding speedup and of total drafting time.
Dataset
Configuration Regime
NTP Accuracy
Acceptance Rate
β
Target Steps
SciQA
1. No SFT reference
68.40%
22.36%
2.51
12,334
2. Target Fine-Tuned Only (SFT)
99.23%
8.96%
1.60
19,423
3. Joint Fine-Tuning (SFT + MTP)
99.37%
28.64%
2.96
10,595
4. Two-Stage Distillation
99.23%
26.48%
2.80
11,155
GSM8K
1. No SFT reference
74.83%
40.17%
3.73
8,414
2. Target Fine-Tuned Only (SFT)
88.01%
28.67%
2.95
10,435
Table 4: Speculative Decoding Domain Adaptation Performance (Gemma 4 E4B). Fine-tuning the model without updating the MTP (row 2) can lead to regression in Block Efficiency ( β ). Joint fine-tuning (row 3) matches or exceeds the β of the no SFT reference (row 1), and two-stage distillation (row 4) recovers about 96% of it.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Model
γ
Avg BE
Baseline (tokens/s)
G-MTP Speculative (tokens/s)
Throughput Speedup
Gemma 4 E2B
2
2.15
278.8
504.7
1.81 ×
3
2.47
278.8
531.5
1.91 ×
4
2.71
278.8
550.1
1.97 ×
5
2.88
278.8
550.1
1.97 ×
6
3.02
278.8
540.8
1.94 ×
7
3.11
278.8
533.9
1.92 ×
Appendix
Table 5: Throughput across lookahead horizons γ (Figure 3 extends to γ=15 ).
Figure 3: Empirical scaling of theoretical Block Efficiency (BE) and speculative wall-clock throughput across varying lookahead horizons ( γ ). While longer draft sequences monotonically increase token acceptance, throughput degrades for some models (e.g., 26B MoE) as drafting and verification costs exceed generation gains.
Table 7: On-device decoding throughput of Gemma 4 E2B and E4B with LiteRT-LM on the GPU of a Samsung Galaxy S26 Ultra ( γ=3 , greedy decoding, full vocabulary projection). Acceptance rate and throughput values are medians over the 76 prompts; the acceptance rate is the draft acceptance rate reported by the runtime. Speedups are ratios of median throughput with and without G-MTP, over all prompts and per task type.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Speculative decoding (SD) accelerates large language model (LLM) inference by using a smaller model to draft future tokens, which are then verified by the target LLM. This preserves generation quality by accepting only aligned tokens. However, individual drafters, often trained for specific tasks or domains, exhibit limited effectiveness across diverse applications. To address this, we introduce MetaSD, a unified framework that integrates multiple drafters into the SD process. MetaSD dynamically allocates computational resources to heterogeneous drafters by leveraging alignment feedback and framing drafter selection as a multi-armed bandit problem. Extensive experiments show MetaSD consistently outperforms single-drafter approaches.
Multi-token prediction has been shown to increase data density during training, improve downstream text-generation quality, and serves as the defacto approach for self-speculative decoding. Existing foundation and open source models that use MTP heads commit to a static tree-based attention topology throughout the entire generation sequence, meaning the speculation depth, and thus the compute required during verification, stays constant regardless of the context. This is fundamentally misaligned with the entropy patterns of natural language where low-entropy regions often support reliable multi-step drafting, while high-entropy regions require more conservative speculation. To address this, we propose Entropy-guided Multi-Token Prediction (EntMTP), a training-free scheduler that toggles between tree-based attention topologies from a set of task-specific pareto-optimal trees conditioned on a running estimate of local generation entropy. By matching speculation depth to context predictability, EntMTP maximizes expected accepted-token throughput across the full distribution of generated text without sacrificing generation quality. When evaluated across Humaneval, ShareGPT, GSM8k, and Litbench benchmarks, EntMTP consistently achieves a 1.15x speedup against Hydra and peak speedup of 1.36x against Medusa baselines respectively.
Multi-token prediction (MTP) can accelerate language-model decoding by drafting several tokens and verifying them in a single target-model pass. However, the best speculation horizon changes with context: longer drafts offer greater amortization when reliable, but can waste compute on suffixes discarded after a rejection. Unlike autoregressive drafting, where a stopping rule can decide token by token, MTP must fix the horizon before drafting, as all heads fire in the same forward pass. We therefore formulate horizon selection as an online control problem, using verification feedback to learn how far the draft can be trusted in each context, and introduce Adaptive Multi-Token Prediction (AMTP): a lightweight controller that requires no retraining and leaves the exact verifier unchanged. To guide horizon selection, we characterize the expected number of tokens committed at each horizon via acceptance survival, the probability that a draft prefix is fully accepted. Before each round, AMTP predicts how far the draft is likely to survive and picks the horizon with the highest expected tokens per second, learning freely from every verification pass. Theoretically, we show that any online learner's prediction error translates into throughput regret against the best horizon for each context, and in turn into a bound on wall-clock completion time. With online gradient descent, this regret is sublinear. Across five draft-target pairings built on Llama-3.1-8B-Instruct with Medusa heads, Meta's MTP models, and Qwen3-4B-Inst-2507-MTP, evaluated on AIME 2025 and GSM8K, AMTP decodes up to 4.1x faster than standard decoding and matches or outperforms the fixed window in 8 of 10 settings, with throughput gains of up to 10%.