Speculative decoding has emerged as a critical technique for accelerating Large Language Model (LLM) inference by decoupling generation into a drafting phase and a parallel verification phase. A crucial challenge is therefore designing a drafter that is both lightweight and accurate enough to approximate the predictions of the target model. We present a highly optimized Multi-Token Prediction (MTP) architecture that extends existing backbone LLMs to efficiently produce high-quality drafts. Built around a single shared drafting head, our approach features three core architectural optimizations: target activation injection to anchor semantic planning, zero-copy KV-cache cross-attention to bypass prefill compute and reduce memory overhead, and an efficient clustered vocabulary projection to reduce the cost of full-vocabulary decoding. Testing these improvements with the Gemma 4 model family, our G-MTP shows out-of-the-box speedups for both large server-side setups (up to 3.1×) and smaller models targeting memory-constrained on-device environments (up to 2.3× on H100 and up to 2.2× on a smartphone GPU). In addition, we present parameter-efficient fine-tuning recipes for G-MTP that keep the drafter aligned with the backbone LLM under domain shift, requiring up to 1.83× fewer target forward passes than tuning only the backbone.
Figures & tables
Figure 1: Architecture of the optimized Multi-Token Prediction (G-MTP) module. The input to the drafter is constructed by concatenating the token embedding ( e(x) ) with the backbone hidden state ( h(L) ), both of dimension dtarget , and reducing to the drafter dimension ddraft . The draft tokens are then propagated through the MTP layers, utilizing cross-attention against the backbone’s frozen KV Cache to bypass prefill computations before recursively triggering the next forward step. For subsequent drafting steps, the drafter’s output state is up-projected ( Wup ), concatenated with the embedding of the sampled draft token, and down-projected ( Wdown ) to form the next input (Equation 3 ).
Figure 2: The autoregressive G-MTP drafter (blue blocks) is initialized with activations from the main model (gray blocks) and cross-attends to its KV cache. Across successive drafting steps, the positional encoding ( p1 ) remains constant, as the drafter repeatedly queries a static, pre-computed KV snapshot.
Model
Drafter Dimension
Drafter Params
Optimal γ
Avg BE
Baseline Throughput
Throughput with G-MTP
Speedup
Gemma 4 E2B
256
76M
4
2.71
278.8
550.1
1.97 ×
Gemma 4 E4B
256
77M
9
3.45
178.0
415.5
2.33 ×
Gemma 4 12B
1024
400M
6
3.69
93.5
249.7
2.67 ×
Gemma 4 26B MoE
1024
430M
4
3.27
248.9
334.0
1.34 ×
Gemma 4 31B
1024
500M
8
4.26
73.2
230.1
3.14 ×
Table 1: Parameter counts and generation speedup across the Gemma 4 family (throughput measured on H100, in tokens per second).
Model
Mode
γ
Block Eff.
Output TPS
Speedup
Gemma 4 31B
Baseline
N/A
1.00
73.2
1.00 ×
Gemma 4 31B
DFlash
2
2.21
142.2
1.94 ×
Gemma 4 31B
DFlash
4
2.76
173.6
2.37 ×
Gemma 4 31B
DFlash
8
3.15
190.5
2.60 ×
Gemma 4 31B
EAGLE-3
2
2.08
136.4
1.86 ×
Gemma 4 31B
EAGLE-3
4
2.52
153.8
2.10 ×
Table 2: Comparison of speedup for three different Gemma 4 31B drafters (2 × H100, utilizing FlashAttention 4 kernels, batch size 1) on SPEED-Bench quality dataset.
Model
no-CE Speedup vs. Baseline
CE Speedup vs. Baseline
CE vs. no-CE Speedup
CE vs. no-CE Drafter Time
Gemma 4 E2B
1.89 ×
1.92 ×
1.01 ×
0.81 ×
Gemma 4 E4B
1.96 ×
1.98 ×
1.01 ×
0.79 ×
Table 3: Full (no-CE) vs. clustered (CE) vocabulary projection on the Edge TPU of a Google Pixel 11 (SPEED-Bench qualitative, 456 prompts, γ=3 , greedy decoding). Values are medians over prompts; speedups are ratios of median decoding throughput. CE vs. no-CE columns are ratios of decoding speedup and of total drafting time.
Dataset
Configuration Regime
NTP Accuracy
Acceptance Rate
β
Target Steps
SciQA
1. No SFT reference
68.40%
22.36%
2.51
12,334
2. Target Fine-Tuned Only (SFT)
99.23%
8.96%
1.60
19,423
3. Joint Fine-Tuning (SFT + MTP)
99.37%
28.64%
2.96
10,595
4. Two-Stage Distillation
99.23%
26.48%
2.80
11,155
GSM8K
1. No SFT reference
74.83%
40.17%
3.73
8,414
2. Target Fine-Tuned Only (SFT)
88.01%
28.67%
2.95
10,435
Table 4: Speculative Decoding Domain Adaptation Performance (Gemma 4 E4B). Fine-tuning the model without updating the MTP (row 2) can lead to regression in Block Efficiency ( β ). Joint fine-tuning (row 3) matches or exceeds the β of the no SFT reference (row 1), and two-stage distillation (row 4) recovers about 96% of it.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Model
γ
Avg BE
Baseline (tokens/s)
G-MTP Speculative (tokens/s)
Throughput Speedup
Gemma 4 E2B
2
2.15
278.8
504.7
1.81 ×
3
2.47
278.8
531.5
1.91 ×
4
2.71
278.8
550.1
1.97 ×
5
2.88
278.8
550.1
1.97 ×
6
3.02
278.8
540.8
1.94 ×
7
3.11
278.8
533.9
1.92 ×
Appendix
Table 5: Throughput across lookahead horizons γ (Figure 3 extends to γ=15 ).
Figure 3: Empirical scaling of theoretical Block Efficiency (BE) and speculative wall-clock throughput across varying lookahead horizons ( γ ). While longer draft sequences monotonically increase token acceptance, throughput degrades for some models (e.g., 26B MoE) as drafting and verification costs exceed generation gains.
Table 7: On-device decoding throughput of Gemma 4 E2B and E4B with LiteRT-LM on the GPU of a Samsung Galaxy S26 Ultra ( γ=3 , greedy decoding, full vocabulary projection). Acceptance rate and throughput values are medians over the 76 prompts; the acceptance rate is the draft acceptance rate reported by the runtime. Speedups are ratios of median throughput with and without G-MTP, over all prompts and per task type.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.