Adaptive Multi-Token Prediction: How far should we go?

Emile Anand; Mohammed Rehan Parwani; Xinyuan Cao

cs.AIOct 20, 2026

Adaptive Multi-Token Prediction: How far should we go?

Authors: Emile Anand, Mohammed Rehan Parwani, Xinyuan Cao

Organizations: Georgia Institute of Technology

Abstract

Multi-token prediction (MTP) can accelerate language-model decoding by drafting several tokens and verifying them in a single target-model pass. However, the best speculation horizon changes with context: longer drafts offer greater amortization when reliable, but can waste compute on suffixes discarded after a rejection. Unlike autoregressive drafting, where a stopping rule can decide token by token, MTP must fix the horizon before drafting, as all heads fire in the same forward pass. We therefore formulate horizon selection as an online control problem, using verification feedback to learn how far the draft can be trusted in each context, and introduce Adaptive Multi-Token Prediction (AMTP): a lightweight controller that requires no retraining and leaves the exact verifier unchanged. To guide horizon selection, we characterize the expected number of tokens committed at each horizon via acceptance survival, the probability that a draft prefix is fully accepted. Before each round, AMTP predicts how far the draft is likely to survive and picks the horizon with the highest expected tokens per second, learning freely from every verification pass. Theoretically, we show that any online learner's prediction error translates into throughput regret against the best horizon for each context, and in turn into a bound on wall-clock completion time. With online gradient descent, this regret is sublinear. Across five draft-target pairings built on Llama-3.1-8B-Instruct with Medusa heads, Meta's MTP models, and Qwen3-4B-Inst-2507-MTP, evaluated on AIME 2025 and GSM8K, AMTP decodes up to 4.1x faster than standard decoding and matches or outperforms the fixed window in 8 of 10 settings, with throughput gains of up to 10%.

Figures & tables

Figure 1: MTP with exact verification. Proposal heads condition on earlier sampled tokens. One target pass scores $p_{j}(\cdot|H_{t},Y_{<j})$ , and circles denote prefix-ordered acceptance. Rejection emits a residual correction $Z$ . The dashed bonus branch requires full acceptance.

Figure 2: AMTP’s online control loop. Before drafting, it picks the horizon with the highest predicted throughput. Verification feedback at every drafted position, including those after a rejection, enters the weighted loss in Equation 8 , and an OGD step updates the hazard predictor.

		AIME 2025		GSM8K
Model	Decoding	tok/s	Speed-up	tok/s	Speed-up
Self-distilled Qwen3-4B	Plain (next-token)	18.8	1.00 $\times$	16.2	1.00 $\times$
	Fixed $k{=}16$	34.0	1.81 $\times$	38.6	2.39 $\times$
	Adaptive (ours)	34.5	1.83 $\times$	38.3	2.37 $\times$
Meta MTP 7B	Plain (next-token)	12.5	1.00 $\times$	4.7 b	1.00 $\times$
	Fixed $k{=}4$	37.7	3.01 $\times$	19.2 b	4.09 $\times$

Table 1: Decoding throughput (tokens/s) and speed-up over plain decoding for each draft/target pairing on AIME 2025 and GSM8K, with AMTP in bold if it exceeds the best fixed window by more than 1%. a Plain decoding run reached 1,166 of 1,319 GSM8K problems before its timeout: speed-up computed over that subset. b Meta MTP’s base checkpoints do not emit an end-of-sequence token after answering, so GSM8K runs for this pairing are capped at 100 problems, 100 rounds each.

Model	Eval	Fixed	Adapt.	Fixed	Adapt.	Plain	Fixed	Adapt.
		Accept. (%)		Tokens/round		Accuracy (%)
Self-distilled 4B	GSM8K	23	32	4.67	4.63	89.4	85.0	85.7
Medusa 8B	GSM8K	33	48	2.99	2.90	83.2	79.6	79.2
Self-dist. 4B $\to$ 14B	GSM8K	21	29	4.33	4.33	92.8 a	92.9	93.5
Self-distilled 4B	AIME	18	34	3.84	3.83	20.0	30.0	26.7
Meta MTP 7B	AIME	50	57	3.00	3.03	0.0	0.0	0.0

Table 2: Draft acceptance rate, tokens committed per verification round, and accuracy for plain decoding, the best fixed window and AMTP on AIME 2025 and GSM8K. a Plain decoding run reached 1,166 of 1,319 GSM8K problems before its time limit; accuracy computed over that subset.

Model	Eval	Mean $k_{t}$	Std.	Range	Distinct $k$
Self-distilled 4B	GSM8K	11.21	2.73	1–16	16
Meta MTP 7B	AIME	3.54	0.69	1–4	4
Meta MTP 13B	AIME	2.93	0.99	1–4	4
Medusa 8B	GSM8K	3.95	1.02	1–6	6

Table 3: Distribution of the draft window

k_{t}

chosen by AMTP over all verification rounds of a run. We remark here that the window is not constant: it varies across rounds within a single generation and across generations.

Panel of Figure 3 — Figure 3: Draft window chosen by AMTP, averaged over generations in bins of 10 verification rounds (points; bars show one standard deviation across rounds in the bin). The window adapts as generation proceeds instead of staying at a single value.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

#	Family	Pairing	Draft	Target	Plain baseline	Max $k$
1	Self-distilled	Self-speculative	Qwen3-4B-Inst-2507-MTP	Same checkpoint	Same checkpoint	16
2	Meta MTP	Self-speculative	multi-token-prediction 7B	Same checkpoint	Llama-2-7B	4
3	Medusa	Self-speculative	Llama-3.1-8B-Instruct + Medusa heads	Same checkpoint	Llama-3.1-8B-Instruct	6
4	Self-distilled	Cross-model	Qwen3-4B-Inst-2507-MTP	Qwen3-14B	Qwen3-14B	16
5	Meta MTP	Cross-model	multi-token-prediction 7B	Llama-2-13B	Llama-2-13B	4

Appendix

Table 4: MTP draft/target experiment matrix, run without KV-caches. Rows 1-3 are the core self-speculative pairings; rows 4-5 are the cross-model scaling pairings. Each row is evaluated on GSM8K and AIME 2025 with a fixed window, AMTP, and plain (next-token) decoding of the target.

Quantity	Value
Maximum window $K$	4 / 6 / 16 (Meta / Medusa / self-dist.)
Hazard model	affine (linear, no sigmoid link), 5 inputs
Hazard optimiser, learning rate	Projected OGD, $0.01$
Hazard projection radius $D$	$10.0$
Exploration rate $\varepsilon$	$0.1$
Cost optimiser, learning rate	Adam, $0.01$

Appendix

Table 5: AMTP configuration used in all experiments. No value was tuned per model or per task.

Model / eval	$k{=}1$	$2$	$3$	$4$	$5$ – $6$	$\geq 7$
Meta MTP 7B, AIME	1.4	7.3	27.0	64.3	0.0	0.0
Meta MTP 13B, AIME	9.5	24.6	29.5	36.5	0.0	0.0
Meta MTP 7B, GSM8K $\ast$	0.3	1.4	19.0	79.2	0.0	0.0
Meta MTP 13B, GSM8K $\ast$	0.5	2.9	16.7	79.9	0.0	0.0
Medusa 8B, GSM8K	0.1	4.7	29.2	43.5	22.5	0.0
Self-dist. 4B, GSM8K	0.0	0.0	0.0	0.2	2.9	96.9

Appendix

Table 6: Share of verification rounds (%) in which the control block chose each window size, over the full round population. The distribution is wide for every model, and the mode differs by model and by task. ∗ Capped runs (first 100 problems, first 100 rounds).

Draft $\to$ target	Condition	Rounds	Tok./round	Accept. (%)	Tok./s	Gen. (h)	Acc. (%)
Self-dist. Qwen3-4B	Plain	81,791	1.00	–	18.8	1.21	20.0
	Fixed $k{=}16$	25,549	3.84	17.7	34.0	0.80	30.0
	Adaptive	25,927	3.83	33.9	34.5	0.80	26.7
Self-dist. 4B $\to$ Qwen3-14B	Qwen3-14B	60,091	1.00	–	10.4	1.61	20.0
	Fixed $k{=}16$	18,244	3.23	14.0	22.5	0.73	20.0
	Adaptive	16,525	3.20	28.2	24.7	0.60	23.3

Appendix

Table 7: AIME 2025 evaluations (

n{=}30

problems in each row). Rounds: total verification rounds over all problems (one round is one token for plain decoding). Tok./s: total committed tokens per total measured round wall time. Gen. (h): summed round wall time. Acc.: fraction of problems answered correctly; it is

0

for every 7–13B model (Meta and Llama), so it carries no signal there.

Draft $\to$ target	Condition	Problems	Rounds	Tok./round	Accept. (%)	Tok./s	Gen. (h)	Acc. (%)
Self-dist. Qwen3-4B	Plain	1319	279,050	1.00	–	16.2	4.80	89.4
	Fixed $k{=}16$	1319	64,008	4.67	23.0	38.6	2.15	85.0
	Adaptive	1319	65,183	4.63	32.4	38.3	2.19	85.7
Self-dist. 4B $\to$ Qwen3-14B	Qwen3-14B $\dagger$	1166	179,720	1.00	–	6.4	7.80	92.8
	Fixed $k{=}16$	1319	47,643	4.33	20.8	21.3	2.69	92.9
	Adaptive	1319	47,529	4.33	28.5	21.5	2.66	93.5

Appendix

Table 8: Complete GSM8K results. Self-distilled 4B and Qwen3-14B target rows use all 1,319 problems where finished; Medusa uses the first 250 problems;

\ast

Meta MTP runs are capped (first 100 problems, first 100 rounds;

\sim

300 tokens for plain decoding) because the Meta base models do not terminate after answering, so their scored accuracy (1–4%, driven almost entirely by truncation) is not meaningful and is omitted.

\dagger

The Qwen3-14B plain run reached 1,166 of 1,319 problems before its time limit; its accuracy is over those completed problems.

Panel of Figure 4 — Figure 4: Top: decoding throughput of plain decoding, the best fixed window and AMTP. Every multi-token-prediction condition is faster than plain decoding. Middle: throughput of AMTP relative to the best fixed window, for self-speculative pairings and the self-distilled cross-model pairing — effects are modest in both directions, within about 15% either way. Bottom: the Meta MTP cross-model pairing, shown separately because its gain ( $+268\%$ AIME, $+147\%$ GSM8K) is an order of magnitude larger than every other condition — draft/target cost asymmetry combined with long, variable-length reasoning produces the largest effect in this validation.

Figure 5: Four consecutive real verification rounds (648–651) from one adaptive-control-block run on AIME 2025 (self-distilled Qwen3-4B). Green tokens were accepted, the red token was rejected and replaced by the correction shown on the right (resampled from the residual), and grey tokens were drafted but never verified because the round already stopped at the rejection. The control block re-selects the window $k_{t}$ every round (shown on the left). All tokens, windows, and corrections are taken directly from logged telemetry.

Round	$k_{t}$	Draft	Accepted	Correction
648	13	2 7 6 8 - ␣ 4 8 6 4 = ␣ 2	13/13 (all)	—
649	12	9 0 4 Yes But ␣ ␣of ␣of ␣number ␣subsets	4/12	Yes
650	14	. ␣So , ␣ $␣number ␣of 1 ␣subsets ␣subsets ␣with ␣is ␣** ␣$ ␣elements	1/14	So
651	14	, ␣the ␣number ␣ $␣of ␣$ \leq ␣the 4 0 4 ␣with ^ ␣\	3/14	␣of
652	16	␣non ␣with B B ␣have ␣B (S ␣are ␣2 0 ^ $ ␣= 2 2	1/16	empty
653	8	␣subsets ␣$ S ␣L CM ␣= ␣ 2	3/8	$

Appendix

Table 9: Seven consecutive real verification rounds (648–654) from one adaptive-control-block run. ␣ marks a leading space in a token; $ and \ are literal draft tokens (the model was mid-LaTeX). All data is taken directly from logged round telemetry.

Round	$k_{t}$	Draft	Accepted	Bonus
101	3	␣the ␣number ␣of	3/3 (all)	␣non

Appendix

Table 10: One real verification round (101) with full acceptance, from the same adaptive-control-block run as Table 9 . The bonus token is sampled from the target’s distribution at the position immediately after the draft, since no rejection occurred.

Multi-Token Prediction

Multi-token prediction trains models to forecast several future tokens at once, supporting more efficient generation and richer predictive representations.

See all 42 papers

Aug 1, 2026

AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction

Multi-Token Prediction (MTP) has emerged as an effective paradigm that augments a shared Large Language Model backbone with auxiliary heads, training the model to predict several future tokens in parallel to enrich its supervision signal and accelerate inference. However, existing training frameworks adopt a rigid, fixed-length prediction horizon, disregarding the highly non-uniform information density of natural language and code. Forcing the auxiliary heads to predict across high-entropy semantic boundaries injects noisy, conflicting training signals; because these heads share the backbone's latent representations, the resulting gradients backpropagate and interfere with the model's core capabilities. We propose AdaMTP, an adaptive training paradigm that dynamically aligns the prediction horizon with the intrinsic predictability of the sequence. At its core, an entropy-based segmentation algorithm leverages the base model to detect sudden surges in uncertainty as semantic boundaries, partitioning sequences into variable-length groups. Each token is assigned an adaptive prediction depth, and a dynamically masked MTP objective suppresses the loss for predictions that cross these boundaries, attenuating the noisy gradients that degrade the backbone. Across mathematical reasoning, code generation, and general benchmarks on three backbones (Llama-3.1-8B, Qwen-2.5-7B, Gemma-3-12B), AdaMTP consistently outperforms standard MTP in both task performance and inference speedup.

Ziqiang Cui, Han Shi, Bowei He +8

City University of Hong Kong · Huawei Technologies · Mohamed bin Zayed University of Artificial Intelligence +2

Multi-Token Prediction LLM Inference Acceleration LLM Training

Ask PDF

Jun 25, 2026

EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction

Multi-token prediction has been shown to increase data density during training, improve downstream text-generation quality, and serves as the defacto approach for self-speculative decoding. Existing foundation and open source models that use MTP heads commit to a static tree-based attention topology throughout the entire generation sequence, meaning the speculation depth, and thus the compute required during verification, stays constant regardless of the context. This is fundamentally misaligned with the entropy patterns of natural language where low-entropy regions often support reliable multi-step drafting, while high-entropy regions require more conservative speculation. To address this, we propose Entropy-guided Multi-Token Prediction (EntMTP), a training-free scheduler that toggles between tree-based attention topologies from a set of task-specific pareto-optimal trees conditioned on a running estimate of local generation entropy. By matching speculation depth to context predictability, EntMTP maximizes expected accepted-token throughput across the full distribution of generated text without sacrificing generation quality. When evaluated across Humaneval, ShareGPT, GSM8k, and Litbench benchmarks, EntMTP consistently achieves a 1.15x speedup against Hydra and peak speedup of 1.36x against Medusa baselines respectively.

Carrie Chen

Cornell University

Multi-Token Prediction LLM Inference Scheduling Speculative Decoding LLM Inference Acceleration

Ask PDF

Oct 1, 2026

Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Heads

Multi-token prediction (MTP) improves the throughput of autoregressive generation by enabling the language model to draft multiple next tokens per forward pass, while a verification step over draft tokens ensures that token distribution of the backbone is preserved. Every open MTP-family release (MiMo-7B, DeepSeek-V3, Qwen3) trains its heads jointly with the backbone over the full pretraining run of tens of trillions of tokens, thus setting the drafter quality at pretraining time. We ask whether a lightweight post-training pass on target-generated chain-of-thought is enough to reach the same expected throughput speedup on a frozen reasoning model, and study how a serving-time system built on such a checkpoint can be optimized. We present three findings. 1) On a frozen Qwen3-8B with

K{=}3

chained MTP heads, we show that a post-training recipe with plain cross-entropy on

\approx\!2.5

B tokens reaches or exceeds the expected speedup of jointly trained MiMo-7B on math, coding and knowledge benchmarks. Our post-training recipe utilizes

10^3

10^4\times

less MTP-training tokens as compared with joint pre-training of MiMO-7B MTP baseline. 2) We propose a chain-aware relaxation of draft token verification rule that allows a bounded drift from backbone language model token distribution. We show that this relaxation lifts expected speedups by

+12

+16\%

per benchmark while preserving task accuracy. 3) We propose an adaptive controller that dynamically chooses the number of MTP heads to be engaged at inference time and demonstrate recovery of upto

11

14\%

loss in speedup using fixed maximum MTP draft length.

Prachi Badarayani, Aidan Jay, Chenghui Zhou +8

Microsoft

Multi-Token Prediction Speculative Decoding LLM Inference Acceleration Language Model Post-Training

Ask PDF

Abstract

Multi-Token Prediction

Explore similar work

AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction

EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction

Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Heads