Adaptive Multi-Token Prediction: How far should we go?

Emile Anand; Mohammed Rehan Parwani; Xinyuan Cao

cs.AIOct 20, 2026

Adaptive Multi-Token Prediction: How far should we go?

Authors: Emile Anand, Mohammed Rehan Parwani, Xinyuan Cao

Organizations: Georgia Institute of Technology

Abstract

Multi-token prediction (MTP) can accelerate language-model decoding by drafting several tokens and verifying them in a single target-model pass. However, the best speculation horizon changes with context: longer drafts offer greater amortization when reliable, but can waste compute on suffixes discarded after a rejection. Unlike autoregressive drafting, where a stopping rule can decide token by token, MTP must fix the horizon before drafting, as all heads fire in the same forward pass. We therefore formulate horizon selection as an online control problem, using verification feedback to learn how far the draft can be trusted in each context, and introduce Adaptive Multi-Token Prediction (AMTP): a lightweight controller that requires no retraining and leaves the exact verifier unchanged. To guide horizon selection, we characterize the expected number of tokens committed at each horizon via acceptance survival, the probability that a draft prefix is fully accepted. Before each round, AMTP predicts how far the draft is likely to survive and picks the horizon with the highest expected tokens per second, learning freely from every verification pass. Theoretically, we show that any online learner's prediction error translates into throughput regret against the best horizon for each context, and in turn into a bound on wall-clock completion time. With online gradient descent, this regret is sublinear. Across five draft-target pairings built on Llama-3.1-8B-Instruct with Medusa heads, Meta's MTP models, and Qwen3-4B-Inst-2507-MTP, evaluated on AIME 2025 and GSM8K, AMTP decodes up to 4.1x faster than standard decoding and matches or outperforms the fixed window in 8 of 10 settings, with throughput gains of up to 10%.

Figures & tables

Figure 1: MTP with exact verification. Proposal heads condition on earlier sampled tokens. One target pass scores $p_{j}(\cdot|H_{t},Y_{<j})$ , and circles denote prefix-ordered acceptance. Rejection emits a residual correction $Z$ . The dashed bonus branch requires full acceptance.

Figure 2: AMTP’s online control loop. Before drafting, it picks the horizon with the highest predicted throughput. Verification feedback at every drafted position, including those after a rejection, enters the weighted loss in Equation 8 , and an OGD step updates the hazard predictor.

		AIME 2025		GSM8K
Model	Decoding	tok/s	Speed-up	tok/s	Speed-up
Self-distilled Qwen3-4B	Plain (next-token)	18.8	1.00 $\times$	16.2	1.00 $\times$
	Fixed $k{=}16$	34.0	1.81 $\times$	38.6	2.39 $\times$
	Adaptive (ours)	34.5	1.83 $\times$	38.3	2.37 $\times$
Meta MTP 7B	Plain (next-token)	12.5	1.00 $\times$	4.7 b	1.00 $\times$
	Fixed $k{=}4$	37.7	3.01 $\times$	19.2 b	4.09 $\times$

Table 1: Decoding throughput (tokens/s) and speed-up over plain decoding for each draft/target pairing on AIME 2025 and GSM8K, with AMTP in bold if it exceeds the best fixed window by more than 1%. a Plain decoding run reached 1,166 of 1,319 GSM8K problems before its timeout: speed-up computed over that subset. b Meta MTP’s base checkpoints do not emit an end-of-sequence token after answering, so GSM8K runs for this pairing are capped at 100 problems, 100 rounds each.

Model	Eval	Fixed	Adapt.	Fixed	Adapt.	Plain	Fixed	Adapt.
		Accept. (%)		Tokens/round		Accuracy (%)
Self-distilled 4B	GSM8K	23	32	4.67	4.63	89.4	85.0	85.7
Medusa 8B	GSM8K	33	48	2.99	2.90	83.2	79.6	79.2
Self-dist. 4B $\to$ 14B	GSM8K	21	29	4.33	4.33	92.8 a	92.9	93.5
Self-distilled 4B	AIME	18	34	3.84	3.83	20.0	30.0	26.7
Meta MTP 7B	AIME	50	57	3.00	3.03	0.0	0.0	0.0

Table 2: Draft acceptance rate, tokens committed per verification round, and accuracy for plain decoding, the best fixed window and AMTP on AIME 2025 and GSM8K. a Plain decoding run reached 1,166 of 1,319 GSM8K problems before its time limit; accuracy computed over that subset.

Model	Eval	Mean $k_{t}$	Std.	Range	Distinct $k$
Self-distilled 4B	GSM8K	11.21	2.73	1–16	16
Meta MTP 7B	AIME	3.54	0.69	1–4	4
Meta MTP 13B	AIME	2.93	0.99	1–4	4
Medusa 8B	GSM8K	3.95	1.02	1–6	6

Table 3: Distribution of the draft window

k_{t}

chosen by AMTP over all verification rounds of a run. We remark here that the window is not constant: it varies across rounds within a single generation and across generations.

Panel of Figure 3 — Figure 3: Draft window chosen by AMTP, averaged over generations in bins of 10 verification rounds (points; bars show one standard deviation across rounds in the bin). The window adapts as generation proceeds instead of staying at a single value.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

#	Family	Pairing	Draft	Target	Plain baseline	Max $k$
1	Self-distilled	Self-speculative	Qwen3-4B-Inst-2507-MTP	Same checkpoint	Same checkpoint	16
2	Meta MTP	Self-speculative	multi-token-prediction 7B	Same checkpoint	Llama-2-7B	4
3	Medusa	Self-speculative	Llama-3.1-8B-Instruct + Medusa heads	Same checkpoint	Llama-3.1-8B-Instruct	6
4	Self-distilled	Cross-model	Qwen3-4B-Inst-2507-MTP	Qwen3-14B	Qwen3-14B	16
5	Meta MTP	Cross-model	multi-token-prediction 7B	Llama-2-13B	Llama-2-13B	4

Appendix

Table 4: MTP draft/target experiment matrix, run without KV-caches. Rows 1-3 are the core self-speculative pairings; rows 4-5 are the cross-model scaling pairings. Each row is evaluated on GSM8K and AIME 2025 with a fixed window, AMTP, and plain (next-token) decoding of the target.

Quantity	Value
Maximum window $K$	4 / 6 / 16 (Meta / Medusa / self-dist.)
Hazard model	affine (linear, no sigmoid link), 5 inputs
Hazard optimiser, learning rate	Projected OGD, $0.01$
Hazard projection radius $D$	$10.0$
Exploration rate $\varepsilon$	$0.1$
Cost optimiser, learning rate	Adam, $0.01$

Appendix

Table 5: AMTP configuration used in all experiments. No value was tuned per model or per task.

Model / eval	$k{=}1$	$2$	$3$	$4$	$5$ – $6$	$\geq 7$
Meta MTP 7B, AIME	1.4	7.3	27.0	64.3	0.0	0.0
Meta MTP 13B, AIME	9.5	24.6	29.5	36.5	0.0	0.0
Meta MTP 7B, GSM8K $\ast$	0.3	1.4	19.0	79.2	0.0	0.0
Meta MTP 13B, GSM8K $\ast$	0.5	2.9	16.7	79.9	0.0	0.0
Medusa 8B, GSM8K	0.1	4.7	29.2	43.5	22.5	0.0
Self-dist. 4B, GSM8K	0.0	0.0	0.0	0.2	2.9	96.9

Appendix

Table 6: Share of verification rounds (%) in which the control block chose each window size, over the full round population. The distribution is wide for every model, and the mode differs by model and by task. ∗ Capped runs (first 100 problems, first 100 rounds).

Draft $\to$ target	Condition	Rounds	Tok./round	Accept. (%)	Tok./s	Gen. (h)	Acc. (%)
Self-dist. Qwen3-4B	Plain	81,791	1.00	–	18.8	1.21	20.0
	Fixed $k{=}16$	25,549	3.84	17.7	34.0	0.80	30.0
	Adaptive	25,927	3.83	33.9	34.5	0.80	26.7
Self-dist. 4B $\to$ Qwen3-14B	Qwen3-14B	60,091	1.00	–	10.4	1.61	20.0
	Fixed $k{=}16$	18,244	3.23	14.0	22.5	0.73	20.0
	Adaptive	16,525	3.20	28.2	24.7	0.60	23.3

Appendix

Table 7: AIME 2025 evaluations (

n{=}30

problems in each row). Rounds: total verification rounds over all problems (one round is one token for plain decoding). Tok./s: total committed tokens per total measured round wall time. Gen. (h): summed round wall time. Acc.: fraction of problems answered correctly; it is

0

for every 7–13B model (Meta and Llama), so it carries no signal there.

Draft $\to$ target	Condition	Problems	Rounds	Tok./round	Accept. (%)	Tok./s	Gen. (h)	Acc. (%)
Self-dist. Qwen3-4B	Plain	1319	279,050	1.00	–	16.2	4.80	89.4
	Fixed $k{=}16$	1319	64,008	4.67	23.0	38.6	2.15	85.0
	Adaptive	1319	65,183	4.63	32.4	38.3	2.19	85.7
Self-dist. 4B $\to$ Qwen3-14B	Qwen3-14B $\dagger$	1166	179,720	1.00	–	6.4	7.80	92.8
	Fixed $k{=}16$	1319	47,643	4.33	20.8	21.3	2.69	92.9
	Adaptive	1319	47,529	4.33	28.5	21.5	2.66	93.5

Appendix

Table 8: Complete GSM8K results. Self-distilled 4B and Qwen3-14B target rows use all 1,319 problems where finished; Medusa uses the first 250 problems;

\ast

Meta MTP runs are capped (first 100 problems, first 100 rounds;

\sim

300 tokens for plain decoding) because the Meta base models do not terminate after answering, so their scored accuracy (1–4%, driven almost entirely by truncation) is not meaningful and is omitted.

\dagger

The Qwen3-14B plain run reached 1,166 of 1,319 problems before its time limit; its accuracy is over those completed problems.

Panel of Figure 4 — Figure 4: Top: decoding throughput of plain decoding, the best fixed window and AMTP. Every multi-token-prediction condition is faster than plain decoding. Middle: throughput of AMTP relative to the best fixed window, for self-speculative pairings and the self-distilled cross-model pairing — effects are modest in both directions, within about 15% either way. Bottom: the Meta MTP cross-model pairing, shown separately because its gain ( $+268\%$ AIME, $+147\%$ GSM8K) is an order of magnitude larger than every other condition — draft/target cost asymmetry combined with long, variable-length reasoning produces the largest effect in this validation.

Figure 5: Four consecutive real verification rounds (648–651) from one adaptive-control-block run on AIME 2025 (self-distilled Qwen3-4B). Green tokens were accepted, the red token was rejected and replaced by the correction shown on the right (resampled from the residual), and grey tokens were drafted but never verified because the round already stopped at the rejection. The control block re-selects the window $k_{t}$ every round (shown on the left). All tokens, windows, and corrections are taken directly from logged telemetry.

Round	$k_{t}$	Draft	Accepted	Correction
648	13	2 7 6 8 - ␣ 4 8 6 4 = ␣ 2	13/13 (all)	—
649	12	9 0 4 Yes But ␣ ␣of ␣of ␣number ␣subsets	4/12	Yes
650	14	. ␣So , ␣ $␣number ␣of 1 ␣subsets ␣subsets ␣with ␣is ␣** ␣$ ␣elements	1/14	So
651	14	, ␣the ␣number ␣ $␣of ␣$ \leq ␣the 4 0 4 ␣with ^ ␣\	3/14	␣of
652	16	␣non ␣with B B ␣have ␣B (S ␣are ␣2 0 ^ $ ␣= 2 2	1/16	empty
653	8	␣subsets ␣$ S ␣L CM ␣= ␣ 2	3/8	$

Appendix

Table 9: Seven consecutive real verification rounds (648–654) from one adaptive-control-block run. ␣ marks a leading space in a token; $ and \ are literal draft tokens (the model was mid-LaTeX). All data is taken directly from logged round telemetry.

Round	$k_{t}$	Draft	Accepted	Bonus
101	3	␣the ␣number ␣of	3/3 (all)	␣non

Appendix

Table 10: One real verification round (101) with full acceptance, from the same adaptive-control-block run as Table 9 . The bonus token is sampled from the target’s distribution at the position immediately after the draft, since no rejection occurred.

Multi-Token Prediction

Multi-token prediction trains models to forecast several future tokens at once, supporting more efficient generation and richer predictive representations.

See all 42 papers

AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction
Aug 1, 2026Ziqiang Cui, Han Shi, Bowei He +8Multi-Token Prediction LLM Inference Acceleration
City University of Hong Kong · Huawei Technologies · Mohamed bin Zayed University of Artificial Intelligence +2
EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction
Jun 25, 2026Carrie ChenMulti-Token Prediction LLM Inference Scheduling
Cornell University
Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Heads
Oct 1, 2026Prachi Badarayani, Aidan Jay, Chenghui Zhou +8Multi-Token Prediction Speculative Decoding
Microsoft

Aug 1, 2026J/K move · Enter open · S save

AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction

Ziqiang Cui, Han Shi, Bowei He, Yu Pan+7

City University of Hong Kong · Huawei Technologies · Mohamed bin Zayed University of Artificial Intelligence+2

Multi-Token Prediction LLM Inference Acceleration LLM Training

Open paper AskPDF arXiv

Multi-Token Prediction (MTP) has emerged as an effective paradigm that augments a shared Large Language Model backbone with auxiliary heads, training the model to predict several future tokens in parallel to enrich its supervision signal and accelerate inference. However, existing training frameworks adopt a rigid, fixed-length prediction horizon, disregarding the highly non-uniform information density of natural language and code. Forcing the auxiliary heads to predict across high-entropy semantic boundaries injects noisy, conflicting training signals; because these heads share the backbone's latent representations, the resulting gradients backpropagate and interfere with the model's core capabilities. We propose AdaMTP, an adaptive training paradigm that dynamically aligns the prediction horizon with the intrinsic predictability of the sequence. At its core, an entropy-based segmentation algorithm leverages the base model to detect sudden surges in uncertainty as semantic boundaries, partitioning sequences into variable-length groups. Each token is assigned an adaptive prediction depth, and a dynamically masked MTP objective suppresses the loss for predictions that cross these boundaries, attenuating the noisy gradients that degrade the backbone. Across mathematical reasoning, code generation, and general benchmarks on three backbones (Llama-3.1-8B, Qwen-2.5-7B, Gemma-3-12B), AdaMTP consistently outperforms standard MTP in both task performance and inference speedup.

Abstract

Multi-Token Prediction

Explore similar work

AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction

EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction

Match the Distribution, Not the Compute: Post-Training Multi-Token Prediction Heads