Self-Distillation for Multi-Token Prediction

Guoliang Zhao; Ruobing Xie; An Wang; Shuaipeng Li; Huaibing Xie; Xingwu Sun

cs.CLMar 25, 2026

Self-Distillation for Multi-Token Prediction

Authors: Guoliang Zhao, Ruobing Xie, An Wang, Shuaipeng Li, Huaibing Xie, Xingwu Sun

Organizations: Large Language Model Department, Tencent

Abstract

As Large Language Models (LLMs) scale up, inference efficiency becomes a critical bottleneck. Multi-Token Prediction (MTP) could accelerate LLM inference by predicting multiple future tokens in parallel. However, existing MTP approaches still face two challenges: limited acceptance rates of MTP heads, and difficulties in jointly training multiple MTP heads. Therefore, we propose MTP-D, a simple yet effective self-distillation method with minimal additional training cost, which boosts MTP head acceptance rates (+7.5%) while maximumly preserving main-head performance. We also introduce a looped extension strategy for MTP-D, enabling effective and economical MTP head extension and further significant inference speedup to 1-head MTP (+220.4%). Moreover, we systematically explore and validate key insights on the distillation strategies and the potential scalability of MTP through extensive experiments on seven benchmarks. These results demonstrate that our MTP-D and looped extension strategy effectively enhance MTP-head performance and inference efficiency, facilitating the practical usage of MTP in LLMs.

Figures & tables

Figure 1: Overview of the gradient-detached, Top $N$ -logits-selected self-distillation method.

Figure 2: Illustration of the training strategy for looped extension of MTP-D. The gray blocks represent the frozen main model and the trained MTP heads from 1 to $m$ . The weights of the MTP heads from 1 to $m$ are copied to initialize the MTP heads from $m{+}1$ to $2m$ . The orange blocks denote the trainable MTP heads.

Panel of Figure 3 — Figure 3: Acceptance rate and cumulative acceptance rate of MTP heads for different models under training-free looped extension. Using the AGIEval-en benchmark as an example, (a) shows the cumulative acceptance rate as the loop is extended to 8 MTP heads for different models, while (b) presents the acceptance rate of each MTP head.

				General	Math		Knowledge			STEM
K	Model	Method (MTP)	H	AGIEval en	GSM8K	MATH	Natural Questions	Simple QA	TriviaQA	Super GPQA
1	2B Dense	MTP	1	85.75	84.86	85.66	90.91	94.28	82.32	85.10
	2B Dense	MTP-D	1	88.98	88.54	89.58	94.40	94.30	86.78	87.93
	A1B MoE	MTP	1	85.41	85.24	85.25	91.34	90.65	80.18	82.51
	A1B MoE	MTP-D	1	90.98	89.36	87.62	93.80	93.99	86.27	87.38
4	2B Dense	MTP	1	81.88 / 81.88	90.32 / 90.32	81.27 / 81.27	89.23 / 89.23	85.17 / 85.17	82.09 / 82.09	82.62 / 82.62

Table 1: Cumulative acceptance rate and acceptance rate of MTP heads for the 2B Dense and A1B MoE models across multiple benchmarks. Here,

H

denotes the index of the MTP head. For the

K=1

group, the CAR is identical to the AR. For the

K=4

group, each cell reports CAR/AR (%) values. The best CARs of different MTP heads are highlighted in bold . For 2B Dense (A1B) with 1 head, MTP-D improves the average acceptance rate by 3.09% (4.11%). With 4 heads, the improvement is 3.91% (4.73%) for the fourth head.

Figure 4: Speedup ratios of the 2B Dense model under different MTP methods and

K

settings across multiple benchmarks, where the inference speed of 1 head MTP serves as the baseline.

Panel of Figure 5 — Figure 5: Comparison of performance across different looped extension strategies with up to 8 loops. The 1-head MTP serves as the baseline.

		General	Math		Knowledge			STEM
Strategy		AGIEval en	GSM8K	MATH	Natural Questions	Simple QA	TriviaQA	Super GPQA	Mean
MTP-D		19.20 \| 88.98	1.74 \| 88.54	1.70 \| 89.58	13.00 \| 94.30	2.22 \| 94.30	37.08 \| 86.78	6.81 \| 87.93	11.68 \| 90.06
no detach		19.60 \| 93.67	1.74 \| 93.70	1.50 \| 93.78	11.58 \| 96.88	2.01 \| 96.40	30.42 \| 92.31	4.45 \| 93.61	10.19 \| 94.34
Top $N$	10000x2	18.18 \| 89.80	1.44 \| 90.12	1.40 \| 89.84	13.24 \| 94.19	2.41 \| 93.74	34.72 \| 87.33	7.53 \| 88.53	11.27 \| 90.51
	1000	20.64 \| 90.44	1.36 \| 88.63	1.50 \| 89.52	12.77 \| 94.73	2.29 \| 94.06	34.86 \| 88.75	6.88 \| 87.14	11.47 \| 90.47
	1	18.91 \| 87.55	1.82 \| 87.00	1.65 \| 85.76	13.38 \| 92.06	2.24 \| 92.56	36.11 \| 83.11	6.16 \| 83.72	11.47 \| 87.39

Table 2: Ablation results for MTP-D with 1 head across multiple benchmarks. Each cell shows two numbers separated by “|” (left: main-head accuracy; right: MTP-head AR). The MTP-D configuration is as follows: detach , Top

N=10{,}000

\beta_{k}=1.0

, and the KL function is the forward KL. Underlined entries indicate strategies that significantly reduce main-head performance.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	2B Dense	N10BA1B MoE
dim	2048	1536
n_heads	16	32
n_kv_heads	4	4
head_dim	128	128
embedding_tie	False	False
qk_norm	True	True

Appendix

Table 3: Detailed description of model configurations.

Hyperparameter	MTP-D	Looped MTP
Optimizer	AdamW
Adam $(\beta_{1},\beta_{2})$	(0.9, 0.95)
Adam $\epsilon$	$1\times 10^{-8}$
Weight decay	0.1
Clip grad norm	1.0
Max lr	$3.0\times 10^{-4}$

Appendix

Table 4: Training hyperparameter details for MTP-D and looped MTP

Panel of Figure 6 — Figure 6: Illustration of the probability distributions of the main head logits for the $t$ -th token under different Top $N$ settings.

Figure 7: Illustration of the training loss curves for a 2B dense LLM equipped with 4 heads. The final losses of the main head and MTP heads 1 to 4 are 2.47 and [2.73, 2.85, 2.92, 2.96], respectively.

Figure 8: Speedup ratios of the A1B MoE model under different MTP methods and

K

settings across multiple benchmarks, where the inference speed of 2B Dense DeepSeek MTP with

K=1

serves as the baseline.

K	Model	Method (MTP)	Mean
1	2B Dense	MTP	11.28
	2B Dense	MTP-D	11.68
	A1B MoE	MTP	13.75
	A1B MoE	MTP-D	14.15
4	2B Dense	MTP	11.06
	2B Dense	MTP-D	10.96

Appendix

Table 5: Mean accuracy of the main head for the 2B Dense and A1B MoE models across benchmarks using different MTP methods.

Panel of Figure 9 — Figure 9: Ablation experiments for MTP-D and its variants with loss as metrics. Orange indicates losses better than MTP-D, while green indicates worse performance.

Figure 10: Training loss curves for the 2B Dense model with 1 head using the whole vocabulary.

Strategy	Mean
MTP-D	10.96
$\beta_{k}=0.1$	10.93
$\beta_{k}=0.3$	11.13
$\beta_{k}=1.0$	10.62
step weights	9.39
ensemble mean	11.22

Appendix

Table 6: Main-head mean accuracy of ablation experiments for MTP-D with 4 heads.

		General	Math		Knowledge			STEM
Strategy	H	AGIEval en	GSM8K	MATH	Natural Questions	Simple QA	TriviaQA	Super GPQA
MTP-D	1	85.86 / 85.86	85.20 / 85.20	86.20 / 86.20	91.69 / 91.69	90.16 / 90.16	84.73 / 84.73	87.77 / 87.77
	2	71.96 / 83.81	70.97 / 83.29	71.96 / 83.48	84.63 / 92.30	81.15 / 90.01	69.46 / 81.98	71.16 / 84.00
	3	61.25 / 85.12	57.57 / 81.12	62.57 / 86.95	78.33 / 92.55	74.71 / 92.06	56.75 / 81.70	58.57 / 82.31
	4	52.96 / 86.46	46.27 / 80.38	54.98 / 87.94	72.76 / 92.89	71.22 / 95.34	46.42 / 81.81	48.52 / 82.85
$\beta_{k}=0.3$	1	83.57 / 83.57	85.42 / 85.42	83.38 / 83.38	91.39 / 91.39	86.67 / 86.67	82.24 / 82.24	83.86 / 83.86

Appendix

Table 7: MTP-head AR and CAR for MTP-D with 4 heads across benchmarks, comparing three groups with similar main-head performance.

Panel of Figure 11 — Figure 11: Cumulative acceptance rates on multiple pretraining benchmarks with the training-free looped MTP scaled up to 8.

Panel of Figure 12 — Figure 12: Acceptance rates on multiple pretraining benchmarks with the training-free looped MTP scaled up to 8.

Panel of Figure 13 — Figure 13: Cumulative acceptance rates on multiple pretraining benchmarks with the looped MTP scaled up to 8.

Panel of Figure 14 — Figure 14: Cumulative acceptance rates on multiple pretraining benchmarks with the looped MTP scaled up to 16.

		General	Math		Knowledge			STEM
Loop	Method	AGIEval en	GSM8K	MATH	Natural Questions	Simple QA	TriviaQA	Super GPQA	Mean
1	MTP-D	1.128	1.142	1.188	1.212	1.196	1.114	1.016	1.142
4	MTP-D	1.940	2.107	2.330	2.856	2.594	2.782	1.994	2.372
1 to 8	MTP-D ∗	1.337	1.451	1.726	2.356	1.971	2.263	1.280	1.769
	MTP	1.764	2.034	2.343	3.835	3.440	3.353	1.733	2.644
	MTP-D	1.964	2.197	2.741	4.193	3.505	3.455	1.865	2.846

Appendix

Table 8: Speedup ratios of different loop strategies for MTP-D and MTP during continued pre-training. Except for MTP-D with 350B data sizes, all other continued pre-training experiments use a data size of 70B tokens. * indicates Training Free.

Multi-Token Prediction

Multi-token prediction trains models to forecast several future tokens at once, supporting more efficient generation and richer predictive representations.

See all 42 papers

EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction
Jun 25, 2026Carrie ChenMulti-Token Prediction LLM Inference Scheduling
Cornell University
CLP: Collocation-Length Prediction for Zero-Loss Adaptive Multi-Token Inference
Jun 9, 2026Xuezhen Xie, Zhiqiang ZhouMulti-Token Prediction Language Model Decoding
AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction
Aug 1, 2026Ziqiang Cui, Han Shi, Bowei He +8Multi-Token Prediction LLM Inference Acceleration
City University of Hong Kong · Huawei Technologies · Mohamed bin Zayed University of Artificial Intelligence +2

Jun 25, 2026J/K move · Enter open · S save

EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction

Carrie Chen

Cornell University

Multi-Token Prediction LLM Inference Scheduling Speculative Decoding LLM Inference Acceleration

Open paper AskPDF arXiv

Multi-token prediction has been shown to increase data density during training, improve downstream text-generation quality, and serves as the defacto approach for self-speculative decoding. Existing foundation and open source models that use MTP heads commit to a static tree-based attention topology throughout the entire generation sequence, meaning the speculation depth, and thus the compute required during verification, stays constant regardless of the context. This is fundamentally misaligned with the entropy patterns of natural language where low-entropy regions often support reliable multi-step drafting, while high-entropy regions require more conservative speculation. To address this, we propose Entropy-guided Multi-Token Prediction (EntMTP), a training-free scheduler that toggles between tree-based attention topologies from a set of task-specific pareto-optimal trees conditioned on a running estimate of local generation entropy. By matching speculation depth to context predictability, EntMTP maximizes expected accepted-token throughput across the full distribution of generated text without sacrificing generation quality. When evaluated across Humaneval, ShareGPT, GSM8k, and Litbench benchmarks, EntMTP consistently achieves a 1.15x speedup against Hydra and peak speedup of 1.36x against Medusa baselines respectively.

		General	Math		Knowledge			STEM
Strategy		AGIEval en	GSM8K	MATH	Natural Questions	Simple QA	TriviaQA	Super GPQA	Mean
MTP-D		19.20 \| 88.98	1.74 \| 88.54	1.70 \| 89.58	13.00 \| 94.30	2.22 \| 94.30	37.08 \| 86.78	6.81 \| 87.93	11.68 \| 90.06
no detach		19.60 \| 93.67	1.74 \| 93.70	1.50 \| 93.78	11.58 \| 96.88	2.01 \| 96.40	30.42 \| 92.31	4.45 \| 93.61	10.19 \| 94.34
Top $N$	10000x2	18.18 \| 89.80	1.44 \| 90.12	1.40 \| 89.84	13.24 \| 94.19	2.41 \| 93.74	34.72 \| 87.33	7.53 \| 88.53	11.27 \| 90.51
	1000	20.64 \| 90.44	1.36 \| 88.63	1.50 \| 89.52	12.77 \| 94.73	2.29 \| 94.06	34.86 \| 88.75	6.88 \| 87.14	11.47 \| 90.47
	1	18.91 \| 87.55	1.82 \| 87.00	1.65 \| 85.76	13.38 \| 92.06	2.24 \| 92.56	36.11 \| 83.11	6.16 \| 83.72	11.47 \| 87.39

Abstract

Multi-Token Prediction

Explore similar work

EntMTP: Accelerating LLM Inference with Entropy Guided Multi Token Prediction

CLP: Collocation-Length Prediction for Zero-Loss Adaptive Multi-Token Inference

AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction