LightMTP: Lightweight Latent Multi-Token Prediction

Tamara Czinczoll; Julie Kallini; Gerard de Melo; Chen Shani

cs.CLOct 5, 2026

LightMTP: Lightweight Latent Multi-Token Prediction

Authors: Tamara Czinczoll, Julie Kallini, Gerard de Melo, Chen Shani

Organizations: Hasso Plattner Institute / University of Potsdam · Stanford University · Tel-Aviv University

Abstract

Next-token prediction (NTP) is the standard pretraining objective for large language models, yet it provides an explicit training signal only for the immediate next token, which can lead models to exploit local patterns instead of capturing longer-range structure and ideas. Multi-token prediction (MTP) addresses this by training models to predict several future tokens. However, existing MTP methods often introduce a large number of new parameters with limited improvements in downstream performance. Latent MTP approaches address this efficiency issue by encoding future tokens into a vector representation. However, these approaches usually rely on external helper models for future token encoding. We propose LightMTP, a lightweight, i.e., parameter-efficient, latent MTP approach that bootstraps the future token representations from the model's own hidden states. Our two LightMTP variants extend supervision to more future tokens without requiring the additional computational overhead of conventional MTP nor the external supervision latent MTP normally relies on. LightMTP adds at most 1% extra parameters, retains better performance on general language modeling benchmarks, and achieves similar gains in planning, coding, and reasoning.

Figures & tables

Figure 1: LightMTP passes the input through a shared transformer trunk and uses the last hidden state at each position to 1) perform next-token prediction via a linear NTP head, and 2) predict a latent representation of the next

k

target vectors stemming from the model’s own future hidden states via an MTP head. a) In our approach’s single-target variant, the model predicts the single hidden state vector in

k

steps. b) In the version with concatenated targets, each intermediate vector is additionally used to supervise the model.

Figure 2: LightMTP pays fewer bits-per-byte – Plot of anticipation cost, which quantifies how many bits-per-byte the models pay per total future token prediction improvement up to position

k

for

k{\leq}10

(lower is better). The top four most cost-efficient models are LightMTP variants.

Method	FineWeb-Edu (PPL)	Lambada (PPL)	Lambada (Acc)	CORE-avg
NTP	1.683 ±0.001	16.00 ±0.72	43.59 ±0.85	40.43 ±0.68
Meta	1.712 ±0.003	18.66 ±0.84	42.14 ±0.91	39.31 ±0.69
Meta-L	1.698 ±0.001	18.13 ±0.48	42.84 ±0.70	39.89 ±0.42
DeepSeek	1.762 ±0.038	31.49 ±11.42	35.68 ±4.28	35.81 ±2.60
Future Summaries	1.685 ±0.002	15.57 ±0.33	44.43 ±0.45	40.25 ±0.37
LightMTP-C	1.683 ±0.000	16.35 ±0.20	43.22 ±0.42	40.30 ±0.23

Table 1: LightMTP retains or even improves general LM benchmark performance – perplexity on FineWeb-Edu and Lambada, and accuracy on Lambada and CORE, each result averaged over three seeds. FineWeb-Edu results report byte-level perplexity

2^{\text{bpb}}

on a held-out shard. The Lambada columns show the FineWeb-Edu-pretrained base model applied to Lambada’s validation split, measuring perplexity and accuracy at the target word. In each column, the best value is bolded and the runner-up underlined .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Architecture (d24)
Layers / width / heads	24 / 1536 / 12 (head dim. 128, no GQA)
MLP	$4\times$ (6,144), ReLU 2 , no biases
Normalization	parameter-free RMSNorm (pre-norm, embedding, final, QK)
Position encoding	RoPE, base $10^{4}$
Attention windows	SSSL pattern (S = 1,024, L = 2,048 tokens), last layer L
Context length	2,048

Appendix

Table 5: Backbone architecture and training hyperparameters for the 24-layer models, inherited unchanged from nanochat .

Model	Total	Scaling	Non-scaling	MTP head	$\Delta$ params	Tokens	Steps	H100 GPU-h
d12 $768\times 12\times 6$
NTP	286.3M	110.1M	176.2M	—	—	1.32B	2 520	0.7
Meta-L ‡	361.8M	185.6M	176.2M	75.5M	+26.4%	2.23B	4 248	2.0
Meta	307.5M	131.3M	176.2M	21.2M	+7.4%	1.58B	3 006	1.6
DeepSeek MTP	311.0M	134.9M	176.2M	24.8M	+8.7%	1.62B	3 087	1.6
Future Summaries	287.4M(+286.3M)	111.3M	176.2M	1.2M	+0.4%	1.34B	2 547	0.9

Appendix

Table 6: Scaling and non-scaling (token and value embeddings) parameter counts for models with 12, 18, and 24 transformer layers. Training tokens and optimizer steps for each architecture. “MTP head” is the auxiliary head(s) each variant adds on top of the NTP model. “

\Delta

params” denotes the percentage of extra parameters the method adds compared to just NTP. Token and step budgets follow nanochat’s speedrun.sh : a data:parameter ratio of

12

over

n_{\text{matrix}}+n_{\text{head}}

(transformer matrices plus output head, excluding token and value embeddings). Because the ratio is applied to a parameter count that grows with the MTP head size, variants that add more parameters are trained for proportionally more steps. Future Summaries requires an already right-to-left-trained, NTP-based helper model to generate informative latent targets. “H100 GPU-h” is the approximate pretraining cost in GPU-hours on one NVIDIA H100.

Figure 6: Probing top-1 accuracy on the models’ frozen last hidden states over three seeds for

k=1,\dots,20

. All MTP models exhibit higher future token anticipation than NTP, but this often comes with a performance trade-off at

k=1

. While for

2\leq k\leq 5

, Meta’s MTP-L exhibits the strongest anticipation effect, its anticipation then gradually anneals to the other MTP models’ performance until it dips below NTP for

k\geq 12

. LightMTP models anticipation effect for early

k

is moderate but it benefits from a mitigated degradation at

k=1

Over all completions
	NTP	Meta	Meta-L	Future Summaries	LightMTP-C	LightMTP-S
parses as Python	96.00 $\pm$ 1.11	96.44 $\pm$ 1.16	96.07 $\pm$ 1.01	95.33 $\pm$ 1.49	96.41 $\pm$ 1.43	95.70 $\pm$ 0.17
solved	4.07 $\pm$ 0.56	4.48 $\pm$ 1.12	6.37 $\pm$ 0.17	5.44 $\pm$ 0.97	5.70 $\pm$ 0.53	4.00 $\pm$ 0.29
Structural failures (% of failing completions)
NameError	8.27 $\pm$ 3.66	7.93 $\pm$ 2.37	7.16 $\pm$ 0.44	5.21 $\pm$ 0.37	6.36 $\pm$ 1.11	5.90 $\pm$ 1.41
SyntaxError	4.13 $\pm$ 0.99	3.68 $\pm$ 0.59	3.72 $\pm$ 0.65	4.61 $\pm$ 1.34	3.66 $\pm$ 1.29	4.20 $\pm$ 1.53

Appendix

Table 7: MBPP failure-mode breakdown, on the MBPP validation split (tasks 511–600): 90 problems

\times

10 samples

=

900 completions per model/seed, temperature

0.7

, mean

\pm

std over seeds 42, 7, and 1337. Parses and solved are percentages of all completions. Each failure row reports a percentage of that model’s failing completions, re-executed against the problem’s asserts and labeled by the exception raised. Structural sums the four exception types attributable to local naming and syntax consistency rather than task or data understanding.

Panel of Figure 7 — Figure 7: Effect of the prediction horizon $k$ for LightMTP at target layer 16 (head layer 23), relative to NTP. NTP uses 5 seeds and every other configuration 3. All sweep runs use target layer 16 without VarReg ( $k{=}4$ is the LightMTP L16 configuration).

Baseline
	World Knowledge						Commonsense							Passage Comprehension				Symbolic reasoning
Method	jeopardy	bb-qa-wikidata	arc-easy	arc-challenge	openbook-qa	bb-language-identification	copa	commonsense-qa	piqa	hellaswag	hellaswag-zeroshot	winograd	winogrande	squad	coqa	boolq	lambada-openai	bb-dyck-languages	bb-cs-algorithms	bb-repeat-copy-logic	bb-operators	lsat-ar	CORE avg	MMLU (ICL)	GSM8K (p@5)	GSM8K (strict)	Lambada (PPL)
NTP	18.61 $\pm$ 1.36	51.60 $\pm$ 0.79	66.44 $\pm$ 1.13	37.30 $\pm$ 0.88	37.96 $\pm$ 1.01	25.42 $\pm$ 0.42	66.80 $\pm$ 1.10	28.70 $\pm$ 6.61	71.58 $\pm$ 0.28	51.27 $\pm$ 0.36	51.11 $\pm$ 0.31	69.60 $\pm$ 1.55	56.76 $\pm$ 0.97	35.89 $\pm$ 1.26	25.79 $\pm$ 1.10	52.97 $\pm$ 6.59	43.49 $\pm$ 0.87	10.86 $\pm$ 0.96	40.44 $\pm$ 3.65	3.12 $\pm$ 0.00	17.81 $\pm$ 1.92	25.91 $\pm$ 2.45	40.43 $\pm$ 0.68	24.97 $\pm$ 0.49	41.97 $\pm$ 2.29	4.94 $\pm$ 1.44	16.00 $\pm$ 0.72
Concat-targets (target layer 16)
Concat $k{=}1$	17.89 $\pm$ 0.80	52.86 $\pm$ 0.53	67.21 $\pm$ 0.53	37.63 $\pm$ 0.15	39.87 $\pm$ 1.33	25.18 $\pm$ 0.56	67.67 $\pm$ 2.31	28.31 $\pm$ 0.74	71.11 $\pm$ 0.29	51.29 $\pm$ 0.39	50.85 $\pm$ 0.32	67.77 $\pm$ 0.73	55.80 $\pm$ 1.10	35.12 $\pm$ 1.11	26.59 $\pm$ 1.18	46.97 $\pm$ 4.96	43.11 $\pm$ 0.57	11.40 $\pm$ 2.05	39.80 $\pm$ 2.77	2.08 $\pm$ 3.61	17.14 $\pm$ 0.95	27.83 $\pm$ 1.15	40.16 $\pm$ 0.38	24.37 $\pm$ 0.55	42.53 $\pm$ 1.91	4.60 $\pm$ 0.24	16.60 $\pm$ 0.09
Concat $k{=}2$	16.97 $\pm$ 2.10	51.90 $\pm$ 0.59	65.80 $\pm$ 0.49	36.58 $\pm$ 0.64	38.60 $\pm$ 0.20	25.19 $\pm$ 0.74	67.33 $\pm$ 2.89	23.31 $\pm$ 3.44	70.62 $\pm$ 1.23	50.52 $\pm$ 1.37	50.11 $\pm$ 1.24	67.28 $\pm$ 2.44	56.35 $\pm$ 1.16	35.46 $\pm$ 1.67	25.98 $\pm$ 0.18	51.06 $\pm$ 4.71	43.21 $\pm$ 0.52	10.30 $\pm$ 2.76	41.39 $\pm$ 3.43	5.21 $\pm$ 1.80	17.62 $\pm$ 1.26	27.10 $\pm$ 2.39	39.90 $\pm$ 0.45	24.41 $\pm$ 0.19	42.66 $\pm$ 2.08	5.10 $\pm$ 1.51	16.92 $\pm$ 0.81

Appendix

Table 8: depth-24 horizon sweep with target layer 16: latent MTP with concatenated next-

k

hidden-state targets (Concat) or the single hidden state at

t{+}k

(Single-target), for

k\in\{1,2,4,8,10\}

, against NTP. NTP is averaged over five seeds, as in the main results tables; every other row is three seeds. At

k{=}1

, the two objectives coincide. Full CORE suite: per-task in-context accuracy (%), grouped into closed-book knowledge, everyday commonsense, passage comprehension, and symbolic reasoning, followed by the CORE average over its 22 tasks, and MMLU and GSM8K pass@5, which are reported outside the CORE average. Cells color-coded per column, green (improvement) to red (regression) vs NTP; in each column the best value is bolded and the runner-up underlined . CORE (centered) averages each of the 22 CORE tasks as

(\mathrm{acc}-b)/(1-b)

with respect to random baseline

b

Baselines
	World Knowledge						Commonsense							Passage Comprehension				Symbolic reasoning
Method	jeopardy	bb-qa-wikidata	arc-easy	arc-challenge	openbook-qa	bb-language-identification	copa	commonsense-qa	piqa	hellaswag	hellaswag-zeroshot	winograd	winogrande	squad	coqa	boolq	lambada-openai	bb-dyck-languages	bb-cs-algorithms	bb-repeat-copy-logic	bb-operators	lsat-ar	CORE avg	MMLU (ICL)	GSM8K (p@5)	GSM8K (strict)	Lambada (PPL)
NTP	18.61 $\pm$ 1.36	51.60 $\pm$ 0.79	66.44 $\pm$ 1.13	37.30 $\pm$ 0.88	37.96 $\pm$ 1.01	25.42 $\pm$ 0.42	66.80 $\pm$ 1.10	28.70 $\pm$ 6.61	71.58 $\pm$ 0.28	51.27 $\pm$ 0.36	51.11 $\pm$ 0.31	69.60 $\pm$ 1.55	56.76 $\pm$ 0.97	35.89 $\pm$ 1.26	25.79 $\pm$ 1.10	52.97 $\pm$ 6.59	43.49 $\pm$ 0.87	10.86 $\pm$ 0.96	40.44 $\pm$ 3.65	3.12 $\pm$ 0.00	17.81 $\pm$ 1.92	25.91 $\pm$ 2.45	40.43 $\pm$ 0.68	24.97 $\pm$ 0.49	41.97 $\pm$ 2.29	4.94 $\pm$ 1.44	16.00 $\pm$ 0.72
Meta	14.49 $\pm$ 1.62	49.84 $\pm$ 0.89	64.49 $\pm$ 0.36	34.95 $\pm$ 0.49	37.67 $\pm$ 0.92	24.88 $\pm$ 0.25	68.00 $\pm$ 1.00	22.88 $\pm$ 2.74	70.18 $\pm$ 0.36	48.50 $\pm$ 0.72	48.24 $\pm$ 0.60	66.06 $\pm$ 0.76	56.01 $\pm$ 1.34	35.19 $\pm$ 1.68	26.13 $\pm$ 1.00	59.06 $\pm$ 3.60	42.12 $\pm$ 0.86	12.27 $\pm$ 1.01	39.32 $\pm$ 1.51	3.12 $\pm$ 3.12	18.25 $\pm$ 0.99	23.19 $\pm$ 3.29	39.31 $\pm$ 0.69	24.65 $\pm$ 0.97	41.45 $\pm$ 1.43	4.09 $\pm$ 0.08	18.66 $\pm$ 0.84
Meta-L	18.76 $\pm$ 0.95	50.69 $\pm$ 0.58	65.76 $\pm$ 0.29	37.30 $\pm$ 0.83	38.48 $\pm$ 0.86	24.89 $\pm$ 0.26	67.20 $\pm$ 2.17	26.22 $\pm$ 2.31	70.95 $\pm$ 0.58	51.02 $\pm$ 0.20	50.79 $\pm$ 0.29	67.69 $\pm$ 1.11	56.56 $\pm$ 1.67	38.53 $\pm$ 0.68	28.16 $\pm$ 0.37	45.22 $\pm$ 6.03	42.80 $\pm$ 0.68	11.38 $\pm$ 0.82	37.14 $\pm$ 2.95	5.62 $\pm$ 1.40	18.00 $\pm$ 1.23	24.43 $\pm$ 2.77	39.89 $\pm$ 0.42	24.54 $\pm$ 0.25	41.91 $\pm$ 0.88	4.82 $\pm$ 1.19	18.13 $\pm$ 0.48
DeepSeek MTP	6.15 $\pm$ 4.58	43.79 $\pm$ 5.14	59.78 $\pm$ 3.76	30.49 $\pm$ 2.62	33.76 $\pm$ 2.04	25.03 $\pm$ 0.39	63.20 $\pm$ 2.77	28.68 $\pm$ 4.94	67.52 $\pm$ 1.43	41.48 $\pm$ 4.25	41.73 $\pm$ 4.21	62.34 $\pm$ 3.45	52.64 $\pm$ 1.78	21.98 $\pm$ 11.46	19.71 $\pm$ 4.66	57.86 $\pm$ 3.32	35.77 $\pm$ 4.33	11.62 $\pm$ 2.64	39.12 $\pm$ 1.55	1.25 $\pm$ 1.71	14.57 $\pm$ 3.98	29.30 $\pm$ 1.52	35.81 $\pm$ 2.60	25.14 $\pm$ 0.48	39.17 $\pm$ 2.11	3.85 $\pm$ 0.68	31.49 $\pm$ 11.42

Appendix

Table 9: Full depth-24 language modeling benchmark results: full CORE benchmark suite (per-task ICL accuracy), CORE average, MMLU (ICL), GSM8K pass@5 under strict and relaxed matchers and Lambada perplexity and accuracy; mean

\pm

std over three seeds for every row, five for NTP. Every MTP model uses

k{=}4

targets. Every run is trained with parameter-matched budget, so the number of steps follows each model’s parameter count. Meta-L trains on 20.7%, DeepSeek MTP on 13.6%, and Meta on 11.6% more tokens than NTP. CORE (centered) averages each of the 22 CORE tasks as

(\mathrm{acc}-b)/(1-b)

with respect to random baseline

b

. LAMBADA columns. The base model (not a Lambada-finetuned variant) on the LAMBADA test split: target-word perplexity (lower is better) and accuracy.

Baselines
	World Knowledge						Commonsense							Passage Comprehension				Symbolic reasoning
Method	jeopardy	bb-qa-wikidata	arc-easy	arc-challenge	openbook-qa	bb-language-identification	copa	commonsense-qa	piqa	hellaswag	hellaswag-zeroshot	winograd	winogrande	squad	coqa	boolq	lambada-openai	bb-dyck-languages	bb-cs-algorithms	bb-repeat-copy-logic	bb-operators	lsat-ar	CORE avg	MMLU (ICL)	GSM8K (p@5)	GSM8K (strict)	Lambada (PPL)
NTP	4.11 $\pm$ 2.27	44.14 $\pm$ 1.95	59.83 $\pm$ 0.53	32.05 $\pm$ 0.74	34.93 $\pm$ 1.10	25.40 $\pm$ 0.34	64.00 $\pm$ 0.00	29.59 $\pm$ 3.79	67.03 $\pm$ 0.09	41.47 $\pm$ 0.57	41.77 $\pm$ 0.42	61.54 $\pm$ 2.29	53.09 $\pm$ 1.51	23.64 $\pm$ 1.90	19.66 $\pm$ 1.07	57.43 $\pm$ 2.69	36.10 $\pm$ 1.22	11.97 $\pm$ 1.44	41.26 $\pm$ 1.87	2.08 $\pm$ 1.80	15.71 $\pm$ 0.48	26.09 $\pm$ 2.17	36.04 $\pm$ 0.25	24.32 $\pm$ 0.60	38.46 $\pm$ 1.45	4.12 $\pm$ 0.65	27.85 $\pm$ 1.20
Meta	5.94 $\pm$ 1.38	43.98 $\pm$ 0.53	59.30 $\pm$ 0.36	29.10 $\pm$ 1.02	34.87 $\pm$ 1.47	24.91 $\pm$ 0.36	64.67 $\pm$ 0.58	26.73 $\pm$ 5.17	67.10 $\pm$ 0.03	41.29 $\pm$ 0.23	41.27 $\pm$ 0.21	63.74 $\pm$ 2.29	53.75 $\pm$ 0.16	26.53 $\pm$ 0.30	21.68 $\pm$ 0.21	53.43 $\pm$ 2.80	36.57 $\pm$ 1.23	12.03 $\pm$ 0.45	41.69 $\pm$ 2.39	1.04 $\pm$ 1.80	16.35 $\pm$ 0.27	24.20 $\pm$ 3.75	35.92 $\pm$ 0.17	24.14 $\pm$ 0.65	39.55 $\pm$ 3.83	4.32 $\pm$ 0.66	29.26 $\pm$ 2.29
Meta-L	6.69 $\pm$ 0.91	43.16 $\pm$ 0.71	59.64 $\pm$ 0.99	31.14 $\pm$ 0.68	35.33 $\pm$ 1.10	25.01 $\pm$ 0.52	63.00 $\pm$ 3.00	23.40 $\pm$ 2.53	66.85 $\pm$ 0.28	42.20 $\pm$ 0.31	42.23 $\pm$ 0.51	62.27 $\pm$ 1.68	54.88 $\pm$ 1.35	24.01 $\pm$ 1.48	21.02 $\pm$ 1.44	51.80 $\pm$ 2.46	36.35 $\pm$ 0.32	9.60 $\pm$ 1.71	37.30 $\pm$ 2.18	2.08 $\pm$ 1.80	14.44 $\pm$ 2.40	24.35 $\pm$ 3.56	35.31 $\pm$ 0.14	24.30 $\pm$ 0.26	38.62 $\pm$ 1.61	3.74 $\pm$ 0.09	29.21 $\pm$ 0.86
DeepSeek MTP	1.45 $\pm$ 0.88	35.31 $\pm$ 3.22	53.58 $\pm$ 1.39	27.13 $\pm$ 1.20	31.27 $\pm$ 1.67	25.37 $\pm$ 0.41	60.33 $\pm$ 2.08	27.27 $\pm$ 2.68	64.35 $\pm$ 0.69	33.53 $\pm$ 1.20	33.67 $\pm$ 1.12	59.34 $\pm$ 0.97	52.14 $\pm$ 0.92	11.56 $\pm$ 3.64	13.05 $\pm$ 1.21	54.86 $\pm$ 8.45	27.41 $\pm$ 1.45	11.97 $\pm$ 3.95	41.09 $\pm$ 0.38	2.08 $\pm$ 1.80	12.70 $\pm$ 4.05	24.64 $\pm$ 1.26	32.00 $\pm$ 0.80	24.21 $\pm$ 0.52	36.95 $\pm$ 2.07	3.06 $\pm$ 0.61	67.80 $\pm$ 10.47

Appendix

Table 11: depth-18 results: full CORE benchmark suite (per-task ICL accuracy), CORE average, MMLU (ICL), GSM8K pass@5 under both matchers and Lambada perplexity and accuracy; mean

\pm

std over three seeds for every row. Every MTP model uses

k{=}4

targets; “LightMTP” layer 12 is d24’s target layer 16 scaled to depth 18, layer 2 is kept as is. Every run is trained with a parameter-matched budget, so the number of steps follows each model’s parameter count; Meta MTP therefore trains on 34.9% and DeepSeek MTP on 17.2% more tokens than NTP. Cells color-coded vs NTP. CORE (centered) averages each of the 22 CORE tasks as

(\mathrm{acc}-b)/(1-b)

with respect to random baseline

b

. Lambada columns. The base model (not a Lambada-finetuned variant) on the LAMBADA test split: target-word perplexity (lower is better) and accuracy.

Explore similar work

Aug 6, 2026cs.CL

Hierarchical Latent Prediction for Language Models

While standard Next-Token Prediction (NTP) lays the foundation of language model pre- training, its teacher-forced training paradigm may not be optimal for long-horizon reasoning and planning. Recent works such as Multi-Token Prediction (MTP) and Next-Latent prediction (NextLat) try to mitigate the problem through predicting multiple future tokens and self-supervised prediction in the latent space. However, those auxiliary objectives either have a limited horizon or suffer from compounding error from multi-step rollout. We introduce Hierarchical Latent Prediction (HiLP), which introduces an auxiliary higher-level abstract latent to help reduce the error accumulation effect in latent-space rollouts. Experiments show that HiLP can lead to longer-horizon coherent belief state representation and demonstrate the effectiveness of our method across coding and multi-step reasoning benchmarks, and offers more speculative decoding efficiency.

Chang Shi, Tim Pearce, Manan Tomar +2

University of Texas at Austin · Microsoft Research

Next-Token Prediction Latent Prediction Long-Horizon Task Planning

Ask PDF

May 24, 2026cs.CL

NITP: Next Implicit Token Prediction for LLM Pre-training

Standard next-token prediction (NTP) supervises language models solely through discrete labels in the output logit space. We argue that this sparse one-hot supervision leaves the latent representation space under-constrained, allowing hidden states to drift into degenerate and anisotropic configurations that can limit generalization. To address this issue, we propose Next Implicit Token Prediction (NITP), which augments discrete prediction with dense continuous supervision directly in the representation space. NITP trains the model to predict the implicit semantic content of the next token, using shallow-layer representations from the same model as stable self-supervised targets. We provide theoretical analysis showing that NITP regularizes the optimization landscape by mitigating under-constrained degrees of freedom and encouraging a compact, structured representation geometry. Empirically, across dense and MoE models ranging from 0.5B to 9B parameters, NITP consistently improves downstream performance with negligible computational overhead. On a 9B MoE model, NITP achieves a 5.7% absolute improvement on MMLU-Pro, along with gains of 6.4% on C3 and 4.3% on CommonsenseQA, with approximately 2% additional training FLOPs and no additional inference cost. Our implementation is available at https://github.com/aHapBean/NITP.

Xiangdong Zhang, Debing Zhang, Shaofeng Zhang +3

School of AI, Shanghai Jiao Tong University · Dots Studio, Xiaohongshu Inc. · University of Science and Technology of China +1

Next-Token Prediction Token-Level Supervision Large Language Model Pretraining Compact Latent Sensor Tokens Latent Prediction

Ask PDF

Aug 4, 2026cs.CL

LoopMTP: A looped transformer guided by latent multi-token prediction

Looped transformers have emerged as a parameter-efficient alternative to scaling depth for strong reasoning. By reusing one stack of layers across

T

iterations, they attain the effective depth and reasoning capabilities of larger models at a fixed parameter count. Yet existing approaches suffer from latent overthinking and undifferentiated computation, largely because intermediate representations receive no guidance across loops. Multi-token prediction (MTP) supplies exactly the dense, forward-looking supervision the loop is missing. We propose \textsc{LoopMTP}, which links the two through a structural correspondence in latent space: a model that loops

T

times can anticipate

T

future tokens. \textsc{LoopMTP} realizes this by softly aligning the hidden state of loop

t

with the embedding of the token

t

steps ahead, while a lightweight gate preserves useful information across iterations. \textsc{LoopMTP} improves average accuracy by up to 8.1% (relative) over the non-looped baseline, with training remaining stable for up to 15 loops.

Behzad Shomali, Markus Frey, David Berghaus +2

Lamarr Institute · University of Bonn · Fraunhofer IAIS

Multi-Token Prediction Transformer Architectures Compact Latent Sensor Tokens Latent Space

Ask PDF