Next-Latent Prediction Transformers Learn Compact World Models
Authors: Jayden Teoh, Manan Tomar, Kwangjun Ahn, Edward S. Hu, Tim Pearce, Pratyusha Sharma, Akshay Krishnamurthy, Riashat Islam, Alex Lamb, John Langford
Transformers replace recurrence with a memory that grows with sequence length and self-attention that enables ad-hoc lookups over past tokens. Consequently, they lack an inherent incentive to compress history into compact latent states with consistent transition rules. This often leads to learning solutions that generalize poorly. We introduce Next-Latent Prediction (NextLat), which extends standard next-token training with self-supervised predictions in the latent space. Specifically, NextLat trains a transformer to learn latent representations that are predictive of its next latent state given the next token. Theoretically, we show that these latents converge towards belief states, compressed information about the history necessary to predict the future. This simple auxiliary objective injects a recurrent inductive bias into transformers while leaving their architecture, parallel training efficiency, and inference unchanged. NextLat effectively encourages transformers to form compact internal world models with coherent belief states and transition dynamics -- crucial properties not guaranteed by standard next-token prediction alone. Empirically, across benchmarks in world modeling, reasoning, planning, and language modeling, NextLat demonstrates significant gains over standard next-token prediction and other baselines in downstream accuracy, representation compression, and lookahead planning. Furthermore, NextLat enables variable-length self-speculative decoding, accelerating inference by up to 3.3x in language modeling. NextLat offers a simple yet effective paradigm for learning compact, predictive representations in transformers that generalize better. Our code is available at https://github.com/JaydenTeoh/NextLat.
Figures & tables
Figure 1: Illustration of different predictive mechanisms. Other methods supervise only in token space, leaving intermediate latent representations implicit. In contrast, NextLat explicitly trains the model to predict the next latent state ( ht+1 ) given the current latent state ( ht ) and next token ( Xt+1 ). Token supervision is then applied to the latent prediction ( h^t+1 ). Therefore, accurate multi-token predictions emerge from faithful latent dynamics modeling.
Figure 2: Illustration comparing self-speculative decoding with multi-token prediction (MTP) vs. NextLat. MTP uses a fixed draft length per step (e.g., d=2 tokens here), whereas NextLat enables variable-length self-speculative decoding, reducing draft-verification cycles and accelerating inference.
Next-Token Test (↑)
Valid Trajectories (↑)
Sequence Compression (↑)
Effective Latent Rank (↓)
Detour Robustness (↑)
GPT
100%
97.0%
0.65
160.1
85.0%
MTP
100%
98.1%
0.64
57.7
95.0%
JTP
100%
97.1%
0.32
215.8
87.0%
NextLat
100%
98.7%
0.71
52.7
95.0%
True world model
100%
100%
1.00
—
100%
Table 1: Comparison of GPT, MTP, JTP, and NextLat trained on Manhattan taxi rides against the true world model across several metrics.
Figure 3: Reconstructed maps from transformers trained on Manhattan taxi rides using different objectives.
Model
Horizon ( d )
Accuracy (%)
GPT
–
33.1
BST
–
42.3
MTP
1
39.2
4
49.7
8
57.3
JTP
1
39.0
Table 2: Performance on Countdown. Best result is bolded , and second best is underlined .
Figure 4: Validity of equations (i.e., LHS = RHS) generated on Countdown. All models in this plot use d=1 .
Figure 7
Figure 7: Cross-entropy loss difference relative to GPT, obtained from linear probes trained on frozen hidden states to predict tokens at varying token offsets (x-axis values) ahead. Lower values indicate better predictive performance.
Model
FW-Edu ppl ↓
Wiki. ppl ↓
LAMB. ppl ↓
LAMB. acc ↑
PIQA acc ↑
HellaS. acc ↑
Wino. acc ↑
ARC-e acc ↑
ARC-c acc ↑
SIQA acc ↑
SciQ acc ↑
Avg.
GPT
10.52
17.93
20.26
42.07
73.45
58.79
60.46
68.18
39.16
42.32
86.10
58.82
JTP (d=1)
11.08
19.28
21.88
41.35
74.92
57.43
58.64
68.73
39.25
42.99
87.30
58.83
JTP (d=2)
11.18
19.60
22.11
41.37
73.34
56.84
59.98
68.86
38.57
43.35
86.70
58.63
MTP (d=1)
10.90
18.82
20.23
41.26
74.32
58.05
60.54
68.52
38.91
42.84
85.40
58.73
MTP (d=2)
11.00
18.61
18.34
43.43
72.80
57.92
59.35
68.35
39.08
41.97
86.60
58.69
NextLat (d=1)
10.83
18.39
19.77
41.08
73.07
58.35
59.27
69.65
39.68
43.24
86.00
58.79
Table 3: Downstream language modeling evaluation on 1.3B-parameter models trained on 100B FineWeb-Edu tokens. Best scores are in bold and second-best are underlined.
Wikipedia
Books
Code
Math
Model
Speedup
Accepted Tokens
Speedup
Accepted Tokens
Speedup
Accepted Tokens
Speedup
Accepted Tokens
JTP (d=1)
1.46
0.96
1.47
0.97
1.47
0.98
1.46
0.97
JTP (d=2)
1.88
1.84
1.90
1.89
1.88
1.85
1.89
1.86
MTP (d=1)
1.38
0.91
1.39
0.95
1.40
0.97
1.39
0.95
MTP (d=2)
1.68
1.72
1.72
1.83
1.75
1.91
1.72
1.84
NextLat (d=1)
2.68
3.52
2.72
3.64
2.29
2.66
2.30
2.72
Table 4: Relative speedup and average accepted tokens per drafting step over diverse domains. Note that “Accepted Tokens” excludes the next-token prediction which is always accepted.
Figure 8: (a) Inference speedup ratio of JTP, MTP, and NextLat on the FineWeb-Edu validation set. (b) Cumulative acceptance rate of draft tokens (product of per-position acceptance probabilities), indicating how likely the entire speculative draft remains valid (i.e., all tokens accepted) as draft length increases.
GPT
BST
MTP
JTP
NextLat
Training parameters ( d=1,2,8 )
1.32B
2.57B
1.37B / 1.42B / 1.72B
1.34B
1.40B
Inference parameters
1.32B
1.32B / 2.57B
1.32B
1.34B
1.32B
Training steps/second ( d=1,2,8 )
3.09
0.89
2.80 / 2.58 / 1.70
3.15 / 2.92 / 2.02
3.09 / 2.79 / 1.73
Gradients
O(T)
O(T2)
O(Td)
O(Td)
O(Td)
Table 5: Comparison of training speed, parameter count and gradient signals provided on FineWeb-Edu pretraining. For training parameter numbers and training iterations/second, we report values for prediction horizons d=1,2,8 . If only a single value is shown, it means that the values are the same for all horizons. All training speeds are measured on a single NVIDIA B200 GPU with batch size of 33k tokens.
Figure 9: Length Generalization for A5 word problem.
Figure 10: Accuracy on G5,5 Path-Star graph task. NextLat improves the performance of Mamba.
Figure 11: A multi-token belief state with window size w=3 . The latent dynamics model predicts h^5 from the last three hidden states and the next token X5 .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Illustration of the latent transition model pψ .
Model
Accuracy (%)
GPT
33.1
JTP
39.0
MTP
39.2
BST
42.3
NextLat ( LKL only)
38.2
NextLat ( Lnext-h only)
50.7
Appendix
Table 6: Accuracy of d=1 models on Countdown. Best result is bolded , and second best is underlined .
Figure 13: KL and Smooth L1 Loss Ablations: (a) TinyStories probe performance under ablations of the KL and Smooth L1 losses. (b) TinyStories validation Smooth L1 and KL losses.
Figure 14: KL and CE Loss Ablations: (a) TinyStories probe performance when replacing the KL loss in NextLat with a CE loss. (b) TinyStories validation Smooth L1 and KL losses.
Figure 15: Stop-Gradient Ablations: (a) TinyStories probe performance under different stop-gradient configurations. (b) TinyStories validation Smooth L1 and KL losses. Note that all models here use both KL and Smooth L1 losses and differ only in where stop-gradients are applied in those loss components. “Stop-Gradient (Smooth L1)” applies stop-gradient only to the target hidden state in the Smooth L1 loss, “Stop-Gradient (KL)” applies stop-gradient only to the output head in the KL loss, and “Stop-Gradient (KL + Smooth L1)” applies stop-gradients to both components. All models use d=1 here.
Figure 16: FineWeb-Edu Pretraining Curves: (a) Validation Smooth L1 loss during training. (b) Validation KL loss during training. (c) Speculative decoding acceptance rate over a 5-token lookahead horizon (excluding the next-token prediction).
Manhattan
Countdown
Path-Star
TinyStories
A5
FineWeb-Edu
Training Parameters
Steps
400k (6 epochs)
100k (204 epochs)
20k (51 epochs)
100k (8.5 epochs)
400k (410 epochs)
191k (100B tokens)
Batch Size
256
1024
512
256
1024
512
Learning Rate
1e-4
3e-4
5e-4
3e-4
1e-4
4e-4
Learning Rate Schedule
Constant
Constant
Constant
Constant
Constant
WSD
Weight Decay
0.01
0.1
0.1
0.1
0.01
0.1
Appendix
Table 7: Training, Model, and NextLat hyperparameters across all benchmarks.
Figure 17: Full plot version of Figure 7 that shows probe performance across all 20 token offsets.
Model
FW-Edu ppl ↓
Wiki. ppl ↓
LAMB. ppl ↓
LAMB. acc ↑
PIQA acc ↑
HellaS. acc ↑
Wino. acc ↑
ARC-e acc ↑
ARC-c acc ↑
SIQA acc ↑
SciQ acc ↑
Avg.
GPT
10.52
17.93
20.26
42.07
73.45
58.79
60.46
68.18
39.16
42.32
86.10
58.82
JTP (d=1)
11.08
19.28
21.88
41.35
74.92
57.43
58.64
68.73
39.25
42.99
87.30
58.83
JTP (d=2)
11.18
19.60
22.11
41.37
73.34
56.84
59.98
68.86
38.57
43.35
86.70
58.63
JTP (d=4)
11.29
20.45
20.65
41.90
73.45
56.58
57.70
69.73
39.68
42.53
88.50
58.76
MTP (d=1)
10.90
18.82
20.23
41.26
74.32
58.05
60.54
68.52
38.91
42.84
85.40
58.73
MTP (d=2)
11.00
18.61
18.34
43.43
72.80
57.92
59.35
68.35
39.08
41.97
86.60
58.69
Appendix
Table 8: Downstream language modeling evaluation on 1.3B-parameter models trained on 100B FineWeb-Edu tokens. Best scores are in bold and second-best are underlined.
Figure 18: Inference speedup ratio on FineWeb-Edu.
Wikipedia
Books
Code
Math
Model
Speedup
Accepted Tokens
Speedup
Accepted Tokens
Speedup
Accepted Tokens
Speedup
Accepted Tokens
JTP (d=1)
1.46
0.96
1.47
0.97
1.47
0.98
1.46
0.97
JTP (d=2)
1.88
1.84
1.90
1.89
1.88
1.85
1.89
1.86
JTP (d=4)
2.58
3.32
2.62
3.43
2.61
3.43
2.42
3.02
MTP (d=1)
1.38
0.91
1.39
0.95
1.40
0.97
1.39
0.95
MTP (d=2)
1.68
1.72
1.72
1.83
1.75
1.91
1.72
1.84
Appendix
Table 9: Relative speedup and average accepted tokens per drafting step over diverse domains. Note that “Accepted Tokens” excludes the next-token prediction which is always accepted.