cs.LGNov 8, 2025

Next-Latent Prediction Transformers Learn Compact World Models

Authors: Jayden Teoh, Manan Tomar, Kwangjun Ahn, Edward S. Hu, Tim Pearce, Pratyusha Sharma, Akshay Krishnamurthy, Riashat Islam, 

Organizations: Microsoft Research

Abstract

Transformers replace recurrence with a memory that grows with sequence length and self-attention that enables ad-hoc lookups over past tokens. Consequently, they lack an inherent incentive to compress history into compact latent states with consistent transition rules. This often leads to learning solutions that generalize poorly. We introduce Next-Latent Prediction (NextLat), which extends standard next-token training with self-supervised predictions in the latent space. Specifically, NextLat trains a transformer to learn latent representations that are predictive of its next latent state given the next token. Theoretically, we show that these latents converge towards belief states, compressed information about the history necessary to predict the future. This simple auxiliary objective injects a recurrent inductive bias into transformers while leaving their architecture, parallel training efficiency, and inference unchanged. NextLat effectively encourages transformers to form compact internal world models with coherent belief states and transition dynamics -- crucial properties not guaranteed by standard next-token prediction alone. Empirically, across benchmarks in world modeling, reasoning, planning, and language modeling, NextLat demonstrates significant gains over standard next-token prediction and other baselines in downstream accuracy, representation compression, and lookahead planning. Furthermore, NextLat enables variable-length self-speculative decoding, accelerating inference by up to 3.3x in language modeling. NextLat offers a simple yet effective paradigm for learning compact, predictive representations in transformers that generalize better. Our code is available at https://github.com/JaydenTeoh/NextLat.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Thinking into the Future: Latent Lookahead Training for Transformers

    Mar 3, 2026Lorenzo Noci, Gregor Bachmann, Seyed-Mohsen Moosavi-Dezfooli +1Autoregressive Language ModelingSequence Prediction

  2. Hierarchical Latent Prediction for Language Models

    Aug 6, 2026Chang Shi, Tim Pearce, Manan Tomar +2Language Model PretrainingMulti-Token Prediction

  3. LightMTP: Lightweight Latent Multi-Token Prediction

    Oct 5, 2026Tamara Czinczoll, Julie Kallini, Gerard de Melo +1Language Model PretrainingMulti-Token Prediction