Predictive Latents for Autoregressive Language Models: Possibility, Gauge, and Obstruction

Yuzhou Hong

cs.CVOct 20, 2026

Predictive Latents for Autoregressive Language Models: Possibility, Gauge, and Obstruction

Authors: Yuzhou Hong

Organizations: Zhejiang Sci-Tech University

Abstract

Autoregressive language models are trained by next-token cross-entropy, yet several recent systems also iterate a hidden state without emitting a token: continuous thoughts, pause steps, and recurrent depth. This note separates three risks on one latent process. The token risk $R_{\mathrm{tok}}$ scores a readout. The teacher-forced latent risk $R_1$ scores a one-step map that sees the true hidden state. The open-loop risk $R_H$ scores the same map after it has been iterated on its own outputs. If the Bayes next-token posterior is a function of a $k$ -dimensional statistic with full-rank covariance, a width- $k$ latent, a linear least-squares transition, and a refit unembedding match the Bayes token predictor. The statistic is identified only up to an invertible gauge: whitening the residual stream does not change the token law once the unembedding is refit. The matching obstructions are sharp. A constant latent drives $R_1$ to zero while token cross-entropy falls back to the marginal entropy. A perfect token predictor can still hide a null-space walk whose open-loop error grows linearly in the horizon. A small next-latent loss does not certificate next-token prediction, and a small next-token loss does not certificate free latent rollout, unless a rank or injectivity condition is stated with the number.

Figures & tables

Panel of Figure 1 — Figure 1: The two scores on one hidden state. The top row reads $z_{t}$ out to the next token and never leaves the data prefix. The bottom row iterates $f$ on its own outputs. $R_{1}$ is only the first arrow of the bottom row.

Panel of Figure 2 — Figure 2: Open-loop risk when the one-step residual is already $1$ . The optimal linear map ( $a=0.9$ ) rises toward $5.26$ . The unit-Lipschitz walk ( $a=1$ ) keeps growing.

Small number	Conclusion	Side condition
$R_{1}$	$R_{\mathrm{tok}}$ near $R_{\star}$	sufficiency, Thm. 5
$R_{1}=0$	token Bayes	fails under collapse, Thm. 12
$R_{\mathrm{tok}}=R_{\star}$	small $R_{H}$	fails in a null direction, Thm. 14
$R_{1}$	small $R_{H}$ for all $H$	$L<1$ , Thm. 16
isotropy of $z$	a new token law	none; it is a gauge, Thm. 11

Table 1: Which small number implies which conclusion. The side condition is required for the implication.

Autoregressive Language Modeling

Autoregressive language modeling predicts sequences one step at a time, using prior tokens to generate text, speech, code, and other data.

See all 131 papers

Thinking into the Future: Latent Lookahead Training for Transformers
Mar 3, 2026Lorenzo Noci, Gregor Bachmann, Seyed-Mohsen Moosavi-Dezfooli +1Autoregressive Language Modeling Sequence Prediction
Apple
Decoupling Token Roles in Autoregressive Pretraining
Sep 27, 2026Suqin Yuan, Runqi Lin, Kevin Qinghong Lin +4Language Model Pretraining Autoregressive Language Modeling
University of Sydney · University of Oxford · Southeast University
Hierarchical Latent Prediction for Language Models
Aug 6, 2026Chang Shi, Tim Pearce, Manan Tomar +2Language Model Pretraining Multi-Token Prediction
University of Texas at Austin · Microsoft Research

Mar 3, 2026J/K move · Enter open · S save

Thinking into the Future: Latent Lookahead Training for Transformers

Lorenzo Noci, Gregor Bachmann, Seyed-Mohsen Moosavi-Dezfooli, Moin Nabi

Apple

Autoregressive Language Modeling Sequence Prediction Autoregressive Transformer LLM Planning Continuous Latent Reasoning

Open paper AskPDF arXiv

Autoregressive language models trained with next-token prediction generate text by sampling one discrete token at a time. Although very scalable, this objective forces the model to commit at every step, preventing it from exploring or reflecting upon multiple plausible continuations. Furthermore, the compute allocation across tokens is uniform; every token is formed based on a single forward-pass, potentially limiting the model's expressiveness in cases where difficult tokens require inherently more compute. Towards addressing these limitations, we introduce latent lookahead, a training strategy that enables models to "think" before generating: at selected positions in the sequence, before committing to the next token, the model performs a multi-step lookahead in latent space. More precisely, instead of sampling future tokens, we leverage the network's latent space by recursively feeding its hidden states back into the context for

τ

steps, investing more compute on predicting that token. This produces

τ

latent predictions that are supervised against the next

τ

ground-truth tokens, encouraging the model to "lookahead" and refine its prediction. We show that latent lookahead substantially outperforms both autoregressive and non-autoregressive baselines on planning tasks such as maze solving, Sudoku, and ProsQA, where foresight is essential.

Abstract

Autoregressive Language Modeling

Explore similar work

Thinking into the Future: Latent Lookahead Training for Transformers

Decoupling Token Roles in Autoregressive Pretraining

Hierarchical Latent Prediction for Language Models