LS-AR: Future-Predictive Latent Steering in Autoregressive LLMs

Anubha Gupta; Eduardo Pignatelli

cs.LGOct 2, 2026

LS-AR: Future-Predictive Latent Steering in Autoregressive LLMs

Authors: Anubha Gupta, Eduardo Pignatelli

Organizations: Electronic & Electrical Engineering University College London London, UK

Abstract

Standard autoregressive (AR) models process high-level task instructions, state history, and transient tokens within a single shared sequence of tokens. Consequently, they lack the architectural mechanisms needed to isolate macro-objectives from context noise. To overcome this single-channel limitation, we introduce Latent-Steered Autoregressive (LS-AR), a dual-channel architecture that decouples continuous goal steering from discrete token decoding via FiLM conditioning. We evaluate a Static Goal Encoder (P_0) for persistent macro-objective retention across long rollouts and a Dynamic State Tracker (P_t) for recurrent latent updates during generation. On long-horizon retrieval past context limits (H=1024, W=500), LS-AR (Static) achieves 100% target recall where parameter-matched baselines collapse (0%), while increasing throughput by ~35% and cutting peak VRAM by 52.8%. In Blocksworld planning under forced perturbations (k=1), LS-AR (Dynamic) sustains an 89.0% completion rate vs. 71.0% for the baseline, though zero-shot entity scaling (N -> N+1) exposes single-vector capacity limits (0%). Finally, dual-channel authority analysis shows that text goal dropout establishes latent-dominant control, offering structural defence against text prompt injection while introducing a latent vector attack surface.

Figures & tables

Figure 1: Static predictor alignment (Phase 2). Predictor $E_{\text{pred}}$ maps mean-pooled prompt representations to target trajectory embeddings $Z_{\text{future}}=\text{MeanPool}(E_{\text{base}}(x_{\text{future}}))$ via VICReg loss while keeping base encoder $E_{\text{base}}$ frozen.

Figure 2: Dynamic state tracker. At trigger steps $i$ , the tracker updates the latent $P_{i-1}$ using hidden state $s_{i}=E_{\text{base}}(x_{1:i})_{i}$ to form step-aligned prior $P_{i}$ .

Figure 3: Conditioned decoding. At step $i$ , steering vector $P_{i}$ modulates decoder layer normalisations via FiLM scale $\gamma$ and shift $\beta$ parameters to condition token prediction $x_{i+1}$ .

Horizon ( $H$ )	Architecture	Target Retrieval (Semantic) $\uparrow$	Passkey Retrieval (Numeric) $\uparrow$
$H=128$ (In-Context)	Baseline (AR GPT-2)	$100.0\pm 0.0\%$	$39.8\pm 4.44\%$
$H=128$ (In-Context)	LS-AR (Static)	$\mathbf{100.0\pm 0.0\%}$	$\mathbf{92.8\pm 1.48\%}$
$H=1024$ (Out-of-Context)	Baseline (AR GPT-2)	$0.0\pm 0.0\%$	$0.0\pm 0.0\%$
$H=1024$ (Out-of-Context)	LS-AR (Static)	$\mathbf{100.0\pm 0.0\%}$	$\mathbf{84.4\pm 4.51\%}$

Table 1: Semantic and numeric memory retrieval (

W=500

) across in-context (

H=128

) and out-of-context (

H=1024

) generation horizons. Values represent

\text{Mean}\pm\text{SEM}

across 5 random seeds.

Panel of Figure 4 — Figure 4: Performance and representational analysis across context boundary ( $W=500$ ). (a) Semantic target retrieval across expanding horizons ( $H$ ). The baseline collapses when $H>W$ , while LS-AR maintains $100\%$ retrieval regardless of sequence length. (b) t-SNE of decoder hidden states immediately preceding terminal retrieval at $H=1024$ ( $N=100$ ). Baseline states collapse into distractor noise, whereas LS-AR forms tightly separated clusters by target identity.

Context Size	Architecture	Throughput ( $\uparrow$ )	$\Delta$ Speed	Peak VRAM ( $\downarrow$ )	$\Delta$ Memory
$W=128$	Baseline-Small ∗ (38M)	$6030.46\pm 316.21$	–	$1056.00\pm 0.00$	–
	Baseline-Large (89M Anchor)	$2381.51\pm 56.59$	Anchor	$1538.74\pm 0.00$	Anchor
	LS-AR (Static 85M)	$\mathbf{3196.66\pm 164.55}$	+34.23%	$\mathbf{1039.28\pm 0.00}$	-32.46%
$W=256$	Baseline-Small ∗ (38M)	$5934.56\pm 306.78$	–	$1103.05\pm 0.00$	–
	Baseline-Large (89M Anchor)	$2190.98\pm 35.06$	Anchor	$1858.41\pm 0.00$	Anchor
	LS-AR (Static 85M)	$\mathbf{3085.42\pm 125.18}$	+40.82%	$\mathbf{1059.43\pm 0.00}$	-42.99%

Table 2: Decoding efficiency of LS-AR (Static) across context windows (

W

\Delta

values are anchored strictly against the parameter-matched Baseline-Large. Values represent

\text{Mean}\pm\text{SEM}

across 5 random seeds.

Architecture	In-Distribution ( $N=5$ )			OOD generalisation
Architecture	$K=0$ (Optimal)	$K=1$ (1 Mistake)	$K=2$ (Zero-Shot)	$N=6$ (Zero-Shot)
Baseline (AR GPT-2)	$96.0\%\pm 0.0$	$71.0\%\pm 0.0$	$6.0\%\pm 0.0$	$0.0\%\pm 0.0$
LS-AR (Static)	$95.0\%\pm 0.0$	$66.0\%\pm 0.0$	$5.0\%\pm 0.0$	$0.0\%\pm 0.0$
LS-AR (Dynamic)	$\mathbf{98.0\%\pm 0.0}$	$\mathbf{89.0\%\pm 0.0}$	$\mathbf{7.0\%\pm 0.0}$	$0.0\%\pm 0.0$
Random FiLM Ablation	0.0%	–	–	–

Table 3: Blocksworld planning, resilience, and OOD generalisation. Goal success rates across optimal trajectories (

K=0

), forced perturbation steps (

K\in\{1,2\}

), and zero-shot entity scaling (

N=6

). Values represent

\text{Mean}\pm\text{SEM}

across 5 random seeds.

Context Size	Architecture	Decoder Depth	Throughput ( $\uparrow$ )	$\Delta$ Speed	Peak VRAM ( $\downarrow$ )	$\Delta$ Memory
$W=32$	Baseline-Small (13M)	4 Layers	$277.5\pm 31.7$	–	$431.6\pm 0.0$	–
	(Decoder-Matched)
	Baseline-Large (32M)	10 Layers	$180.0\pm 2.7$	Anchor	$432.1\pm 0.0$	Anchor
	(Parameter-Matched)
	LS-AR (Dynamic 32M)	4 Layers (Dec)	$\mathbf{240.0\pm 23.0}$	+33.56%	$\mathbf{433.8\pm 0.0}$	0.41%
		+ 4 Layers ( $E_{\text{base}}$ )

Table 4: Inference efficiency of LS-AR (Dynamic) (

W=32

32\text{M}

parameters).

\Delta

metrics are anchored strictly against the parameter-matched Baseline-Large (

32\text{M}

). Sparsely executing

E_{\text{pred}}

yields

+33.56\%

higher throughput while preserving peak memory footprint. Values represent

\text{Mean}\pm\text{SEM}

across 5 random seeds.

Condition	Architecture	Standard Curriculum		Latent-Forced Curriculum
Textual Goal, Injected Latent Goal		Goal A Success	Goal B Success	Goal A Success	Goal B Success
Control ∗ ( $G_{A}$ , $G_{A}$ )	Baseline	$63.6\%\pm 0.94$	0.0%	–	–
Control ∗ ( $G_{A}$ , $G_{A}$ )	LS-AR (Dynamic)	$60.4\%\pm 1.07$	0.0%	$72.2\%\pm 0.85$	0.0%
Control ∗ ( $G_{B}$ , $G_{B}$ )	Baseline	0.0%	$62.2\%\pm 0.85$	–	–
Control ∗ ( $G_{B}$ , $G_{B}$ )	LS-AR (Dynamic)	0.0%	$61.4\%\pm 1.12$	0.0%	$73.2\%\pm 0.72$
Adversarial † ( $G_{A}$ , $G_{B}$ )	LS-AR (Dynamic)	$\mathbf{56.2\%\pm 1.39}$	$0.4\%\pm 0.36$	$1.4\%\pm 0.54$	$\mathbf{67.2\%\pm 1.65}$

Table 5: Latent controllability and modality authority . Under Standard training, LS-AR prioritises textual authority; under Latent-Forced training, steering vectors act as high-authority control overrides. Values represent

\text{Mean}\pm\text{SEM}

across 5 random seeds.

Autoregressive Language Modeling

Autoregressive language modeling predicts sequences one step at a time, using prior tokens to generate text, speech, code, and other data.

See all 128 papers

Thinking into the Future: Latent Lookahead Training for Transformers
Mar 3, 2026Lorenzo Noci, Gregor Bachmann, Seyed-Mohsen Moosavi-Dezfooli +1Autoregressive Language Modeling Sequence Prediction
Apple
AutoCRAT: Within-trajectory Joint Control of Stochasticity and Compute for LLM Reasoning
Aug 30, 2026Hanjun Luo, Qiushi Liu, Jingya Zhang +8Adaptive Inference LLM Inference Acceleration
New York University · New York University Abu Dhabi · University of Washington Seattle +4
Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning
Feb 24, 2026Justin Lovelace, Christian Belardi, Sofian Zalouk +3Autoregressive Language Modeling Diffusion Language Models
Department of Computer Science, Cornell University

Mar 3, 2026J/K move · Enter open · S save

Thinking into the Future: Latent Lookahead Training for Transformers

Lorenzo Noci, Gregor Bachmann, Seyed-Mohsen Moosavi-Dezfooli, Moin Nabi

Apple

Autoregressive Language Modeling Sequence Prediction Autoregressive Transformer LLM Planning Continuous Latent Reasoning

Open paper AskPDF arXiv

Autoregressive language models trained with next-token prediction generate text by sampling one discrete token at a time. Although very scalable, this objective forces the model to commit at every step, preventing it from exploring or reflecting upon multiple plausible continuations. Furthermore, the compute allocation across tokens is uniform; every token is formed based on a single forward-pass, potentially limiting the model's expressiveness in cases where difficult tokens require inherently more compute. Towards addressing these limitations, we introduce latent lookahead, a training strategy that enables models to "think" before generating: at selected positions in the sequence, before committing to the next token, the model performs a multi-step lookahead in latent space. More precisely, instead of sampling future tokens, we leverage the network's latent space by recursively feeding its hidden states back into the context for

τ

steps, investing more compute on predicting that token. This produces

τ

latent predictions that are supervised against the next

τ

ground-truth tokens, encouraging the model to "lookahead" and refine its prediction. We show that latent lookahead substantially outperforms both autoregressive and non-autoregressive baselines on planning tasks such as maze solving, Sudoku, and ProsQA, where foresight is essential.

Abstract

Autoregressive Language Modeling

Explore similar work

Thinking into the Future: Latent Lookahead Training for Transformers

AutoCRAT: Within-trajectory Joint Control of Stochasticity and Compute for LLM Reasoning

Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning