The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook

Xinlei Yu; Zhangquan Chen; Yongbo He; Tianyu Fu; Guanting Dong; Cheng Yang; Chengming Xu; Yue Ma

cs.AIApr 2, 2026

The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook

Authors: Xinlei Yu, Zhangquan Chen, Yongbo He, Tianyu Fu, Guanting Dong, Cheng Yang, Chengming Xu, Yue Ma,

Organizations: National University of Singapore · Fudan University · Tsinghua University · Zhejiang University · Shanghai Artificial Intelligence Laboratory · Renmin University of China · The Chinese University of Hong Kong · The Hong Kong University of Science and Technology · DeepWisdom · Nanjing University · Shanghai Jiatong University · Nanyang Technological University · Tencent Hunyuan · QuantaAlpha · Beijing University of Posts and Telecommunications · Zhejiang Lab · University of Chinese Academy of Sciences · Hong Kong University of Science and Technology (Guangzhou) · Sun Yat-sen University

Abstract

Latent space is rapidly emerging as a native substrate for language-based models. While modern systems are still commonly understood through explicit token-level generation, an increasing body of work shows that many critical internal processes are more naturally carried out in continuous latent space than in human-readable verbal traces. This shift is driven by the structural limitations of explicit-space computation, including linguistic redundancy, discretization bottlenecks, sequential inefficiency, and semantic loss. This survey aims to provide a unified and up-to-date landscape of latent space in language-based models. We organize the survey into five sequential perspectives: Foundation, Evolution, Mechanism, Ability, and Outlook. We begin by delineating the scope of latent space, distinguishing it from explicit or verbal space and from the latent spaces commonly studied in generative visual models. We then trace the field's evolution from early exploratory efforts to the current large-scale expansion. To organize the technical landscape, we examine existing work through the complementary lenses of mechanism and ability. From the perspective of Mechanism, we identify four major lines of development: Architecture, Representation, Computation, and Optimization. From the perspective of Ability, we show how latent space supports a broad capability spectrum spanning Reasoning, Planning, Modeling, Perception, Memory, Collaboration, and Embodiment. Beyond consolidation, we discuss the key open challenges, and outline promising directions for future research. We hope this survey serves not only as a reference for existing work, but also as a foundation for understanding latent space as a general computational and systems paradigm for next-generation intelligence.

Figures & tables

Figure 1: Overview of the latent space methods classified by two axes: four main Mechanisms (Section 4 ) and seven key Abilities (Section 5 ). Within our classification system, a single method may be affiliated with one or more mechanisms and capabilities. For the visualization in this figure, we adopt the most appropriate classification for each method; a comprehensive elaboration of these categories will be presented in the main text.

Figure 2: Outline of the survey, including five sections and sequential questions: Foundation: What is Latent Space? (Section 2 ), Evolution: How Did Latent Space Develop? (Section 3 ), Mechanism: How Does Latent Space Work? (Section 4 ), Ability: What Does Latent Space Enable? (Section 5 ), and Outlook: What is Next? (Section 6 )

Figure 3: Comparison of the explicit space and latent space of the language models, including their representational properties and functional capabilities.

Figure 4: Timeline of representative works in the evolution of latent space research, organized into four developmental stages: Prototype (Section 3.1 ), Formation (Section 3.2 ), Expansion (Section 3.3 ), and Outbreak (Section 3.4 ) stages, where the horizontal axis denotes the month, and vertical axis indicates the number of the latent-level works.

Symbol	Space	Description
$\mathcal{V}$	token space	discrete vocabulary/token IDs
$\mathcal{H}$	latent space	continuous hidden space $\mathbb{R}^{d}$
$\mathcal{M}$	model space	set of models/modules/components
$\mathcal{W}$	parameter space	parameters within the model
$\mathcal{D}$	data space	the whole training and test data
$\mathbf{x}$	$\mathcal{V}$	input

Table 1: General notations used throughout the paper.

Figure 5: Representative works operate in accordance with latent space mechanisms. We classify all methods into four lines based on diverse ways of utilizing the latent space, including: Architecture (Section 4.1 ), Representation (Section 4.2 ), Computation (Section 4.3 ), and Optimization (Section 4.4 ).

Date	Backbone	Paper	Code	Dimension	Layer	Size	Feature
01/25	Heima [ 173 ]	Paper	Code	4096	72	19B	encoder-decoder/progressive/adaptive decoding
02/25	Huginn [ 50 ]	Paper	Code	5280	8	3.5B	decoder-only/recurrent depth/shared recurrent block/test-time
02/25	Looped Trans. [ 167 ]	Paper	-	5120	24	1.5B	decoder-only/looped model/looping-based regularization
03/25	MoLAE [ 127 ]	Paper	-	512	12	0.1B	mixture of latent experts/shared projection/lower dimension
04/25	PHD-Trans. [ 223 ]	Paper	-	2048	16	1.2B	decoder-only/cache management/sliding window attention
09/25	PonderLM2 [ 267 ]	Paper	Code	2048	24	0.5B/1.4B	decoder-only/iterative refinement/Jacobi-style parallel updates

Table 2: Overview of the Backbone (Section 4.1.1 ) based architecture. We compare the hidden dimension, layer, size, and architectural feature of these backbones.

Date	Method	Paper	Code	Modality	Backbone	Module/Model	Function	Scenario
Component
12/24	AURORA [ 11 ]	Paper	Code	vision	LLaVA1.5-13B	VQ-VAE	vision generation	visual perception
01/25	LF-Steering [ 250 ]	Paper	-	text	LLaMA2-7B	SAE	semantic projection	semantic consistency
02/25	AlignVLM [ 137 ]	Paper	-	vision	LLaMA3.2-1B/3B	mixed layers	vision alignment	document understanding
02/25	CoCoMix [ 186 ]	Paper	Code	text	GPT2-0.1B/0.4B/1.4B	SAE	concept extraction	interpretability/efficiency
02/25	IMM [ 148 ]	Paper	-	text	GPT2-0.1B	vector library	memory bank	long-horizon/safety

Table 3: Overview of the Component (Section 4.1.2 ) and Auxiliary Model (Section 4.1.3 ) based architecture, respectively. We compare the modality, backbone, the type of the component module or external model, function, and scenario. Here, (VQ)-AE, SAE, MLP, Q-Former, LoRA, and JEPA denote (vector quantized) variational autoencoder, multilayer perceptron, querying transformer, low-rank adaptation, and joint embedding predictive architecture, respectively.

Figure 6: The schematic diagram of Representation mechanism, including four sub-types: Internal (Section 4.2.1 ), External (Section 4.2.2 ), Learnable (Section 4.2.3 ), and Hybrid (Section 4.2.4 ).

Date	Method	Paper	Code	Modality	Backbone	Subject	Scenario
Internal
10/24	CoE [ 217 ]	Paper	Code	text	LLaMA2-7B/other 6	embedding/hidden state	label-free self-evaluation
10/24	VTI [ 119 ]	Paper	Code	vision	LLaVA1.5-7B/other 2	hidden state	hallucination mitigation
12/24	COCONUT [ 58 ]	Paper	Code	text	GPT2-0.1B	the last hidden state	latent reasoning/efficiency
05/25	Soft Thinking [ 287 ]	Paper	-	text	LLaMA3.1-8B/other 3	weighted embedding	latent reasoning/efficiency
05/25	CoT2 [ 52 ]	Paper	Code	text	GPT2-0.1B	weighted embedding	exploration/complex reasoning

Table 4: Overview of the Internal (Section 4.2.1 ), External (Section 4.2.2 ), Learnable (Section 4.2.3 ), and Hybrid (Section 4.2.4 ) representation. We compare the modality, backbone, representation subject, and scenario.

Figure 7: The schematic diagram of Computation mechanism, including four sub-types: Compressed (Section 4.3.1 ), Expanded (Section 4.3.2 ), Adaptive (Section 4.3.3 ), and Interleaved (Section 4.3.4 ).

Date	Method	Paper	Code	Modality	Backbone	Operation	Scenario
Compressed
09/24	HCoT [ 122 ]	Paper	-	text	LLaMA2-7B/13B	semantic alignment	efficiency/generalization
12/24	CCoT [ 31 ]	Paper	-	text	LLaMA2-7B	variable sequence	latent reasoning/efficiency
02/25	SoftCoT [ 243 ]	Paper	Code	text	LLaMA3.1-8B	semantic projection	efficiency/generalization/zero-shot
02/25	CODI [ 174 ]	Paper	Code	text	GPT2-0.1B/LLaMA3.2-1B	self distillation	latent reasoning/efficiency/generalization
05/25	CoLaR [ 188 ]	Paper	Code	text	LLaMA3.2-1B	dynamic compaction	dynamic reasoning/efficiency

Table 5: Overview of the Compressed (Section 4.3.1 ), Expanded (Section 4.3.2 ), Adaptive (Section 4.3.3 ), and Interleaved (Section 4.3.4 ) computation. We compare the modality, backbone, computational operation, and scenario.

Date	Method	Paper	Code	Modality	Backbone	Supervision	Objective	Scenario
Pre-training
10/24	LAPA [ 255 ]	Paper	-	action	LWM-7B	Recon.	action quantization/prediction	embodied manipulation/transferring
02/25	CoCoMix [ 186 ]	Paper	Code	text	GPT2-0.1B/0.4B/1.4B	CE/Recon.	multi-task prediction	interpretability/efficiency
02/25	Looped Trans. [ 167 ]	Paper	-	text	decoder-only-1.5B	CE/regularization	next-token prediction	latent reasoning/complex reasoning
04/25	PHD-Trans. [ 223 ]	Paper	-	text	decoder-only-1.2B	CE	next-token prediction	scaling/efficiency/exploration
05/25	LARES [ 112 ]	Paper	-	text	transformer-based	InfoNCE/CE/reward	alignment/next-token prediction	sequential recommendation

Table 6: Overview of the Pre-training (Section 4.4.1 ), Post-training (Section 4.4.2 ), and Inference (Section 4.4.3 ) optimization. We compare the modality, backbone, computational operation, and scenario. Here, Recon., CE, InfoNCE, KL, MSE, and CL denote reconstruction, cross-entropy, information noise-contrastive estimation, Kullback–Leibler divergence, mean squared error, and contrastive learning, respectively.

Figure 8: Core abilities brought by the latent space, including: Reasoning (Section 5.1 ), Planning (Section 5.2 ), Modeling (Section 5.3 ), Perception (Section 5.4 ), Memory (Section 5.5 ), Collaboration (Section 5.6 ), and Embodiment (Section 5.7 ).

Continuous Latent Reasoning

Continuous latent reasoning studies how models can compute through continuous hidden states instead of spelling out every reasoning step.

See all 80 papers

From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMs
Mar 27, 2026Jiyuan An, Liner Yang, Mengyan Wang +3Spatial Reasoning in Language Models Spatial Reasoning
National Language Resources Monitoring and Research Center for Print Media, Beijing Language and Culture University, Beijing, China · School of Information Science, Beijing Language and Culture University, Beijing, China · School of artificial intelligence, Beijing normal university, Beijing, China
Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning
May 22, 2025Xinghao Chen, Anhao Zhao, Heming Xia +7CoT Reasoning LLM Reasoning
Department of Computing, The Hong Kong Polytechnic University · Ningbo Digital Twin Institute, Eastern Institute of Technology, Ningbo, China
The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Models
Apr 7, 2026Michael Rizvi-Martel, Guillaume Rabusseau, Marius MosbachContinuous Latent Reasoning LLM Interpretability
Mila – Quebec AI Institute · Universit´e de Montr´eal · McGill University

Mar 27, 2026J/K move · Enter open · S save

From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMs

Jiyuan An, Liner Yang, Mengyan Wang, Luming Lu+2

National Language Resources Monitoring and Research Center for Print Media, Beijing Language and Culture University, Beijing, China · School of Information Science, Beijing Language and Culture University, Beijing, China · School of artificial intelligence, Beijing normal university, Beijing, China

Spatial Reasoning in Language Models Spatial Reasoning LLM Interpretability

Open paper AskPDF arXiv

As spatial intelligence becomes an increasingly important capability for foundation models, it remains unclear whether large language models' (LLMs) performance on spatial reasoning benchmarks reflects structured internal spatial representations or reliance on linguistic heuristics. We address this question from a mechanistic perspective by examining how spatial information is internally represented and used. Drawing on computational theories of human spatial cognition, we decompose spatial reasoning into three primitives, relational composition, representational transformation, and stateful spatial updating, and design controlled task families for each. We evaluate multilingual LLMs in English, Chinese, and Arabic under single pass inference, and analyze internal representations using linear probing, sparse autoencoder based feature analysis, and causal interventions. We find that task relevant spatial information is encoded in intermediate layers and can causally influence behavior, but these representations are transient, fragmented across task families, and weakly integrated into final predictions. Cross linguistic analysis further reveals mechanistic degeneracy, where similar behavioral performance arises from distinct internal pathways. Overall, our results suggest that current LLMs exhibit limited and context dependent spatial representations rather than robust, general purpose spatial reasoning, highlighting the need for mechanistic evaluation beyond benchmark accuracy.

Abstract

Continuous Latent Reasoning

Explore similar work

From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMs

Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning

The Illusion of Superposition? A Principled Analysis of Latent Thinking in Language Models