Organizations: National University of Singapore · Fudan University · Tsinghua University · Zhejiang University · Shanghai Artificial Intelligence Laboratory · Renmin University of China · The Chinese University of Hong Kong · The Hong Kong University of Science and Technology · DeepWisdom · Nanjing University · Shanghai Jiatong University · Nanyang Technological University · Tencent Hunyuan · QuantaAlpha · Beijing University of Posts and Telecommunications · Zhejiang Lab · University of Chinese Academy of Sciences · Hong Kong University of Science and Technology (Guangzhou) · Sun Yat-sen University
Latent space is rapidly emerging as a native substrate for language-based models. While modern systems are still commonly understood through explicit token-level generation, an increasing body of work shows that many critical internal processes are more naturally carried out in continuous latent space than in human-readable verbal traces. This shift is driven by the structural limitations of explicit-space computation, including linguistic redundancy, discretization bottlenecks, sequential inefficiency, and semantic loss. This survey aims to provide a unified and up-to-date landscape of latent space in language-based models. We organize the survey into five sequential perspectives: Foundation, Evolution, Mechanism, Ability, and Outlook. We begin by delineating the scope of latent space, distinguishing it from explicit or verbal space and from the latent spaces commonly studied in generative visual models. We then trace the field's evolution from early exploratory efforts to the current large-scale expansion. To organize the technical landscape, we examine existing work through the complementary lenses of mechanism and ability. From the perspective of Mechanism, we identify four major lines of development: Architecture, Representation, Computation, and Optimization. From the perspective of Ability, we show how latent space supports a broad capability spectrum spanning Reasoning, Planning, Modeling, Perception, Memory, Collaboration, and Embodiment. Beyond consolidation, we discuss the key open challenges, and outline promising directions for future research. We hope this survey serves not only as a reference for existing work, but also as a foundation for understanding latent space as a general computational and systems paradigm for next-generation intelligence.
Figures & tables
Figure 1: Overview of the latent space methods classified by two axes: four main Mechanisms (Section 4 ) and seven key Abilities (Section 5 ). Within our classification system, a single method may be affiliated with one or more mechanisms and capabilities. For the visualization in this figure, we adopt the most appropriate classification for each method; a comprehensive elaboration of these categories will be presented in the main text.
Figure 2: Outline of the survey, including five sections and sequential questions: Foundation: What is Latent Space? (Section 2 ), Evolution: How Did Latent Space Develop? (Section 3 ), Mechanism: How Does Latent Space Work? (Section 4 ), Ability: What Does Latent Space Enable? (Section 5 ), and Outlook: What is Next? (Section 6 )
Figure 3: Comparison of the explicit space and latent space of the language models, including their representational properties and functional capabilities.
Figure 4: Timeline of representative works in the evolution of latent space research, organized into four developmental stages: Prototype (Section 3.1 ), Formation (Section 3.2 ), Expansion (Section 3.3 ), and Outbreak (Section 3.4 ) stages, where the horizontal axis denotes the month, and vertical axis indicates the number of the latent-level works.
Symbol
Space
Description
V
token space
discrete vocabulary/token IDs
H
latent space
continuous hidden space Rd
M
model space
set of models/modules/components
W
parameter space
parameters within the model
D
data space
the whole training and test data
x
V
input
Table 1: General notations used throughout the paper.
Figure 5: Representative works operate in accordance with latent space mechanisms. We classify all methods into four lines based on diverse ways of utilizing the latent space, including: Architecture (Section 4.1 ), Representation (Section 4.2 ), Computation (Section 4.3 ), and Optimization (Section 4.4 ).
Table 2: Overview of the Backbone (Section 4.1.1 ) based architecture. We compare the hidden dimension, layer, size, and architectural feature of these backbones.
Date
Method
Paper
Code
Modality
Backbone
Module/Model
Function
Scenario
Component
12/24
AURORA [ 11 ]
Paper
Code
vision
LLaVA1.5-13B
VQ-VAE
vision generation
visual perception
01/25
LF-Steering [ 250 ]
Paper
-
text
LLaMA2-7B
SAE
semantic projection
semantic consistency
02/25
AlignVLM [ 137 ]
Paper
-
vision
LLaMA3.2-1B/3B
mixed layers
vision alignment
document understanding
02/25
CoCoMix [ 186 ]
Paper
Code
text
GPT2-0.1B/0.4B/1.4B
SAE
concept extraction
interpretability/efficiency
02/25
IMM [ 148 ]
Paper
-
text
GPT2-0.1B
vector library
memory bank
long-horizon/safety
Table 3: Overview of the Component (Section 4.1.2 ) and Auxiliary Model (Section 4.1.3 ) based architecture, respectively. We compare the modality, backbone, the type of the component module or external model, function, and scenario. Here, (VQ)-AE, SAE, MLP, Q-Former, LoRA, and JEPA denote (vector quantized) variational autoencoder, multilayer perceptron, querying transformer, low-rank adaptation, and joint embedding predictive architecture, respectively.
Figure 6: The schematic diagram of Representation mechanism, including four sub-types: Internal (Section 4.2.1 ), External (Section 4.2.2 ), Learnable (Section 4.2.3 ), and Hybrid (Section 4.2.4 ).
Date
Method
Paper
Code
Modality
Backbone
Subject
Scenario
Internal
10/24
CoE [ 217 ]
Paper
Code
text
LLaMA2-7B/other 6
embedding/hidden state
label-free self-evaluation
10/24
VTI [ 119 ]
Paper
Code
vision
LLaVA1.5-7B/other 2
hidden state
hallucination mitigation
12/24
COCONUT [ 58 ]
Paper
Code
text
GPT2-0.1B
the last hidden state
latent reasoning/efficiency
05/25
Soft Thinking [ 287 ]
Paper
-
text
LLaMA3.1-8B/other 3
weighted embedding
latent reasoning/efficiency
05/25
CoT2 [ 52 ]
Paper
Code
text
GPT2-0.1B
weighted embedding
exploration/complex reasoning
Table 4: Overview of the Internal (Section 4.2.1 ), External (Section 4.2.2 ), Learnable (Section 4.2.3 ), and Hybrid (Section 4.2.4 ) representation. We compare the modality, backbone, representation subject, and scenario.
Figure 7: The schematic diagram of Computation mechanism, including four sub-types: Compressed (Section 4.3.1 ), Expanded (Section 4.3.2 ), Adaptive (Section 4.3.3 ), and Interleaved (Section 4.3.4 ).
Date
Method
Paper
Code
Modality
Backbone
Operation
Scenario
Compressed
09/24
HCoT [ 122 ]
Paper
-
text
LLaMA2-7B/13B
semantic alignment
efficiency/generalization
12/24
CCoT [ 31 ]
Paper
-
text
LLaMA2-7B
variable sequence
latent reasoning/efficiency
02/25
SoftCoT [ 243 ]
Paper
Code
text
LLaMA3.1-8B
semantic projection
efficiency/generalization/zero-shot
02/25
CODI [ 174 ]
Paper
Code
text
GPT2-0.1B/LLaMA3.2-1B
self distillation
latent reasoning/efficiency/generalization
05/25
CoLaR [ 188 ]
Paper
Code
text
LLaMA3.2-1B
dynamic compaction
dynamic reasoning/efficiency
Table 5: Overview of the Compressed (Section 4.3.1 ), Expanded (Section 4.3.2 ), Adaptive (Section 4.3.3 ), and Interleaved (Section 4.3.4 ) computation. We compare the modality, backbone, computational operation, and scenario.
Date
Method
Paper
Code
Modality
Backbone
Supervision
Objective
Scenario
Pre-training
10/24
LAPA [ 255 ]
Paper
-
action
LWM-7B
Recon.
action quantization/prediction
embodied manipulation/transferring
02/25
CoCoMix [ 186 ]
Paper
Code
text
GPT2-0.1B/0.4B/1.4B
CE/Recon.
multi-task prediction
interpretability/efficiency
02/25
Looped Trans. [ 167 ]
Paper
-
text
decoder-only-1.5B
CE/regularization
next-token prediction
latent reasoning/complex reasoning
04/25
PHD-Trans. [ 223 ]
Paper
-
text
decoder-only-1.2B
CE
next-token prediction
scaling/efficiency/exploration
05/25
LARES [ 112 ]
Paper
-
text
transformer-based
InfoNCE/CE/reward
alignment/next-token prediction
sequential recommendation
Table 6: Overview of the Pre-training (Section 4.4.1 ), Post-training (Section 4.4.2 ), and Inference (Section 4.4.3 ) optimization. We compare the modality, backbone, computational operation, and scenario. Here, Recon., CE, InfoNCE, KL, MSE, and CL denote reconstruction, cross-entropy, information noise-contrastive estimation, Kullback–Leibler divergence, mean squared error, and contrastive learning, respectively.
As spatial intelligence becomes an increasingly important capability for foundation models, it remains unclear whether large language models' (LLMs) performance on spatial reasoning benchmarks reflects structured internal spatial representations or reliance on linguistic heuristics. We address this question from a mechanistic perspective by examining how spatial information is internally represented and used. Drawing on computational theories of human spatial cognition, we decompose spatial reasoning into three primitives, relational composition, representational transformation, and stateful spatial updating, and design controlled task families for each. We evaluate multilingual LLMs in English, Chinese, and Arabic under single pass inference, and analyze internal representations using linear probing, sparse autoencoder based feature analysis, and causal interventions. We find that task relevant spatial information is encoded in intermediate layers and can causally influence behavior, but these representations are transient, fragmented across task families, and weakly integrated into final predictions. Cross linguistic analysis further reveals mechanistic degeneracy, where similar behavioral performance arises from distinct internal pathways. Overall, our results suggest that current LLMs exhibit limited and context dependent spatial representations rather than robust, general purpose spatial reasoning, highlighting the need for mechanistic evaluation beyond benchmark accuracy.
Jiyuan An, Liner Yang, Mengyan Wang +3
National Language Resources Monitoring and Research Center for Print Media, Beijing Language and Culture University, Beijing, China · School of Information Science, Beijing Language and Culture University, Beijing, China · School of artificial intelligence, Beijing normal university, Beijing, China
Large Language Models (LLMs) have shown impressive performance on complex tasks through Chain-of-Thought (CoT) reasoning. However, conventional CoT relies on explicitly verbalized intermediate steps, which constrains its broader applicability, particularly in abstract reasoning tasks beyond language. To address this, there has been growing research interest in latent CoT reasoning, where the reasoning process is embedded within latent spaces. By decoupling reasoning from explicit language generation, latent CoT offers the promise of richer cognitive representations and facilitates more flexible, faster inference. This paper aims to present a comprehensive overview of this emerging paradigm and establish a systematic taxonomy. We analyze recent advances in methods, categorizing them from token-wise horizontal approaches to layer-wise vertical strategies. We then provide in-depth discussions of these methods, highlighting their design principles, applications, and remaining challenges. We hope that our survey provides a structured foundation for advancing this promising direction in LLM reasoning. The relevant papers will be regularly updated at https://github.com/EIT-NLP/Awesome-Latent-CoT.
Xinghao Chen, Anhao Zhao, Heming Xia +7
Department of Computing, The Hong Kong Polytechnic University · Ningbo Digital Twin Institute, Eastern Institute of Technology, Ningbo, China
Latent reasoning via continuous chain-of-thoughts (Latent CoT) has emerged as a promising alternative to discrete CoT reasoning. Operating in continuous space increases expressivity and has been hypothesized to enable superposition: the ability to maintain multiple candidate solutions simultaneously within a single representation. Despite theoretical arguments, it remains unclear whether language models actually leverage superposition when reasoning using latent CoTs. We investigate this question across three regimes: a training-free regime that constructs latent thoughts as convex combinations of token embeddings, a fine-tuned regime where a base model is adapted to produce latent thoughts, and a from-scratch regime where a model is trained entirely with latent thoughts to solve a given task. Using Logit Lens and entity-level probing to analyze internal representations, we find that only models trained from scratch exhibit signs of using superposition. In the training-free and fine-tuned regimes, we find that the superposition either collapses or is not used at all, with models discovering shortcut solutions instead. We argue that this is due to two complementary phenomena: i) pretraining on natural language data biases models to commit to a token in the last layers ii) capacity has a huge effect on which solutions a model favors. Together, our results offer a unified explanation for when and why superposition arises in continuous chain-of-thought reasoning, and identify the conditions under which it collapses.
Michael Rizvi-Martel, Guillaume Rabusseau, Marius Mosbach
Mila – Quebec AI Institute · Universit´e de Montr´eal · McGill University