Latent CoT Reasoning

CoT: Chain-of-Thought

Overview

Latent chain-of-thought reasoning asks whether language models can perform multi-step computation in hidden states or compact learned tokens, potentially reducing the cost of generating long verbal explanations. Approaches include recurrent hidden-state reasoning, discrete or continuous latent tokens, and methods that compress or selectively retain explicit steps; training uses distillation, process supervision, reinforcement learning, and alignment with written reasoning traces. Work spans math and symbolic benchmarks as well as multimodal, structured-data, and tool-using tasks, with methods increasingly combining latent exploration with explicit verification. A central open question is whether latent states genuinely guide answers and remain monitorable, rather than merely correlating with success, while preserving the accuracy and flexibility of explicit chain-of-thought.

Latest papers

31 papers1 paper in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

May 22, 2025

Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoning

Large Language Models (LLMs) have shown impressive performance on complex tasks through Chain-of-Thought (CoT) reasoning. However, conventional CoT relies on explicitly verbalized intermediate steps, which constrains its broader applicability, particularly in abstract reasoning tasks beyond language. To address this, there has been growing research interest in latent CoT reasoning, where the reasoning process is embedded within latent spaces. By decoupling reasoning from explicit language generation, latent CoT offers the promise of richer cognitive representations and facilitates more flexible, faster inference. This paper aims to present a comprehensive overview of this emerging paradigm and establish a systematic taxonomy. We analyze recent advances in methods, categorizing them from token-wise horizontal approaches to layer-wise vertical strategies. We then provide in-depth discussions of these methods, highlighting their design principles, applications, and remaining challenges. We hope that our survey provides a structured foundation for advancing this promising direction in LLM reasoning. The relevant papers will be regularly updated at https://github.com/EIT-NLP/Awesome-Latent-CoT.