The viability of chain-of-thought (CoT) monitoring hinges on models being unable to reason effectively in their latent representations. Yet little is known about the limits of such latent reasoning in LLMs. We test these limits by studying whether models can discover multi-step planning strategies without supervision on intermediate steps and execute them latently, within a single forward pass. Using graph path-finding tasks that precisely control the number of required latent planning steps, we uncover a striking limitation unresolved by massive scaling: tiny transformers trained from scratch discover strategies requiring up to three latent steps, fine-tuned GPT-4o and Qwen3-32B reach five, and GPT-5.4 attains seven under few-shot prompting. Although the maximum latent planning depth models can learn during training is five, the discovered strategy generalizes up to eight latent steps at test-time. This reveals a dissociation between the ability to discover a latent strategy under final-answer supervision alone and the ability to execute it once discovered. If similar limits hold more broadly, strategies requiring multiple coordinated latent planning steps may need to be explicitly taught or externalized, lending credence to CoT monitoring.
Figures & tables
Figure 1: (Right) We study latent planning using path-finding on star graphs. Shown is an example of a star graph G(3,3) together with the corresponding input tokens and target label for training the transformer. (Left) Latent planning capacity across models and evaluation settings. Despite scaling from a 1.6M-parameter transformer to GPT-4o, the maximum LPC discovered during training increases by only two steps. However, LLMs can then generalize this strategy by a few more steps at test-time, up to 8 for the best models. As training is not publicly available for GPT-5.4, we report its few-shot LPC as a lower bound on the discovery ceiling, since fine-tuning strictly matches or exceeds few-shot performance for all other LLMs evaluated.
Figure 2
Depth
Branch Factor ( k )
2
3
4
5
10
m=3
0.50
0.51
0.78
0.73
0.10
m=4
0.49
0.34
0.25
0.20
0.10
Table 1: Average backtracking ratio (BR) of the trained transformer on the test set. Blue-highlighted bold text indicates configurations where the model successfully learned a latent planning strategy, while gray text indicates failure.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Train Size
Validation Size
Test Size
Transformer
100000
2048
2048
Qwen 2.5
xxx - 7B
10000
N/A
2048
xxx - 32B
2000
N/A
2048
Qwen 3
xxx - 8B
10000
N/A
2048
Appendix
Table 2: Dataset statistics across models.
Hyper-parameter
From-Scratch Transformer
Open-Source LLMs
Closed-Source
Latent / CoT
ICoT
7B / 8B
32B
GPT-4o
Epochs
500
1000
1
1
10
Learning Rate
1×10−4
1×10−4
3×10−6
3×10−6
Auto
Batch Size
1024
1024
4
1
1
Dropout
0.1
0.1
N/A
N/A
-
GPUs
1
1
8
8
-
Appendix
Table 3: Hyper-parameters for training models on star graph tasks. Note that the reported batch size represents the per-GPU batch size.
Figure 4: Training dynamics of the from-scratch transformer across planning depths m∈{3,4} and branch factors k∈K . The model exhibits a two-stage learning process: validation accuracy first rises to the random baseline ( k1 ) as the model learns to predict valid neighbors, then either jumps to near-perfect accuracy (successful strategy discovery) or stagnates while training loss continues to decrease (overfitting without strategy discovery).
Model
m=7
m=8
2
3
4
5
10
2
3
4
5
10
GPT-5.4
xxx - Zero Shot
0.22
0.07
0.17
-0.03
0.02
-0.08
0.07
-0.03
-0.03
-0.01
xxx - Few Shot
0.50
0.25
0.27
0.23
0.09
0.00
0.12
0.17
0.01
0.06
Appendix
Table 7: To further probe the latent planning capability (LPC) of GPT-5.4, we additionally evaluate it on more challenging star graph tasks with planning depths m∈{7,8} and branch factors k∈{2,3,4,5,10} . We report skill scores for zero-shot and few-shot settings. Blue-highlighted bold text indicates configurations where the model successfully rejects random guessing, while gray text indicates failure.
Figure 5: Training dynamics of Qwen 2.5 models (7B and 32B) across graph configurations. Both models exhibit the same two-stage learning process observed in the from-scratch transformer, with failures occurring exclusively in the second stage.
Figure 6: Training dynamics of Qwen 3 models (8B and 32B) across graph configurations. The same two-stage pattern persists, confirming that the discovery bottleneck is consistent across model families and scales.
Model
Average Skill over K at Depth m
3
4
5
6
20
Transformer
1.00
0.78
1.00
0.75
0.78
Qwen 2.5
xxx - 7B
1.00
1.00
1.00
1.00
1.00
xxx - 32B
1.00
0.99
1.00
0.99
0.99
Qwen 3
Appendix
Table 8: Average skill on star graph tasks across planning depths m∈{3,4,5,6,20} .
Figure 7: Training loss under explicit chain-of-thought supervision. All models are trained with dense supervision that provides the full backtracking trace from vtarget to vsource . Under this setting, all LLMs converge within approximately 20 training updates across all configurations, confirming that the task itself is not inherently difficult.
m
k∈K
2
3
4
5
10
3
1.00
1.00
1.00
1.00
-0.01
4
0.99
0.01
0.00
-0.01
-0.11
5
1.00
0.02
-0.33
0.00
-0.11
6
1.00
-0.01
-0.33
-0.25
-0.11
7
-1.00
-0.50
-0.33
-0.25
-0.11
Appendix
Table 9: Skill of ICoT on star graph tasks across branching factors k∈K={2,3,4,5,10} and planning depths m∈{3,4,5,6,7} . Blue-highlighted bold text indicates configurations where the model successfully rejects random guessing, while gray text indicates failure.
Figure 8: Attention visualization for successful configurations ( m=3 , k∈{2,3,4,5} ). For each configuration, the graph structure is shown on the left, where the orange node denotes the target and the violet node denotes the prediction, and the attention weights from the final token to all input tokens at each transformer layer are shown on the right. As k increases from 2 to 5, the attention progressively concentrates on nodes along the path from vtarget to vsource , with shallower layers attending to the target and deeper layers tracing back toward the source.
Figure 9: Attention visualization for failed configurations ( m=3 with k=10 , and m=4 with k∈{2,3,4,5,10} ), following the same convention as Figure 8 : orange denotes the target node and violet denotes the predicted node. No structured attention pattern emerges across any of these configurations, confirming that the model has not discovered a strategy.
Implicit reasoning in language models studies how models solve problems through internal computation rather than, or alongside, written chains of thought.
Chain-of-Thought (CoT) empowers Large Language Models (LLMs) to tackle complex problems, but remains constrained by the computational cost and early token commitments in discrete reasoning traces. Recent latent reasoning approaches attempt to optimize efficiency by performing reasoning within continuous hidden states. However, many such methods optimize latent states end to end without a trained interface for intermediate textual readout, and several representative configurations use a pre-defined number of latent steps during inference. In this work, we introduce PLaT (Planning with Latent Thoughts), a framework that decouples latent planning from verbalization. The Planner deterministically evolves latent planning states, while an independent Decoder provides textual readouts when needed. Answer-aware textual stopping allows the latent rollout to use a problem-dependent number of groups rather than a pre-specified chain length. PLaT achieves competitive coverage at larger k in several mathematical settings, with lower Pass@1: on Llama-1B GSM8K, it reaches 80.59% Pass@128 versus CODI's 72.37%. These results support PLaT as a candidate-generation interface supplying multiple textual readouts for downstream verification or reranking.
Large language models often answer complex reasoning questions without revealing intermediate steps, raising whether they reason latently or complete patterns. We propose the Hidden CoT Detection Score (HCDS), a comparative behavioral and mechanistic signal measuring whether neutral-prompt behavior aligns more closely with explicit CoT or explicit no- CoT. Here, hidden CoT operationally denotes this neutral-prompt CoT-like alignment; HCDS does not directly observe or prove an unexposed reasoning trace. On GSM8K, HCDS is significantly positive for both Qwen3-4B variants (Thinking +1.87, p=1.2×10−7; Instruct +1.41, p=1.9×10−4), replicates across a different inference stack and quantization within 0.08 (+1.80 and +1.45), and is not significantly positive in seven of eight length-adjusted calibration-control cells. The unadjusted score produces large positive scores on single-step arithmetic and numeric factual lookup. The variants also respond differently to no-CoT instructions: Instruct complies from the prompt alone, whereas Thinking continues reasoning and requires intervention. These findings show stronger, less prompt-conditional CoT-like behavior in the reasoning-tuned model, consistent with but not proof of latent reasoning. HCDS thus investigates latent reasoning without relying on models' self-reported traces.
Armaan Singh, Ryan Trinh Le, Jasmine Kaur +5
1Stanford University · 2York University · 3Facebook
Language models typically reason via explicit chain-of-thought (CoT), generating intermediate steps token-by-token. Latent CoT offers an alternative: it performs multi-step reasoning in the model's hidden states, replacing decoded tokens with continuous representations for greater efficiency. However, existing latent CoT methods underperform explicit CoT beyond 1B parameters, and the gap widens with scale. Looped, or recurrent-depth, Transformers, which reuse their weights to increase computation depth without adding parameters, are a natural fit for latent reasoning. We therefore ask whether looped Transformers can bridge this gap. We answer affirmatively with a simple recipe: a looped padded Transformer that processes K latent blocks in parallel for R iterations, with a cross-entropy loss on each latent position's gold CoT-step token, similar to explicit CoT supervision. We instantiate it as LOTUS (Looped Transformers with parallel supervision on latents). LOTUS is, to our knowledge, the first latent-CoT method to bridge the gap to explicit CoT at the 3B scale, while cutting thought-phase latency by 2.5x-6.9x from compact math expressions to natural language. Projecting LOTUS's post-loop latents through the base LM head recovers the gold reasoning steps and even surfaces alternative valid intermediate steps, evidence that its latent space is interpretable and CoT-aligned. Ablations confirm that both the looped backbone and the parallel supervision on gold CoT tokens are essential. Code is available at https://github.com/yingfan-bot/lotus.