The viability of chain-of-thought (CoT) monitoring hinges on models being unable to reason effectively in their latent representations. Yet little is known about the limits of such latent reasoning in LLMs. We test these limits by studying whether models can discover multi-step planning strategies without supervision on intermediate steps and execute them latently, within a single forward pass. Using graph path-finding tasks that precisely control the number of required latent planning steps, we uncover a striking limitation unresolved by massive scaling: tiny transformers trained from scratch discover strategies requiring up to three latent steps, fine-tuned GPT-4o and Qwen3-32B reach five, and GPT-5.4 attains seven under few-shot prompting. Although the maximum latent planning depth models can learn during training is five, the discovered strategy generalizes up to eight latent steps at test-time. This reveals a dissociation between the ability to discover a latent strategy under final-answer supervision alone and the ability to execute it once discovered. If similar limits hold more broadly, strategies requiring multiple coordinated latent planning steps may need to be explicitly taught or externalized, lending credence to CoT monitoring.
Figures & tables
Figure 1: (Right) We study latent planning using path-finding on star graphs. Shown is an example of a star graph G(3,3) together with the corresponding input tokens and target label for training the transformer. (Left) Latent planning capacity across models and evaluation settings. Despite scaling from a 1.6M-parameter transformer to GPT-4o, the maximum LPC discovered during training increases by only two steps. However, LLMs can then generalize this strategy by a few more steps at test-time, up to 8 for the best models. As training is not publicly available for GPT-5.4, we report its few-shot LPC as a lower bound on the discovery ceiling, since fine-tuning strictly matches or exceeds few-shot performance for all other LLMs evaluated.
Figure 2
Depth
Branch Factor ( k )
2
3
4
5
10
m=3
0.50
0.51
0.78
0.73
0.10
m=4
0.49
0.34
0.25
0.20
0.10
Table 1: Average backtracking ratio (BR) of the trained transformer on the test set. Blue-highlighted bold text indicates configurations where the model successfully learned a latent planning strategy, while gray text indicates failure.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Train Size
Validation Size
Test Size
Transformer
100000
2048
2048
Qwen 2.5
xxx - 7B
10000
N/A
2048
xxx - 32B
2000
N/A
2048
Qwen 3
xxx - 8B
10000
N/A
2048
Appendix
Table 2: Dataset statistics across models.
Hyper-parameter
From-Scratch Transformer
Open-Source LLMs
Closed-Source
Latent / CoT
ICoT
7B / 8B
32B
GPT-4o
Epochs
500
1000
1
1
10
Learning Rate
1×10−4
1×10−4
3×10−6
3×10−6
Auto
Batch Size
1024
1024
4
1
1
Dropout
0.1
0.1
N/A
N/A
-
GPUs
1
1
8
8
-
Appendix
Table 3: Hyper-parameters for training models on star graph tasks. Note that the reported batch size represents the per-GPU batch size.
Figure 4: Training dynamics of the from-scratch transformer across planning depths m∈{3,4} and branch factors k∈K . The model exhibits a two-stage learning process: validation accuracy first rises to the random baseline ( k1 ) as the model learns to predict valid neighbors, then either jumps to near-perfect accuracy (successful strategy discovery) or stagnates while training loss continues to decrease (overfitting without strategy discovery).
Model
m=7
m=8
2
3
4
5
10
2
3
4
5
10
GPT-5.4
xxx - Zero Shot
0.22
0.07
0.17
-0.03
0.02
-0.08
0.07
-0.03
-0.03
-0.01
xxx - Few Shot
0.50
0.25
0.27
0.23
0.09
0.00
0.12
0.17
0.01
0.06
Appendix
Table 7: To further probe the latent planning capability (LPC) of GPT-5.4, we additionally evaluate it on more challenging star graph tasks with planning depths m∈{7,8} and branch factors k∈{2,3,4,5,10} . We report skill scores for zero-shot and few-shot settings. Blue-highlighted bold text indicates configurations where the model successfully rejects random guessing, while gray text indicates failure.
Figure 5: Training dynamics of Qwen 2.5 models (7B and 32B) across graph configurations. Both models exhibit the same two-stage learning process observed in the from-scratch transformer, with failures occurring exclusively in the second stage.
Figure 6: Training dynamics of Qwen 3 models (8B and 32B) across graph configurations. The same two-stage pattern persists, confirming that the discovery bottleneck is consistent across model families and scales.
Model
Average Skill over K at Depth m
3
4
5
6
20
Transformer
1.00
0.78
1.00
0.75
0.78
Qwen 2.5
xxx - 7B
1.00
1.00
1.00
1.00
1.00
xxx - 32B
1.00
0.99
1.00
0.99
0.99
Qwen 3
Appendix
Table 8: Average skill on star graph tasks across planning depths m∈{3,4,5,6,20} .
Figure 7: Training loss under explicit chain-of-thought supervision. All models are trained with dense supervision that provides the full backtracking trace from vtarget to vsource . Under this setting, all LLMs converge within approximately 20 training updates across all configurations, confirming that the task itself is not inherently difficult.
m
k∈K
2
3
4
5
10
3
1.00
1.00
1.00
1.00
-0.01
4
0.99
0.01
0.00
-0.01
-0.11
5
1.00
0.02
-0.33
0.00
-0.11
6
1.00
-0.01
-0.33
-0.25
-0.11
7
-1.00
-0.50
-0.33
-0.25
-0.11
Appendix
Table 9: Skill of ICoT on star graph tasks across branching factors k∈K={2,3,4,5,10} and planning depths m∈{3,4,5,6,7} . Blue-highlighted bold text indicates configurations where the model successfully rejects random guessing, while gray text indicates failure.
Figure 8: Attention visualization for successful configurations ( m=3 , k∈{2,3,4,5} ). For each configuration, the graph structure is shown on the left, where the orange node denotes the target and the violet node denotes the prediction, and the attention weights from the final token to all input tokens at each transformer layer are shown on the right. As k increases from 2 to 5, the attention progressively concentrates on nodes along the path from vtarget to vsource , with shallower layers attending to the target and deeper layers tracing back toward the source.
Figure 9: Attention visualization for failed configurations ( m=3 with k=10 , and m=4 with k∈{2,3,4,5,10} ), following the same convention as Figure 8 : orange denotes the target node and violet denotes the predicted node. No structured attention pattern emerges across any of these configurations, confirming that the model has not discovered a strategy.
Implicit reasoning in language models studies how models solve problems through internal computation rather than, or alongside, written chains of thought.