Existing latent reasoners typically run for a preset latent horizon during inference. Intuitively, a longer horizon should provide more opportunity for reasoning. However, in this work, we find that extending latent Chain-of-Thought (CoT) to the full preset horizon may in fact degrade reasoning performance. We observe that latent CoT evolves through distinct functional stages, with earlier computation forming frontier reasoning states, while later computation mainly reorganizes existing candidates and can weaken gold candidate support. These findings motivate a rethink of fixed-horizon latent inference. To address this, we characterize the reasoning benefits of latent CoT by realized and remaining constructive progress. Empirically, we find that higher realized constructive progress corresponds to higher current reasoning accuracy, while more remaining progress corresponds to larger future gains from continued latent CoT. To this end, we introduce a lightweight method that estimates the remaining reasoning progress and adaptively adjusts the latent horizon during inference. Across representative latent CoT reasoners and multiple arithmetic benchmarks, our method improves reasoning accuracy by up to 7.2 percentage points while reducing latent computation by up to 29.0%, without retraining the LLM.
Figures & tables
Figure 1: The full latent horizon can bury better reasoning outcomes. We evaluate six trained latent CoT reasoners across GPT-2 ( Radford et al., 2019 ) and Llama-1B ( Grattafiori et al., 2024 ) backbones, covering Coconut, CODI, and SIM-CoT variants. For each problem, we retrospectively stop latent reasoning at the first depth that produces a correct answer, if such a depth exists. Gray bars show accuracy at the preset full horizon, while colored bars show accuracy under this retrospective stopping point.
Figure 2: An example of latent reasoning progress. We use the symbolic Coconut model from Zhu et al. (2026b) and roll it out for six latent steps to trace the reasoning progress. We report the top-6 candidates at each step, with highlighted nodes indicating high support.
Figure 3: Latent progression and revision. (a) Gold candidate formation; (b) Gold candidate support. For trajectories whose answer is formed at the third latent step, (c) Correction and reversal rates, and (d) Accuracy over revision vs. the best accuracy reached along the trajectory.
GSM8K
SVAMP
MultiArith
Latent step
Method
Full- K
Ours
Gain
Full- K
Ours
Gain
Full- K
Ours
Gain
saving (%)
GPT-2
CODI
42.1
43.4
+1.3
41.0
43.1
+2.1
94.4
95.0
+0.6
20.6
SIM-CoT
39.2
41.5
+2.3
40.0
43.5
+3.5
91.7
92.2
+0.5
29.0
Coconut
27.1
27.7
+0.6
31.4
33.7
+2.3
71.7
78.9
+7.2
11.0
SIM-CoT Coconut
44.7
44.7
+0.0
40.6
42.2
+1.6
92.8
93.3
+0.5
25.1
Table 1: Adaptive latent CoT inference. Full- K denotes the preset latent horizon while ours uses adaptive latent inference. Latent step saving (%) reports the average step reduction.
Model / Budget policy
Accuracy (%)
GSM8K
SVAMP
MultiArith
(a) CODI family
4 steps
5 steps
6 steps
4 steps
5 steps
6 steps
4 steps
5 steps
6 steps
GPT-2
CODI
Fixed depth
36.6
41.2
42.1
41.5
41.0
41.0
92.8
93.3
94.4
Adaptive
41.5
43.2
43.4
42.2
42.2
43.1
92.8
93.3
95.0
SIM-CoT
Fixed depth
35.3
39.1
39.2
41.3
40.6
40.0
89.4
91.1
91.7
Table 2: Comparison of latent CoT performance under adaptive and fixed-preset inference. We report task accuracy (%) under maximum latent budgets of 4, 5, and 6 steps.
Figure 4: Constructive progress completion across latent CoT . Bars show how constructive progress completion is distributed. The top strips summarize completion status at latent step t=3 , contrasting completed and still progressing. Triangles mark the first latent step where at least 75% of examples have completed constructive progress. Completion marks the first latent step where the latent CoT realizes all constructive progress.
Backbone
Method
GSM8K
SVAMP
MultiArith
Lower
Higher
Lower
Higher
Lower
Higher
GPT-2
CODI
29.3
45.0
16.6
70.8
81.0
96.7
SIM-CoT
10.8
53.3
13.2
73.4
25.3
94.1
Llama-1B
CODI
24.9
69.8
54.4
77.0
95.1
99.0
SIM-CoT
34.7
67.2
45.3
80.3
94.5
98.4
GPT-2
Coconut
11.4
32.8
21.1
45.3
31.4
38.7
Table 3: Reasoning performance across constructive progress. At latent depth t=3 , we compare task accuracy (%) across different levels of realized constructive progress, as defined in Section 4.2 .
Accuracy (%)
Upper bound
Compute
Dataset
Backbone
Method
Ours
ALR
DEER
Dynamic K-Head
Ours with true Ω
Per-item oracle
Avg. steps
GSM8K
GPT-2
CODI
41.1
38.4 − 2.7
37.3 − 3.8
38.4 − 2.7
42.6
47.0
4.37
SIM-CoT
39.2
35.1 − 4.1
34.3 − 4.9
34.3 − 4.9
41.8
44.7
4.28
Llama-1B
CODI
54.5
53.1 − 1.4
54.4 − 0.1
52.5 − 2.0
55.6
58.8
4.57
SIM-CoT
54.7
52.3 − 2.4
52.8 − 1.9
52.9 − 1.8
55.5
58.2
4.72
SVAMP
GPT-2
SIM-CoT
40.2
39.9 − 0.3
39.6 − 0.6
39.8 − 0.4
41.5
42.9
4.68
Table 6: Comparison with transferred adaptive latent methods. All methods run on the same frozen reasoners with matched average latent computation, and none stops before our adaptation step. The upper bounds replace our estimate with the true remaining progress, or give each item its best stopping step under the same total computation. Subscripts indicate the difference relative to ours.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
(a) Regression target
Remaining progress
Future gain
Current value
Shuffled
Signal free
Accuracy
47.2
46.9 − 0.3
45.3 − 1.9
45.1 − 2.1
45.1 − 2.1
Appendix
Table 11: Remaining-progress targets under matched latent computation. Each comparison keeps the trajectory representation, regressor, offline trajectories, and adaptive controller fixed, and differs only in the regression target. Panel (a) compares regression targets under the default design, with average latent computation matched at 4.49 steps. Panels (b) – (d) compare remaining progress with the current reasoning value under alternative designs. We report final task accuracy on GSM8K averaged over the four CODI-family reasoners, with subscripts indicating the difference relative to remaining progress.
Explicit chain-of-thought (CoT) reasoning substantially improves the reasoning ability of large language models (LLMs), but incurs high inference cost due to lengthy autoregressive traces. Existing latent reasoning methods offer a promis- ing alternative, yet they often treat reasoning as uniformly compressible, causing precision-critical intermediate steps to be overly compressed and thereby degrading reasoning accuracy. In this work, we propose Selective Latent Thinking (SLT), a reasoning paradigm that selectively compresses redundant reasoning spans into latent representations while preserving precision-critical spans as explicit CoT within the same reasoning trajectory. Our primary instantiation uses a lightweight decoder to anticipate a short upcoming reasoning span and applies confidence- based gating to determine the longest span that can be reliably compressed; the resulting selective compression policy is further optimized with trajectory-level reinforcement learning to balance answer correctness against reasoning cost. Ex- periments across four mathematical reasoning benchmarks show that which spans are compressed is critical: with the same compressor, compressing arbitrary spans reduces the GSM-Hard accuracy of Qwen3-4B from 54.7% to 17.3%, whereas keeping numerical tokens explicit retains 51.8%. SLT reduces explicit reasoning length by 58.4% with only a 2.8-point accuracy drop compared to explicit CoT on Llama-3.2-1B. Notably, the principle extends beyond this design: instantiated within existing latent reasoning models via a learned compression gate, it improves accuracy at equal reasoning length (+2.6 points over Latent-SFT) and at shorter length (+5.6 points over Latent-GRPO with 13% fewer tokens).
Hui Xie, Jie Liu, Ziyue Qiao +1
Eindhoven University of Technology, Netherlands · School of Computing and Information Technology, Great Bay University, China
Chain-of-thought (CoT) prompting improves reasoning in large language models (LLMs) by externalizing intermediate computation as discrete text tokens, but this textual interface also introduces redundancy and inference overhead. Latent reasoning offers a promising alternative by carrying part of the computation in continuous representations. However, existing methods typically predefine when latent computation is invoked and how it is allocated during decoding, leaving a key problem unresolved: when to invoke latent computation, what type of computation to perform, and how much budget to allocate. We propose Typed Latent Reasoning (Tyler), a typed and budget-aware framework for latent reasoning during autoregressive decoding. Tyler learns a policy that, at each decoding step, chooses between emitting a text token and switching to a latent computation module specialized for a particular reasoning function. Once invoked, an operator maps the current reasoning state into latent tokens that support global planning, local state updates, or reusable procedural abstraction. Across extensive experiments on three backbone LLMs, Tyler improves accuracy by up to 14.49 points over CoT and by up to 4.30 points over the strongest competing baseline. It further generalizes across diverse reasoning domains and achieves the best final-stage performance with the lowest forgetting.
Chain-of-thought (CoT) reasoning often improves language-model performance by giving models additional computation before answering. However, explicit CoT expresses this computation as a sequence of autoregressively generated tokens. Latent reasoning replaces these tokens with compact continuous states, but most autoregressive latent-reasoning methods retain a left-to-right dependency among latent vectors. We introduce LLoCoT: a looped latent-reasoning framework that replaces left-to-right latent generation with iterative refinement of a compact latent workspace. A shared transformer is reapplied for a small number of refinement iterations, jointly updating the latent slots based on the prompt and the evolving workspace state. Using the refined state, a probabilistic head predicts a distribution from which latent tokens are sampled in parallel and used to condition an autoregressive decoder for answer generation. Training uses continuous representations derived from explicit CoT together with a final-answer prediction loss and likelihood-based supervision of the latent states. Across HumanEval and MBPP, LLoCoT achieves the highest mean among the evaluated methods, performing on par in accuracy with Reasoning SFT, our explicit-CoT baseline, while outperforming the base model, answer-only SFT and NF-CoT. Relative to Reasoning SFT, LLoCoT reduces time to the first answer token by approximately 36× and reasoning-phase latency by approximately 42×, while increasing end-to-end throughput by 9.2%. This design replaces serial thought generation with parallel latent-slot refinement while retaining probabilistic latent modeling and autoregressive answer decoding.