Large Language Models can perform multi-step reasoning and improve task performance through different forms of intermediate computation, from token-based traces to computation carried out in latent space. However, a question remains open: do these different forms of thinking rely on the same underlying mechanism? To address this, we train and compare five variants of the same GPTNeoX backbone from scratch on an extended multi-hop reasoning task (ProsQA-Ext): a vanilla model, a Chain-of-Thought (CoT) model, a Pause Token model, and two latent-reasoning models that are optimized end-to-end without intermediate reasoning traces. We find that, strong in-distribution (ID) performance does not guarantee depth generalization. Vanilla, CoT, and Pause Token models solve ID problems well, but rely largely on local graph features and generalize poorly to out-of-distribution (OOD) problems with longer hops. In contrast, latent variants generalize better and show internal dynamics consistent with forward reachability propagation on the graph. Causal interventions and circuit analysis localize this computation to a sparse recurrent search circuit in the bottleneck latent model: an attention head retrieves graph relations, an MLP and the residual stream update the reachability state across recurrent steps, while multiple attention heads together then do the candidate matching. Together, these results show that different thinking mechanisms can learn distinct computational solutions, even at similar ID performance. In this setting, latent recurrence supports a reusable forward-search algorithm that generalizes beyond the training depth.
Figures & tables
Figure 1: ProsQA-Ext task and reasoning performance. A shows an example ( H=8 ) with the correct path highlighted in green and distractor edges in gray. B shows the five model variants. C and D show free-generation accuracy on ID and OOD problems, respectively (error bars are SEM).
Figure 2: Representational alignment with forward graph propagation. A and B show RSA heatmaps for five variants on ID (4-hop) and OOD (8-hop) problems, respectively, using the same examples across variants. Each entry shows the Spearman correlation between pairwise model-representation dissimilarities and pairwise graph-frontier dissimilarities at depth d . For G=(V,E) with query root r , the propagation frontiers are defined by F0={r} and Fd+1={v∈V:∃u∈Fd,(u,v)∈E} . Gray hatched cells denote undefined correlations. Color scales are shared across rows within each variant. C shows diagonality across 3–12-hop problems, with colors indicating variants. The dashed line separates ID (3–6 hops) from OOD (7–12 hops).
Figure 3: Effects of candidate and successor degree on model predictions. A shows matched pairs that reverse the candidates’ relative in-degree while preserving the proof path and correct answer. B and C show the final-answer accuracy on problems with short and long reasoning depths. Solid and hatched bars indicate that the correct candidate has lower and higher in-degree, respectively. D shows matched pairs that switch which immediate successor of the query root leads to the correct candidate while preserving all node degrees and the correct final answer. E and F show final-answer accuracy for all five models and first-successor accuracy for CoT on problems with short and long reasoning depths. Solid and hatched bars indicate that the correct successor has a lower or higher in-degree minus out-degree, respectively. In schematics, (r) denotes the query root, (+) and (-) denote the correct and incorrect candidates, and green arrows indicate the proof path.
Figure 4: Controlled interventions probe the recurrent computation in latent reasoning models. A shows how swapping premises shifts the query root r ’s correct candidate from A to B . In B , the top row shows the normalized cosine distance between paired latent states for 8-hop problems with connectivity swaps at depth d , and the bottom row shows the fraction of pairs for which transplanting the latent state at step t redirects the answer to y′ . C shows OOD accuracy with one fewer ( K=5 ), the trained number ( K=6 ), or one additional ( K=7 ) thinking position. Shaded region indicates 95% CI.
Figure 5: Localization of the recurrent circuit in Bottleneck-latent . Replacement based pruning ( A ) retains eight of twenty recurrent components ( B ). The selected circuit largely preserves candidate choices ( C ) and causal state-transfer effects ( D ), whereas removing it or retaining random size-matched components does not. Error bars indicate pointwise 95% bootstrap confidence intervals over base graphs; the gray band shows the 10th–90th percentiles across twenty random circuits.
Figure 6: Functional analysis of the recurrent circuit in Bottleneck-latent . A illustrates the LHS and RHS swaps. Interventions within the pruned circuit distinguish L4H1’s key/value routing ( B ), query-dependent selection of reading depth ( C ), and component contributions to the next query and final answer ( D ). Fixing or transplanting the L4 MLP response ( E ) separates its contribution from the residual pathway. F tests candidate matching through Q/K interventions in L4H1/H2/H4. Error bars indicate pointwise 95% bootstrap confidence intervals.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1: Representational alignment with forward graph propagation across task depths. RSA heatmaps for five model variants on 3 to 12-hop problems (rows), across ID (3–6 hops) and OOD (7–12 hops) conditions. Entries represent Spearman correlations between model-representation and frontier dissimilarities, as shown in Fig. 2 .
Figure A.2: Diagonality of alignment with parallel backward search. The same diagonality measure is applied to RSA against the joint backward frontier for all five model variants. The dashed line separates ID from OOD problems.
Figure A.3: Effect of hop perturbation on pruned circuit in Bottleneck-latent model.
Figure A.4: Head contributions during recurrence and answer readout. Individual interventions compare recurrent Q/K matching (left), candidate-value exchange (middle), and final answer readout (right) for L4H1, L4H2, and L4H4. Error bars indicate pointwise 95% bootstrap confidence intervals.
Chain-of-thought reasoning unfolds in discrete token space: each step is committed as text, errors propagate, and eliciting good traces presupposes traces to imitate. Reasoning instead in a model's continuous representation space - where intermediate states are vectors rather than words - sidesteps these constraints, but leaves open how those latent states should be computed. We approach this along two axes. First, we keep a large language model (LLM) frozen and use it for what it is already good at - modeling and decoding sequences - while a small auxiliary network supplies continuous latent thoughts as input. Second, we produce those latents by recurrence: a tiny recurrent reasoner refines them over many steps, decoupling the depth of computation from the size of the model, so that the latents are a product of iterative processing rather than a single forward pass. We instantiate this as Latent Recurrent Thoughts (LRT): a task-dedicated proposer supplies base latents, a recurrent reasoner refines them through bounded residual corrections, and the frozen LLM decodes the answer. On symbolic reasoning with answer supervision but no reasoning traces (Countdown-4, Sudoku) and on natural-language reasoning (HumanEval, MBPP, StrategyQA), LRT substantially outperforms prior frozen-decoder continuous-space reasoning methods under an identical decoder, prompt, data, and training budget, and outperforms non-thinking-mode chain-of-thought prompting on the same backbone at a small fraction of its inference compute.
Large language models solve complex problems by generating lengthy chains of explicit reasoning tokens. While effective, this makes reasoning expensive, length-sensitive, and constrained to (discrete) natural language. While latent reasoning offers a continuous alternative, determining useful structures for intermediate latent states is an open challenge. In this paper, we formulate latent reasoning as a geometric path-approximation problem within the model's pretrained token-embedding space. We introduce Geometric Latent Reasoning (GLR), which uses a lightweight transition head to predict iterative direction updates in embedding space. Using textual chain-of-thought traces as anchors, GLR learns to approximate discrete reasoning trajectories while permitting continuous deviations from exact token embeddings. Evaluations on mathematical reasoning benchmarks using Qwen3 models reveal an emergent phenomenon: geometric latent reasoning induces substantially shorter generations without an explicit length objective. By replacing early explicit reasoning with continuous latent steps, models often reach correct answers using substantially fewer total generation steps. These findings suggest that continuous trajectories act as compact intermediate reasoning states, exposing a new tradeoff between latent computation budget, output length, and accuracy.
Shashi Kumar, Yacouba Kaloga, Petr Motlicek +2
Idiap Research Institute, Switzerland · EPFL, Switzerland · BUT, Czech Republic
Standard Transformers have a fixed computational depth, limiting their ability to generalize to tasks that require variable-depth reasoning. The usual remedy, Chain-of-Thought (CoT), spends tokens to reason, inflating the key--value cache and making latency grow with the step count, so memory becomes the limiting cost when reasoning is served over large query batches. We study a depth-recurrent Transformer that decouples computational depth from parameter count by iterating a shared-weight block, so that each added reasoning step costs flat memory and linear latency, with no token generation. Three ingredients keep the recurrence stable for 20+ thinking steps: a silent thinking objective that supervises only the final output, LayerScale initialization, and an identity-biased gate that opens a gradient highway across steps. We characterize it on three compositional domains with decreasing structural bias: graph reachability (adjacency masking), nested boolean logic (relative positioning), and unstructured relational text (no positional cue). We find a \emph{computational frontier}: accuracy climbs once the thinking-step count meets the task's complexity, reaching near-perfect performance on the two structured tasks and a lower plateau on unstructured text. How it climbs depends on the structural bias---abruptly from chance on the graph task, gradually on the other two. Depth recurrence extrapolates beyond the training range: it succeeds on the graph task where fixed-depth models barely extrapolate, and on the two sequence tasks comes within two points of fixed-depth Transformers that use 4--6.4× more parameters. On the graph task, whose adjacency mask makes propagation depth verifiable, intermediate per-step supervision---a standard recipe for deep iterative models---consistently \emph{harms} this extrapolation. We release the code for reproducibility.
Hung-Hsuan Chen
Computer Science and Information Engineering National Central University Taoyuan, Taiwan