Reasoning has become a central capability in large language models. Recent research has shown that reasoning performance can be improved by looping an LLM's layers in the latent dimension, resulting in looped reasoning language models. Despite promising results, few works have investigated how their internal dynamics differ from those of standard feedforward models. In this paper, we conduct a mechanistic analysis of the latent states in looped language models, focusing in particular on how the stages of inference observed in feedforward models compare to those observed in looped ones. To this end, we analyze cyclic recurrence and show that for many of the studied models each layer in the cycle converges to a distinct fixed point; consequently, the recurrent block follows a consistent cyclic trajectory in the latent space. We provide evidence that as these fixed points are reached, attention-head behavior stabilizes, leading to constant behavior across recurrences. Empirically, we discover that recurrent blocks learn stages of inference that closely mirror those of feedforward models, repeating these stages in depth with each iteration. We study how recurrent block size, input injection, and normalization influence the emergence and stability of these cyclic fixed points. We believe these findings help translate mechanistic insights into practical guidance for architectural design.
Figures & tables
Figure 1: Latent states after each block in a recurrent model frequently tend towards separate fixed points , meaning that the application of a recurrent block tends towards a consistent trajectory in latent space.
Figure 2: Frobenius norm between attention patterns at different depths , averaged across the batch and head dimensions. Depth index visualized on each axis, cells show the norms between attention patterns at each pair of depth indices. Left: Ouro 1.4B ( Zhu et al., 2025 ) . Center: Retrofitted Llama ( McLeish et al., 2025 ) . Right: Huginn-0125 ( Geiping et al., 2025 ) . All models looped 8 times.
Figure 3: Norm of the difference between the residual stream after successive recurrences of the same layer.
Figure 4: Norm of the difference between the residual stream after each layer in the recurrent block and its “approximate fixed point” - the residual stream after that layer in the 128th recurrence. While Huginn-0125 and retrofitted Llama quickly reach a fixed point, Ouro does not - despite small successive differences evidenced by Fig. 3 .
Figure 5: Retrofitted Llama ( McLeish et al., 2025 ) latent space trajectory traced out by the hidden states of the final sequence position on a single test prompt; reduced to two dimensions by computing PCA over all final sequence position embeddings. Trajectories perfectly overlap in the second plot, demonstrating that a cyclic fixed point has been reached.
Figure 6: Cosine similarity between residual streams after each layer and the approximate fixed point for a range of norms, with and without input injection. Each model is randomly initialized with 12 layers. Cosine similarity is taken between the residual stream after the first layer at each recurrence and left: the approximate fixed point of the first layer, right: the approximate fixed point of the layer with the lowest cosine similarity to the first layer.
Figure 7: Stages of inference for retrofitted Llama ( McLeish et al., 2025 ) with 8 recurrences. Individual layers are visualized as solid lines; successive layers in the looped Transformer as a dashed black line. Individual layers quickly converge towards constant behavior, and the cyclic action of these layers results in cyclic stages of inference.
Figure 8: Stages of inference for each recurrent loop in left: Ouro 1.4B ( Zhu et al., 2025 ) center: retrofitted Llama and right : retrofitted OLMo ( McLeish et al., 2025 ) . Ouro 1.4B resembles Llama stages of inference, and the two retrofitted to their associated base models.
Figure 9: Norms of the residual stream for a range of models, demonstrating that Huginn-0125 is unable to develop the activation magnitude changes required for stages of inference due to its repeated normalization of the residual stream.
Figure 10: ColSum concentrations for small-scale trained Looped Transformers with a simplified loss function and constant train recurrence schedule of 4 recurrences . Also visualized in red is a “control” feedforward Transformer of depth 12. All models have 2 prelude and 2 coda layers with no input injection, left: 4 recurrent layers, center: 8 recurrent layers, right: 12 recurrent layers.
Figure 11: Colsum concentration of each layer with successive recurrences for left: retrofitted Llama and right: Ouro 1.4B, both using 128 recurrences. While the layers of retrofitted Llama quickly converge to constant ColSum concentration, the layers of Ouro continually change throughout the recurrences tested.
Figure 12: Colsum concentration of each layer vs the percentage depth at which that layer appears in the recurrent block. Left: retrofitted Llama and right: Ouro 1.4B, both using 128 recurrences. Feedforward Llama shown in dashed red.
Appendix figures & tables55 assets
Supplementary material from the paper’s appendix.
Appendix
Model Name
Structure
Train Loops
Block Norm
Norm After Loop?
Ouro 1.4B ( Zhu et al., 2025 )
(0,24,0)
4
X^X′=X+n(\attn(n(X))=X^+n(MLP(n(X^)))
Yes
Huginn-0125 ( Geiping et al., 2025 )
(2,4,2)I
32
X^X′=n(X+\attn(n(X))=n(X^+MLP(n(X^)))
Yes
Retrofitted ( McLeish et al., 2025 )
(4,6,4)I / (4,8,4)I
32
X^X′=X+\attn(n(X)=X^+MLP(n(X^))
No
Appendix
Table 1: Looped Transformer architecture summary. Retrofitted Llama and OLMo use 6 layers in their recurrent block, TinyLlama uses 8.
Model
Num Params
Huggingface ID
Base Model
Prepends BOS
Notes
Ouro 1.4B
1.4B
ByteDance/Ouro-1.4B
-
\crossproduct
-
Ouro 2.6B
2.6B
ByteDance/Ouro-2.6B
ByteDance/Ouro-1.4B
\crossproduct
“Upcycled” from Ouro 1.4B by repeating the same layers after the first training phase.
Huginn-0125
3.5B
tomg-group-umd/huginn-0125
-
✓
-
Retrofitted Llama
1B
smcleish/Recurrent-Llama-3.2-train-recurrence-32
meta-llama/Llama-3.2-1B
✓
Models trained with fewer recurrences exhibit similar mixing patterns.
Table 2: Additional Huggingface details on pretrained Looped models used.
Figure 13: Visualizing the component parts of the Orbit detection algorithm of Algorithm 1 . The input sequence (cosine similarities for the residual streams of a given token and layer and successive recursions, as compared to their final residual stream) is visualized in the leftmost column. The center column visualizes the effect of windowing and de-trending, and the rightmost column shows the FFT magnitudes. The top row visualizes the detected Orbit with the largest amplitude for the Huginn-0125 model (which occurs with the “Long Persona” prompt) and the bottom row visualizes the largest amplitude for the Retrofitted Llama model (which occurs with no system prompt). We note the appearance of complex, multi-frequency oscillation in this latter case.
Model
System Prompt
Non-Fixed-Point %
Orbit %
Slider %
Unknown %
Huginn
Long Persona
0.14
0.13
0.00
0.01
Huginn
Long Persona (Padded)
0.05
0.02
0.02
0.01
Huginn
No System Prompt
0.02
0.01
0.01
0.00
Huginn
Short Math
0.01
0.00
0.00
0.00
Retrofitted-Llama
Long Persona
0.00
0.00
0.00
0.00
Retrofitted-Llama
Long Persona (Padded)
0.00
0.00
0.00
0.00
Appendix
Table 3: Percentage of tokens exhibiting each behavior type, by model and system prompt.
Model
System Prompt
Non-Fixed-Point %
Orbit %
Slider %
Unknown %
Huginn
Long Persona
2.81
2.50
0.15
1.06
Huginn
Long Persona (Padded)
0.83
0.23
0.61
0.15
Huginn
No System Prompt
0.76
0.53
0.15
0.08
Huginn
Short Math
0.45
0.38
0.08
0.00
Retrofitted-Llama
Long Persona
0.00
0.00
0.00
0.00
Retrofitted-Llama
Long Persona (Padded)
0.00
0.00
0.00
0.00
Appendix
Table 4: Percentage of examples exhibiting each behavior type at least once on any question token, by model and system prompt.
Figure 14: Conditional probabilities of co-occurrence for the different limiting behaviors.
Figure 15: PCA trajectories in the intermediate layers of Huginn-0125: this reproduces the leftmost column of Fig. 16 in Geiping et al. (2025) (the first two principal components) and additionally plots the latent trajectories for the intermediate layers in the recurrent block.
Figure 16: Stages of inference metrics (sink rate, mixing score and colsum concentration) for Huginn-0125 across 128 recurrences, for the GSM8k test prompt that exhibited the largest orbit amplitude. Visualized in realized depth.
Figure 17: Stages of inference metrics (sink rate, mixing score and colsum concentration) for Huginn-0125 across 128 recurrences, for the GSM8k test prompt that exhibited the largest orbit amplitude: only the final 64 recurrences are visualized, to isolate the impact of the orbit. Visualized in percentage block depth.
Figure 18: Stages of inference metrics (sink rate, mixing score and colsum concentration) for Retrofitted Llama across 128 recurrences, for the GSM8k test prompt that exhibited the largest orbit amplitude: only the final 64 recurrences are visualized, to isolate the impact of the orbit. Visualized in percentage block depth.
Figure 19: Cosine similarity between residual streams after every pair of layers for different Transformer models , averaged across the batch and sequence dimensions. Left: Ouro 1.4B ( Zhu et al., 2025 ) . Center: Retrofitted Llama ( McLeish et al., 2025 ) . Right: Huginn-0125 ( Geiping et al., 2025 ) . All models looped 8 times. Diagonal patterns indicate that the residual stream after each sub-block is most similar to the same block in the next recurrence; every layer in the recurrent block reaches a different fixed point.
Figure 20: Cosine similarity between residual streams after every pair of layers for different Transformer models, averaged across the batch and sequence dimensions. Left: Huginn-0125 ( Geiping et al., 2025 ) . Center Left: Retrofitted Llama. Center Right: Retrofitted OLMo. Right: Retrofitted TinyLlama ( McLeish et al., 2025 ) . All models looped 32 times. Extended version of Fig. 19 .
Figure 21: Frobenius norm between attention matrices for different Transformer models, averaged across the batch and head dimensions. Left: Huginn-0125 ( Geiping et al., 2025 ) . Center Left: Retrofitted Llama. Center Right: Retrofitted OLMo. Right: Retrofitted TinyLlama ( McLeish et al., 2025 ) . All models looped 32 times. Extended version of Fig. 2 .
Figure 22: Cosine similarity between the residual stream after each layer in the recurrent block and its “approximate fixed point” - the residual stream after that layer in the 128th recursion. While Huginn-0125 and retrofitted Llama quickly reach a fixed point, Ouro does not - even though the cosine similarity between successive recursions tends towards one, as evidenced by Fig. 22 .
Figure 23: Frobenius norm between attention matrices of each layer in the recurrent block and their corresponding “approximate fixed point” ; the attention matrices of the same layer in the 128th recursion. While Huginn-0125 and retrofitted Llama quickly reach a fixed point, Ouro does not.
Figure 24: Cosine similarities between successive recursions of the residual stream after the same layer.
Figure 25: Frobenius norm between attention matrices of each layer between successive recurrences.
Figure 26: Ouro 1.4B ( Zhu et al., 2025 ) latent space trajectory traced out by the hidden states of the final sequence position on the test prompt; reduced to two dimensions by computing PCA over all final sequence position embeddings.
Figure 27: Huginn-0125 ( Geiping et al., 2025 ) latent space trajectory traced out by the hidden states of the final sequence position on the test prompt; reduced to two dimensions by computing PCA over all final sequence position embeddings.
Figure 28: Ouro 1.4B ( Zhu et al., 2025 ) latent space trajectory traced out by the hidden states of the final sequence position on a “maths” test prompt ( The square root of 16 is ); reduced to two dimensions by computing PCA over all final sequence position embeddings.
Figure 29: Norm of the difference between the residual stream after the first layer in the recurrent block for successive recurrences and an “approximate fixed point” – the residual stream in the 128th recurrence. Two fixed point differences are visualized: the difference to the fixed point of the same (first) layer (blue) and the difference to the fixed point which has the greatest norm difference from the first layer (green). Visualized are a range of norm structures (columns), with input injection (top row) and without (bottom row). All models are randomly initialized with 12 layers in the recurrent block, and no prelude or coda.
Figure 30: Cosine similarity between the residual stream after the first layer in the recurrent block for successive recurrences and an “approximate fixed point” – the residual stream in the 128th recurrence. Two fixed point differences are visualized: the difference to the fixed point of the same (first) layer (blue) and the difference to the fixed point which has the lowest cosine similarity to the first layer (green). Visualized are a range of norm structures (columns), with input injection (top row) and without (bottom row). All models are randomly initialized with 12 layers in the recurrent block, and no prelude or coda.
Figure 31: Cosine similarity between the residual stream after the first layer in the recurrent block for successive recurrences and an “approximate fixed point” – the residual stream in the 128th recurrence. Two fixed point differences are visualized: the difference to the fixed point of the same (first) layer (blue) and the difference to the fixed point which has the lowest cosine similarity to the first layer (green). Visualized are a range of norm structures (columns), with input injection (top row) and without (bottom row). Here models of 4 and 16 layers are compared, showing qualitatively identical behavior and demonstrating that the behavior is not particular to 12 layers.
Figure 32: Fraction of prediction and suppression neurons in a selection of feedforward models used throughout the paper .
Figure 33: Fraction of prediction and suppression neurons in a selection of looped models used throughout the paper .
Figure 34: Stages of inference for a selection of Looped transformers, all using 8 recurrences : Huginn-0125 ( Geiping et al., 2025 ) , Ouro 1.4B ( Zhu et al., 2025 ) and Llama with retrofitted recurrences ( McLeish et al., 2025 ) . Note Huginn-0125 and Retrofitted Llama have prelude and coda layers too: each 2 layers in Huginn-0125 and each 4 in Retrofitted Llama.
Figure 35: Stages of inference for each recurrent loop in Ouro 1.4B. The close overlap with feedforward stages of inference is a particularly striking result as this model is trained from scratch with recurrence.
Figure 36: Stages of inference for each recurrent loop in Huginn-0125. This represents a negative result: stages of inference do not occur. We discuss possible causes for this in the main text.
Figure 37: Stages of inference for each recurrent loop in the retrofitted Llama model ( McLeish et al., 2025 ) . Each block demonstrates very similar stages of inference to Llama, the base model from which pretrained layers are taken.
Figure 38: Stages of inference for each recurrent loop in the retrofitted OLMo model ( McLeish et al., 2025 ) . Similarly, each block demonstrates very similar stages of inference to OLMo, the base model from which pretrained layers are taken.
Figure 39: Stages of inference for each recurrent loop in the retrofitted TinyLlama model ( McLeish et al., 2025 ) . Similarly, each block demonstrates very similar stages of inference to TinyLlama, the base model from which pretrained layers are taken.
Figure 40: Stages of inference for each recurrent loop in Ouro 2.6B. For this model we separate out the first and second half of the recurrent block and overlay them, demonstrating that both halves have close alignment with the Llama feedforward stages of inference. We suggest that this likely arises due to the training regime of Zhu et al. (2025) , which first trains a single 24 layer 1.4B parameter model, and then “upcycles” this into a 48 layer 2.6B parameter model by duplicating these layers, consequently duplicating the stages of inference as well.
Figure 41: Stages of inference for each recurrent loop in Retrofitted Llama for which the massive activations have been ablated.
Figure 42: Stages of inference for each recurrent loop in Ouro 1.4B, run on the HellaSwag dataset.
Figure 43: Stages of inference for each recurrent loop in Huginn-0125, run on the HellaSwag dataset.
Figure 44: Stages of inference for each recurrent loop in the retrofitted Llama model ( McLeish et al., 2025 ) , run on the HellaSwag dataset.
Figure 45: Stages of inference for each recurrent loop in the retrofitted OLMo model ( McLeish et al., 2025 ) , run on the HellaSwag dataset. ColSum concentration deviates slightly from its GSM8k counterpart here, but still broadly follows the same stages of inference as the feedforward OLMo model.
Figure 46: Stages of inference for each recurrent loop in the retrofitted TinyLlama model ( McLeish et al., 2025 ) , run on the HellaSwag dataset.
Figure 47: Stages of inference for each of the distinct blocks in Ouro ( Zhu et al., 2025 ) , as they are reapplied throughout the model for 128 recurrences. These consistently change throughout the realized depth of the model, reaching no clear fixed point. Mean and standard deviation are over separate inputs to the model, taken over the GSM8k subset.
Figure 48: Stages of inference for each of the distinct blocks in Huginn-0125 ( Geiping et al., 2025 ) , as they are reapplied throughout the model for 128 recurrences. These converge to constant behavior. Mean and standard deviation are over separate inputs to the model, taken over the GSM8k subset.
Figure 49: Stages of inference for each of the distinct blocks in retrofitted Llama ( McLeish et al., 2025 ) , as they are reapplied throughout the model for 128 recurrences. These converge to constant behavior. Mean and standard deviation are over separate inputs to the model, taken over the GSM8k subset.
Figure 50: Stages of inference for Ouro with each of 128 recurrences , visualized over percentage recurrent depth. These consistently change with successive recurrences, deviating significantly from the stages of inference seen with train-time recurrences.
Figure 51: Stages of inference for Huginn-0125 with each of 128 recurrences , visualized over percentage recurrent depth. These quickly reach a fixed point and do not deviate far from their starting stages of inference.
Figure 52: Stages of inference for retrofitted Llama with each of 128 recurrences , visualized over percentage recurrent depth. These quickly reach a fixed point and do not deviate far from their starting stages of inference.
Figure 53: Stages of inference for retrofitted OLMo-2 with each of 128 recurrences , visualized over percentage recurrent depth. These quickly reach a fixed point and do not deviate far from their starting stages of inference.
Figure 54: Stages of inference metrics for a small-scale Looped Transformer of configuration (2,4⊗4,2) , compared to a “control” feedforward Transformer with the same training configuration and depth 12.
Figure 55: Stages of inference metrics for a small-scale Looped Transformer of configuration (2,8⊗4,2) , compared to a “control” feedforward Transformer with the same training configuration and depth 12.
Figure 56: Stages of inference metrics for a small-scale Looped Transformer of configuration (2,12⊗4,2) , compared to a “control” feedforward Transformer with the same training configuration and depth 12.
Figure 57: Stages of inference metrics for a small-scale Looped Transformer of configuration (2,4⊗4,2)I , compared to a “control” feedforward Transformer with the same training configuration and depth 12.
Figure 58: Stages of inference metrics for a small-scale Looped Transformer of configuration (2,8⊗4,2)I , compared to a “control” feedforward Transformer with the same training configuration and depth 12.
Figure 59: Stages of inference metrics for a small-scale Looped Transformer of configuration (2,12⊗4,2)I , compared to a “control” feedforward Transformer with the same training configuration and depth 12.
Figure 60: Stages of inference metrics for a small-scale Looped Transformer of configuration (0,4⊗4,0) , compared to a “control” feedforward Transformer with the same training configuration and depth 12.
Figure 61: Stages of inference metrics for a small-scale Looped Transformer of configuration (0,8⊗4,0) , compared to a “control” feedforward Transformer with the same training configuration and depth 12.
Figure 62: Stages of inference metrics for a small-scale Looped Transformer of configuration (0,12⊗4,0) , compared to a “control” feedforward Transformer with the same training configuration and depth 12.
Figure 63: Entire attention pattern floorplan for the retrofitted Llama model , illustrating the cyclic similarity between recurrences.
Looped language models (LoopLMs) increase computational depth through parameter sharing, offering a path to scale inference computation without adding parameters. However, it remains unclear when additional recurrence is beneficial and how architectural choices affect its effectiveness. Through controlled experiments, we systematically examine (1) when recurrence helps, (2) where it should be applied, and (3) how its conditioning affects performance. Our evaluation covers inference budgets below, within, and beyond the training horizon under knowledge and reasoning tasks. (1) We find that recurrence can improve reasoning beyond the training horizon while degrading knowledge performance, but harder reasoning instances do not consistently benefit more. (2) Performance also depends on how distinct layers and recurrent iterations are allocated, showing that effective depth alone is insufficient to predict behavior. Non-recurrent output layers improve robustness to under-unrolling, while the preferred placement of input and output layers varies with inference budget. (3) Finally, we find that conventional initial-state injection offers limited robustness to varying recurrence depth. We therefore propose history-state injection as an alternative, and show that channel-wise history-state injection combined with timestep conditioning offers a low-cost and more effective design, better preserving knowledge under extended unrolling while improving robustness across inference budgets. Overall, our results clarify when recurrent computation helps, where it fails, and offer practical guidelines for designing LoopLMs across variable inference budgets.
Looped language models (LMs) reuse one block of layers several times per token generation, promising more latent computation at no additional storage cost. Despite their rising popularity, the analysis of looped LMs is still in its infancy: little is known about whether the iterative transformations actually improve their capabilities, or whether these models remain stable under prompt perturbation. We construct an inspection protocol rooted in Mechanistic Interpretability comprising five diverse lenses. We apply it to examine eight looped LMs from four families. Models differ in many aspects but the key one is whether the prompt is transmitted only within the looped state (carry loops: Ouro, LoopUS) or re-supplied at every iteration (re-read loops: Parcae, and Rec-OLMo & Huginn from the Raven family). The results of our study provide valuable insights concerning safety and efficient use of looped language models.
Pawel Olszowiec, Paulina Tomaszewska, Maciej Żelaszczyk +2
Samsung AI Center, Warsaw, Poland · IFTR, Polish Academy of Sciences, Warsaw, Poland · University of Warsaw, Poland
Repeating a small block of middle layers increases a language model's effective inference depth without adding parameters or generating extra tokens, and recent work shows that this latent recurrence improves reasoning. However, two design choices limit these gains. Each iteration sees only the previous output and cannot directly access earlier computations. Moreover, a fixed loop count wastes depth on easy inputs while leaving hard ones with too little computation. We introduce RecurTrace, which addresses both limitations using the loop's own trajectory. Specifically, Loop Memory Attention lets each looped layer attend to its own states from previous iterations along the loop-time axis, so the model can revisit earlier computations instead of relying on the latest state alone. A halting head then reads the loop state and predicts whether to continue, with supervision from an oracle that identifies when additional depth still reduces loss. In a controlled MathQA comparison on the same looped backbone, RecurTrace achieves 56.9% accuracy with an average of 2.0 loops, exceeding the best fixed loop depth by 2.2 points at matched compute. By comparison, ACT and PonderNet collapse to one loop, and CALM reaches only 54.1% with 5.6 loops, while the stronger LoopUS-Conf and TaH-Mismatch baselines reach 55.3% at 3.2 loops and 55.7% at 2.1 loops. Finally, RecurTrace improves generation accuracy over same-budget fine-tuned baselines at 0.6B, 1.7B, 4B, and 8B, with the gain growing with model size from 0.6 to 3.4 points.
Yuxiang Wang, Kunyu Feng, Yingda Shen +3
The Chinese University of Hong Kong, Shenzhen · The Chinese University of Hong Kong · Tianjin University +2