A Mechanistic Analysis of Looped Reasoning Language Models

Hugh Blayney; Álvaro Arroyo; Johan Obando-Ceron; Pablo Samuel Castro; Aaron Courville; Michael M. Bronstein; Xiaowen Dong

Model Name	Structure	Train Loops	Block Norm	Norm After Loop?
Ouro 1.4B ( Zhu et al., 2025 )	$(0,24,0)$	4	$\begin{aligned} \hat{{\bm{X}}}&={\bm{X}}+n\left(\attn(n({\bm{X}})\right)\\ {{\bm{X}}}^{\prime}&=\hat{{\bm{X}}}+n\left(\text{MLP}(n(\hat{{\bm{X}}}))\right)\\ \end{aligned}$	Yes
Huginn-0125 ( Geiping et al., 2025 )	$(2,4,2)_{I}$	32	$\begin{aligned} \hat{{\bm{X}}}&=n\left({\bm{X}}+\attn(n({\bm{X}})\right)\\ {{\bm{X}}}^{\prime}&=n\left(\hat{{\bm{X}}}+\text{MLP}(n(\hat{{\bm{X}}}))\right)\end{aligned}$	Yes
Retrofitted ( McLeish et al., 2025 )	$(4,6,4)_{I}$ / $(4,8,4)_{I}$	32	$\begin{aligned} \hat{{\bm{X}}}&={\bm{X}}+\attn(n({\bm{X}})\\ {{\bm{X}}}^{\prime}&=\hat{{\bm{X}}}+\text{MLP}(n(\hat{{\bm{X}}}))\end{aligned}$	No

Appendix

Table 1: Looped Transformer architecture summary. Retrofitted Llama and OLMo use 6 layers in their recurrent block, TinyLlama uses 8.

Model	Num Params	Huggingface ID	Base Model	Prepends BOS	Notes
Ouro 1.4B	1.4B	ByteDance/Ouro-1.4B	-	$\crossproduct$	-
Ouro 2.6B	2.6B	ByteDance/Ouro-2.6B	ByteDance/Ouro-1.4B	$\crossproduct$	“Upcycled” from Ouro 1.4B by repeating the same layers after the first training phase.
Huginn-0125	3.5B	tomg-group-umd/huginn-0125	-	$\checkmark$	-
Retrofitted Llama	1B	smcleish/Recurrent-Llama-3.2-train-recurrence-32	meta-llama/Llama-3.2-1B	$\checkmark$	Models trained with fewer recurrences exhibit similar mixing patterns.
Retrofitted OLMo-2	1B	smcleish/Recurrent-OLMo-2-0425-train-recurrence-32	allenai/OLMo-2-0425-1B	$\crossproduct$	As above.
Retrofitted TinyLlama	0.8B	smcleish/Recurrent-TinyLlama-3T-train-recurrence-32	TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T	$\checkmark$	As above.

Appendix

Table 2: Additional Huggingface details on pretrained Looped models used.

Figure 13: Visualizing the component parts of the Orbit detection algorithm of Algorithm 1 . The input sequence (cosine similarities for the residual streams of a given token and layer and successive recursions, as compared to their final residual stream) is visualized in the leftmost column. The center column visualizes the effect of windowing and de-trending, and the rightmost column shows the FFT magnitudes. The top row visualizes the detected Orbit with the largest amplitude for the Huginn-0125 model (which occurs with the “Long Persona” prompt) and the bottom row visualizes the largest amplitude for the Retrofitted Llama model (which occurs with no system prompt). We note the appearance of complex, multi-frequency oscillation in this latter case.

Model	System Prompt	Non-Fixed-Point %	Orbit %	Slider %	Unknown %
Huginn	Long Persona	0.14	0.13	0.00	0.01
Huginn	Long Persona (Padded)	0.05	0.02	0.02	0.01
Huginn	No System Prompt	0.02	0.01	0.01	0.00
Huginn	Short Math	0.01	0.00	0.00	0.00
Retrofitted-Llama	Long Persona	0.00	0.00	0.00	0.00
Retrofitted-Llama	Long Persona (Padded)	0.00	0.00	0.00	0.00

Appendix

Table 3: Percentage of tokens exhibiting each behavior type, by model and system prompt.

Model	System Prompt	Non-Fixed-Point %	Orbit %	Slider %	Unknown %
Huginn	Long Persona	2.81	2.50	0.15	1.06
Huginn	Long Persona (Padded)	0.83	0.23	0.61	0.15
Huginn	No System Prompt	0.76	0.53	0.15	0.08
Huginn	Short Math	0.45	0.38	0.08	0.00
Retrofitted-Llama	Long Persona	0.00	0.00	0.00	0.00
Retrofitted-Llama	Long Persona (Padded)	0.00	0.00	0.00	0.00

Appendix

Table 4: Percentage of examples exhibiting each behavior type at least once on any question token, by model and system prompt.

Figure 14: Conditional probabilities of co-occurrence for the different limiting behaviors.

Figure 15: PCA trajectories in the intermediate layers of Huginn-0125: this reproduces the leftmost column of Fig. 16 in Geiping et al. (2025) (the first two principal components) and additionally plots the latent trajectories for the intermediate layers in the recurrent block.

Figure 16: Stages of inference metrics (sink rate, mixing score and colsum concentration) for Huginn-0125 across 128 recurrences, for the GSM8k test prompt that exhibited the largest orbit amplitude. Visualized in realized depth.

Figure 17: Stages of inference metrics (sink rate, mixing score and colsum concentration) for Huginn-0125 across 128 recurrences, for the GSM8k test prompt that exhibited the largest orbit amplitude: only the final 64 recurrences are visualized, to isolate the impact of the orbit. Visualized in percentage block depth.

Figure 18: Stages of inference metrics (sink rate, mixing score and colsum concentration) for Retrofitted Llama across 128 recurrences, for the GSM8k test prompt that exhibited the largest orbit amplitude: only the final 64 recurrences are visualized, to isolate the impact of the orbit. Visualized in percentage block depth.

Figure 19: Cosine similarity between residual streams after every pair of layers for different Transformer models , averaged across the batch and sequence dimensions. Left: Ouro 1.4B ( Zhu et al., 2025 ) . Center: Retrofitted Llama ( McLeish et al., 2025 ) . Right: Huginn-0125 ( Geiping et al., 2025 ) . All models looped 8 times. Diagonal patterns indicate that the residual stream after each sub-block is most similar to the same block in the next recurrence; every layer in the recurrent block reaches a different fixed point.

Figure 20: Cosine similarity between residual streams after every pair of layers for different Transformer models, averaged across the batch and sequence dimensions. Left: Huginn-0125 ( Geiping et al., 2025 ) . Center Left: Retrofitted Llama. Center Right: Retrofitted OLMo. Right: Retrofitted TinyLlama ( McLeish et al., 2025 ) . All models looped 32 times. Extended version of Fig. 19 .

Figure 21: Frobenius norm between attention matrices for different Transformer models, averaged across the batch and head dimensions. Left: Huginn-0125 ( Geiping et al., 2025 ) . Center Left: Retrofitted Llama. Center Right: Retrofitted OLMo. Right: Retrofitted TinyLlama ( McLeish et al., 2025 ) . All models looped 32 times. Extended version of Fig. 2 .

Figure 22: Cosine similarity between the residual stream after each layer in the recurrent block and its “approximate fixed point” - the residual stream after that layer in the 128th recursion. While Huginn-0125 and retrofitted Llama quickly reach a fixed point, Ouro does not - even though the cosine similarity between successive recursions tends towards one, as evidenced by Fig. 22 .

Figure 23: Frobenius norm between attention matrices of each layer in the recurrent block and their corresponding “approximate fixed point” ; the attention matrices of the same layer in the 128th recursion. While Huginn-0125 and retrofitted Llama quickly reach a fixed point, Ouro does not.

Figure 24: Cosine similarities between successive recursions of the residual stream after the same layer.

Figure 25: Frobenius norm between attention matrices of each layer between successive recurrences.

Figure 26: Ouro 1.4B ( Zhu et al., 2025 ) latent space trajectory traced out by the hidden states of the final sequence position on the test prompt; reduced to two dimensions by computing PCA over all final sequence position embeddings.

Figure 27: Huginn-0125 ( Geiping et al., 2025 ) latent space trajectory traced out by the hidden states of the final sequence position on the test prompt; reduced to two dimensions by computing PCA over all final sequence position embeddings.

Figure 28: Ouro 1.4B ( Zhu et al., 2025 ) latent space trajectory traced out by the hidden states of the final sequence position on a “maths” test prompt ( The square root of 16 is ); reduced to two dimensions by computing PCA over all final sequence position embeddings.

Figure 29: Norm of the difference between the residual stream after the first layer in the recurrent block for successive recurrences and an “approximate fixed point” – the residual stream in the 128th recurrence. Two fixed point differences are visualized: the difference to the fixed point of the same (first) layer (blue) and the difference to the fixed point which has the greatest norm difference from the first layer (green). Visualized are a range of norm structures (columns), with input injection (top row) and without (bottom row). All models are randomly initialized with 12 layers in the recurrent block, and no prelude or coda.

Figure 30: Cosine similarity between the residual stream after the first layer in the recurrent block for successive recurrences and an “approximate fixed point” – the residual stream in the 128th recurrence. Two fixed point differences are visualized: the difference to the fixed point of the same (first) layer (blue) and the difference to the fixed point which has the lowest cosine similarity to the first layer (green). Visualized are a range of norm structures (columns), with input injection (top row) and without (bottom row). All models are randomly initialized with 12 layers in the recurrent block, and no prelude or coda.

Figure 31: Cosine similarity between the residual stream after the first layer in the recurrent block for successive recurrences and an “approximate fixed point” – the residual stream in the 128th recurrence. Two fixed point differences are visualized: the difference to the fixed point of the same (first) layer (blue) and the difference to the fixed point which has the lowest cosine similarity to the first layer (green). Visualized are a range of norm structures (columns), with input injection (top row) and without (bottom row). Here models of 4 and 16 layers are compared, showing qualitatively identical behavior and demonstrating that the behavior is not particular to 12 layers.

Figure 32: Fraction of prediction and suppression neurons in a selection of feedforward models used throughout the paper .

Figure 33: Fraction of prediction and suppression neurons in a selection of looped models used throughout the paper .

Figure 34: Stages of inference for a selection of Looped transformers, all using 8 recurrences : Huginn-0125 ( Geiping et al., 2025 ) , Ouro 1.4B ( Zhu et al., 2025 ) and Llama with retrofitted recurrences ( McLeish et al., 2025 ) . Note Huginn-0125 and Retrofitted Llama have prelude and coda layers too: each 2 layers in Huginn-0125 and each 4 in Retrofitted Llama.

Figure 35: Stages of inference for each recurrent loop in Ouro 1.4B. The close overlap with feedforward stages of inference is a particularly striking result as this model is trained from scratch with recurrence.

Figure 36: Stages of inference for each recurrent loop in Huginn-0125. This represents a negative result: stages of inference do not occur. We discuss possible causes for this in the main text.

Figure 37: Stages of inference for each recurrent loop in the retrofitted Llama model ( McLeish et al., 2025 ) . Each block demonstrates very similar stages of inference to Llama, the base model from which pretrained layers are taken.

Figure 38: Stages of inference for each recurrent loop in the retrofitted OLMo model ( McLeish et al., 2025 ) . Similarly, each block demonstrates very similar stages of inference to OLMo, the base model from which pretrained layers are taken.

Figure 39: Stages of inference for each recurrent loop in the retrofitted TinyLlama model ( McLeish et al., 2025 ) . Similarly, each block demonstrates very similar stages of inference to TinyLlama, the base model from which pretrained layers are taken.

Figure 40: Stages of inference for each recurrent loop in Ouro 2.6B. For this model we separate out the first and second half of the recurrent block and overlay them, demonstrating that both halves have close alignment with the Llama feedforward stages of inference. We suggest that this likely arises due to the training regime of Zhu et al. (2025) , which first trains a single 24 layer 1.4B parameter model, and then “upcycles” this into a 48 layer 2.6B parameter model by duplicating these layers, consequently duplicating the stages of inference as well.

Figure 41: Stages of inference for each recurrent loop in Retrofitted Llama for which the massive activations have been ablated.

Figure 42: Stages of inference for each recurrent loop in Ouro 1.4B, run on the HellaSwag dataset.

Figure 43: Stages of inference for each recurrent loop in Huginn-0125, run on the HellaSwag dataset.

Figure 44: Stages of inference for each recurrent loop in the retrofitted Llama model ( McLeish et al., 2025 ) , run on the HellaSwag dataset.

Figure 45: Stages of inference for each recurrent loop in the retrofitted OLMo model ( McLeish et al., 2025 ) , run on the HellaSwag dataset. ColSum concentration deviates slightly from its GSM8k counterpart here, but still broadly follows the same stages of inference as the feedforward OLMo model.

Figure 46: Stages of inference for each recurrent loop in the retrofitted TinyLlama model ( McLeish et al., 2025 ) , run on the HellaSwag dataset.

Figure 47: Stages of inference for each of the distinct blocks in Ouro ( Zhu et al., 2025 ) , as they are reapplied throughout the model for 128 recurrences. These consistently change throughout the realized depth of the model, reaching no clear fixed point. Mean and standard deviation are over separate inputs to the model, taken over the GSM8k subset.

Figure 48: Stages of inference for each of the distinct blocks in Huginn-0125 ( Geiping et al., 2025 ) , as they are reapplied throughout the model for 128 recurrences. These converge to constant behavior. Mean and standard deviation are over separate inputs to the model, taken over the GSM8k subset.

Figure 49: Stages of inference for each of the distinct blocks in retrofitted Llama ( McLeish et al., 2025 ) , as they are reapplied throughout the model for 128 recurrences. These converge to constant behavior. Mean and standard deviation are over separate inputs to the model, taken over the GSM8k subset.

Figure 50: Stages of inference for Ouro with each of 128 recurrences , visualized over percentage recurrent depth. These consistently change with successive recurrences, deviating significantly from the stages of inference seen with train-time recurrences.

Figure 51: Stages of inference for Huginn-0125 with each of 128 recurrences , visualized over percentage recurrent depth. These quickly reach a fixed point and do not deviate far from their starting stages of inference.

Figure 52: Stages of inference for retrofitted Llama with each of 128 recurrences , visualized over percentage recurrent depth. These quickly reach a fixed point and do not deviate far from their starting stages of inference.

Figure 53: Stages of inference for retrofitted OLMo-2 with each of 128 recurrences , visualized over percentage recurrent depth. These quickly reach a fixed point and do not deviate far from their starting stages of inference.