Looped language models (LMs) reuse one block of layers several times per token generation, promising more latent computation at no additional storage cost. Despite their rising popularity, the analysis of looped LMs is still in its infancy: little is known about whether the iterative transformations actually improve their capabilities, or whether these models remain stable under prompt perturbation. We construct an inspection protocol rooted in Mechanistic Interpretability comprising five diverse lenses. We apply it to examine eight looped LMs from four families. Models differ in many aspects but the key one is whether the prompt is transmitted only within the looped state (carry loops: Ouro, LoopUS) or re-supplied at every iteration (re-read loops: Parcae, and Rec-OLMo & Huginn from the Raven family). The results of our study provide valuable insights concerning safety and efficient use of looped language models.
Figures & tables
Lenses (L1-L5)
All models (both architectures)
Carry loops (Ouro, LoopUS)
Re-read loops (Huginn, Rec-OLMo, Parcae)
Trajectory : do hidden states settle?
Harder problems do not change the dynamics
Settle only for large Ouro-2.6B or smoothed LoopUS
Settle at 98-100% of tokens
Commitment : when is the top-1 token final?
Token mostly decided at iteration 1-2
Exit gates fire late
Huginn decides gradually, other models - early
Readout : which iteration predicts best?
Majority vote never helps
Best depth ≤ training depth
Monotonous improvement, a plateau
Attention Heads : do heads change jobs across the loop?
Heads keep their roles
Small shared drift
Heads barely move
Noise Injection : does the loop recover from an injection at entry?
Isotropic noise rarely jailbreaks, at most at ≤ 4% of safety answers
Answers are intact up to α=1 for Ouro ( 0.5 for LoopUS). At α=2 requests are misread and refusals become unreliable. Collapse at α=4 (prompt amnesia, script switching)
Almost unchanged even at α=50
Table 1: Investigation protocol and findings through the five lenses.
Figure 1: Looped architectures (a) Carry loop: the prompt embedding is the starting state h0 , later iterations see only ht . (b) Re-read loop: a prelude embeds the prompt into e , which is supplied at every iteration and the loop starts from a random state s0 . Red boxes mark where Lens 5 perturbs each loop.
Model
Family
Loop
T run
trained depth
block layers
other layers
width
exit gate
noise enters at
Ouro-1.4B
Ouro
carry
6
4 (fixed)
24
–
2048
yes
embeddings
Ouro-1.4B-Thinking
Ouro
carry
6
4 (fixed)
24
–
2048
yes
embeddings
Ouro-2.6B
Ouro
carry
6
4 (fixed)
48
–
2048
yes
embeddings
Ouro-2.6B-Thinking
Ouro
carry
6
4 (fixed)
48
–
2048
yes
embeddings
LoopUS-1.7B
LoopUS
carry
20
≤ 20 (halting)
25
2 + 1
2048
yes
encoder output
Parcae-1.3B
Parcae
re-read
8
mean 8 (random)
8
8 + 8
1536
no
random start
Table 2: Model descriptions Loop : carry or re-read. T run : iterations per token at inference. Trained depth : fixed, adaptive, or random (mean shown). Block/other layers : number of layers inside the loop vs. non-iterated. Width : hidden state size. Exit gate : trained halting head. Noise enters at : tensor perturbed in Lens 5.
Figure 2: Lens 1: hidden-state trajectory and attention focus. (a) Share of tokens whose hidden state settles. (b) Top-1 flips per token. How often next most likely token changes along the loop. In (a) and (b) dots are means over MATH, arithmetic, harmful/benign prompts. Faded bars are the ranges. (c) Attention (log-scale) from the last prompt token to the answer span on HotpoQA, relative to uniform attention, against relative loop position t/(T−1) . The value of 1 corresponds to uniform attention. Solid lines are carry loops and dashed are re-read loops.
Figure 3: Lens 2: when a token is decided. Share of MATH tokens whose top-1 prediction is already settled vs. relative loop position for: (a) prompt tokens and (b) answer tokens. A curve that starts high means most decisions are made in the first iteration. A curve that climbs slowly means the loop keeps changing its mind. All four generative sets are shown in Fig. 8 .
Figure 4: Lens 3: how well each iteration reads the prompt. Results on MATH prompts, every iteration predicts the next prompt token. The X-axis represents the relative loop position. (a) Accuracy. (b) The log-loss relative to its value at the first iteration. A value below 1 means the loop has improved the prediction, a value above 1 suggests a deterioration in the answer. (c) The expected calibration error (ECE, see App. B ). Lower value is better.
Figure 5: Lens 4: do attention heads switch roles across iterations? For every head of the last layer of the looped block we compute the attention fingerprint (App. B ). (a) Movement ratio: how far fingerprints move across iterations relative to the gap between heads (value 0 is a fixed role, 1 means no identity). (b) Movement ratio after removing the drift shared by all heads. (c) Cross-prompt consistency of head changes: if the behavior of a head changes in the same way on every prompt (1) or differently on each prompt (0). Dots are means over the five datasets, faded bars are their ranges.
Figure 6: Lens 5: how the answers change when the loop’s starting state is injected with random noise. X-axis: noise amplitude α ( 0 denotes a clean run with no noise). (a) Word overlap with clean answer. (b) Broken answers on adversarial prompts 2 2 2 The higher share of broken outputs for Parcae (0.52) and Rec-OLMo (0.36), reflect that they are released as base models without specific instruction or safety tuning. Under greedy decoding such models might continue a request repetitively instead of answering it. . (c) Share of fluent, on-topic answers. Solid lines are carry loops, dashed are re-read loops.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
share of tokens that converge
top-1 flips per token
needle attention (× uniform)
Model
MATH
Arith.
Harm/Ben.
MATH
Arith.
Harm/Ben.
first iter.
last iter.
Ouro-1.4B
0.10
0.19
0.18
0.40
0.54
0.57
0.12
1.95
Ouro-1.4B-T
0.40
0.43
0.62
0.36
0.59
0.65
0.12
1.63
Ouro-2.6B
0.94
0.93
0.96
0.26
0.39
0.30
0.33
1.45
Ouro-2.6B-T
0.92
0.86
0.92
0.24
0.42
0.43
0.85
1.15
LoopUS
1.00
1.00
1.00
0.72
0.82
0.78
0.20
0.01
Appendix
Table 3: Lens 1 per prompt set: settling during token generation, flips and answer attention. Share of settling tokens, top-1 flips per token, and (HotpotQA) attention to the answer span at first and last iteration, for MATH, arithmetic, and harmful/benign. The ranking is the same on every set, so settling and flipping are model properties. Ouro-1.4B settles least (10-19%); all others settle on 86-100%. Attention to the answer grows significantly only in Ouro and Huginn, while LoopUS ends below uniform. Whether a loop focuses on the relevant input is model-specific, so it has to be measured and cannot be taken for granted from the loop design alone.
Figure 7: Does difficulty change loop dynamics? (a) Top-1 flips. (b) Settling share for MATH levels 1-5. (c) Top-1 changes per token for arithmetic problems with one-, two- and three-digit operands. An adaptive loop should work harder on harder inputs. The curves are flat: across five MATH levels, flips change by < 0.08 and settling by < 3 points. The one exception is Huginn on arithmetic (flips grow 30%). In investigated looped LLMs, difficulty is not visibly translated into loop dynamics and the extra depth is spent the same way on easy and hard problems. Adaptive computation would need to be trained explicitly.
answer tokens decided at iter. 1 (%)
iteration by which 90% are decided
prompt
post-dec.
Model
T
MATH
GSM8K
Arith.
Harm/Ben.
MATH
GSM8K
Arith.
Harm/Ben.
at iter. 1 (%)
KL share
Ouro-1.4B
6
82
85
82
70
3
2
3
5
66
0.77
Ouro-1.4B-T
6
83
82
78
61
2
2
3
4
63
0.73
Ouro-2.6B
6
89
90
92
83
2
2
1
2
65
0.82
Ouro-2.6B-T
6
90
89
87
73
2
2
2
3
66
0.80
LoopUS
20
34
36
29
32
2
2
2
2
39
0.35
Appendix
Table 4: Lens 2 per prompt set: decision timing and post-decision refinement. Per generative set: answer tokens decided at iteration 1. In subsequent columns are presented averaged values of: prompt tokens decided at iteration 1, post-decision KL share, and oracle compute savings. A high post-decision share means the loop mostly sharpens an already-made choice. Six models decide 61-93% at iteration 1, 73-88% of distribution change comes after the decision. LoopUS decides 90% by iteration 2 with the lowest post-decision share (0.35). Huginn needs iterations 8-11 (share 0.46).
Model
T
decision
gate fires
gap
Ouro-1.4B
6
1.79
5.86
4.07
Ouro-1.4B-T
6
1.76
5.58
3.82
Ouro-2.6B
6
1.44
4.78
3.34
Ouro-2.6B-T
6
1.51
4.71
3.20
LoopUS
20
2.05
18.66
16.62
Appendix
Table 5: Exit gates fire long after the answer is decided. Analysis performed only for gated models. The columns present the averaged values of: the decision iteration, gate-firing iteration, and the gap between these two. Both gates are very conservative: answers are fixed after 1.4-2.1 iterations on average, but gates wait until near the loop’s end. Ouro’s gate fires on averaged at iterations 3.2-4.1. LoopUS’s at iteration 18.7 of 20.
Figure 8: Decision timing for every generative set. Similarly to Fig. 3 , this presents the share of tokens with final top-1 at each relative loop position. At the top: statistics on generated answer tokens. At the bottom: statistics on prompt tokens. In the main part of the paper we present Fig. 3 on MATH dataset only. These plots confirm that the early decision generalizes to other datasets and is not specific to MATH prompts. On every set, most answer tokens of most models are decided at iteration 1. Huginn rises slowly, LoopUS jumps at iteration 2, Ouro, Parcae, and Rec-OLMo decide very early.
next-token accuracy
best iteration
log-loss
ECE
fixed
broken
Model
T
iter. 1
best
last
per set
last / first
iter. 1
last
(%)
(%)
Ouro-1.4B
6
0.534
0.605
0.580
4,3,4,4,4
0.73
0.181
0.087
7.0
2.4
Ouro-1.4B-T
6
0.484
0.505
0.463
1,5,1,1,5
1.05
0.132
0.181
7.0
9.1
Ouro-2.6B
6
0.561
0.604
0.602
4,4,4,4,4
0.80
0.164
0.080
7.1
3.1
Ouro-2.6B-T
6
0.476
0.486
0.452
2,2,1,1,4
1.22
0.136
0.213
3.9
6.3
LoopUS
20
0.240
0.421
0.420
12,7,2,2,6
0.48
0.375
0.096
20.1
2.1
Appendix
Table 6: Lens 3 in numbers: is the last iteration the best reader? All values on prompt tokens, averaged over five sets. Next-token accuracy at first, best, and last iteration. Best iteration per set presents for datasets MATH, GSM8K, Arithmetic, Harm/Benign, HotpotQA the iteration (counted from 1) with the highest accuracy. Log-loss last/first : value below 1 means that additional loops help. ECE is the expected calibration error at the first and last iteration. Fixed ( broken ): share of tokens wrong (right) at iteration 1 but right (wrong) at the end. Prompt tokens give a fair test of every iteration. A loop works best when the best and the last accuracy agree, log-loss is well below 1 and percentage of "fixed" prompts exceeds "broken" which is true for base Ouro, LoopUS, Huginn and false for both Ouro-Thinking models. LoopUS and Huginn fix ∼ 20% of tokens while breaking only 2%.
Figure 9: Prompt-reading accuracy per set. Plots at the top: next-token accuracy on prompt tokens. At the bottom: expected calibration error (ECE, lower is better). Both presented along the relative position in the loop on five datasets. There are three main-text patterns, i.e. peaking at training depth, overthinking after reasoning tuning, monotone improvement in re-read loops which hold on every dataset. The arithmetic task is least informative due to short and predictable prompts. HotpotQA plots depicts the largest gains (Huginn 0.30 → 0.56, LoopUS 0.23 → 0.47). We conclude that the choice of the best readout iteration is a model-dependent artifact.
Model
MATH
GSM8K
Arith.
Harm/Ben.
HotpotQA
Ouro-1.4B
0.63/0.70/0.68
0.53/0.59/0.58
0.55/0.61/0.56
0.51/0.57/0.54
0.45/0.56/0.54
Ouro-1.4B-T
0.60/0.60/0.57
0.49/0.52/0.51
0.47/0.47/0.38
0.46/0.46/0.40
0.40/0.48/0.46
Ouro-2.6B
0.66/0.71/0.70
0.56/0.60/0.60
0.57/0.57/0.57
0.53/0.58/0.57
0.49/0.57/0.56
Ouro-2.6B-T
0.59/0.60/0.56
0.50/0.52/0.50
0.41/0.41/0.35
0.41/0.41/0.37
0.46/0.49/0.47
LoopUS
0.36/0.56/0.56
0.20/0.48/0.48
0.25/0.27/0.26
0.17/0.32/0.32
0.23/0.47/0.47
Parcae
0.43/0.48/0.48
0.41/0.47/0.46
0.29/0.29/0.28
0.31/0.34/0.34
0.40/0.44/0.44
Appendix
Table 7: Prompt-token accuracy per set: at first, best and last iterations. Each cell presents three accuracies on prompt tokens, at the first iteration, at the best iteration and at the last iteration on the five datasets. The "best-last" gap is what a model loses by reading at the end. The "best-first" gap is the loop’s maximum gain. For re-read loops, best and last agree within 0.01 on every set, they never lose by looping longer. For carry loops they often differ. Results show strong dependence on the model and the task. However, looping to the end is not harmful only for the re-read models.
ECE: first / lowest (iteration) / last
oracle
majority
Model
MATH
GSM8K
Arith.
accuracy
vote
Ouro-1.4B
0.129 / 0.039 (4) / 0.054
0.192 / 0.058 (4) / 0.062
0.255 / 0.105 (5) / 0.192
0.651
0.604
Ouro-1.4B-T
0.081 / 0.081 (1) / 0.146
0.081 / 0.081 (1) / 0.133
0.257 / 0.230 (3) / 0.266
0.568
0.494
Ouro-2.6B
0.102 / 0.034 (3) / 0.053
0.140 / 0.045 (5) / 0.050
0.279 / 0.105 (5) / 0.155
0.646
0.604
Ouro-2.6B-T
0.102 / 0.102 (1) / 0.169
0.103 / 0.103 (1) / 0.161
0.255 / 0.255 (1) / 0.331
0.545
0.477
LoopUS
0.270 / 0.062 (5) / 0.065
0.416 / 0.043 (19) / 0.043
0.478 / 0.165 (7) / 0.221
0.449
0.420
Appendix
Table 8: Calibration along the loop and iteration-combination rules. The expected calibration error (ECE) at the first iteration, at the best-calibrated iteration (in brackets) when ECE is the lowest and at the last iteration. Results on prompts from MATH, GSM8K and arithmetic datasets. The last two columns are averaged over all five datasets and present the: (a) oracle accuracy , i.e. a token counts as correct if any iteration predicts it, and (b) the accuracy based on a majority vote over the top-1 tokens of all iterations. If different iterations make different mistakes, then combining all iterations by an oracle or majority voting might beat the last iteration. We note that the best-calibrated iteration is rarely the last for carry loops. The right answer is often visited, then abandoned, but the majority vote approach does not recover it. The oracle accuracy is 3-8 points above the best single iteration, but the vote never beats it. We conclude that there is useful information in the intermediate iterations, the extraction of which would require training a selecting model rather than simply averaging or voting.
movement ratio (raw)
drift-
weakest
role
cross-
corr. with
Model
MATH
GSM8K
Arith.
Harm/Ben.
HotpotQA
removed
z
churn
prompt
state drift
Ouro-1.4B
0.40
0.43
0.42
0.40
0.52
0.24
-20
0.06
0.95
0.95
Ouro-1.4B-T
0.33
0.33
0.36
0.38
0.40
0.19
-20
0.12
0.96
0.96
Ouro-2.6B
0.29
0.31
0.30
0.27
0.36
0.14
-22
0.12
0.95
0.94
Ouro-2.6B-T
0.28
0.28
0.29
0.28
0.31
0.12
-22
0.09
0.97
0.95
LoopUS
0.42
0.42
0.44
0.41
0.43
0.24
-59
0.02
0.98
0.99
Appendix
Table 9: Lens 4 per prompt set: heads keep their roles.
Figure 10: Attention fingerprints: one row per head, one column per iteration. Raw fingerprints for MATH prompts, before any standardization. Each small heat map is one of the eight features (for example attention to the first token, to the previous token, or the entropy of the attention) and rows are the heads of the last layer of the looped block, columns are iterations. Top: Ouro-1.4B (6 iterations). Bottom: Huginn (16 iterations). The summary numbers of Lens 4 do not give a broad view as much as the raw maps which show directly what a head keeps its role means. Both maps are horizontal stripes: each head keeps its own feature mix from first to last iteration. In Ouro-1.4B the one visible change (first-token attention jumping from zero to its final level) is the shared drift subtracted in the summary ratios. Huginn’s stripes are even more uniform. A looped block’s heads are best described by one fixed role each plus a small common adjustment at the loop’s start.
Figure 11: Effect of noise on safety-prompt answers. Stacked bars on three safety datasets (AdvBench, CValues, WildChat jailbreaks). For carry loops: answers with no noise injected (clean) and with noise of amplitude α=1 , 2 and 8 times. For re-read loops: noise-free and with noise amplitutde α=8 , on exactly the same prompts. Each answer is sorted into one of four groups: refuses , on-topic without a refusal phrase , fluent but off-topic , broken . Up to α=1 the carry loops answer as they do when clean. At α=2 the refusals of Ouro-1.4B shrink from 73% to 24%, but only a minority of the new on-topic non-refusal answers follow the harmful request (Sec. 4.5 ). The Thinking Ouro models refuse more, because they skip their reasoning and answer with a one-line refusal. At α=8 noise removes refusals from carry loops but produces broken or off-topic text, not harmful answers: e.g. clean Ouro-1.4B/2.6B refuse ∼ 70% and under noise injection 63-73% of the answers become broken. Re-read loops are unaffected.
Model
measure
clean
α=0.25
0.5
1
2
4
8
12
17
33
50
Ouro-1.4B
similarity
1.00
0.74
0.65
0.48
0.16
0.02
0.02
0.02
–
–
–
broken
0.02
0.02
0.01
0.03
0.29
0.76
0.72
0.66
–
–
–
fluent & on-topic
0.75
0.75
0.75
0.82
0.51
0.01
0.00
0.01
–
–
–
non-Latin script
0.01
0.00
0.00
0.00
0.00
0.02
0.03
0.03
–
–
–
refusal (safety)
0.73
0.72
0.72
0.66
0.25
0.00
0.01
0.00
–
–
–
reasoning trace (safety)
0.02
0.02
0.02
0.12
0.22
0.02
0.02
0.03
–
–
–
Appendix
Table 10: Noise sensitivity at every amplitude. Table presents six measures for noise-free runs and for each noise amplitude on three safety datasets (AdvBench, CValues, WildChat): word similarity, broken answers, fluent and on-topic answers, non-Latin-script answers, answers with a refusal phrase and answers that open a reasoning trace. We note that in carry loops every metrics collapses at amplitude α=4 . In re-read loops the metrics on clean and noisy runs are almost equal. For carry models we note that they keep ≤ 4% similarity, the broken text rises to 46-89% and refusals almost disappear. Re-read models retain 53-78% similarity even at α=50 .
Figure 13: Refusals and response styles of carry loops on safety prompts. Ratio of answers with different response styles depending on the noise amplitude. (a) Answers containing a refusal phrase. (b) Answers opening a reasoning trace e.g. "Okay, the user is asking …". (c) Answers starting with a refusal phrase. From α=0 up to α=1 nothing changes qualitatively. At α=2 we observe that the Ouro Thinking models stop reasoning and start their answer with a refusal - their refusal rate rises. On the other hand, Ouro-1.4B starts to reason and refuses less. For α≥4 , no carry loop refuses.
safety
follows
another
reasoning
role-
Model
prompts
read
harmful request
request
trace
play
other
Ouro-1.4B
300
73
12 (4.0%)
24
25
5
7
Ouro-2.6B
300
21
5 (1.7%)
12
0
0
4
Ouro-1.4B-T
200
27
6 (3.0%)
13
0
0
8
Ouro-2.6B-T
200
14
2 (1.0%)
2
0
0
10
Appendix
Table 11: What the noisy answers at α=2 say. We analyze the answers generated with noise at amplitude α=2 on prompts on which refusal disappeared (on-topic without a refusal phrase) in spite of noise-free generation being a refusal. Safety prompts : all prompts with generated answers at α=2 on the safety sets (AdvBench, CValues, WildChat). Read : answers selected (on-topic without a refusal phrase). Follows harmful request : the answer that does what was asked (percentage of all answers in brackets). Another request : a harmless answer to a misread or invented request. Reasoning trace : answers addressing the request but not reaching a conclusion within 128 tokens. Role-play : a WildChat-Jailbreak role-play whose content is not harmful but which the model refused to answer on a noise-free run. Other : echoed prompts, invented system prompts, hedges without a refusal phrase, harmless prompts answered helpfully and noisy answers whose noise-free generation counterparts were not refusals.
Figure 14: Does the loop amplify or forget a perturbation? This figure presents the change of amplification pertt/pert0 rate of the hidden-state perturbation along the subsequent iterations on AdvBench prompts. pertt/pert0 is the mean distance between the noisy and the clean hidden state at iteration t , divided by its value at the first iteration ( it0 ). A value above 1 means the difference between the two runs grows, a value below 1 suggest that the difference shrinks. The color of the line denotes the noise amplitude (the amplitudes differ between panels). We note that carry loops (a-c) amplify the perturbation, re-read loops (d-f) shrink it.
Figure 15: Exit gates and next-token distributions under noise injection on AdvBench dataset. Plots (a,b) present the averaged change - noisy run minus clean run on the same prompts as the percentage of tokens whose exit probability exceeds 0.5 at each iteration. Noise injection makes Ouro-2.6B less willing to stop the iterations. Conversely, it makes LoopUS more willing to exit the loop earlier. Plots (c,d) present the Jensen-Shannon divergence between the noise-free and the noisy next-token distribution at each generated token position. For the carry loop, the divergence reaches high values from the very first tokens, while for the re-read loop it starts low and grows as the generated answers drift apart.
Figure 16: Analysis of internal states at the last iteration of the loop. The figure presents the averages over all generated answer tokens of noisy runs over all the investigated datasets. (a) Value of the hidden state norm relative to the corresponding values of noise-free runs. (b) Decay ratio (see Lens 1, B ), computed from the mean step sizes ∥ht−ht−1∥ as the mean of the last w steps over the mean of the first w steps. Dotted horizontal line at value 0.25 corresponds to the chosen settling threshold. (c) Top-1 probability of the next-token distribution during the generation phase. (d) Attention of a generated token to itself. (e) Attention entropy (relative to noise-free runs). (f) Percentage of the variance of the hidden states on their first principal component. Parcae is not presented on purpose - in every run the first principal component of its state carries 47-99.8% of the whole variance. Carry loops are presented with solid lines and re-read loops with dashed lines. Carry loops do not change much up to α=1 , start to change significantly (bifurcate) between α=1 and α=2 and are in a completely different internal state for α≥4 for Ouro (in LoopUS’s case - for α≥8 ). Re-read loops stay close to their noise-free counterpart values even for large noise amplitudes α .
Figure 17: Evolution of hidden states along the iterations. Top plots present the norm of the hidden state along the iterations. Plots in the middle present the step length ∥ht−ht−1∥ . At the bottom - cosine similarity between consecutive states. One line in the plot is drawn per amplitude and the black line corresponds to noise-free runs. Ouro contracts the initial hidden states at every noise amplitude. For α≥8 , the LoopUS model does not shrink the steps and the norm of the hidden state grows with every iteration.
Figure 18: Next-token probability distribution along the loop. Top plots present entropy while bottom plots display the top-1 probability of the distribution read out at each iteration. For α≤2 , the first iterations of Ouro models are less certain than on noise-free runs. For α≥4 , Ouro models are more certain than noise-free runs at every iteration. For α≥8 , the LoopUS model becomes less certain with every iteration.
Figure 19: Top-1 token changes under noise injection. Plots at the top present the percentage of the generated tokens whose top-1 token at iteration t is the same as at t−1 . At the bottom: KL(pt∥pt−1) , the Kullback-Leibler divergence between consecutive next-token probability distributions after iteration t . For α≥4 , the Ouro models fix their top-1 token earlier than for noise-free runs. On the other hand, the LoopUS model decides early on the top-1 token. For α≥8 , it keeps changing its top-1 tokens along the loop.
Figure 20: Where does the generated token attend? The plots present the attention of a generated token to itself (top) and to the first generated token of an answer (bottom). The values are the averages over the heads of the last layer of the looped block. Attention is measured among the generated tokens only. For α≥4 , noise runs of Ouro models halve their self-attention in comparison to noise-free runs. Moreover, at the first iteration Ouro models put 13-43% of their attention on the first generated token. On the other hand, LoopUS shows a weaker and later version of both changes at α=8−17 .
Figure 21: Broadness and similarity of the heads’ attention. Plots at the top present attention entropy averaged over the heads. At the bottom - mean pairwise cosine similarity of their attention maps (agreement of the heads) at the first four iterations. For α≥4 , the heads of Ouro attend similarly at the first iteration (due to the attention sink - Fig. 20 ) and more broadly than noise-free runs.
Figure 22: Geometry of the hidden states. Plots at the top present the percentage of the variance of the hidden states, over all generated tokens, on their first principal component. At the bottom - the cosine similarity of the state after iteration t to the state after the first iteration. Under noise injection the states concentrate on fewer directions - the percentage of the variance on the first principal component rises two to three times with respect to the noise-free counterpart for Ouro from α≥4 . For α≥8 the hidden state of LoopUS moves away from its first iteration.
Figure 24: Exit gates under noise injection. Plots at the top: percentage of generated tokens whose exit probability (the sigmoid of the gate output) exceeds 0.5. At the bottom: mean exit probability. We note that the LoopUS model generates tokens with a default halting threshold of 0.9, but in our analysis we deliberately use a threshold 0.5 for all the three models. Under noise injection, the gate of Ouro-2.6B stops fewer tokens after the second iteration, despite its predictions becoming more confident, while the halting head of LoopUS stops more often.
Figure 25: Small noise injections: token by token analysis of answer drif. The plot presents the comparison of noisy runs with noise-free runs at the same position of the answer (for the sake of demonstration, we truncate the answers to 128 tokens). The presented values are averages over the loop iterations and the seven prompt sets with α=0.25 and α=0.5 . (a) For each token position - the percentage of prompts in which both runs (noise and noise-free) have the same top-1 token. (b) Jensen-Shannon divergence between noisy and noise-free runs on the next-token distributions. (c) Distance between hidden states relative to the norm of noise-free runs. We note that at the first positions the two runs nearly agree. Later, since every token that is decided differently changes the context of the subsequent tokens, the disagreement accumulates along the answer - faster for LoopUS, slower for Ouro.
Implicit reasoning in language models studies how models solve problems through internal computation rather than, or alongside, written chains of thought.
Reasoning has become a central capability in large language models. Recent research has shown that reasoning performance can be improved by looping an LLM's layers in the latent dimension, resulting in looped reasoning language models. Despite promising results, few works have investigated how their internal dynamics differ from those of standard feedforward models. In this paper, we conduct a mechanistic analysis of the latent states in looped language models, focusing in particular on how the stages of inference observed in feedforward models compare to those observed in looped ones. To this end, we analyze cyclic recurrence and show that for many of the studied models each layer in the cycle converges to a distinct fixed point; consequently, the recurrent block follows a consistent cyclic trajectory in the latent space. We provide evidence that as these fixed points are reached, attention-head behavior stabilizes, leading to constant behavior across recurrences. Empirically, we discover that recurrent blocks learn stages of inference that closely mirror those of feedforward models, repeating these stages in depth with each iteration. We study how recurrent block size, input injection, and normalization influence the emergence and stability of these cyclic fixed points. We believe these findings help translate mechanistic insights into practical guidance for architectural design.
Hugh Blayney, Álvaro Arroyo, Johan Obando-Ceron +4
University of Oxford · Mila – Quebec AI Institute · Universit´e de Montréal +1
Looped language models increase effective depth by repeatedly applying a shared block of layers, but existing large-scale recipes require multi-stage training over trillions of tokens, while the benefits of recurrence remain difficult to separate from differences in data and training. In this work, we establish practical training recipes for looped language models, with three main results. (1) We develop a compute-efficient from-scratch pipeline that reduces the training budget from 7.7T tokens in Ouro to 310B tokens while retaining strong reasoning performance. Pretraining followed by high-quality mid-training, together with learning-rate warmup and stronger exit-gate regularization, enables stable recurrent training without prior multi-stage schedules. (2) Under controlled comparisons, our 1.4B LoopLM outperforms a parameter-matched dense model trained on the same data and token budget on all 12 evaluated benchmarks, including +14 points on GSM8K, +10 on MATH, and +22 on DROP. At matched inference compute, it approaches a 3.9B dense model on mathematical reasoning and reading comprehension while using only 36% as many parameters. (3) We introduce a minimal recipe for converting pretrained dense models into looped ones: a single learned input-mixing scalar and a smoothed exit loss, with no step-specific parameters. Applied to Qwen3-1.7B-Base, Looped Qwen improves over an identically continued dense baseline on every evaluated benchmark across two data regimes, with statistically clear gains on GSM8K, MATH, and MMLU-Pro on the curated mixture. Together, these results make looped language models substantially cheaper to train from scratch and practical to introduce into existing pretrained checkpoints, while isolating the gains due to recurrence itself.
Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin +5
Chain-of-thought (CoT) monitoring provides a promising approach for detecting undesirable model behavior. Looped language models (LoopLMs) repeatedly apply shared transformer layers, increasing effective computational depth and enabling additional latent computation without increasing model size. However, the effect of looped architectures on CoT monitorability remains largely unexplored. In this work, we provide the first systematic evaluation of CoT monitorability in LoopLMs. We study two complementary settings: (1) varying the loop depth within the same LoopLM family to isolate the effect of additional recurrent computation, and (2) comparing LoopLMs with non-looped language models matched by parameter size, transformer-layer count, or effective depth to study whether LoopLMs are less monitorable. Across eight tasks from MonitorBench and both standard and stress-test settings, we observe task-dependent reductions in CoT monitorability under stress tests on specific Logic/Science/Engineering Cue Answer tasks, while other tasks exhibit weaker or qualitatively different trends. Our diagnosis suggests that these declines are not fully explained by task difficulty, verification pass rate, or generated token length; qualitative examples further suggest changes in how deeper-loop models explicitly use or attribute provided cues. Our cross-model comparison finds no evidence that LoopLMs are systematically less monitorable than non-looped language models matched on size or depth. Overall, our results suggest that deeper loop depth can reduce CoT monitorability in some tasks under stress tests, but looped transformer architecture alone does not necessarily imply lower monitorability.