cs.AIOct 13, 2026
SaveMonitoring latent reasoning in looped language models
Organizations: Samsung AI Center, Warsaw, Poland · IFTR, Polish Academy of Sciences, Warsaw, Poland · University of Warsaw, Poland
Abstract
Looped language models (LMs) reuse one block of layers several times per token generation, promising more latent computation at no additional storage cost. Despite their rising popularity, the analysis of looped LMs is still in its infancy: little is known about whether the iterative transformations actually improve their capabilities, or whether these models remain stable under prompt perturbation. We construct an inspection protocol rooted in Mechanistic Interpretability comprising five diverse lenses. We apply it to examine eight looped LMs from four families. Models differ in many aspects but the key one is whether the prompt is transmitted only within the looped state (carry loops: Ouro, LoopUS) or re-supplied at every iteration (re-read loops: Parcae, and Rec-OLMo & Huginn from the Raven family). The results of our study provide valuable insights concerning safety and efficient use of looped language models.
Figures & tables
| Lenses (L1-L5) | All models (both architectures) | Carry loops (Ouro, LoopUS) | Re-read loops (Huginn, Rec-OLMo, Parcae) |
|---|---|---|---|
| Trajectory : do hidden states settle? | Harder problems do not change the dynamics | Settle only for large Ouro-2.6B or smoothed LoopUS | Settle at 98-100% of tokens |
| Commitment : when is the top-1 token final? | Token mostly decided at iteration 1-2 | Exit gates fire late | Huginn decides gradually, other models - early |
| Readout : which iteration predicts best? | Majority vote never helps | Best depth training depth | Monotonous improvement, a plateau |
| Attention Heads : do heads change jobs across the loop? | Heads keep their roles | Small shared drift | Heads barely move |
| Noise Injection : does the loop recover from an injection at entry? | Isotropic noise rarely jailbreaks, at most at 4% of safety answers | Answers are intact up to for Ouro ( for LoopUS). At requests are misread and refusals become unreliable. Collapse at (prompt amnesia, script switching) | Almost unchanged even at |
Table 1: Investigation protocol and findings through the five lenses.
Figure 1: Looped architectures (a) Carry loop: the prompt embedding is the starting state , later iterations see only . (b) Re-read loop: a prelude embeds the prompt into , which is supplied at every iteration and the loop starts from a random state . Red boxes mark where Lens 5 perturbs each loop.
| Model | Family | Loop | run | trained depth | block layers | other layers | width | exit gate | noise enters at |
| Ouro-1.4B | Ouro | carry | 6 | 4 (fixed) | 24 | – | 2048 | yes | embeddings |
| Ouro-1.4B-Thinking | Ouro | carry | 6 | 4 (fixed) | 24 | – | 2048 | yes | embeddings |
| Ouro-2.6B | Ouro | carry | 6 | 4 (fixed) | 48 | – | 2048 | yes | embeddings |
| Ouro-2.6B-Thinking | Ouro | carry | 6 | 4 (fixed) | 48 | – | 2048 | yes | embeddings |
| LoopUS-1.7B | LoopUS | carry | 20 | 20 (halting) | 25 | 2 + 1 | 2048 | yes | encoder output |
| Parcae-1.3B | Parcae | re-read | 8 | mean 8 (random) | 8 | 8 + 8 | 1536 | no | random start |
Table 2: Model descriptions Loop : carry or re-read. run : iterations per token at inference. Trained depth : fixed, adaptive, or random (mean shown). Block/other layers : number of layers inside the loop vs. non-iterated. Width : hidden state size. Exit gate : trained halting head. Noise enters at : tensor perturbed in Lens 5.
Figure 2: Lens 1: hidden-state trajectory and attention focus. (a) Share of tokens whose hidden state settles. (b) Top-1 flips per token. How often next most likely token changes along the loop. In (a) and (b) dots are means over MATH, arithmetic, harmful/benign prompts. Faded bars are the ranges. (c) Attention (log-scale) from the last prompt token to the answer span on HotpoQA, relative to uniform attention, against relative loop position . The value of corresponds to uniform attention. Solid lines are carry loops and dashed are re-read loops.
Figure 3: Lens 2: when a token is decided. Share of MATH tokens whose top-1 prediction is already settled vs. relative loop position for: (a) prompt tokens and (b) answer tokens. A curve that starts high means most decisions are made in the first iteration. A curve that climbs slowly means the loop keeps changing its mind. All four generative sets are shown in Fig. 8 .
Figure 4: Lens 3: how well each iteration reads the prompt. Results on MATH prompts, every iteration predicts the next prompt token. The X-axis represents the relative loop position. (a) Accuracy. (b) The log-loss relative to its value at the first iteration. A value below 1 means the loop has improved the prediction, a value above 1 suggests a deterioration in the answer. (c) The expected calibration error (ECE, see App. B ). Lower value is better.
Figure 5: Lens 4: do attention heads switch roles across iterations? For every head of the last layer of the looped block we compute the attention fingerprint (App. B ). (a) Movement ratio: how far fingerprints move across iterations relative to the gap between heads (value 0 is a fixed role, 1 means no identity). (b) Movement ratio after removing the drift shared by all heads. (c) Cross-prompt consistency of head changes: if the behavior of a head changes in the same way on every prompt (1) or differently on each prompt (0). Dots are means over the five datasets, faded bars are their ranges.
Figure 6: Lens 5: how the answers change when the loop’s starting state is injected with random noise. X-axis: noise amplitude ( denotes a clean run with no noise). (a) Word overlap with clean answer. (b) Broken answers on adversarial prompts 2 2 2 The higher share of broken outputs for Parcae (0.52) and Rec-OLMo (0.36), reflect that they are released as base models without specific instruction or safety tuning. Under greedy decoding such models might continue a request repetitively instead of answering it. . (c) Share of fluent, on-topic answers. Solid lines are carry loops, dashed are re-read loops.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| share of tokens that converge | top-1 flips per token | needle attention (× uniform) | ||||||
| Model | MATH | Arith. | Harm/Ben. | MATH | Arith. | Harm/Ben. | first iter. | last iter. |
| Ouro-1.4B | 0.10 | 0.19 | 0.18 | 0.40 | 0.54 | 0.57 | 0.12 | 1.95 |
| Ouro-1.4B-T | 0.40 | 0.43 | 0.62 | 0.36 | 0.59 | 0.65 | 0.12 | 1.63 |
| Ouro-2.6B | 0.94 | 0.93 | 0.96 | 0.26 | 0.39 | 0.30 | 0.33 | 1.45 |
| Ouro-2.6B-T | 0.92 | 0.86 | 0.92 | 0.24 | 0.42 | 0.43 | 0.85 | 1.15 |
| LoopUS | 1.00 | 1.00 | 1.00 | 0.72 | 0.82 | 0.78 | 0.20 | 0.01 |
Table 3: Lens 1 per prompt set: settling during token generation, flips and answer attention. Share of settling tokens, top-1 flips per token, and (HotpotQA) attention to the answer span at first and last iteration, for MATH, arithmetic, and harmful/benign. The ranking is the same on every set, so settling and flipping are model properties. Ouro-1.4B settles least (10-19%); all others settle on 86-100%. Attention to the answer grows significantly only in Ouro and Huginn, while LoopUS ends below uniform. Whether a loop focuses on the relevant input is model-specific, so it has to be measured and cannot be taken for granted from the loop design alone.
Figure 7: Does difficulty change loop dynamics? (a) Top-1 flips. (b) Settling share for MATH levels 1-5. (c) Top-1 changes per token for arithmetic problems with one-, two- and three-digit operands. An adaptive loop should work harder on harder inputs. The curves are flat: across five MATH levels, flips change by 0.08 and settling by 3 points. The one exception is Huginn on arithmetic (flips grow 30%). In investigated looped LLMs, difficulty is not visibly translated into loop dynamics and the extra depth is spent the same way on easy and hard problems. Adaptive computation would need to be trained explicitly.
| answer tokens decided at iter. 1 (%) | iteration by which 90% are decided | prompt | post-dec. | ||||||||
| Model | MATH | GSM8K | Arith. | Harm/Ben. | MATH | GSM8K | Arith. | Harm/Ben. | at iter. 1 (%) | KL share | |
| Ouro-1.4B | 6 | 82 | 85 | 82 | 70 | 3 | 2 | 3 | 5 | 66 | 0.77 |
| Ouro-1.4B-T | 6 | 83 | 82 | 78 | 61 | 2 | 2 | 3 | 4 | 63 | 0.73 |
| Ouro-2.6B | 6 | 89 | 90 | 92 | 83 | 2 | 2 | 1 | 2 | 65 | 0.82 |
| Ouro-2.6B-T | 6 | 90 | 89 | 87 | 73 | 2 | 2 | 2 | 3 | 66 | 0.80 |
| LoopUS | 20 | 34 | 36 | 29 | 32 | 2 | 2 | 2 | 2 | 39 | 0.35 |
Table 4: Lens 2 per prompt set: decision timing and post-decision refinement. Per generative set: answer tokens decided at iteration 1. In subsequent columns are presented averaged values of: prompt tokens decided at iteration 1, post-decision KL share, and oracle compute savings. A high post-decision share means the loop mostly sharpens an already-made choice. Six models decide 61-93% at iteration 1, 73-88% of distribution change comes after the decision. LoopUS decides 90% by iteration 2 with the lowest post-decision share (0.35). Huginn needs iterations 8-11 (share 0.46).
| Model | decision | gate fires | gap | |
|---|---|---|---|---|
| Ouro-1.4B | 6 | 1.79 | 5.86 | 4.07 |
| Ouro-1.4B-T | 6 | 1.76 | 5.58 | 3.82 |
| Ouro-2.6B | 6 | 1.44 | 4.78 | 3.34 |
| Ouro-2.6B-T | 6 | 1.51 | 4.71 | 3.20 |
| LoopUS | 20 | 2.05 | 18.66 | 16.62 |
Table 5: Exit gates fire long after the answer is decided. Analysis performed only for gated models. The columns present the averaged values of: the decision iteration, gate-firing iteration, and the gap between these two. Both gates are very conservative: answers are fixed after 1.4-2.1 iterations on average, but gates wait until near the loop’s end. Ouro’s gate fires on averaged at iterations 3.2-4.1. LoopUS’s at iteration 18.7 of 20.
Figure 8: Decision timing for every generative set. Similarly to Fig. 3 , this presents the share of tokens with final top-1 at each relative loop position. At the top: statistics on generated answer tokens. At the bottom: statistics on prompt tokens. In the main part of the paper we present Fig. 3 on MATH dataset only. These plots confirm that the early decision generalizes to other datasets and is not specific to MATH prompts. On every set, most answer tokens of most models are decided at iteration 1. Huginn rises slowly, LoopUS jumps at iteration 2, Ouro, Parcae, and Rec-OLMo decide very early.
| next-token accuracy | best iteration | log-loss | ECE | fixed | broken | |||||
| Model | iter. 1 | best | last | per set | last / first | iter. 1 | last | (%) | (%) | |
| Ouro-1.4B | 6 | 0.534 | 0.605 | 0.580 | 4,3,4,4,4 | 0.73 | 0.181 | 0.087 | 7.0 | 2.4 |
| Ouro-1.4B-T | 6 | 0.484 | 0.505 | 0.463 | 1,5,1,1,5 | 1.05 | 0.132 | 0.181 | 7.0 | 9.1 |
| Ouro-2.6B | 6 | 0.561 | 0.604 | 0.602 | 4,4,4,4,4 | 0.80 | 0.164 | 0.080 | 7.1 | 3.1 |
| Ouro-2.6B-T | 6 | 0.476 | 0.486 | 0.452 | 2,2,1,1,4 | 1.22 | 0.136 | 0.213 | 3.9 | 6.3 |
| LoopUS | 20 | 0.240 | 0.421 | 0.420 | 12,7,2,2,6 | 0.48 | 0.375 | 0.096 | 20.1 | 2.1 |
Table 6: Lens 3 in numbers: is the last iteration the best reader? All values on prompt tokens, averaged over five sets. Next-token accuracy at first, best, and last iteration. Best iteration per set presents for datasets MATH, GSM8K, Arithmetic, Harm/Benign, HotpotQA the iteration (counted from 1) with the highest accuracy. Log-loss last/first : value below 1 means that additional loops help. ECE is the expected calibration error at the first and last iteration. Fixed ( broken ): share of tokens wrong (right) at iteration 1 but right (wrong) at the end. Prompt tokens give a fair test of every iteration. A loop works best when the best and the last accuracy agree, log-loss is well below 1 and percentage of "fixed" prompts exceeds "broken" which is true for base Ouro, LoopUS, Huginn and false for both Ouro-Thinking models. LoopUS and Huginn fix 20% of tokens while breaking only 2%.
Figure 9: Prompt-reading accuracy per set. Plots at the top: next-token accuracy on prompt tokens. At the bottom: expected calibration error (ECE, lower is better). Both presented along the relative position in the loop on five datasets. There are three main-text patterns, i.e. peaking at training depth, overthinking after reasoning tuning, monotone improvement in re-read loops which hold on every dataset. The arithmetic task is least informative due to short and predictable prompts. HotpotQA plots depicts the largest gains (Huginn 0.30 0.56, LoopUS 0.23 0.47). We conclude that the choice of the best readout iteration is a model-dependent artifact.
| Model | MATH | GSM8K | Arith. | Harm/Ben. | HotpotQA |
|---|---|---|---|---|---|
| Ouro-1.4B | 0.63/0.70/0.68 | 0.53/0.59/0.58 | 0.55/0.61/0.56 | 0.51/0.57/0.54 | 0.45/0.56/0.54 |
| Ouro-1.4B-T | 0.60/0.60/0.57 | 0.49/0.52/0.51 | 0.47/0.47/0.38 | 0.46/0.46/0.40 | 0.40/0.48/0.46 |
| Ouro-2.6B | 0.66/0.71/0.70 | 0.56/0.60/0.60 | 0.57/0.57/0.57 | 0.53/0.58/0.57 | 0.49/0.57/0.56 |
| Ouro-2.6B-T | 0.59/0.60/0.56 | 0.50/0.52/0.50 | 0.41/0.41/0.35 | 0.41/0.41/0.37 | 0.46/0.49/0.47 |
| LoopUS | 0.36/0.56/0.56 | 0.20/0.48/0.48 | 0.25/0.27/0.26 | 0.17/0.32/0.32 | 0.23/0.47/0.47 |
| Parcae | 0.43/0.48/0.48 | 0.41/0.47/0.46 | 0.29/0.29/0.28 | 0.31/0.34/0.34 | 0.40/0.44/0.44 |
Table 7: Prompt-token accuracy per set: at first, best and last iterations. Each cell presents three accuracies on prompt tokens, at the first iteration, at the best iteration and at the last iteration on the five datasets. The "best-last" gap is what a model loses by reading at the end. The "best-first" gap is the loop’s maximum gain. For re-read loops, best and last agree within 0.01 on every set, they never lose by looping longer. For carry loops they often differ. Results show strong dependence on the model and the task. However, looping to the end is not harmful only for the re-read models.
| ECE: first / lowest (iteration) / last | oracle | majority | |||
| Model | MATH | GSM8K | Arith. | accuracy | vote |
| Ouro-1.4B | 0.129 / 0.039 (4) / 0.054 | 0.192 / 0.058 (4) / 0.062 | 0.255 / 0.105 (5) / 0.192 | 0.651 | 0.604 |
| Ouro-1.4B-T | 0.081 / 0.081 (1) / 0.146 | 0.081 / 0.081 (1) / 0.133 | 0.257 / 0.230 (3) / 0.266 | 0.568 | 0.494 |
| Ouro-2.6B | 0.102 / 0.034 (3) / 0.053 | 0.140 / 0.045 (5) / 0.050 | 0.279 / 0.105 (5) / 0.155 | 0.646 | 0.604 |
| Ouro-2.6B-T | 0.102 / 0.102 (1) / 0.169 | 0.103 / 0.103 (1) / 0.161 | 0.255 / 0.255 (1) / 0.331 | 0.545 | 0.477 |
| LoopUS | 0.270 / 0.062 (5) / 0.065 | 0.416 / 0.043 (19) / 0.043 | 0.478 / 0.165 (7) / 0.221 | 0.449 | 0.420 |
Table 8: Calibration along the loop and iteration-combination rules. The expected calibration error (ECE) at the first iteration, at the best-calibrated iteration (in brackets) when ECE is the lowest and at the last iteration. Results on prompts from MATH, GSM8K and arithmetic datasets. The last two columns are averaged over all five datasets and present the: (a) oracle accuracy , i.e. a token counts as correct if any iteration predicts it, and (b) the accuracy based on a majority vote over the top-1 tokens of all iterations. If different iterations make different mistakes, then combining all iterations by an oracle or majority voting might beat the last iteration. We note that the best-calibrated iteration is rarely the last for carry loops. The right answer is often visited, then abandoned, but the majority vote approach does not recover it. The oracle accuracy is 3-8 points above the best single iteration, but the vote never beats it. We conclude that there is useful information in the intermediate iterations, the extraction of which would require training a selecting model rather than simply averaging or voting.
| movement ratio (raw) | drift- | weakest | role | cross- | corr. with | |||||
| Model | MATH | GSM8K | Arith. | Harm/Ben. | HotpotQA | removed | churn | prompt | state drift | |
| Ouro-1.4B | 0.40 | 0.43 | 0.42 | 0.40 | 0.52 | 0.24 | -20 | 0.06 | 0.95 | 0.95 |
| Ouro-1.4B-T | 0.33 | 0.33 | 0.36 | 0.38 | 0.40 | 0.19 | -20 | 0.12 | 0.96 | 0.96 |
| Ouro-2.6B | 0.29 | 0.31 | 0.30 | 0.27 | 0.36 | 0.14 | -22 | 0.12 | 0.95 | 0.94 |
| Ouro-2.6B-T | 0.28 | 0.28 | 0.29 | 0.28 | 0.31 | 0.12 | -22 | 0.09 | 0.97 | 0.95 |
| LoopUS | 0.42 | 0.42 | 0.44 | 0.41 | 0.43 | 0.24 | -59 | 0.02 | 0.98 | 0.99 |
Table 9: Lens 4 per prompt set: heads keep their roles.
Figure 10: Attention fingerprints: one row per head, one column per iteration. Raw fingerprints for MATH prompts, before any standardization. Each small heat map is one of the eight features (for example attention to the first token, to the previous token, or the entropy of the attention) and rows are the heads of the last layer of the looped block, columns are iterations. Top: Ouro-1.4B (6 iterations). Bottom: Huginn (16 iterations). The summary numbers of Lens 4 do not give a broad view as much as the raw maps which show directly what a head keeps its role means. Both maps are horizontal stripes: each head keeps its own feature mix from first to last iteration. In Ouro-1.4B the one visible change (first-token attention jumping from zero to its final level) is the shared drift subtracted in the summary ratios. Huginn’s stripes are even more uniform. A looped block’s heads are best described by one fixed role each plus a small common adjustment at the loop’s start.
Figure 11: Effect of noise on safety-prompt answers. Stacked bars on three safety datasets (AdvBench, CValues, WildChat jailbreaks). For carry loops: answers with no noise injected (clean) and with noise of amplitude , 2 and 8 times. For re-read loops: noise-free and with noise amplitutde , on exactly the same prompts. Each answer is sorted into one of four groups: refuses , on-topic without a refusal phrase , fluent but off-topic , broken . Up to the carry loops answer as they do when clean. At the refusals of Ouro-1.4B shrink from 73% to 24%, but only a minority of the new on-topic non-refusal answers follow the harmful request (Sec. 4.5 ). The Thinking Ouro models refuse more, because they skip their reasoning and answer with a one-line refusal. At noise removes refusals from carry loops but produces broken or off-topic text, not harmful answers: e.g. clean Ouro-1.4B/2.6B refuse 70% and under noise injection 63-73% of the answers become broken. Re-read loops are unaffected.
| Model | measure | clean | 0.5 | 1 | 2 | 4 | 8 | 12 | 17 | 33 | 50 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Ouro-1.4B | similarity | 1.00 | 0.74 | 0.65 | 0.48 | 0.16 | 0.02 | 0.02 | 0.02 | – | – | – |
| broken | 0.02 | 0.02 | 0.01 | 0.03 | 0.29 | 0.76 | 0.72 | 0.66 | – | – | – | |
| fluent & on-topic | 0.75 | 0.75 | 0.75 | 0.82 | 0.51 | 0.01 | 0.00 | 0.01 | – | – | – | |
| non-Latin script | 0.01 | 0.00 | 0.00 | 0.00 | 0.00 | 0.02 | 0.03 | 0.03 | – | – | – | |
| refusal (safety) | 0.73 | 0.72 | 0.72 | 0.66 | 0.25 | 0.00 | 0.01 | 0.00 | – | – | – | |
| reasoning trace (safety) | 0.02 | 0.02 | 0.02 | 0.12 | 0.22 | 0.02 | 0.02 | 0.03 | – | – | – |
Table 10: Noise sensitivity at every amplitude. Table presents six measures for noise-free runs and for each noise amplitude on three safety datasets (AdvBench, CValues, WildChat): word similarity, broken answers, fluent and on-topic answers, non-Latin-script answers, answers with a refusal phrase and answers that open a reasoning trace. We note that in carry loops every metrics collapses at amplitude . In re-read loops the metrics on clean and noisy runs are almost equal. For carry models we note that they keep 4% similarity, the broken text rises to 46-89% and refusals almost disappear. Re-read models retain 53-78% similarity even at .
Figure 13: Refusals and response styles of carry loops on safety prompts. Ratio of answers with different response styles depending on the noise amplitude. (a) Answers containing a refusal phrase. (b) Answers opening a reasoning trace e.g. "Okay, the user is asking …". (c) Answers starting with a refusal phrase. From up to nothing changes qualitatively. At we observe that the Ouro Thinking models stop reasoning and start their answer with a refusal - their refusal rate rises. On the other hand, Ouro-1.4B starts to reason and refuses less. For , no carry loop refuses.
| safety | follows | another | reasoning | role- | |||
|---|---|---|---|---|---|---|---|
| Model | prompts | read | harmful request | request | trace | play | other |
| Ouro-1.4B | 300 | 73 | 12 (4.0%) | 24 | 25 | 5 | 7 |
| Ouro-2.6B | 300 | 21 | 5 (1.7%) | 12 | 0 | 0 | 4 |
| Ouro-1.4B-T | 200 | 27 | 6 (3.0%) | 13 | 0 | 0 | 8 |
| Ouro-2.6B-T | 200 | 14 | 2 (1.0%) | 2 | 0 | 0 | 10 |
Table 11: What the noisy answers at say. We analyze the answers generated with noise at amplitude on prompts on which refusal disappeared (on-topic without a refusal phrase) in spite of noise-free generation being a refusal. Safety prompts : all prompts with generated answers at on the safety sets (AdvBench, CValues, WildChat). Read : answers selected (on-topic without a refusal phrase). Follows harmful request : the answer that does what was asked (percentage of all answers in brackets). Another request : a harmless answer to a misread or invented request. Reasoning trace : answers addressing the request but not reaching a conclusion within 128 tokens. Role-play : a WildChat-Jailbreak role-play whose content is not harmful but which the model refused to answer on a noise-free run. Other : echoed prompts, invented system prompts, hedges without a refusal phrase, harmless prompts answered helpfully and noisy answers whose noise-free generation counterparts were not refusals.
Figure 14: Does the loop amplify or forget a perturbation? This figure presents the change of amplification rate of the hidden-state perturbation along the subsequent iterations on AdvBench prompts. is the mean distance between the noisy and the clean hidden state at iteration , divided by its value at the first iteration ( it0 ). A value above 1 means the difference between the two runs grows, a value below 1 suggest that the difference shrinks. The color of the line denotes the noise amplitude (the amplitudes differ between panels). We note that carry loops (a-c) amplify the perturbation, re-read loops (d-f) shrink it.
Figure 15: Exit gates and next-token distributions under noise injection on AdvBench dataset. Plots (a,b) present the averaged change - noisy run minus clean run on the same prompts as the percentage of tokens whose exit probability exceeds 0.5 at each iteration. Noise injection makes Ouro-2.6B less willing to stop the iterations. Conversely, it makes LoopUS more willing to exit the loop earlier. Plots (c,d) present the Jensen-Shannon divergence between the noise-free and the noisy next-token distribution at each generated token position. For the carry loop, the divergence reaches high values from the very first tokens, while for the re-read loop it starts low and grows as the generated answers drift apart.
Figure 16: Analysis of internal states at the last iteration of the loop. The figure presents the averages over all generated answer tokens of noisy runs over all the investigated datasets. (a) Value of the hidden state norm relative to the corresponding values of noise-free runs. (b) Decay ratio (see Lens 1, B ), computed from the mean step sizes as the mean of the last steps over the mean of the first steps. Dotted horizontal line at value 0.25 corresponds to the chosen settling threshold. (c) Top-1 probability of the next-token distribution during the generation phase. (d) Attention of a generated token to itself. (e) Attention entropy (relative to noise-free runs). (f) Percentage of the variance of the hidden states on their first principal component. Parcae is not presented on purpose - in every run the first principal component of its state carries 47-99.8% of the whole variance. Carry loops are presented with solid lines and re-read loops with dashed lines. Carry loops do not change much up to , start to change significantly (bifurcate) between and and are in a completely different internal state for for Ouro (in LoopUS’s case - for ). Re-read loops stay close to their noise-free counterpart values even for large noise amplitudes .
Figure 17: Evolution of hidden states along the iterations. Top plots present the norm of the hidden state along the iterations. Plots in the middle present the step length . At the bottom - cosine similarity between consecutive states. One line in the plot is drawn per amplitude and the black line corresponds to noise-free runs. Ouro contracts the initial hidden states at every noise amplitude. For , the LoopUS model does not shrink the steps and the norm of the hidden state grows with every iteration.
Figure 18: Next-token probability distribution along the loop. Top plots present entropy while bottom plots display the top-1 probability of the distribution read out at each iteration. For , the first iterations of Ouro models are less certain than on noise-free runs. For , Ouro models are more certain than noise-free runs at every iteration. For , the LoopUS model becomes less certain with every iteration.
Figure 19: Top-1 token changes under noise injection. Plots at the top present the percentage of the generated tokens whose top-1 token at iteration is the same as at . At the bottom: , the Kullback-Leibler divergence between consecutive next-token probability distributions after iteration . For , the Ouro models fix their top-1 token earlier than for noise-free runs. On the other hand, the LoopUS model decides early on the top-1 token. For , it keeps changing its top-1 tokens along the loop.
Figure 20: Where does the generated token attend? The plots present the attention of a generated token to itself (top) and to the first generated token of an answer (bottom). The values are the averages over the heads of the last layer of the looped block. Attention is measured among the generated tokens only. For , noise runs of Ouro models halve their self-attention in comparison to noise-free runs. Moreover, at the first iteration Ouro models put 13-43% of their attention on the first generated token. On the other hand, LoopUS shows a weaker and later version of both changes at .
Figure 21: Broadness and similarity of the heads’ attention. Plots at the top present attention entropy averaged over the heads. At the bottom - mean pairwise cosine similarity of their attention maps (agreement of the heads) at the first four iterations. For , the heads of Ouro attend similarly at the first iteration (due to the attention sink - Fig. 20 ) and more broadly than noise-free runs.
Figure 22: Geometry of the hidden states. Plots at the top present the percentage of the variance of the hidden states, over all generated tokens, on their first principal component. At the bottom - the cosine similarity of the state after iteration to the state after the first iteration. Under noise injection the states concentrate on fewer directions - the percentage of the variance on the first principal component rises two to three times with respect to the noise-free counterpart for Ouro from . For the hidden state of LoopUS moves away from its first iteration.
Figure 24: Exit gates under noise injection. Plots at the top: percentage of generated tokens whose exit probability (the sigmoid of the gate output) exceeds 0.5. At the bottom: mean exit probability. We note that the LoopUS model generates tokens with a default halting threshold of 0.9, but in our analysis we deliberately use a threshold 0.5 for all the three models. Under noise injection, the gate of Ouro-2.6B stops fewer tokens after the second iteration, despite its predictions becoming more confident, while the halting head of LoopUS stops more often.
Figure 25: Small noise injections: token by token analysis of answer drif. The plot presents the comparison of noisy runs with noise-free runs at the same position of the answer (for the sake of demonstration, we truncate the answers to 128 tokens). The presented values are averages over the loop iterations and the seven prompt sets with and . (a) For each token position - the percentage of prompts in which both runs (noise and noise-free) have the same top-1 token. (b) Jensen-Shannon divergence between noisy and noise-free runs on the next-token distributions. (c) Distance between hidden states relative to the norm of noise-free runs. We note that at the first positions the two runs nearly agree. Later, since every token that is decided differently changes the context of the subsequent tokens, the disagreement accumulates along the answer - faster for LoopUS, slower for Ouro.
More in this topic
Implicit Reasoning in Language Models
Implicit reasoning in language models studies how models solve problems through internal computation rather than, or alongside, written chains of thought.
See all 95 papers