Looped language models can stop iterating early, and recent work shows that post-hoc readouts of the loop trajectory often rival the learned exit gate of the looped model Ouro. Such comparisons reduce each method to one operating point chosen by a selection rule. We ask whether that rule decides the outcome. On a frozen Ouro-1.4B with four loops, we compared a trained stability probe (π1-x) with Ouro's native gate, adapted to a per-question stopping decision, in pre-registered studies; the confirmatory study was designed after an exploratory finding on development data and evaluated once on a sealed test set of 8,517 questions. In the pre-registered confirmatory comparison, which selected operating points with a bootstrap-screened 1 pp accuracy-loss rule, the probe saved more of the four-loop budget than the native gate (+0.053 gross macro saving over three benchmarks, 95% CI [+0.049, +0.057]), mostly from CommonsenseQA, where the probe's test upper bound on accuracy loss exceeded 1 pp. Under a point-only rule, a pre-specified secondary analysis on the same test set reversed the preference (-0.022 [-0.026, -0.018]), with operating points that exceeded the budget. In pre-specified secondary comparisons, the probe showed no demonstrated advantage over an untrained KL divergence between successive loops (+0.00125 [-0.0041, +0.0065]); we emphasise this result post hoc and claim no equivalence. The probe also broke its budget on one of two out-of-distribution benchmarks (3.26 pp loss on Social IQa). Savings are estimated from stored full-depth traces, not measured inference time, and our screen carries no formal risk-control guarantee. We conclude that halting methods should be compared as signals together with selection procedures, with strong untrained baselines and out-of-distribution checks.
Figures & tables
Benchmark
ρscore
ρrest
ρtotal
One loop (ms)
OpenBookQA
0.036
0.014
0.050
20.7
CommonsenseQA
0.010
0.004
0.014
79.9
HellaSwag
0.013
0.003
0.015
148.8
Table 1: Freeze-time overhead of the probe, as a fraction of one loop’s time. Measured on one GPU, batch size 1, 200 development questions per benchmark.
Set
Questions
Role
Development (Phase 1 train + DEV+)
17,615 + 2,635 = 20,250
fitting, selection, freeze
TEST, OpenBookQA
3,000
primary
TEST, CommonsenseQA
1,000
primary
TEST, HellaSwag
750
primary
TEST, PIQA
1,837
out of distribution
TEST, Social IQa
1,930
out of distribution
Table 2: Data split.
Rule and data
π 1-x − native-pb (gross macro saving)
95% CI
Evidence
Point-only, G3a nested held-out folds on development data
− 0.024
per-fold range − 0.044 to − 0.014
G3a STOP, [E]
Point-only, G3a exposed Phase 1 validation
− 0.029
none computed
[E]
Point-only, TEST
− 0.022
[ − 0.026, − 0.018]
[S]
Bootstrap-screened, TEST
+0.053
[+0.049, +0.057]
[C], verdict PASS
Bootstrap-screened, TEST, net of estimated probe overhead
+0.038
[+0.033, +0.042]
[S], estimated cost
Table 3: π 1-x minus native-pb under the two selection rules. The first two rows are development-data results; the last three are on the sealed TEST set. The G3a value to four decimals is − 0.0236.
Benchmark
Method
mb
Gross saving
Loss (pp)
Upper bound (pp)
Exits at loop 1 / 2 / 3 / 4
OpenBookQA (3,000)
π 1-x
4
0.157
0.17
0.53
6 / 243 / 1,377 / 1,374
OpenBookQA
native-pb
4
0.136
0.20
0.67
0 / 26 / 1,579 / 1,395
OpenBookQA
B_pb
4
0.000
0.00
0.00
0 / 0 / 0 / 3,000
CommonsenseQA (1,000)
π 1-x
4
0.282
0.60
1.10
8 / 349 / 405 / 238
CommonsenseQA
native-pb
4
0.161
− 0.10
0.30
0 / 1 / 643 / 356
CommonsenseQA
B_pb
4
0.000
0.00
0.00
0 / 0 / 0 / 1,000
Table 4: TEST results under the bootstrap-screened rule, per primary benchmark [S, descriptive]. Savings are gross. The upper bound is the one-sided 95% paired-bootstrap bound of the loss on TEST. Bold marks bounds above 1 pp.
Rule
Method
OpenBookQA saving / loss (pp)
CommonsenseQA saving / loss (pp)
HellaSwag saving / loss (pp)
Macro saving
Point-only
π 1-x
0.305 / 1.13
0.357 / 1.10
0.351 / 1.07
0.337
Point-only
native-pb
0.403 / 1.57
0.359 / 1.20
0.316 / 0.67
0.359
Bootstrap-screened
π 1-x
0.157 / 0.17
0.282 / 0.60
0.279 / 0.40
0.239
Bootstrap-screened
native-pb
0.136 / 0.20
0.161 / − 0.10
0.262 / 0.13
0.186
Table 5: TEST operating points selected by each rule [S]. Bold marks point losses above 1 pp. The macro savings under the point-only rule are means of the per-benchmark entries.
Figure 1: TEST operating points selected by each rule, one panel per primary benchmark (OpenBookQA, CommonsenseQA, HellaSwag). Horizontal axis: accuracy loss relative to loop 4, in percentage points. Vertical axis: gross loop saving, as a fraction of the four-loop budget. Circles: π 1-x. Squares: native-pb. Filled markers: bootstrap-screened rule. Open markers: point-only rule. The dashed vertical line marks the 1 pp budget. Each marker is one frozen (τb,mb) selected on development data; the figure shows four operating points per panel and is not a saving-loss frontier, so it does not support a comparison at matched loss. Macro-averaged over the three benchmarks, the screened-rule difference π 1-x − native-pb is +0.053 [+0.049, +0.057] [C] and the point-only difference is − 0.022 [ − 0.026, − 0.018] [S].
Baseline (selected under the same rule)
Macro gross saving
Δ = π 1-x − baseline
95% CI
B_pb
0.083
+0.156
[+0.151, +0.161]
native-pb (primary)
0.186
+0.053
[+0.049, +0.057]
r-conf
0.171
+0.068
[+0.063, +0.073]
r-margin
0.170
+0.069
[+0.064, +0.075]
r-ent
0.179
+0.060
[+0.055, +0.065]
r-dlogit
0.162
+0.077
[+0.071, +0.083]
Table 6: TEST comparison of π 1-x (macro saving 0.239) with every baseline, paired and benchmark-stratified bootstrap, 10,000 resamples [S]. The r-kl row is given to four decimals. The nine intervals are not adjusted for multiple comparisons. All baselines were selected on development data under the bootstrap-screened rule.
Benchmark (n)
Method
Gross saving
Loss (pp)
Upper bound (pp)
Exits at loop 1 / 2 / 3 / 4
PIQA (1,837)
π 1-x
0.308
0.71
1.52
0 / 433 / 1,399 / 5
PIQA
native-pb
0.126
0.11
0.60
0 / 0 / 923 / 914
PIQA
B_pb
0.000
0.00
0.00
0 / 0 / 0 / 1,837
Social IQa (1,930)
π 1-x
0.481
3.26
4.72
0 / 1,785 / 143 / 2
Social IQa
native-pb
0.028
− 0.05
0.16
0 / 0 / 214 / 1,716
Social IQa
B_pb
0.000
0.00
0.00
0 / 0 / 0 / 1,930
Table 7: Out-of-distribution TEST results with one task-agnostic (τ,m) per policy [S]. Bold marks losses and upper bounds above 1 pp. The B_pb row is full depth.
Looped language models run each token through the same layers several times, and each run is one iteration. To save compute, an extra layer, the halting head, gives after each iteration the probability that the token stops there, so that easy tokens can stop early. Pretraining models from scratch, up to 1.3B parameters, we find two problems. First, heads trained on a mix of all iterations predict that tokens need under half the iterations, yet under the standard stopping rule tokens run almost every iteration, because this training never scores the single iteration where a token stops. Scoring every iteration instead fixes this, but in our runs it works only with a strong penalty favoring early stops. Second, savings are commonly reported as an average number of iterations per token, but when sequences are generated in batches that run until their slowest token stops, a model without a head is almost as good at that cost. The reported average can therefore overstate the saving. We recommend reporting the iterations tokens actually run, with the stopping threshold and batch size, and give a test, run before training a head, that bounds its gain.
Looped language models (LMs) reuse one block of layers several times per token generation, promising more latent computation at no additional storage cost. Despite their rising popularity, the analysis of looped LMs is still in its infancy: little is known about whether the iterative transformations actually improve their capabilities, or whether these models remain stable under prompt perturbation. We construct an inspection protocol rooted in Mechanistic Interpretability comprising five diverse lenses. We apply it to examine eight looped LMs from four families. Models differ in many aspects but the key one is whether the prompt is transmitted only within the looped state (carry loops: Ouro, LoopUS) or re-supplied at every iteration (re-read loops: Parcae, and Rec-OLMo & Huginn from the Raven family). The results of our study provide valuable insights concerning safety and efficient use of looped language models.
Pawel Olszowiec, Paulina Tomaszewska, Maciej Żelaszczyk +2
Samsung AI Center, Warsaw, Poland · IFTR, Polish Academy of Sciences, Warsaw, Poland · University of Warsaw, Poland
Can a language model read the quality of its ongoing computation, and can an external intervention turn that readout into better outcomes? We test both questions in a frozen 2.6B looped transformer, Ouro-RLTT. On GSM8K, a strict pre-answer probe excludes the answer region and gold value yet predicts success: hidden states plus length/log-probability features reach AUROC 0.797 versus 0.731 for those surface features alone (increment +0.066; task-clustered 95% CI [+0.021,+0.112]; 170 tasks). On Horizon Logic, a prospectively extended task-disjoint study gives an increment of +0.111 (CI [+0.056,+0.169]), independently replicated on the new cohort (+0.095) and robust to an adversarial malformed-sibling shortcut. Recurrence also moves candidate-quality readability to progressively earlier physical depth; the trend replicates across the Ouro family and qualitatively in out-of-family Huginn, although their transfer geometry differs. The readout converts into validated decision-level gains. Hidden-state-based scores improve risk-coverage over shortcut-only scores in four sealed selective-prediction arms, and terminal selection beats matched random even when every candidate is well formed (27/32 correct selections versus 64.8% expected; p = 0.0086). Generative control does not convert: directional steering is negative, a branch screen is bounded, and exact-compute loop allocation and minimal LoRA direction-binding detect no gain. These tests run through bit-exact branch/carry/prune machinery over Ouro's 192-slot recurrent cache, including a suffix-recompute splice saving up to 88% of per-branch layer passes. We call this decision-usable but not generatively controllable property operational proto-introspection. All load-bearing values use source-item-disjoint splits and antisymmetrized pairwise evaluation.