When Does Adaptive Halting Win? Selection Rules Matter in Looped Language Models
Organizations: Aston University, Birmingham, UK
Abstract
Looped language models can stop iterating early, and recent work shows that post-hoc readouts of the loop trajectory often rival the learned exit gate of the looped model Ouro. Such comparisons reduce each method to one operating point chosen by a selection rule. We ask whether that rule decides the outcome. On a frozen Ouro-1.4B with four loops, we compared a trained stability probe (-x) with Ouro's native gate, adapted to a per-question stopping decision, in pre-registered studies; the confirmatory study was designed after an exploratory finding on development data and evaluated once on a sealed test set of 8,517 questions. In the pre-registered confirmatory comparison, which selected operating points with a bootstrap-screened 1 pp accuracy-loss rule, the probe saved more of the four-loop budget than the native gate (+0.053 gross macro saving over three benchmarks, 95% CI [+0.049, +0.057]), mostly from CommonsenseQA, where the probe's test upper bound on accuracy loss exceeded 1 pp. Under a point-only rule, a pre-specified secondary analysis on the same test set reversed the preference (-0.022 [-0.026, -0.018]), with operating points that exceeded the budget. In pre-specified secondary comparisons, the probe showed no demonstrated advantage over an untrained KL divergence between successive loops (+0.00125 [-0.0041, +0.0065]); we emphasise this result post hoc and claim no equivalence. The probe also broke its budget on one of two out-of-distribution benchmarks (3.26 pp loss on Social IQa). Savings are estimated from stored full-depth traces, not measured inference time, and our screen carries no formal risk-control guarantee. We conclude that halting methods should be compared as signals together with selection procedures, with strong untrained baselines and out-of-distribution checks.
Figures & tables
| Benchmark | One loop (ms) | |||
|---|---|---|---|---|
| OpenBookQA | 0.036 | 0.014 | 0.050 | 20.7 |
| CommonsenseQA | 0.010 | 0.004 | 0.014 | 79.9 |
| HellaSwag | 0.013 | 0.003 | 0.015 | 148.8 |
| Set | Questions | Role |
|---|---|---|
| Development (Phase 1 train + DEV+) | 17,615 + 2,635 = 20,250 | fitting, selection, freeze |
| TEST, OpenBookQA | 3,000 | primary |
| TEST, CommonsenseQA | 1,000 | primary |
| TEST, HellaSwag | 750 | primary |
| TEST, PIQA | 1,837 | out of distribution |
| TEST, Social IQa | 1,930 | out of distribution |
| Rule and data | 1-x native-pb (gross macro saving) | 95% CI | Evidence |
|---|---|---|---|
| Point-only, G3a nested held-out folds on development data | 0.024 | per-fold range 0.044 to 0.014 | G3a STOP, [E] |
| Point-only, G3a exposed Phase 1 validation | 0.029 | none computed | [E] |
| Point-only, TEST | 0.022 | [ 0.026, 0.018] | [S] |
| Bootstrap-screened, TEST | +0.053 | [+0.049, +0.057] | [C], verdict PASS |
| Bootstrap-screened, TEST, net of estimated probe overhead | +0.038 | [+0.033, +0.042] | [S], estimated cost |
| Benchmark | Method | Gross saving | Loss (pp) | Upper bound (pp) | Exits at loop 1 / 2 / 3 / 4 | |
|---|---|---|---|---|---|---|
| OpenBookQA (3,000) | 1-x | 4 | 0.157 | 0.17 | 0.53 | 6 / 243 / 1,377 / 1,374 |
| OpenBookQA | native-pb | 4 | 0.136 | 0.20 | 0.67 | 0 / 26 / 1,579 / 1,395 |
| OpenBookQA | B_pb | 4 | 0.000 | 0.00 | 0.00 | 0 / 0 / 0 / 3,000 |
| CommonsenseQA (1,000) | 1-x | 4 | 0.282 | 0.60 | 1.10 | 8 / 349 / 405 / 238 |
| CommonsenseQA | native-pb | 4 | 0.161 | 0.10 | 0.30 | 0 / 1 / 643 / 356 |
| CommonsenseQA | B_pb | 4 | 0.000 | 0.00 | 0.00 | 0 / 0 / 0 / 1,000 |
| Rule | Method | OpenBookQA saving / loss (pp) | CommonsenseQA saving / loss (pp) | HellaSwag saving / loss (pp) | Macro saving |
|---|---|---|---|---|---|
| Point-only | 1-x | 0.305 / 1.13 | 0.357 / 1.10 | 0.351 / 1.07 | 0.337 |
| Point-only | native-pb | 0.403 / 1.57 | 0.359 / 1.20 | 0.316 / 0.67 | 0.359 |
| Bootstrap-screened | 1-x | 0.157 / 0.17 | 0.282 / 0.60 | 0.279 / 0.40 | 0.239 |
| Bootstrap-screened | native-pb | 0.136 / 0.20 | 0.161 / 0.10 | 0.262 / 0.13 | 0.186 |
| Baseline (selected under the same rule) | Macro gross saving | = 1-x baseline | 95% CI |
|---|---|---|---|
| B_pb | 0.083 | +0.156 | [+0.151, +0.161] |
| native-pb (primary) | 0.186 | +0.053 | [+0.049, +0.057] |
| r-conf | 0.171 | +0.068 | [+0.063, +0.073] |
| r-margin | 0.170 | +0.069 | [+0.064, +0.075] |
| r-ent | 0.179 | +0.060 | [+0.055, +0.065] |
| r-dlogit | 0.162 | +0.077 | [+0.071, +0.083] |
| Benchmark (n) | Method | Gross saving | Loss (pp) | Upper bound (pp) | Exits at loop 1 / 2 / 3 / 4 |
|---|---|---|---|---|---|
| PIQA (1,837) | 1-x | 0.308 | 0.71 | 1.52 | 0 / 433 / 1,399 / 5 |
| PIQA | native-pb | 0.126 | 0.11 | 0.60 | 0 / 0 / 923 / 914 |
| PIQA | B_pb | 0.000 | 0.00 | 0.00 | 0 / 0 / 0 / 1,837 |
| Social IQa (1,930) | 1-x | 0.481 | 3.26 | 4.72 | 0 / 1,785 / 143 / 2 |
| Social IQa | native-pb | 0.028 | 0.05 | 0.16 | 0 / 0 / 214 / 1,716 |
| Social IQa | B_pb | 0.000 | 0.00 | 0.00 | 0 / 0 / 0 / 1,930 |