A Tilted Bowl Is Not a Slippery Slope: Compressing Looped Models

Steven Kolawole; Pearse Jim; Opegbemi M. Busoye; Glory Bagai; Virginia Smith

cs.LGSep 30, 2026

A Tilted Bowl Is Not a Slippery Slope: Compressing Looped Models

Authors: Steven Kolawole, Pearse Jim, Opegbemi M. Busoye, Glory Bagai, Virginia Smith

Organizations: Carnegie Mellon University · ML Collective

Abstract

Looped models reason by applying the same block of weights many times, so compressing that block saves memory traffic on every loop. Compressed looped models, however, often collapse, and the collapse is usually blamed on rounding error that accumulates from loop to loop. In this work we test that account on more than 30 models from five families and find, to our surprise, that it holds only for loops that never settle. When a loop settles, a fixed rounding error does not accumulate. It moves the point where the loop settles, much as tilting a bowl moves where a ball comes to rest, and the answer is lost only when the shift is larger than the readout tolerates. This picture lets us predict which models fail from a single label-free measurement, and it tells us why failed models recover: their loops still settle, so a few final loops with 8-bit weights bring the answer back. Motivated by these findings, we build a controller that stops when the model's halting head fires and then finishes with 8-bit loops. On Sudoku-Extreme and Maze-Hard it beats fixed-depth inference by up to 15 points under a third of the weight traffic.

Figures & tables

Figure 1: Thinking longer does not hurt a compressed model whose loop settles; a loop that does not settle piles up error. (a) Compression tilts the bowl, so the loop settles slightly off target. The answer survives while the settling shift stays within the room, and finishing steps level the bowl again. On a slippery slope, every loop carries the ball further. (b) Change in the gap between a 4-bit model and its own full precision as the loops go on past the default depth, one model per family; a value below zero means the gap widens. No interval lies below zero. (c) DDIM, the one loop that does not settle, adds 140 FID over the same range. Paired 95% intervals; the looped language models are in Table 2 ; protocol in App. C .

Family (models)	Models	Task	What loops
Puzzle reasoners (24)	TRM with an MLP or an attention mixer, HRM ( Jolicoeur-Martineau, 2025 ; Wang et al., 2025 ) ; 19 TRMs we train at several widths and seeds	Sudoku-Extreme, Maze-Hard	one block, $H\times L$ cycles per supervision step
Recurrent convnets (1)	DeepThinking ( Bansal et al., 2022 )	mazes of size 13 and 33	a recall block, input re-injected
Implicit models (3)	MDEQ-Small, MDEQ-XL ( Bai et al., 2020 ) ; DEQ-Transformer ( Bai et al., 2019 )	ImageNet, WikiText-103	a solver, run to a fixed point
Looped language models (4)	Huginn-3.5B ( Geiping et al., 2025 ) ; Ouro-1.4B, Ouro-2.6B ( Zhu et al., 2025 ) ; Recurrent-OLMo-2 ( McLeish et al., 2025 )	GSM8K	a core block, 4–32 recurrences; full precision is bf16
Diffusion, the boundary (1)	DDPM UNet, sampled by DDPM ( Ho et al., 2020 ) and DDIM ( Song et al., 2021 )	CIFAR-10	the denoiser; DDPM draws fresh noise each step, DDIM is deterministic

Table 1: Five families, 33 models, one repeated map each. The family is the unit of the claims; the boundary case, diffusion, is the one loop that need not settle. Every model, its default depth and its metric are in Appendix A .

Figure 2: The size of the error sets the cliff; whether the error stays fixed decides whether more loops repair it. (a) Accuracy against noise level for TRM Sudoku-MLP. Zero-mean fresh noise, which has nothing to accumulate, produces the same cliff as fixed noise, only later. (b) Gap to full precision as the supervision steps grow, for fresh error, drawn anew at every call, and fixed error, added to the weights once, matched on fidelity; the number at each line end is the noise level. Fresh error heals with more loops in every draw; fixed error stays, except in some attention draws. Means and 95% intervals over noise draws; protocol in App. D .

Figure 3: A collapsed model comes back, 8-bit finishing is enough, and the gain is predicted in advance. (a) Share of full-precision accuracy that collapsed models retain after

k

finishing steps with full-precision, 8-bit or 4-bit weights;

k{=}0

is the compressed model alone. The 8-bit curve lies on the full-precision curve. Medians over models with the interquartile band; (i) and (ii) are the two platforms. (b) Gain predicted by the finishing law against the observed gain, one point per model; all 20 fall inside the tolerance set before the run (band). Protocol in App. E .

Figure 4: Who fails is predictable before deployment: a model tolerates weight error up to its room divided by its sensitivity. (a) Tolerance predicted from label-free sensitivity and one room shared by all models, against the measured weight error at which accuracy halves; one point per model and task (18 models on 19 pairs,

R^{2}{=}0.72

in log scale). Filled and open markers are the two platforms, ringed markers the seeds held out of the fit; the band is the leave-one-out median error. (b) Measured tolerance of width-512 TRMs, one point per seed. The attention mixer tolerates four times the error of the MLP mixer on both platforms. Fits in App. F .

			w4c		w3a
Model (loops)	Settles	bf16	alone	+precise	alone	+precise
Huginn-3.5B (32)	✓	42	33	37	3	40
Recurrent-OLMo-2 (32)	✓	41	35	46	2	45
Ouro-1.4B (4)	✗	73	7	66	0	0
Ouro-2.6B (4)	✗	81	78	74	0	50

Table 2: Looped language models that settle come back when their last loops run at full precision; Ouro, which does not settle, marks the boundary. GSM8K accuracy (%) at full precision (bf16), for the compressed model alone, and with the second half of its loops at full precision (+precise). One platform,

n{=}100

per entry (Wilson half-width about 10 points); settling tests, fidelity and prompts in App. G .

Method	Accuracy (%)	Cost	Finishing steps
Fixed depth	61.1	55.0	0
+ stop at the halting head	63.0	23.4	0
+ 8-bit finishing = controller	75.8	16.1	1.13
controller $-$ fixed depth	$+14.7$ [ $+14.3$ , $+15.2$ ]
+ latent sampling (add-on)	78.7	28.4	1.45
Controller with full-precision finishing	75.7	25.8	1.20

Table 3: Stopping at the halting head and finishing with 8-bit loops beats fixed depth at under a third of the weight traffic. Held-out exact accuracy on Sudoku-Extreme, averaged over 20 compressed models; cost in compressed loops per puzzle; finishing steps per puzzle. The first block adds one lever at a time. The second block holds the alternatives: uniform refinement finishes every puzzle and confidence-gated refinement finishes the puzzles whose halting logit is low, both at full precision. The last block tests the head’s choice of puzzles to finish against a random set of the same size. Right, the same rows on the cost–accuracy plane; the path is the first block, the dotted step the add-on, the open grey points the alternatives. Paired 95% intervals; protocol in App. H .

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Model	#	What loops	Default depth	Task, metric	$n$
TRM, MLP mixer (released)	1	one 2-layer block, reused for the $z_{L}$ and $z_{H}$ updates; token-mixing SwiGLU	$H{=}3$ , $L{=}6$ , 16 supervision steps	Sudoku-Extreme (97 positions), exact	1000; 1800 (depth to 512)
TRM, attention mixer (released)	2	as above, self-attention with RoPE; one checkpoint per task	as above	Sudoku-Extreme; Maze-Hard $30{\times}30$ (916 positions), exact and valid path	1000 / 500
HRM (released)	2	separate high- and low-level modules (4 layers each); one checkpoint per task	$H{=}2$ , $L{=}2$ , 16 supervision steps	Sudoku-Extreme, Maze-Hard	1000 / 500
Our TRMs, MLP mixer	10	as TRM; width 256 (seeds 0–2), 512 (seeds 0–5), 768 (seed 0)	as TRM	Sudoku-Extreme, exact	1000
Our TRMs, attention mixer	9	as TRM; width 256 (seeds 0–2), 512 (seeds 0–5)	as TRM	Sudoku-Extreme, exact	1000
DeepThinking	1	recall block, input concatenated at every iteration (width 128)	300 iterations; swept to 1000	mazes of size 13 and 33, exact	per size

Appendix

Table 4: Every model, the map it repeats, its default depth and its metric; 33 models in all. “What loops” is the weight-tied map; “default depth” is the released depth;

n

is the number of evaluation examples per compressed model in the main experiments of § 3 .

	Dense		25% pruned		50% pruned		Distilled
Task	exact	cell	exact	cell	exact	cell	exact	cell
ARC	36.0	87.6	0.0	0.3	0.0	0.6	–	–
Maze	86.8	99.5	0.0	86.4	0.0	0.0	0.0	87.3
Sudoku	69.1	87.5	0.0	50.0	0.0	37.8	0.0	54.8

Appendix

Table 5: Pruning and distillation leave the puzzle reasoners unable to solve a single puzzle. Exact puzzle accuracy and cell accuracy (%) of the dense model, after structured pruning, and of a distilled student (Maze teacher 6.82M, student 855K parameters; Sudoku teacher 5.03M, student 745K). Released checkpoints at

H{=}3

with 10 supervision steps (ARC: 16 steps with test-time augmentation), full test sets, one run.

Compressed format	w4	w3	w3g32	w2	w2g32	w8
$b_{\text{eff}}$	4	3	3.5	2	2.5	8
$r$ (full-precision loop)	8	10.7	9.1	16	12.8	4

Appendix

Table 6: One full-precision loop costs 4 to 16 compressed loops.

r=32/b_{\text{eff}}

, with

b_{\text{eff}}

the bits read per weight including group scales. Weight traffic only.

	Endpoints			Trend
Run (series)	$n_{\text{sup}}$	$H$	$L$	$n_{\text{sup}}$	$H$	$L$
A100 (22)	1 / 16 / 5	0 / 18 / 4	3 / 14 / 5	0 / 15 / 7	0 / 19 / 3	5 / 12 / 5
L40S replicate (23)	0 / 15 / 8	0 / 20 / 3	–	0 / 18 / 5	0 / 20 / 3	–
L40S independent (23)	2 / 18 / 3	0 / 21 / 2	–	1 / 18 / 4	0 / 21 / 2	–

Appendix

Table 7: Along supervision steps and outer cycles the gap of a surviving model almost never widens. Number of surviving series that widen / stay flat / narrow, by endpoints and by trend (95% paired bootstrap, 2000 resamples). Sudoku

n{=}1000

, Maze

n{=}500

. Each row comes from one platform. The L40S replicate did not sweep

L

; its Maze

H

series cover

H

1–3 in the audited records, and records to deeper

H

leave every verdict unchanged.

Model	Format	$n_{\text{sup}}$	$H$	$L$
TRM Sudoku, MLP	w4c	+1.1 [ $-$ 1.9, +4.1] =	$-$ 1.2 [ $-$ 3.3, +1.1] =	+1.4 [ $-$ 1.9, +4.8] =
	w4g32	$-$ 0.2 [ $-$ 2.4, +2.1] =	$-$ 1.2 [ $-$ 3.3, +0.9] =	$-$ 1.3 [ $-$ 4.4, +1.8] =
	W4A4 g32	+5.0 [+2.2, +7.9] $\downarrow$	$-$ 0.3 [ $-$ 2.5, +1.9] =	$-$ 7.5 [ $-$ 10.7, $-$ 4.5] $\uparrow$
	W4A4 MXInt4	+7.5 [+4.1, +10.9] $\downarrow$	+0.1 [ $-$ 2.3, +2.3] =	$-$ 11.3 [ $-$ 14.4, $-$ 7.9] $\uparrow$
	W8A8 per-tensor	+13.8 [+10.8, +16.9] $\downarrow$	+2.5 [+0.2, +5.0] $\downarrow$	+7.3 [+3.8, +10.8] $\downarrow$
TRM Sudoku, attn.	w4c	+0.7 [ $-$ 1.0, +2.4] =	+0.3 [ $-$ 2.0, +2.6] =	+4.7 [+2.3, +7.1] $\downarrow$

Appendix

Table 8: A100 platform: no surviving series widens along outer cycles, and one widens along supervision steps. Change of the gap from the shallowest to the deepest point, in points of exact accuracy [95% paired bootstrap interval];

\uparrow

widens,

=

flat,

\downarrow

narrows (endpoints).

n_{\text{sup}}

\to

32;

H

\to

6 (HRM 1

\to

4);

L

\to

8 (HRM 1

\to

4); default

H

L

and 16 supervision steps on the other axes. Sudoku

n{=}1000

, Maze

n{=}500

Noise / format	$k{=}8$	$k{=}16$	$k{=}26$	$k{=}100$	Cosine to full precision	Converged / median steps
fp32	72.2	73.9	73.8	73.8	1	99.7% / 24
$\sigma$ 0.05	71.4	72.9	73.1 [72.7, 73.7]	73.0	0.978	99.7% / 24
$\sigma$ 0.1	67.3	69.0	68.9 [67.1, 70.7]	68.9	0.923	99.8% / 24
$\sigma$ 0.2	46.0	49.1	49.1 [47.1, 52.9]	49.1	0.779	99.8% / 25
$\sigma$ 0.3	12.6	14.6	14.8 [10.5, 18.4]	14.8	0.614	99.8% / 25
$\sigma$ 0.5	0.3	0.2	0.3	0.3	0.334	99.5% / 26

Appendix

Table 9: MDEQ-Small on ImageNet: accuracy stops changing once the solver converges, at every noise level, and the solver converges on collapsed models too. Top-1 % by Broyden budget

k

(26 = default); cosine to the full-precision equilibrium; fraction of images with residual below

10^{-2}

k{=}100

; median steps to

10^{-2}

. Noise rows: mean [min, max] over 3 noise directions at

k{=}26

; quantizer rows give the mean relative weight error.

n{=}5000

validation images; L40S platform.

Noise / format	$k{=}10$	$k{=}30$	$k{=}60$	Residual, $k{=}60$	Cosine, $k{=}60$
fp32	91.2	24.48	23.14	2.5e-2	1
$\sigma$ 0.1	92.3	25.23	23.84 [23.82, 23.86]	2.4e-2	0.981
$\sigma$ 0.3	98.5	34.59	32.65 [32.63, 32.70]	2.3e-2	0.846
$\sigma$ 0.5	219	124.7	127.1 [124.2, 132.0]	2.9e-2	0.55
$\sigma$ 0.7	923	1071	1182 [1142, 1204]	6.1e-2	0.29
w8c (0.008)	90.2	24.48	23.14	2.5e-2	1.000

Appendix

Table 10: DEQ-Transformer on WikiText-103: extra iterations help as long as the model survives and hurt only after it has collapsed; the collapsed quantizer has the smallest residual. Test perplexity by Anderson iteration cap

k

(lower is better), median relative residual and cosine to full precision at

k{=}60

. Noise rows: mean [min, max] over 3 noise directions at

k{=}60

; quantizer rows give the relative weight error. Full test set (245,569 tokens); L40S platform.

	DDIM steps				DDPM steps
$\sigma$ / format	10	25	50	100	50	100	250	1000
$\sigma$ 0.01	7.4	6.6	6.1	6.7	1.5	2.1	2.6	–
$\sigma$ 0.03	26.5	17.0	17.2	49.8	6.2	7.1	7.9	–
$\sigma$ 0.05	45.7	21.2	61.4	189.5	12.7	13.3	13.8	–
$\sigma$ 0.1	95.6	136.5	345.8	397.6	43.2	42.8	42.1	46.1
$\sigma$ 0.3	386	358	356	312	213	255	290	–

Appendix

Table 11: The deterministic DDIM sampler compounds with steps; ancestral DDPM keeps a step-flat cost inside its tolerance. FID-5k against the full-precision samples of the same sampler, drawn from the same starting and step noise (lower is closer). Noise rows: mean over 3 noise directions; quantizer rows give the relative weight error. DDPM CIFAR-10 UNet; L40S platform. The quantizer rows of the DDPM-1000 column need no noise draw.

Clipping	Quantized layers	Maze exact	Maze valid path	Sudoku-MLP exact	$\rho$ Maze / MLP
max over calls (ours)	all linears	0.0 [0.0–0.0]	0.0	0.0 [0.0–0.0]	0.293 / 0.466
max over calls	block only	0.0 [0.0–0.0]	0.0	0.0 [0.0–0.0]	0.293 / 0.465
99.99th percentile	all linears	82.2 [82.2–82.2]	98.4	0.8 [0.4–1.0]	0.145 / 0.270
99.99th percentile	block only	82.3 [82.2–82.4]	98.4	0.8 [0.5–1.0]	0.144 / 0.269

Appendix

Table 12: The clipping rule alone decides whether per-tensor W4A4 survives on Maze. Exact % at 16 supervision steps, mean [min–max] over 3 calibration draws, each with its own full-precision reference;

\rho

= per-call output error. Maze-attention

n{=}500

(full precision 88.4), Sudoku-MLP

n{=}1000

(full precision 71.4); A100 platform.

Model	Error	$\sigma$ ( $\varphi_{H}$ )	Gap 16 / 64 / 512	$D$	Per-draw regimes
TRM MLP	fresh	0.15 (0.89)	$-4.4$ / $-1.4$ / $-0.1$	$+4.3$ [ $+3.8$ , $+4.7$ ]	full catch-up 5
	fresh	0.2 (0.87)	$-7.8$ / $-3.6$ / $-1.4$	$+6.4$ [ $+5.6$ , $+7.2$ ]	full 1, partial 4
	fresh	0.25 (0.70)	$-17.1$ / $-17.1$ / $-18.4$	$-1.3$ [ $-1.9$ , $-0.8$ ]	constant 5
	fixed	0.08 (0.88)	$-3.4$ / $-2.8$ / $-3.6$	$-0.2$ [ $-0.8$ , $+0.3$ ]	constant 5
	fixed	0.1 (0.85)	$-6.1$ / $-5.6$ / $-6.6$	$-0.4$ [ $-1.4$ , $+0.5$ ]	constant 5
	fixed	0.12 (0.72)	$-17.8$ / $-19.1$ / $-22.0$	$-4.2$ [ $-6.5$ , $-1.9$ ]	widening 5

Appendix

Table 13: Fresh noise catches up with more loops; fixed noise does not, except on some attention draws. Gap to full precision (points) at 16 / 64 / 512 supervision steps and

D=

gap(512)

-

gap(16), mean over 5 noise draws [95%

t

-interval across draws]; fidelity

\varphi_{H}

at 512 steps in parentheses; per-draw regimes by the rule of Appendix B.2 . 1800 stratified Sudoku-Extreme puzzles; L40S platform; each model with its own full-precision reference.

Variant	$\sigma$ 0.1	0.15	0.2	0.3	0.5	$\sigma_{1/2}$	$\rho_{1/2}$
an fresh, zero-mean	70.1 [70–70]	67.4 [67–68]	62.9 [63–63]	32.9 [32–34]	2.3 [2–2]	0.287–0.293	0.080–0.082
af fixed, zero-mean	66.5 [66–67]	58.6 [57–60]	43.8 [41–46]	10.4 [8–13]	0.3	0.216–0.231	0.061–0.064
ab fixed, sign-consistent	64.6 [62–67]	46.5 [35–56]	22.9 [10–31]	3.7 [0–6]	0.0	0.149–0.190	0.094–0.124
wn weight, zero-mean	64.9 [63–66]	26.5 [24–31]	5.5 [4–7]	1.0	0.0	0.135–0.144	0.038–0.042

Appendix

Table 14: TRM Sudoku-MLP: zero-mean fresh noise alone produces the cliff; bias and fixedness move it. Exact % at each

\sigma

, mean [min–max] over 3 noise draws;

\sigma_{1/2}

and

\rho_{1/2}

are where exact accuracy falls below half of full precision (71.4), per draw. Weight bias ( wb , own grid):

\sigma

0.02

\to

66.3 [64–69], 0.05

\to

3.1 [2–4];

\sigma_{1/2}

0.034–0.035,

\rho_{1/2}

0.047–0.049.

n{=}1000

H{=}3

L{=}6

, 16 supervision steps; A100 platform.

Variant	$\sigma$ 0.5	0.7	1.0	1.5	$\sigma_{1/2}$ exact	$\sigma_{1/2}$ valid path	$\rho_{1/2}$
an fresh, zero-mean	87.5 [87–88]	85.5 [85–87]	23.3 [23–24]	0.0	0.897–0.902	1.20–1.21	0.198–0.199
af fixed, zero-mean	77.7 [70–82]	34.9 [20–48]	0.0	0.0	0.60–0.73	0.78–0.83	0.13–0.16
ab fixed, sign-consistent	86.1 [84–88]	79.5 [72–84]	9.3 [0–25]	0.0	0.82–0.90	0.83–0.91	0.18–0.20
wn weight, zero-mean	87.7 [87–88]	–	86.1 [85–87]	27.1 [0–81]	1.24–1.73	1.73–1.75	0.30–0.44

Appendix

Table 15: TRM Maze-attention: the same variants, a wider tolerance. Exact % mean [min–max] over 3 noise draws;

\sigma_{1/2}

on exact match and on valid path; full precision 88.4.

n{=}500

H{=}3

L{=}4

; A100 platform.

Model / format (regime)	Mixer-path error, steps 1 / 4 / 16 / 64 / 512	Whole-step error	Entropy	Top probability
attention, full precision	0	0	0.694 $\to$ 0.643	0.21 $\to$ 0.26
attention w3c (catch-up)	0.210 / 0.281 / 0.287 / 0.290 / 0.291	0.636 $\to$ 0.289	0.719 $\to$ 0.646	0.19 $\to$ 0.26
attention w3a (catch-up)	0.170 / 0.232 / 0.237 / 0.239 / 0.240	0.444 $\to$ 0.229	0.703 $\to$ 0.643	0.20 $\to$ 0.26
attention w3g32 (catch-up)	0.136 / 0.181 / 0.187 / 0.189 / 0.190	0.310 $\to$ 0.168	0.696 $\to$ 0.638	0.21 $\to$ 0.26
attention w4t (constant)	0.169 $\to$ 0.240	0.503 $\to$ 0.229	0.711 $\to$ 0.651	0.20 $\to$ 0.25
MLP w4c (constant)	0.086, flat	0.520 $\to$ 0.220	–	–

Appendix

Table 16: Attention never becomes one-hot, and its error does not shrink in the models that catch up. Values at supervision steps 1

\to

512 (mixer-path error also at 4 / 16 / 64).

n{=}128

Sudoku-Extreme puzzles, 512 supervision steps; L40S platform; hypothesis and pass criterion written before any record they score existed.

Clause	Bar	Outcome	Verdict
(A1) collapsed models with $r\geq 1$	$\geq$ 80%	12%	fails
(A2) surviving models with $r<1$	$\geq$ 80%	91%	holds
(B1) collapsed models with $r<1$	$\geq$ 80%	88%	holds
(B2) zero-parameter width, new seeds	bias in [0.9, 1.1]	$\times$ 8.9 (bias 8.25 / 9.60)	fails
one fitted scale	(reported)	$\times$ 1.57 (bias 1.45 / 1.68)	–
reference: $\delta^{\star}/S$ , $\delta^{\star}/S_{\text{mix}}$	–	$\times$ 1.39, $\times$ 1.16	–

Appendix

Table 17: Collapsed models keep settling: the contraction-loss explanation fails its defining clause. Each row is a clause with its bar, the outcome and the verdict. Width errors are on the six new mixer seeds (§ F.3 ; median

|\log|

ratio, as a factor); bias = predicted/measured tolerance, MLP / attention. L40S platform, 32 record files, 201 unique compressed models (60 collapsed).

ID	Order	Prediction and bar	Outcome
P1	1	A compressed model collapses iff its settled state does not return under the full-precision loop: rule “collapse iff median $\rho>1$ ” balanced accuracy $\geq 0.80$ ; AUROC( $\rho$ ) $>$ AUROC( $-\varphi$ ); leave-one-family-out $\rho\geq\varphi$ ; Spearman(retained, return) $\geq 0.7$ .	Falsified on both platforms. L40S: balanced accuracy 0.63; AUROC $\rho$ 0.70 vs $-\varphi$ 0.98; LOFO 0.72 vs 0.94; 29/64 collapsed models return from the grid cap. A100: 0.61; AUROC 0.60 vs 0.97; LOFO 0.78 vs 0.91; 17/28 at the cap.
P2	1	Finishing law: gain of full-precision finishing = return rate $\times$ gap; supported if $\|$ pred $-$ obs $\|\leq\max(2\text{ pts},25\%)$ on $\geq$ 80% of matched models.	Confirmed : 20/20 matched models, mean $\|$ pred $-$ obs $\|$ 1.52 points, $r=0.997$ (L40S).
P3	1	Sampling: the halting head picks draws with larger $t_{c}$ (sign test $p<0.05$ in each model), and the best-of-16 gain peaks at intermediate return rates.	Falsified . The head picks larger- $t_{c}$ draws in HRM w3c (57/18) and MLP w4c (15/1), not attention w3c (64/67); the gain is largest at the lowest return rate. The head does pick correct draws (picked-correct vs median draw: attention 68/0, HRM 89/0, MLP 62/0 on the L40S platform; 72/0, 83/1, 68/1 on the A100 platform).
P4	1	Radius law: $\sigma_{1/2}\approx 0.1\times$ median $t_{c}$ at noise 0.1 removes the tolerance law’s mixer bias.	Falsified : $R^{2}(\log)=-0.84$ , worse than the laws with $S$ and $S_{\text{mix}}$ .
K1	2	$k\leq 4$ full-precision finishing steps recover $\geq$ 80% of the examples full precision solves, on collapsed models.	Falsified on the L40S platform, partly on the A100 platform; 8–16 steps needed (Table 19 ).
K2	2	w8c finishing within 2 points of full-precision finishing on $\geq$ 90% of compressed models (at $k=4$ and 16).	Partly on the L40S platform (0.84 of models), confirmed on the A100 platform (0.93); median $\|$ w8c $-$ fp32 $\|$ 0.003 (L40S), 0.000 (A100).

Appendix

Table 18: Every prediction and its outcome, the falsified ones included. Order is the order in which the predictions and their scoring rules were written, each before any record it scores existed. P1–P4 are scored on the L40S return records (255 compressed models; 191 scored, 64 collapsed); the A100 replicate covers the released models (148; 91 scored, 28 collapsed) and scores P1 and the head-pick part of P3 only. The two platforms are scored separately. The looped-LM return test (§ G.1 ) has its own predictions, also written before any record they score existed.

Platform	$k$ (full-precision finishing steps)	1	2	4	8	16
L40S	recovery (median model)	0.28	0.50	0.71	0.83	0.89
	retained accuracy	0.29	0.51	0.75	0.88	0.96
A100	recovery (median model)	0.32	0.67	0.80	0.87	0.88
	retained accuracy	0.33	0.69	0.81	0.88	0.96

Appendix

Table 19: Recovery on collapsed models climbs with the number of finishing steps, on both platforms alike. Medians over collapsed models, full-precision finishing; recovery = share of the examples full precision solves whose decoded answer equals full precision’s after the same

k

loops; retained = accuracy after finishing over full-precision accuracy. The compressed model alone retains 0.05 (L40S) and 0.01 (A100); surviving models retain 0.97 and 1.00 already at

k{=}1

. L40S platform: 36 record files, 201 compressed models with full-precision accuracy

\geq

10%, 65 collapsed (TRM released checkpoints, trained width and seed variants, quantization-aware-trained and fine-tuned models, HRM, on Sudoku-Extreme and Maze-Hard; DeepThinking mazes; MDEQ-Small/XL). A100 platform: 12 files, 100 compressed models, 28 collapsed (released models). The two platforms are never pooled.

Model	Family	Full precision	$\sigma_{1/2}$	$S(0.05)$	$S\cdot\sigma_{1/2}$
TRM Sudoku MLP	TRM	0.715	0.14	1.74	0.24
TRM Sudoku attention	TRM	0.727	0.80	0.69	0.55
TRM Maze attention	TRM	0.875	1.71	0.32	0.55
HRM Sudoku	HRM	0.477	1.05	0.35	0.37
HRM Maze	HRM	0.748	1.71	0.23	0.40
DeepThinking maze 13	DeepThinking	1.000	0.37	0.92	0.34

Appendix

Table 20: L40S models: the implied room spans 0.19 to 0.89. full precision = full-precision accuracy (DEQ-Transformer: perplexity, lower is better);

\sigma_{1/2}

= measured tolerance;

S(0.05)

= sensitivity at

\sigma_{0}=0.05

; implied room =

S\cdot\sigma_{1/2}

. L40S platform.

Variant	Full precision	$\sigma_{1/2}$	$S(0.05)$	$S\cdot\sigma_{1/2}$
attention w512 s1	0.629	0.65	0.80	0.52
attention w512 s2	0.534	0.60	0.85	0.51
MLP w256 s0	0.599	0.14	1.46	0.21
MLP w256 s1	0.623	0.13	2.05	0.27
MLP w256 s2	0.640	0.13	1.30	0.17
MLP w512 s0	0.836	0.16	1.50	0.25

Appendix

Table 21: A100 models: the nine scored Sudoku variants, with implied room 0.17 to 0.52. Columns as in Table 20 . Excluded from the fit: attention w256 seeds 0–2 (full-precision accuracy 10–28%, undertrained) and attention w512 s0 (

\sigma_{1/2}>0.7

, not located). A100 platform,

\sigma_{1/2}

from

n=1000

Seed	Full precision	w4t ( $\varphi$ )	predicted $\sigma_{1/2}$	measured $\sigma_{1/2}$	error
attention s3	68.6	27.9 (0.741)	0.537	0.709	$\times$ 1.32
attention s4	71.1	60.1 (0.840)	0.563	0.578	$\times$ 1.03
attention s5	64.3	57.7 (0.901)	0.603	0.723	$\times$ 1.20
MLP s3	82.1	3.9 (0.559)	0.268	0.176	$\times$ 1.52
MLP s4	81.3	6.1 (0.534)	0.284	0.158	$\times$ 1.80
MLP s5	82.4	7.8 (0.498)	0.236	0.147	$\times$ 1.60

Appendix

Table 22: New mixer seeds: attention tolerates about 4

\times

the weight noise of the MLP mixer, and every parameter-free prediction lands inside the band. Full-precision and w4t (per-tensor INT4) accuracy from the seed evaluation,

\varphi

in parentheses; measured

\sigma_{1/2}

from our noise sweep; error = measured/predicted as a factor. L40S platform,

n=1000

, three noise draws.

Law	$R^{2}$	LOO median / worst	new seeds median / worst	bias MLP / attn / HRM
$\sigma_{1/2}=\delta^{\star}/S$	0.84	$\times$ 1.47 / $\times$ 2.07	$\times$ 1.39 / $\times$ 1.61	1.37 / 0.64 / 0.89
$\sigma_{1/2}=\delta^{\star}/S_{\text{mix}}$	0.91	$\times$ 1.35 / $\times$ 1.96	$\times$ 1.16 / $\times$ 1.29	1.19 / 0.76 / 0.98

Appendix

Table 23: The token mixer’s sensitivity shrinks the mixer bias but keeps its sign (post hoc). LOO = leave one model out; errors are factors; bias = predicted/measured tolerance on the fit set, where above 1 over-predicts. L40S platform only; 15 fit models (5 released TRM/HRM and 10 variants with full-precision accuracy

\geq

30%); the six new seeds held out.

Model	$S$	$S_{\text{mix}}$	implied room	pred/meas ( $S$ )
LSQ-QAT s0 / s1 / s2	0.92 / 1.21 / 0.95	0.55 / 0.95 / 0.69	0.21 / 0.28 / 0.22	1.58 / 1.17 / 1.55
fine-tune s0	1.57	1.09	0.21	1.57
MLP w256 s0 / s1 / s2	1.29–2.05	1.05–1.86	0.19 / 0.27 / 0.17	1.73 / 1.22 / 1.97

Appendix

Table 24: LSQ widens tolerance by lowering the sensitivity; the room stays put. Sensitivity

S

and

S_{\text{mix}}

; implied room

\sigma_{1/2}\cdot S

; pred/meas = the law (

\delta^{\star}=0.332

, refit on the 15-model set) over measured tolerance. Comparators, both trained in full precision: a fine-tune of the same length (s0) and the MLP w256 siblings. L40S platform.

Model	Format	bf16	compressed	+precise	$\varphi$	$\rho$	return (teacher-forced)	return (answer)
Huginn	w8c	42	36	41	1.000	0.25	100%	80%
Huginn	w4g32	42	35	39	0.978	0.25	97%	67%
Huginn	w4c	42	33	37	0.929	0.25	91%	55%
Huginn	w3a †	42	3	40	0.782	0.25	85%	47%
Huginn	w4t †	42	2	38	0.696	0.25	89%	54%
Recurrent-OLMo-2	w8c	41	43	43	1.000	0.25	100%	86%

Appendix

Table 25: Return test on all 24 compressed looped language models: in Huginn and Recurrent-OLMo-2, collapsed states return nearly as often as surviving ones. Accuracy %; collapse = retained below 50% (

\dagger

); +precise = second half of the loops in bf16; return (teacher-forced) and return (answer) = teacher-forced and answer-level return. L40S platform, transformers 4.55.4, GSM8K test[:100], batch 16.

Loops	bf16	w8c	w4g32	w4c	w4t
4	44.0	43.0 (1.00)	41.0 (0.98)	31.0 (0.942)	0.0 (0.12)
8	44.0	51.0 (1.00)	40.0 (0.98)	37.0 (0.947)	0.0 (0.11)
16	45.0	47.0 (1.00)	41.0 (0.98)	36.0 (0.947)	0.0 (0.12)
32	42.0	44.0 (1.00)	42.0 (0.98)	38.0 (0.946)	0.0 (0.11)

Appendix

Table 26: Recurrent-OLMo-2 keeps its accuracy and fidelity across loop counts. Accuracy % with

\varphi

in parentheses; teacher-forced KL(bf16

\|

q): w4g32 0.02, w4c 0.09, w4t 9.6. A100 platform, GSM8K test[:100], 2-shot greedy; Wilson half-width

\approx\pm 10

points.

Stopping rule	$\Delta$ accuracy	Mean steps
Halting head, rule frozen per model	$+1.23$	8.15
Step-size threshold, frozen on full precision	$-0.36$	6.04
Step-size threshold, tuned per model	$-0.43$	–
Stop when the step size stops shrinking	$-11.2$	7.2

Appendix

Table 27: Only the halting head stops early and gains accuracy over the protocol depth. Change in held-out exact accuracy (points) against running every puzzle to the 16-step protocol depth, and mean steps per puzzle; means over the 46 models of the grid (compressed, full-precision and weight-noise models). Every rule is capped at 16 steps and frozen on full-precision development rows unless marked. Held-out rows 1000–1999 for Sudoku (

n{=}1000

) and 500–999 for Maze-Hard (

n{=}500

); one platform (A100 platform).

Compressed format	w4	w3	w3g32	w2g32	w2
$b_{\text{eff}}$	4	3	3.5	2.5	2
Full-precision finishing, $r=32/b_{\text{eff}}$	8	10.7	9.1	12.8	16
8-bit finishing, $r=8/b_{\text{eff}}$	2	2.67	2.29	3.2	4

Appendix

Table 28: An 8-bit finishing step costs two to four compressed steps; a full-precision one costs eight to sixteen. Cost

r

of one finishing step in compressed steps,

r=b_{f}/b_{\text{eff}}

, by compressed format;

b_{\text{eff}}

counts the bits read per weight including group scales. Weight traffic only.

Model	Format	$r$	Fixed	Stop	Finish all	Controller	Low confidence	Stop, sample	+Sampling
TRM-MLP	w3g32	9.1	59.1/32	61.5/22	74.2/51	74.2/51	74.1/32	71.2/52	76.4/56
TRM-MLP	w4c	8.0	77.5/64	77.5/19	76.5/48	77.5/19	76.5/18	82.6/36	82.6/36
TRM-MLP	w4t	8.0	7.0/64	7.0/64	67.3/42	67.3/42	66.8/40	7.7/64	67.3/42
TRM-attn	w3c	10.7	82.9/64	80.5/17	80.6/58	80.5/17	79.7/40	86.3/32	86.9/34
TRM-attn	w4c	8.0	83.8/64	83.7/15	79.4/47	83.7/15	79.8/15	86.8/32	86.8/32
HRM	w2g32	12.8	20.8/64	20.5/62	37.0/52	37.0/52	34.6/55	26.0/64	37.0/52

Appendix

Table 29: The controller’s gain sits on the collapsed models; surviving models keep their accuracy at a fraction of the cost. Each entry is held-out accuracy (%) / mean cost in compressed steps, at a budget of 64, with full-precision finishing throughout;

r

is the cost of one full-precision finishing step. Stop: stop at the halting head. Finish all: finish every puzzle. Controller: stop, then finish. Low confidence: finish when the head’s confidence is low. Stop, sample: stop and sample, no finishing. +Sampling: the controller with the sampling add-on. Configurations are chosen on development rows (Sudoku 0–999, mazes 0–499) and scored on held-out rows (Sudoku 1000–1999, mazes 500–999). One platform (A100 platform, A100-40GB cards).

		Full-precision finishing			8-bit finishing
Model	Format	Head	Random	Finished	Head	Random	Finished
TRM-MLP	w3g32	74.1	69.2 [68.2, 70.1]	0.64	74.8	69.4 [68.3, 70.4]	0.64
TRM-MLP	w4c	76.5	74.1 [73.6, 74.6]	0.27	76.8	74.2 [73.6, 74.7]	0.27
TRM-MLP	w4t	66.8	66.8 [66.8, 66.8]	1.00	67.6	67.6 [67.6, 67.6]	1.00
TRM-attn	w3c	76.9	72.4 [71.6, 73.3]	0.27	76.9	72.5 [71.6, 73.3]	0.27
TRM-attn	w4c	79.4	78.7 [78.4, 79.0]	0.21	79.6	78.7 [78.5, 79.1]	0.21

Appendix

Table 30: The head chooses which puzzles to finish better than chance in every model where the gate is active. Gated finishing (finish the puzzles on which the head has not fired by

D_{0}

) against random sets of the same size at identical cost; held-out accuracy (%), budget 64. Random: mean [2.5, 97.5 percentile] over 1000 permutations; “Finished” is the fraction of puzzles finished. Rows at 1.00 finish every puzzle, where the two arms coincide. One platform (A100 platform, A100-40GB cards).

	TRM-MLP w4c	TRM-attn w3c	HRM w3c
Stochastic rounding, best of 16 (held out)	93.7	94.2	79.2
Full precision + latent noise, best-of-16	88.0	87.1	73.3
Difference [95% CI]	$+5.7$ [ $+4.1$ , $+7.4$ ]	$+7.1$ [ $+5.3$ , $+9.0$ ]	$+5.9$ [ $+3.8$ , $+8.0$ ]
Stochastic-rounding draws (rows 0–999)	86.9	86.4	71.9
Gaussian noise at the draws’ variance (rows 0–999)	87.0	85.7	71.7

Appendix

Table 31: Redrawing the rounding and keeping the draw the head trusts most beats full precision sampled with tuned latent noise, and matched Gaussian weight noise does as well as the rounding draws. Exact accuracy (%). Top: held-out rows 1000–1999,

K{=}16

draws,

\sigma

tuned on rows 0–999, with paired bootstrap 95% intervals for the difference. Bottom: development rows 0–999 only. One platform (L40S platform),

n{=}1000

Explore similar work

Sparse Layers are Critical to Scaling Looped Language Models
May 9, 2026Ryan Lee, Jacob Biloki, Edward J. Hu +1Transformer Architectures Layer-Wise
1USC Information Sciences Institute · 2Netflix · 3Independent Researcher
Closing the Loop: Practical Training Recipes for Looped Language Models
Sep 30, 2026Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin +5Recurrent Model Loop
RND NLP, DAIMLD, Russian Federation
Thinking with Looped Flows
Sep 12, 2026Ayhan Suleymanzade, Chanhyuk Lee, Floor Eijkelboom +3Deep Learning Recurrent State
EPFL · AITHYRA · KAIST +4

May 9, 2026, cs.LGJ/K move · Enter open · S save

Sparse Layers are Critical to Scaling Looped Language Models

Ryan Lee, Jacob Biloki, Edward J. Hu, Jonathan May

1USC Information Sciences Institute · 2Netflix · 3Independent Researcher

Transformer Architectures Layer-Wise Localized Lora-Moe Mixture-Of-Experts Peak Memory

Open paper AskPDF arXiv

Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries. However, looped models do not scale as favorably as standard transformers with unique layers. We compare standard and Mixture-of-Experts (MoE) transformers, with and without looping, and find two main results. First, we find Looped-MoE models scale better than the standard baseline while dense looped models do not. We trace this to routing divergence between loops: in Looped-MoE models, different experts are activated on each pass through the same shared layers, recovering expressivity without additional parameters. Our second finding is that looped models have better compute-quality trade-offs with early exits than standard models. Because each loop ends with the same layers that produce the final output, loop boundaries are superior exit points, as confirmed by earlier output convergence at these points. In sum, we provide a clear direction for scaling looped models: a Looped-MoE model with early exits can not only beat standard transformers at scale, but also enable significant memory and inference savings with minimal degradation in quality.

Abstract

Explore similar work

Sparse Layers are Critical to Scaling Looped Language Models

Closing the Loop: Practical Training Recipes for Looped Language Models

Thinking with Looped Flows