Learning to Compute Is Not Learning to Retain or Reuse in Looped Transformers

Peilong Zhou; Hongtao Zhang; Yulong Huang; Cangyuan Li; Mingyu Chen; Ying Wang

cs.LGOct 14, 2026

Learning to Compute Is Not Learning to Retain or Reuse in Looped Transformers

Authors: Peilong Zhou, Hongtao Zhang, Yulong Huang, Cangyuan Li, Mingyu Chen, Ying Wang

Organizations: SKLP, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences · The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Looped transformers compute by applying one block repeatedly, so they can run for more loops than they were trained with. Recurrent reasoners are known to overthink: iterating past the trained depth can turn a correct answer into a wrong one. Such a late error can reflect failed computation, loss of retention, or a state that later computation cannot reuse. We separate these cases in in-context Mealy-machine execution, where a simulator supplies intermediate targets and exact counterfactuals for hidden-state interventions. Models trained to answer at loop 32 are fully correct there but keep only 22% of these answers at loop 64. Retention depends on what training asks for: supervising one randomly chosen position per example, instead of always the last, keeps 99.8%, and randomizing the number of training loops also helps; truncating gradients, as in progressive training against overthinking, does not. The computation itself is exact: position $k$ is computed at loop $k$ , and swapping its loop- $k$ hidden vector for the one from another input redirects every later position to that input's exact continuation (237/237 pairs in each of ten models). Retention, however, is not reuse. Even in models that keep nearly all answers, holding each position $k$ fixed from loop $k$ drops accuracy to 4--6%, whereas holding it from loop $k+4$ is harmless. A released recurrent maze network trained to avoid overthinking behaves the same way. Learning to compute is thus not learning to retain, and avoiding overthinking does not make a state reusable when its answer first appears.

Figures & tables

Figure 1: Task, recurrence, and where the loss is applied. (a) A Mealy machine (edge label: input/output), its in-context encoding, and the simulator targets

q_{k}

y_{k}

. (b) One loop of the shared block

B_{\theta}

;

\diamond

marks where interventions replace

h_{k}^{r}

. (c) Each cell is one vector

h_{k}^{r}

(position

k

, loop

r

). Colored cells receive the loss at the exit

R

; rows above

R

are never trained and are where retention is measured.

Figure 2: Position coverage, exit distribution, and gradient propagation. Endpoint joint accuracy (a,c,d) and whole-trajectory exactness (b: all 32 state–output pairs correct) on 512 length-32 examples per model; thin curves are models, including incomplete learners, and thick curves their means. (a,b) Common cohort: one uniformly sampled supervised position per example preserves late accuracy without adding supervised positions. (c) Original cohort: Slack-4 and Jitter-8 match expected loop budgets. (d) Common cohort: progressive and matched random exit share supervised exits and forward work but differ in gradient truncation and backward work. Shading marks training exit support; the dashed line in (c) is Slack-4’s exit. Neither comparison matches measured training FLOPs.

Figure 3: One swapped state drives the exact remaining execution. (a) Schematic, $H=R=6$ , $c=3$ . The source’s vector at position 3 and loop 3, whose state is $B$ , replaces the base run’s; read at loop $k$ (the frontier), positions 4–6 then give $C$ , $D$ , $A$ , the source’s continuation $\tilde{q}_{k}$ . (b,c) Ten final-only models, $H=32$ , swap at position 8 and loop 8 ( $\diamond$ in c). Shading is the fraction of pairs whose state and output at $(k,r)$ both match the target: the base run’s own trajectory (b) or the source’s continuation after the swap (c); means over 256 pairs and ten models. The bright diagonal is the frontier; the fading above it shows answers decaying after they are computed, and some agreement away from it is chance. 237/237: eligible pairs whose whole suffix ( $k=9,\ldots,32$ ) matches along the frontier.

Figure 4: Answers can be cached when they appear; their states can be frozen only a few loops later. (a) Freezing captures each position’s own hidden vector at loop

k

and writes it back after every later block; other positions update normally, and no other run or label is involved. (b,c) Six final-only , three all-prefix , and three larger final-only models, 512 H32 examples each. (b) Fraction of examples with all 32 state–output pairs correct at loop 32. Cache keeps each position’s loop-

k

prediction outside the model; freeze holds the hidden vectors instead, for all positions or only positions 1–8. Dots are models; short marks are group means. (c) Accuracy of each position at loop 32 after freezing all positions (group means); the dashed line is the cache. (d) sampled-prefix and matched random exit , three seeds each (thin lines) and their means (thick), with capture delayed by

d

loops: endpoint accuracy at the actual capture loop

32+d

, unchanged through loop 64; the dashed line is clean. Clean readouts at

s(k)=k

are not perfectly correct for two matched random exit seeds (Appendix D.3 ).

Appendix figures & tables32 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1: Network details. (a) An example is one sequence: the initial state

q_{0}

, one factorized token

(q,a,q^{\prime},y)

per transition, and the

H

action symbols. It is embedded once as

E(x)

, with

T=P+H

rows of width

d

, and added before every application of

B_{\theta}

. (b)

B_{\theta}

stacks

L

pre-normalized layers with distinct weights:

L=2

d=128

, four heads, and MLP width 512 in the main models;

L=4

and

d=512

in the larger models. There is no dropout, recurrent gate, layer scaling, or positional or loop embedding. (c) Rows are queries and columns keys; filled cells are visible. Prefix tokens attend within the prefix, and action position

k

attends to the prefix and positions

1{:}k

. Schematic with three prefix tokens and five actions; padding is masked. (d) At loop

r

, endpoint accuracy scores position

H

; whole-trajectory exactness requires all

H

positions of an example to be correct at that same loop. A position is correct when its state and output are both correct.

Maximum horizon	Stages	Steps	Machine distribution
1	1	2,000	4 states, 2 inputs, 2 outputs
2	4	7,000	2,000 steps each with 4, then 4/6, then 4/6/8 states; 1,000 steps adding the 4-input cases
3	1	3,000	4/6/8 states, 2/4 inputs, 2 outputs
4	1	8,000	Same full distribution
5–32	28	112,000	4,000 steps per successive maximum horizon

Appendix

Table A1: The 35-stage curriculum. A maximum horizon is sampled with probability .5 after the first stage; the remaining draws come from shorter horizons. All stages include the shorter-length range from 1. Rows grouping several stages retain their individual training allocations below.

Condition	$A_{32}$	$A_{40}$	$A_{64}$	$A_{96}$	$T_{32}$	$T_{64}$	$T_{96}$
final-only	100.00	28.39	22.33	22.07	0.00	0.00	0.00
all-prefix	100.00	100.00	94.27	82.16	100.00	74.28	67.45
sampled-prefix	100.00	100.00	99.80	96.61	100.00	96.03	74.09
matched random exit	100.00	100.00	97.85	86.26	74.09	29.56	11.52
progressive	69.27	70.31	58.53	38.09	12.37	0.00	0.00

Appendix

Table A2: Common-stream H32 training comparison. Three-seed means in percent, all seeds 9–11, 512 examples per model, stream 20260923.

A_{r}

is endpoint joint accuracy and

T_{r}

is whole-trajectory exactness at loop

r

. The serial conditions train at

R=32

; the two exit controls supervise exits in 32–40. All rows use the same evaluated inputs.

Condition	Seed	Peak $A$	$A_{32}$	$A_{40}$	$A_{64}$	$A_{96}$	$T_{32}$	$T_{64}$	$T_{96}$
final-only	9	100.00	100.00	27.15	24.02	23.24	0.00	0.00	0.00
final-only	10	100.00	100.00	33.59	20.51	19.14	0.00	0.00	0.00
final-only	11	100.00	100.00	24.41	22.46	23.83	0.00	0.00	0.00
all-prefix	9	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
all-prefix	10	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
all-prefix	11	100.00	100.00	100.00	82.81	46.48	100.00	22.85	2.34

Appendix

Table A3: Every model in the H32 training comparison. Same metric, stream, and population as Table A2 ; no seed is excluded. Peak

A

is the maximum endpoint accuracy over the complete

r=1

–96 sweep and is reported to distinguish failure to learn from later loss.

Condition	Seed	$A_{64}$	$A_{72}$	$A_{128}$	$A_{192}$	$T_{64}$	$T_{128}$
final-only	9	100.00	21.68	18.16	17.97	0.00	0.00
final-only	10	93.36	33.59	13.67	13.48	0.00	0.00
final-only	11	100.00	27.34	20.31	19.92	0.00	0.00
final-only	Mean	97.79	27.54	17.38	17.12	0.00	0.00
all-prefix	9	100.00	100.00	99.61	96.29	100.00	97.85
all-prefix	10	100.00	100.00	100.00	99.22	100.00	100.00

Appendix

Table A4: H64 extrapolation of all five training conditions. Every seed and its condition mean, 512 examples per model, stream 20260923.

A_{r}

and

T_{r}

are endpoint and whole-trajectory accuracy in percent. H64 exceeds the trained horizon range for every condition. These scores are length extrapolation, not H32 over-budget retention.

Inference loops $r$	32	33	36	40	48	64
final-only	100.00	74.90	52.41	39.78	33.01	30.50
all-prefix	100.00	100.00	100.00	100.00	100.00	97.85

Appendix

Table A5: Paired endpoint accuracy beyond the training exit.

A(32,r)

in percent, averaged over six model seeds 9–14, 512 examples per seed, evaluation stream 20260909. The input length, sample stream, and metric are fixed across columns.

Model seed	9	10	11	12	13	14
final-only , $r=33$	56.25	73.44	51.56	94.14	88.48	85.55
all-prefix , $r=33$	100.00	100.00	100.00	100.00	100.00	100.00
final-only , $r=64$	33.40	23.83	26.37	31.05	36.52	31.84
all-prefix , $r=64$	100.00	100.00	87.30	99.80	100.00	100.00

Appendix

Table A6: Endpoint joint accuracy by model seed. Entries are percentages on 512 length-32 examples per model, evaluation seed 20260909. The model is frozen while the inference budget changes.

Figure A2: Original readability grids for the serial baselines. This historical view uses seeds 9–14 and evaluation stream 20260909; each cell counts models whose state and output marginal accuracies at that position and loop are both at least 99%. This criterion differs from sample-level whole-trajectory exactness in Figure 2 . It is retained separately from the common-stream training comparison.

Contract	Training budget	Interpretation at $H=32$
Serial	$R=H$	Trained exit 32
Double	$R=2H$	Trained exit 64
Slack-4	$R=H+4$	Trained exit 36
Slack-8	$R=H+8$	Trained exit 40
Jitter-8	$R=H+U\{0,\ldots,8\}$	Trained support 32–40
Random-2x	$R=H+U\{0,\ldots,H\}$	Trained support 32–64

Appendix

Table A7: Training budget functions.

U

is discrete uniform. Slack-4 and Jitter-8 have the same conditional expected budget; the other rows do not form an equal-compute comparison.

Inference loops $r$	36	48	64	96
Slack-4	93.98	38.15	30.01	26.79
Jitter-8	87.27	87.17	85.55	79.79

Appendix

Table A8: Endpoint retention at nearly matched recurrent work. H32 endpoint joint accuracy in percent, averaged over all six paired seeds; 512 examples per model, evaluation seed 20260909.

Model seed	9	10	11	12	13	14
Slack-4	32.23	31.25	33.79	34.96	15.04	32.81
Jitter-8	95.51	100.00	100.00	57.23	60.55	100.00

Appendix

Table A9: The loop-64 retention comparison by seed. H32 endpoint joint accuracy in percent, using the same evaluation stream and population size as Table A8 .

Condition	$r=64$	$r=96$
final-only	0.07	0.07
all-prefix	86.39	72.46
Slack-4	0.00	0.00
Jitter-8	54.46	47.40

Appendix

Table A10: Whole-trajectory exactness after the H32 exit. Percent of 512 length-32 examples, averaged over six model seeds 9–14; evaluation stream 20260909. Every action position must have correct state and output. The endpoint means at

r=64

are shown in Tables A5 and A8 ; the two metrics measure different failure modes.

Seed	9	10	11	12	13	14
final-only	88	56	72	56	48	56
all-prefix	72	88	48	48	64	80

Appendix

Table A11: The paired sampled length boundary has mixed changes.

B_{\mathrm{final}}

is defined by final-position joint accuracy, not by all-position trajectory exactness.

Figure A3: Retention training changes the two readouts of an early swap differently, but does not protect a delayed swap. Swap at position 16; six paired models per condition. Exact suffix requires every state and output at positions 17–32 to match the target. (a) Unswapped runs scored against their own trajectory, all 256 pairs. (b) Swapped runs scored against the source continuation, 242 eligible pairs. Frontier reads position

k

at loop

k

; exit reads every position at loop 32. Lines join the two readouts of one model; bars are means. (c) The source vector is taken and inserted

a

loops after the frontier, and position

k

is read at loop

k+a

(solid), next to the matching unswapped run (dashed); means over six models. final-only is faded because its unswapped runs are no longer readable at

k+a

. Clean and source scores use different targets and denominators and are not subtracted.

		final-only		all-prefix
Readout	$a$	No swap	Swap	No swap	Swap
Frontier	0	1.000	1.000	1.000	.939
	1	.249	.169	1.000	.177
	2	.133	.021	1.000	.102
	4	.031	.002	1.000	.000
Exit	0	.007	.002	1.000	.722

Appendix

Table A12: Ordinary retention is essential to interpreting a delayed swap. Exactness of positions 17–32, averaged over six paired models. The shifted-frontier schedule reads position

k

k+a

; the exit schedule reads every position at loop 32.

Source loop	Base loop	final-only	all-prefix
16	16	1.000	.946
20	16	.196	.260
16	20	.267	.000
20	20	.169	.000

Appendix

Table A13: Source collection and base insertion at crossed times. All vectors are at position 16. Position 17 is read one update after insertion, using the same 242 eligible pairs and six model seeds. Entries are joint state–output agreement with the source target.

Recipient vectors replaced at loop 20	final-only	all-prefix
Cycle 16, donor vector from loop 16	.163	.161
Cycle 16	.000	.000
Cycle 20 (current frontier)	1.000	.935
Cycles 16–20	1.000	.934
Cycles 17–20	1.000	.934

Appendix

Table A14: Replacement extent at loop 20. Position 21 is read at loop 21, on 121 eligible pairs. Both clean base runs are correct on every pair. Unless specified, all source vectors are collected at loop 20.

Figure A4: Resetting the downstream position restores the use of a fresh source, but not of an equally readable aged source. (a) The base run is held at loop 20 (gray band). Position 16 may receive the source’s position-16 vector from loop 16 (fresh, solid) or loop 20 (aged, dashed), and position 17 may be reset to the base run’s own loop-16 vector; other positions keep their loop-20 vectors. After one update, position 17 is scored at loop 21 (

\odot

). Shaded cells are readable under all-prefix ; both source vectors decode the correct state and output on every pair. (b) Rows name the replacements at loop 20 (none: unmodified run). Percentage of the 242 eligible pairs whose position-17 state and output both match the counterfactual target, not factual accuracy or suffix exactness; dots are six all-prefix models, bars their means, and the dashed line the early swap. final-only results are in Table A15 and controls in Table A17 .

Intervention	final-only	all-prefix
Fresh source reference	$100.00\pm 0.00$	$94.63\pm 8.14$
Late clean ( $A_{00}$ )	$16.67\pm 3.64$	$0.00\pm 0.00$
Late fresh source only ( $A_{10}$ )	$26.72\pm 26.17$	$0.00\pm 0.00$
Late downstream reset only ( $A_{01}$ )	$19.63\pm 2.97$	$26.17\pm 3.40$
Late fresh source + reset ( $A_{11}$ )	$100.00\pm 0.00$	$94.56\pm 8.10$
Late aged source only	$16.87\pm 3.77$	$0.00\pm 0.00$

Appendix

Table A15: Single-step counterfactual accuracy after downstream recovery. All values are percentages, reported as mean

\pm

sample standard deviation over six model seeds. Each model uses the same 242 eligible pairs. Late arms intervene at loop 20 and read position 17 at loop 21; the fresh reference intervenes at loop 16 and reads at loop 17.

Condition	Late intervention	9	10	11	12	13	14
final-only	Clean	53	37	35	34	33	50
	Source only	193	35	45	33	31	51
	Reset only	41	57	52	51	38	46
	Source + reset	242	242	242	242	242	242
all-prefix	Clean	0	0	0	0	0	0
	Source only	0	0	0	0	0	0

Appendix

Table A16: Per-seed primary outcomes. Entries are successful joint counterfactual predictions out of 242 eligible pairs for every cell. “Source” means the fresh loop-16 source inserted at loop 20.

Intervention	final-only	all-prefix
Fresh clean	$100.00\pm 0.00$	$100.00\pm 0.00$
Fresh source reference	$100.00\pm 0.00$	$95.18\pm 6.49$
Late clean	$47.27\pm 15.78$	$100.00\pm 0.00$
Late fresh source only	$57.23\pm 18.05$	$100.00\pm 0.00$
Late downstream reset only	$33.46\pm 8.66$	$25.13\pm 1.18$
Late fresh source + reset	$100.00\pm 0.00$	$95.18\pm 6.49$

Appendix

Table A17: Same-state/different-history controls. Factual joint accuracy in percent, mean

\pm

sample standard deviation over six seeds, using 256 pairs per model. Source and factual symbolic continuations coincide, so distinguishable counterfactual accuracy is undefined for this population.

Condition	Seed	Clean	Readout cache	Frontier freeze	Positions 1–8	One loop later
final-only	9	100.0	100.0	11.5	11.9	100.0
final-only	10	100.0	100.0	6.2	8.0	100.0
final-only	11	100.0	100.0	1.6	5.1	100.0
final-only	12	100.0	100.0	1.6	1.8	100.0
final-only	13	100.0	100.0	2.7	5.5	77.1
final-only	14	100.0	100.0	1.6	1.6	100.0

Appendix

Table A18: Endpoint joint accuracy (%) at loop 32 for

H=32

, 512 inputs per model. The larger models use

d=512

and four layers per block and a batch size of 32, so they are not paired input by input with the other rows.

Figure A5: Where continuing computation fails in the original freezing cohort. Joint state–output accuracy at each position’s scheduled readout on H32, stream 20260923. Thick curves are means within the three original model groups and thin curves show individual seeds. Positions that remain readable early in the sequence do not prevent failure at later positions under immediate freezing.

Condition	Seed	Clean $A_{32}$	Clean $T_{32}$	Min. diagonal	Cache $T_{32}$
sampled-prefix	9	100.00	100.00	100.00	100.00
sampled-prefix	10	100.00	100.00	100.00	100.00
sampled-prefix	11	100.00	100.00	100.00	100.00
matched random exit	9	99.80	99.02	96.48	94.92
matched random exit	10	100.00	32.03	99.80	99.80
matched random exit	11	100.00	73.83	100.00	100.00

Appendix

Table A19: Clean diagnostics for every new freeze model. H32, stream 20260924, 512 inputs per model; percentages.

A_{32}

and

T_{32}

denote endpoint and whole-trajectory accuracy at loop 32. “Min. diagonal” is the minimum over positions of clean joint accuracy at

r=k

. Cache

T_{32}

requires every saved prediction to be correct.

Condition	Clean	Cache	$+0$	$+1$	$+2$	$+4$	P8	P16
sampled-prefix	100.00	100.00	4.10	80.21	99.80	100.00	7.49	4.36
matched random exit	99.93	99.93	6.25	20.18	87.24	99.80	7.42	6.25

Appendix

Table A20: Endpoint accuracy at the scheduled capture readout. Three-seed means in percent on all 512 H32 examples per model, stream 20260924. The

+d

columns read the endpoint at

r=32+d

; clean, cache, P8, and P16 use

r=32

. P8/P16 freeze only the indicated prefix. Clean endpoint accuracy at

r=33,34,36

is 100% for every model; the clean

r=32

exception is reported in Table A19 .

Seed	Readout	Clean	Cache	$+0$	$+1$	$+2$	$+4$	P8	P16
9	$32$	100.00	100.00	3.91	100.00	100.00	100.00	3.91	3.71
9	$33$	100.00	100.00	3.91	100.00	100.00	100.00	3.91	3.91
9	$64$	100.00	100.00	3.91	100.00	100.00	100.00	4.10	4.10
9	Sched.	100.00	100.00	3.91	100.00	100.00	100.00	3.91	3.71
10	$32$	100.00	100.00	2.54	100.00	100.00	100.00	5.66	2.54
10	$33$	100.00	100.00	2.54	82.81	100.00	100.00	6.45	2.54

Appendix

Table A21: Every sampled-prefix endpoint score under freezing. Percent of 512 H32 examples, stream 20260924. Numerical readouts are common loops. “Sched.” reads the endpoint at

32+d

for

+d

and at 32 for clean/cache/P8/P16. Cache leaves the hidden computation live; P8/P16 freeze only positions 1–8/1–16. All eight arms and all seeds are included.

Seed	Readout	Clean	Cache	$+0$	$+1$	$+2$	$+4$	P8	P16
9	$32$	99.80	99.80	3.91	12.30	96.09	99.80	7.42	4.30
9	$33$	100.00	99.80	3.91	9.18	93.75	100.00	9.38	4.30
9	$64$	100.00	99.80	3.91	9.18	90.23	100.00	7.81	5.08
9	Sched.	99.80	99.80	3.91	9.18	90.23	100.00	7.42	4.30
10	$32$	100.00	100.00	13.28	12.89	97.66	92.77	13.09	12.70
10	$33$	100.00	100.00	13.28	13.28	73.44	97.85	14.26	12.89

Appendix

Table A22: Every matched random exit endpoint score under freezing. Same complete reporting and readout definitions as Table A21 ; 512 H32 examples per model, stream 20260924.

Seed	Readout	Clean	Cache	$+1$	$+2$	$+4$
9	$32$	100.00	100.00	100.00	100.00	100.00
9	$33$	100.00	100.00	100.00	100.00	100.00
9	$64$	100.00	100.00	100.00	100.00	100.00
9	Sched.	100.00	100.00	100.00	100.00	100.00
10	$32$	100.00	100.00	1.56	100.00	100.00
10	$33$	100.00	100.00	1.56	100.00	100.00

Appendix

Table A23: Every sampled-prefix whole-trajectory score under freezing. Percent of 512 H32 examples, stream 20260924. Numerical readouts require all positions to be correct simultaneously. “Sched.” instead reads each position at its own

k+d

for

+d

, or at

k

for clean/cache/P8/P16, on that arm’s trajectory.

Seed	Readout	Clean	Cache	$+2$	$+4$
9	$32$	99.02	94.92	85.74	99.80
9	$33$	99.22	94.92	83.40	100.00
9	$64$	71.88	94.92	82.62	100.00
9	Sched.	94.92	94.92	82.62	100.00
10	$32$	32.03	99.80	14.06	91.99
10	$33$	27.54	99.80	13.28	97.66

Appendix

Table A24: Every matched random exit whole-trajectory score under freezing. Same readout definitions as Table A23 ; 512 H32 examples per model, stream 20260924. Clean scheduled exactness and common-loop exactness differ because clean position states continue to evolve.

Condition	Seed	Frozen prefix
sampled-prefix	9	8
sampled-prefix	9	16
sampled-prefix	10	8
sampled-prefix	10	16
sampled-prefix	11	8
sampled-prefix	11	16

Appendix

Table A25: Exactness of the never-frozen suffix. Percent of all 512 H32 examples per model, stream 20260924. Freezing prefix 1–8 leaves suffix 9–32 live; freezing 1–16 leaves 17–32 live. Every position in that suffix must have both labels correct. The endpoint scores for these same arms appear in Tables A21 and A22 .

Arm	$9\times 9$ @30	$9\times 9$ @120	$33\times 33$ @120
Clean	32	32	19
Answer cache (state untouched)	32	32	19
Capture $+0$	0	0	0
Capture $+1$	3	3	0
Capture $+2$	28	28	2
Capture $+4$	32	32	18

Appendix

Table A26: Solved mazes out of 32 for the released recall network. “Capture

+d

” freezes each pixel

d

iterations after its answer arrival. Freezing every pixel at iteration 30 is harmless where the maze is already solved and fatal where computation is still running.

$d$	Seed	$\|E\|$	31	32	Both	Adjacent	Earlier	Sham	Broken
$128$	9	224	165	61	224	0	52	0	1 / 148 / 0
$128$	10	136	95	4	136	18	12	0	2 / 201 / 0
$128$	11	248	195	53	248	15	46	0	10 / 130 / 0
$128$	12	30	18	3	30	5	2	0	90 / 350 / 0
$128$	13	59	54	13	59	13	11	0	32 / 291 / 0
$128$	14	74	58	9	74	2	10	0	2 / 242 / 0

Appendix

Table A27: Recovered members of

E

at loop 33. Columns: position 31 reset to its loop-31 vector (31), position 32 reset to its loop-31 vector (32), both, position 30 reset to its loop-31 vector (adjacent), position 32 reset to its loop-30 vector (earlier), and a same-phase self-replacement of both positions (sham). The last column counts inputs correct at both loops that each arm breaks at loop 33, for 31 / 32 / both.

Recurrent Transformers

Recurrent Transformers reuse layers or state across steps to extend context, reasoning depth, and sequence modeling with less parameter growth.

See all 180 papers

May 29, 2026

Chain-of-Thought and Compressed Looped Transformers: A Memory-Budget Separation

Chain-of-thought prompting and looped Transformers both give a fixed model more test-time computation, but they differ in what they remember. Chain-of-thought stores intermediate state in generated tokens that remain in the context, whereas a looped Transformer carries state through recurrent hidden activations. We argue that this persistent mutable memory is a central resource for test-time reasoning. We compare three memory regimes, the compressed latent loop, the full sequence-state loop, and the chain-of-thought scratchpad. Our main result shows that a compressed loop is limited by the size of its recurrent state. Running the loop longer adds computation but does not by itself create a growing scratchpad, so a loop with a small recurrent state remains a small-space reasoner even when run for many steps. Under a standard complexity assumption, such loops cannot decide problems that are P-complete under logspace reductions, whereas polynomial-length chain-of-thought can. The separation is specific to compressed loops, as full sequence-state loops carry state at every input position and live in a memory-rich regime closer to explicit scratchpads. Controlled pointer-chasing and associative-recall sweeps illustrate this memory-budget view, with performance sensitive to whether the persistent-state budget matches the task's working-memory demand.

Haozhou Zhang

Department of Mathematics and Statistics University of Idaho

CoT Reasoning Persistent Memory for Language Models Recurrent Transformers

Ask PDF

Sep 27, 2026

Beyond the Training Horizon: Mechanisms and Limits of Length Generalization in Looped Transformers

Looped Transformers can generalize to reasoning chains longer than those encountered during training, but the computations enabling this behavior and limiting its extent remain unclear. We mechanistically compare two looped-Transformer configurations, which we call the Matched-Recurrence Looped Transformer (MR-Loop) and Decoupled-Recurrence Looped Transformer (DR-Loop), reflecting their respective recurrence-training schemes. We evaluate polynomial iteration, finite-state composition, and knowledge-graph traversal using detailed mechanistic analysis. Attention analysis, intermediate-state decoding, and causal interventions reveal distinct mechanisms learned under final-answer supervision. MR-Loop updates an intermediate state at a fixed readout while advancing relation selection through adjacent-token interactions and a transferable progress cue. DR-Loop instead propagates intermediate states across relation positions, forming an advancing computational frontier. However, both mechanisms become unreliable at greater depths: MR-Loop exhibits degradation of its readout state and progress cues, while DR-Loop exhibits declining reliability of state propagation. Limited self-correction allows local errors to persist and compound. Across both models, we uncover a common representational principle: recurrent states encode not only task-relevant content but also its computational status, whether that content remains in a form that can support subsequent computation. Transferable live-consumed and fresh-aged residual directions causally control whether represented information can participate in subsequent computation, including beyond the training horizon. We further show that length generalization need not rely on faithful step-by-step reasoning, as Looped Transformers can exploit task structure without explicitly representing every intermediate state.

Jia Liang, Xi Jin, Liangming Pan

Institute for Computational and Mathematical Engineering, Stanford University · MOE Key Laboratory of Computational Linguistics, School of Computer Science, Peking University

Length Generalization Recurrent Transformers Algorithmic Reasoning Transformer Interpretability

Ask PDF

Mar 9, 2026

Adaptive Loops and Memory in Transformers: Think Harder or Know More?

Chain-of-thought (CoT) prompting enables reasoning in language models but requires explicit verbalization of intermediate steps. Looped transformers offer an alternative by iteratively refining representations within hidden states. This parameter efficiency comes at a cost, as looped models lack the storage capacity of deeper models which use unique weights per layer. In this work, we investigate transformer models that feature both adaptive per-layer looping, where each transformer block learns to iterate its hidden state via a learned halting mechanism, and gated memory banks, that provide additional learned storage. We find that looping primarily benefits mathematical reasoning, while memory banks help recover performance on commonsense tasks compared to parameter and FLOP matched models. Combining both mechanisms yields a model that outperforms an iso-FLOP baseline, with three times the number of layers, across math benchmarks. Analysis of model internals reveals layer specialization: early layers learn to loop minimally and access memory sparingly, while later layers do both more heavily.

Markus Frey, Behzad Shomali, Ali Hamza Bashir +3

Lamarr Institute · Fraunhofer IAIS · University of Bonn

Transformer Memory-Augmented Language Models Implicit Reasoning in Language Models Recurrent Transformers Numerical Reasoning in Language Models

Ask PDF

Abstract

Recurrent Transformers

Explore similar work

Chain-of-Thought and Compressed Looped Transformers: A Memory-Budget Separation

Beyond the Training Horizon: Mechanisms and Limits of Length Generalization in Looped Transformers

Adaptive Loops and Memory in Transformers: Think Harder or Know More?