Inside the Loop: A Mechanistic Study of Weight-Tied Transformers on Depth-Bound Algorithmic Tasks

Magnus Sesodia; Juil Sock; Philip Torr; Christian Schroeder de Witt

cs.LGOct 16, 2026

Inside the Loop: A Mechanistic Study of Weight-Tied Transformers on Depth-Bound Algorithmic Tasks

Authors: Magnus Sesodia, Juil Sock, Philip Torr, Christian Schroeder de Witt

Organizations: University of Oxford · BBC R&D

Abstract

Solving algorithmic tasks with many sequential steps typically requires scaling up transformer depth or chain-of-thought length, incurring unfavorable parameter and inference-compute costs, respectively. Looped transformers, which reuse one block of layers across many iterations, are poised to address this, since they scale effective depth without adding parameters or tokens. Yet, it remains unclear what each iteration is actually computing, or whether the loop merely simulates a deeper untied stack. Here, we study directed permutation traversal, in which a model follows $H$ hops from a starting node and reports the node it reaches, so every intermediate hop has a known ground truth. We find that a minimum effective depth is necessary to solve the task, beyond which the degree of weight tying becomes the determining factor. For instance, a looped transformer with a four-layer block iterated eight times solves the task in every seed, while an identically-trained untied transformer of the same effective depth never does, succeeding only with a lower learning rate or supervision of its intermediate layers. Probing and intervening on the residual stream, we find that the looped model advances through the intermediate nodes gradually, forming a staircase that adapts to its iteration budget, whereas the untied model, at that lower rate, exposes them only in its final layers. Taken together, our results recast looped transformers as iterative program learners, which allocate their iterations to successive steps of the algorithm and solve tasks that identically-trained untied models of the same effective depth do not. Code is available at https://github.com/magnus-sesodia/insidetheloop.

Figures & tables

Figure 1: Looped model forms a staircase. Linear-probe selectivity for each hop of the 8-hop dePo task (illustrated at left), in a $4\times 8$ looped model (left map) and an untied $32$ -layer model (right map), both trained at $\eta=3\times 10^{-4}$ . White dots mark where a hop first reaches selectivity $0.5$ , and crosses mark hops that never do.

Figure 2: Directed permutation traversal. The model reads the edges of a cycle in shuffled order and must follow them for

H

hops from the start node

s

. In this example

s=3

and

H=4

, so the walk visits the reference states

z_{1}=11

z_{2}=6

and

z_{3}=2

and ends at the answer

z_{H}=9

. Straight arrows are edges and curved arrows are hops, and the dashed segment stands for the nine nodes not drawn, which complete the

N=16

cycle.

Figure 3: Capability at learning rate

\eta=10^{-3}

. Each node is a model with

L

layers in its block, iterated

T

times, trained with five seeds. Its label and color give the fraction of seeds that solve all

15

hops, i.e., reach accuracy

0.8

on every hop length from

1

15

. Grey diagonals mark constant effective depth

D_{\mathrm{eff}}{=}LT

. The

T{=}1

column, left of the dashed line, holds the untied models.

Task	$\eta$	Looped $4\times 8$	Untied $32$ -layer
$N{=}16$	$10^{-3}$	$5/5$	$0/5$
$N{=}16$	$3{\times}10^{-4}$	$5/5$	$4/5$
$N{=}16$	$10^{-4}$	$1/5$	$1/5$
$N{=}16$ , untied $16$ layers	$3{\times}10^{-4}$	–	$1/5$
$N{=}16$ , auxiliary loss	$10^{-3}$	–	$2/5$
$N{=}32$	$3{\times}10^{-4}$	$H^{\star}=6,0,1$	$H^{\star}=1,1,1$

Table 1: Looped

4\times 8

against untied

32

-layer. The untied column holds the

32

-layer model unless the row names another depth. Seeds that solve all

15

hops at

N{=}16

, and solved-chain depth

H^{\star}\[email protected]

for each seed at

N{=}32

Figure 4: Looped vs. untied solution structures. Probe selectivity (§ 3.4 ) by layer and hop at $H{=}4$ : (a) the looped $4\times 8$ model at $\eta=10^{-3}$ ; (b) the untied $32$ -layer model at the same rate; (c) the untied model at the lower rate $\eta=3{\times}10^{-4}$ ; (d) the untied model at $\eta=10^{-3}$ with an auxiliary next-token loss on the outputs of layers $8$ , $16$ and $24$ (dashed lines). White dots, joined by black lines, mark the first layer at which each hop reaches selectivity $0.5$ , and faint grey lines in (a) separate iterations. Onsets for the other seeds are shown in Figure A2 .

Figure 5: How the loop spends its iterations. The first iteration at which each reference node becomes decodable (selectivity

0.5

) at

H{=}8

, for looped models with more iterations than hops (

2\times 16

, a), as many (

4\times 8

, b) and fewer (

8\times 4

, c). Fractional positions give the layer within the iteration, light vertical lines separate iterations. Markers in the never column are nodes that never become decodable. Each model was trained with five seeds, and a seed is drawn if its answer accuracy is at least

0.95

(three seeds in a and b, two in c). All models are trained at

\eta=10^{-3}

Figure 6: Logit lens and causal interventions. (a) Logit lens: probability that the looped $4\times 8$ model ( $H{=}8$ , one seed) writes each reference node when its loop is stopped after a given iteration and its state read through the final RMSNorm and unembedding head. (b) Share of answers that are the donor’s (solid) or the receiver’s (dashed) when the receiver’s states at the readout position and first answer digit are replaced by the donor’s after a given iteration ( $4\times 8$ , $H{=}6$ , three seeds, coinciding series offset). Donor and receiver share the permutation and differ in the start node. (c) Schematic of (b) at $H{=}4$ : the receiver walks from $3$ and writes $9$ , but with its state after iteration $2$ replaced by that of a donor starting at $0$ , it finishes the donor’s walk and writes $7$ .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1: Training loss at matched effective depth. Training loss of the untied

32

-layer model and the looped

4\times 8

and

2\times 16

models at

\eta=10^{-3}

, averaged over a trailing window of

2

k steps, for each of five seeds (thin) and their mean (thick). The dashed line marks the end of the hop curriculum at

30

k steps, before which the loss rises as longer chains enter training.

Figure A2: Where each hop first becomes decodable, every seed. First layer at which each hop of the

H{=}4

chain reaches probe selectivity

0.5

, one colour and marker per seed. Crosses in the never column mark hops that never do, and seeds are offset vertically by up to

0.2

hops so that coinciding points stay visible. (a) Looped

4\times 8

\eta=10^{-3}

, seeds

42

46

(row split, read at the end of each iteration, so onsets fall on multiples of

4

), and the prompt-disjoint re-probe of seed

42

(filled). (b) Untied

32

-layer model at

\eta=10^{-3}

(row split). (c) Untied at

\eta=3{\times}10^{-4}

(prompt-disjoint). (d) Untied with the auxiliary loss on layers

8

16

and

24

(grey vertical lines),

\eta=10^{-3}

, the two seeds that solve the task (row split). (e) Looped

4\times 8

\eta=3{\times}10^{-4}

(prompt-disjoint). The grey diagonal in (a) and (e) is one hop per iteration.

Model, $\eta$	Probes	Seed	Hop 1	Hop 2	Hop 3	Answer	Accuracy
Looped $4\times 8$ , $10^{-3}$	row split	42	8	12	16	16	1.000
	row split	43	8	12	16	20	1.000
	row split	44	8	12	16	20	1.000
	row split	45	8	16	20	28	1.000
	row split	46	8	12	–	32	0.650
	prompt-disjoint	42	8	11	15	16	1.000

Appendix

Table A1: First decodable layer and accuracy, every seed. First layer at which each hop at

H{=}4

reaches selectivity

0.5

(“–” never) and full-answer accuracy on the first query, for every seed of Fig. A2 . Looped onsets in the row split are resolved per iteration.

Figure A3: Forward-walk nodes are decodable. Peak probe selectivity for each node

\pi^{d}(s)

at forward distance

d

from the start node

s

, in the looped

2\times 16

model at

H{=}12

(

\eta=10^{-3}

, row split). The dotted line separates the forward walk from the three nodes of the four-step backward route, and the dashed line is the decodability threshold

0.5

Figure A4: Longer chains at

N{=}32

. Per-hop accuracy of the evaluation during training (App. A ) at the final step, for each of three seeds of the looped

4\times 8

model and the untied

32

-layer model at

\eta=3{\times}10^{-4}

. Points are offset slightly in

H

so that overlapping seeds stay visible. The dashed line marks the

0.8

threshold that defines a solved hop.

Figure A5: A looped seed with weak middle hops. (a) Peak probe selectivity per hop at $H{=}4$ for seeds $42$ , $44$ and $46$ of the looped $2\times 16$ model at $\eta=10^{-3}$ (row split, read after each iteration), with the dotted line at the decodability threshold $0.5$ . (b, c) Probability that seed $44$ (b) and seed $42$ (c) write each reference node when the loop is stopped after a given iteration, read through the model’s own head (§ 3.4 ), averaged over $5{,}000$ prompts.

Figure A6: Activation patching, three seeds. Looped

4\times 8

model at

H{=}6

, seeds

42

44

, with the states at the readout position and first answer digit replaced after a given iteration (§ 3.4 ). (a) Same permutation: fraction of the

2{,}000

answers that are the donor’s. (b) Different permutations: fraction of answers that are the node composed from both walks, among the

1{,}721

1{,}780

answers where it differs from both prompts’ answers. A patch after iteration

6

or later makes the composed node the donor’s answer, so (b) ends at iteration

5

. The dashed line is chance (

1/16

), grey lines are an untrained model, and coinciding seeds are offset slightly.

Recurrent Transformers

Recurrent Transformers reuse layers or state across steps to extend context, reasoning depth, and sequence modeling with less parameter growth.

See all 172 papers

When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers
Jul 22, 2026Tong Zhang, Junhao Hu, Yun Peng +1Inference-Time Scaling Length Generalization
Fudan University · Peking University
DeepLoop: Depth Scaling for Looped Transformers
Jul 15, 2026Shuzhen Li, Yifan Zhang, Jiacheng Guo +2Residual Learning Recurrent Transformers
Princeton University · University of California, Los Angeles
Loop- $μ$ P: Residual Scaling and Hyperparameter Transfer in Looped Transformers
Jun 16, 2026Shaowen Wang, Bingrui Li, Tansheng Zhu +4Residual Learning Hyperparameter Transfer
Tsinghua University · ByteDance Seed · M-A-P

Jul 22, 2026J/K move · Enter open · S save

When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers

Tong Zhang, Junhao Hu, Yun Peng, Tao Xie

Fudan University · Peking University

Inference-Time Scaling Length Generalization Recurrent Transformers Neural Network Training Dynamics Algorithmic Reasoning

Open paper AskPDF arXiv

When does a weight-tied looped transformer -- one block applied T times -- implement an actual algorithm? We answer with four findings from controlled populations on group word problems. (1) The budget law: free training installs a linear computation frontier, a mechanism that solves v positions per loop, whose speed is priced by the training contract: v ~ n_train/T_train (exponent 0.98 +/- 0.04, R^2=0.99), exactly unity under T=n training. SGD selects a frontier matching the minimum the contract demands; granting more test-time loops than ever trained rescues late positions at fixed input length, yielding a principled halting rule T* = ceil(n / v-hat). (2) Architecture prior, not expressivity, picks the algorithm: standard-depth transformers learn parallel scans on this family; weight tying flips the selection to the serial frontier, even when positional addressing for a log-depth scan is supplied. At matched depth and parameters, untied models extrapolate worst and fail to learn A5 at all. (3) The walls are not where circuit complexity says: NC1-completeness costs nothing (A5 generalizes fully), while group order does (S5's 120x120 operator deadlocks joint learning) -- and an operator-first curriculum dissolves the wall in every seed. (4) Mechanisms are portable, not mandatable: warm-starting across budget contracts transfers the algorithm in every seed, re-pricing its speed, while imposing seriality through the input schedule fails where free training succeeds. These results are invisible to standard instruments, which provably saturate at the fixed points trained loops converge to. We introduce a head instrument, the convergence-time scaling tau(n,i), validate it causally via damage cones whose slope reproduces v, and show in-distribution head measurements predict out-of-distribution fate where tail metrics do not. Results replicate on the public easy-to-hard benchmark.

Abstract

Recurrent Transformers

Explore similar work

When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers

DeepLoop: Depth Scaling for Looped Transformers

Loop-μμμP: Residual Scaling and Hyperparameter Transfer in Looped Transformers

Loop- $μ$ P: Residual Scaling and Hyperparameter Transfer in Looped Transformers