Looped Transformers increase effective depth by re-executing shared layers, but three limitations hinder their adoption: memory grows with the loop count, compute can only be adjusted in whole-block steps, and the loop count is hard to change after training. We propose LooM (Loop-invariant Memory), which addresses each in turn. In a causal encoder-decoder (CED), LooM loops only the decoder layers after the first, which read a global memory written once per token, and each of them publishes the key-value (KV) entry for a token only on its first visit, so the cache does not grow with the loop count. These layers loop independently, providing every budget from 3 to 21 layer executions per token, and one model trained on randomly sampled budgets covers all of them without a router. With enough loops, LooM outperforms a single-pass baseline trained on the same data on language modeling, zero-shot accuracy and long-context perplexity, while using the same cache size. Our analysis further shows that the training budget distribution decides what extra depth buys. These results suggest that a CED is a natural place to loop: memory is written once, and depth becomes a budget chosen at deployment.
Figures & tables
Figure 1: Loop-invariant memory in LooM. (a) Decoder layer 1 (D1) writes global memory once per token, before any repetition; each of D2–D6 then runs dℓ∈{0,…,4} times (D3 and D5 at least once). Squares are decoder calls on Random-Depth’s locked paths at three budgets. Every visit re-reads global memory (left), but only a layer’s first visit publishes the token’s local KV entry (right); publishing arrows are drawn for the 21-call path only. (b) D2, standing for any of D2–D6: a later token reads one local entry per earlier token however often the layer ran, so the cache (after a 128-token prompt) is the same on every path. CSA2 modes are defined in Section 3.1 .
Model
Decoder depth during training
Single-pass
Always 6 calls; no loop
LooM, Fixed4
Always 21 calls: decoder layers 2–6 run four times each
LooM, Random-Depth
A deep-heavy random budget (75% at 17–21 calls), then a random path
LooM, Self-Distilled
A random path learns from a 21-call teacher
LooM, Planner
A learned router sets each layer’s depth from the token
Grouped memory
As Fixed4, but memory rebuilt at decoder layers 1, 3 and 5
Table 1: Models and controls. LooM models share the looped read stage and publish-once cache; grouped memory writes memory inside the loop. Only the planner routes on token content.
Cache
Test
Zero-shot accuracy (%) ↑
PPL ↓
Model
Calls
(KiB)
CE ↓
LAMB.
HellaS.
PIQA
WinoG.
ARC-E
ARC-C
Avg
LAMB.
Wiki-2
Single-pass (no loop)
6
364.1
3.0506
20.1
30.4
62.6
50.7
51.6
25.4
40.15
180.4
43.20
LooM, Fixed4
21
364.1
3.0465
21.2
30.3
61.3
50.7
52.5
24.8
40.14
173.2
42.75
LooM, Self-Distilled
6
364.1
3.0640
20.8
30.5
61.8
51.5
51.0
24.3
39.98
168.6
43.60
21
364.1
3.0456
20.9
30.7
62.2
51.4
50.8
23.9
39.98
154.1
42.59
LooM, Planner †
17.96
364.1
3.0355
21.4
31.0
61.6
51.9
53.0
24.1
40.52
153.9
42.63
Table 2: Main results. All models are 204M-parameter CEDs trained on the same data and updates; LooM rows differ only in training recipe and use one validation-locked static path per budget ( † native routing, 17.96 calls on average). Cache is measured after a 128-token prompt; CE is in nats per token. Zero-shot scores are accuracy (%), normalized for HellaSwag and ARC-C. As several tasks are near chance, only CE, Avg and PPL are marked (bold red: best; underlined blue: second). CE at all budgets: Figure 2 .
Figure 2: Test CE (nats per token; lower is better) of LooM models on validation-locked static paths; the star is the planner’s native routing. (a) Full range. (b) Zoom: Random-Depth and Self-Distilled both cross the six-call single-pass line between 10 and 11 calls.
Decode, 128-token prompt
Prefill (ms)
Calls
Path
ms/token
GFLOP/token
Test CE ↓
4,096
65,536
3
(1,0,1,0,1,0)
44.74
5.381
4.3787
82.2
3,134.6
6
(1,1,1,1,1,1)
56.52
6.305
3.1941
94.3
3,145.6
11
(1,2,2,2,2,2)
75.07
7.847
3.0465
112.2
3,171.6
16
(1,4,3,3,2,3)
93.15
9.387
3.0316
128.4
3,194.5
21
(1,4,4,4,4,4)
112.54
10.929
3.0307
147.3
3,233.7
Table 3: Cost of Random-Depth’s locked paths at batch size one on one NVIDIA H800 GPU with a Python reference implementation (medians of 24 decode and five prefill runs). Prefill columns use 4,096- and 65,536-token prompts. Full measurements are in Appendix C .
Prompt
LooM
Grouped
Ratio
LooM saves
1,024
0.75
1.19
1.58 ×
37%
4,096
2.19
3.94
1.80 ×
44%
16,384
7.93
14.93
1.88 ×
47%
Table 4: Persistent decode cache (MiB, BF16) of LooM and grouped memory under Fixed4’s 21-call schedule.
Ctx.
Single
RD
RD − Single, 95% CI
4K
25.483
24.934
− 0.0218 [ − 0.0252, − 0.0184]
8K
24.277
23.858
− 0.0174 [ − 0.0211, − 0.0136]
16K
23.986
23.660
− 0.0137 [ − 0.0176, − 0.0097]
32K
23.858
23.568
− 0.0122 [ − 0.0162, − 0.0082]
64K
23.806
23.644
− 0.0068 [ − 0.0110, − 0.0025]
Table 5: PG19 perplexity after the shared 64K continuation. The last column is Random-Depth (RD) minus single-pass NLL in nats, with a whole-book 95% CI; negative favors RD. Bold red: best; underlined blue: second best at each length.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Property
Value
Backbone parameters
204,046,232
Planner parameters (planner model only)
19,159
Hidden / feed-forward width
832 / 2,688
Attention heads × head dimension
64 × 80
Sliding-window width (enc. 1–2; dec.)
256; 128
Indexer heads × width, Top-K
32 × 32, 128
Appendix
Table 6: Architecture of the shared backbone. KV: key–value.
Calls
Deep-heavy
Uniform
3
4.2236
3.4108
6
3.4551
3.3587
11
3.3290
3.3490
16
3.3114
3.3470
21
3.3106
3.3468
Train calls
17.19
13.88
Appendix
Table 7: Two budget distributions in the same recipe, compared early in training: test cross-entropy (CE; nats per token, lower is better). Deep-heavy is Random-Depth’s schedule; the uniform run instead draws the full 21-call path with probability 0.25 and otherwise a budget uniform in 3–20 calls. Both then draw a uniformly random legal path of that budget, skipped layers included. Train calls are expected decoder calls per training token; the uniform run uses fewer, which confounds absolute levels but not the within-model flatness of its curve.
Figure 3: Prefill and subsequent decode for Random-Depth against the budget, one curve per prompt length (one NVIDIA H800, batch size one, median of five). (a) Prefill is set by the prompt, not the budget: at 65,536 tokens the whole 3-to-21-call range costs 3.2% more. (b) Decode is set by the budget, with the four prompt lengths almost on one curve. The persistent cache is identical across paths at every prompt length. Prompts beyond 4,096 tokens are timed for cost only.
Replacement
Cap
Δ CE
Static path
6
− 0.05273 to − 0.05053
11–21
− 0.00037 to +0.00060
Static split
6
− 0.05273 to − 0.05053
11–21
+0.00005 to +0.00063
Mean totals
21
+0.00013 to +0.00014
Shuffled totals
21
+0.00166 to +0.00167
Appendix
Table 8: Allocation controls on two 200-update budget-conditioned planner continuations (seeds 42 and 43), needed because the original planner is poorly calibrated at cap 6. Each value is the range of replacement minus native-routing cross-entropy (CE); negative favors the replacement. At cap 6, static path and static split improve over poorly calibrated native routing. A cap bounds each token’s total calls.
Figure 4: Mechanism analysis on 256 test sequences. (a) The allocation-relevant negative log-likelihood (NLL) gain from 11 to 21 calls by difficulty decile (1: easiest); positive is improvement, and difficulty uses mean NLL over both paths, avoiding selection on one arm’s noise. (b) Relative residual-state updates at visits 1–4 of decoder layers 2–6, labeled D2–D6 in the panel, on the 21-call path.
Ctx.
RD − native
Static − native
4K
+3.9 [+0.6, +7.2]
+0.8 [+0.5, +1.3]
8K
+1.8 [ − 2.2, +5.4]
+1.3 [+0.9, +1.8]
16K
+1.9 [ − 1.8, +5.4]
+1.4 [+0.9, +1.8]
32K
+2.4 [ − 1.2, +5.8]
+1.3 [+0.9, +1.7]
64K
+6.4 [+2.8, +9.8]
+1.4 [+1.0, +1.8]
Appendix
Table 9: Paired negative log-likelihood (NLL) differences in 10−3 nats per token with whole-book 95% confidence intervals; negative favors the first model in each difference. RD denotes Random-Depth; Static and native denote the planner with its static path and with native routing, respectively. Point estimates favor native at every length: static minus native excludes zero at every length, while RD minus native excludes zero only at 4K and 64K.
Length
Single
Planner native
Planner static
RD
4K
61.67
68.33
68.33
68.33
8K
63.33
68.33
71.67
68.33
16K
66.67
70.00
70.00
70.00
32K
61.67
70.00
70.00
71.67
64K
53.33
63.33
65.00
58.33
Overall
61.33
68.00
69.00
67.33
Appendix
Table 10: Three-task needle-in-a-haystack (NIAH) macro accuracy (%, higher is better) by context length in tokens and overall (300 questions per model). Single denotes the single-pass model, and RD denotes Random-Depth. All arms use the same answer-format fine-tuning. Random-Depth and the planner with its static path use the static 21-call path, whereas the planner with native routing uses its learned per-token policy. The table reports point estimates; the overall paired interval for RD minus the single-pass model is [+1.33,+10.33] percentage points.