Looped Transformers increase effective depth by re-executing shared layers, but three limitations hinder their adoption: memory grows with the loop count, compute can only be adjusted in whole-block steps, and the loop count is hard to change after training. We propose LooM (Loop-invariant Memory), which addresses each in turn. In a causal encoder-decoder (CED), LooM loops only the decoder layers after the first, which read a global memory written once per token, and each of them publishes the key-value (KV) entry for a token only on its first visit, so the cache does not grow with the loop count. These layers loop independently, providing every budget from 3 to 21 layer executions per token, and one model trained on randomly sampled budgets covers all of them without a router. With enough loops, LooM outperforms a single-pass baseline trained on the same data on language modeling, zero-shot accuracy and long-context perplexity, while using the same cache size. Our analysis further shows that the training budget distribution decides what extra depth buys. These results suggest that a CED is a natural place to loop: memory is written once, and depth becomes a budget chosen at deployment.
Figures & tables
Figure 1: Loop-invariant memory in LooM. (a) Decoder layer 1 (D1) writes global memory once per token, before any repetition; each of D2–D6 then runs dℓ∈{0,…,4} times (D3 and D5 at least once). Squares are decoder calls on Random-Depth’s locked paths at three budgets. Every visit re-reads global memory (left), but only a layer’s first visit publishes the token’s local KV entry (right); publishing arrows are drawn for the 21-call path only. (b) D2, standing for any of D2–D6: a later token reads one local entry per earlier token however often the layer ran, so the cache (after a 128-token prompt) is the same on every path. CSA2 modes are defined in Section 3.1 .
Model
Decoder depth during training
Single-pass
Always 6 calls; no loop
LooM, Fixed4
Always 21 calls: decoder layers 2–6 run four times each
LooM, Random-Depth
A deep-heavy random budget (75% at 17–21 calls), then a random path
LooM, Self-Distilled
A random path learns from a 21-call teacher
LooM, Planner
A learned router sets each layer’s depth from the token
Grouped memory
As Fixed4, but memory rebuilt at decoder layers 1, 3 and 5
Table 1: Models and controls. LooM models share the looped read stage and publish-once cache; grouped memory writes memory inside the loop. Only the planner routes on token content.
Cache
Test
Zero-shot accuracy (%) ↑
PPL ↓
Model
Calls
(KiB)
CE ↓
LAMB.
HellaS.
PIQA
WinoG.
ARC-E
ARC-C
Avg
LAMB.
Wiki-2
Single-pass (no loop)
6
364.1
3.0506
20.1
30.4
62.6
50.7
51.6
25.4
40.15
180.4
43.20
LooM, Fixed4
21
364.1
3.0465
21.2
30.3
61.3
50.7
52.5
24.8
40.14
173.2
42.75
LooM, Self-Distilled
6
364.1
3.0640
20.8
30.5
61.8
51.5
51.0
24.3
39.98
168.6
43.60
21
364.1
3.0456
20.9
30.7
62.2
51.4
50.8
23.9
39.98
154.1
42.59
LooM, Planner †
17.96
364.1
3.0355
21.4
31.0
61.6
51.9
53.0
24.1
40.52
153.9
42.63
Table 2: Main results. All models are 204M-parameter CEDs trained on the same data and updates; LooM rows differ only in training recipe and use one validation-locked static path per budget ( † native routing, 17.96 calls on average). Cache is measured after a 128-token prompt; CE is in nats per token. Zero-shot scores are accuracy (%), normalized for HellaSwag and ARC-C. As several tasks are near chance, only CE, Avg and PPL are marked (bold red: best; underlined blue: second). CE at all budgets: Figure 2 .
Figure 2: Test CE (nats per token; lower is better) of LooM models on validation-locked static paths; the star is the planner’s native routing. (a) Full range. (b) Zoom: Random-Depth and Self-Distilled both cross the six-call single-pass line between 10 and 11 calls.
Decode, 128-token prompt
Prefill (ms)
Calls
Path
ms/token
GFLOP/token
Test CE ↓
4,096
65,536
3
(1,0,1,0,1,0)
44.74
5.381
4.3787
82.2
3,134.6
6
(1,1,1,1,1,1)
56.52
6.305
3.1941
94.3
3,145.6
11
(1,2,2,2,2,2)
75.07
7.847
3.0465
112.2
3,171.6
16
(1,4,3,3,2,3)
93.15
9.387
3.0316
128.4
3,194.5
21
(1,4,4,4,4,4)
112.54
10.929
3.0307
147.3
3,233.7
Table 3: Cost of Random-Depth’s locked paths at batch size one on one NVIDIA H800 GPU with a Python reference implementation (medians of 24 decode and five prefill runs). Prefill columns use 4,096- and 65,536-token prompts. Full measurements are in Appendix C .
Prompt
LooM
Grouped
Ratio
LooM saves
1,024
0.75
1.19
1.58 ×
37%
4,096
2.19
3.94
1.80 ×
44%
16,384
7.93
14.93
1.88 ×
47%
Table 4: Persistent decode cache (MiB, BF16) of LooM and grouped memory under Fixed4’s 21-call schedule.
Ctx.
Single
RD
RD − Single, 95% CI
4K
25.483
24.934
− 0.0218 [ − 0.0252, − 0.0184]
8K
24.277
23.858
− 0.0174 [ − 0.0211, − 0.0136]
16K
23.986
23.660
− 0.0137 [ − 0.0176, − 0.0097]
32K
23.858
23.568
− 0.0122 [ − 0.0162, − 0.0082]
64K
23.806
23.644
− 0.0068 [ − 0.0110, − 0.0025]
Table 5: PG19 perplexity after the shared 64K continuation. The last column is Random-Depth (RD) minus single-pass NLL in nats, with a whole-book 95% CI; negative favors RD. Bold red: best; underlined blue: second best at each length.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Property
Value
Backbone parameters
204,046,232
Planner parameters (planner model only)
19,159
Hidden / feed-forward width
832 / 2,688
Attention heads × head dimension
64 × 80
Sliding-window width (enc. 1–2; dec.)
256; 128
Indexer heads × width, Top-K
32 × 32, 128
Appendix
Table 6: Architecture of the shared backbone. KV: key–value.
Calls
Deep-heavy
Uniform
3
4.2236
3.4108
6
3.4551
3.3587
11
3.3290
3.3490
16
3.3114
3.3470
21
3.3106
3.3468
Train calls
17.19
13.88
Appendix
Table 7: Two budget distributions in the same recipe, compared early in training: test cross-entropy (CE; nats per token, lower is better). Deep-heavy is Random-Depth’s schedule; the uniform run instead draws the full 21-call path with probability 0.25 and otherwise a budget uniform in 3–20 calls. Both then draw a uniformly random legal path of that budget, skipped layers included. Train calls are expected decoder calls per training token; the uniform run uses fewer, which confounds absolute levels but not the within-model flatness of its curve.
Figure 3: Prefill and subsequent decode for Random-Depth against the budget, one curve per prompt length (one NVIDIA H800, batch size one, median of five). (a) Prefill is set by the prompt, not the budget: at 65,536 tokens the whole 3-to-21-call range costs 3.2% more. (b) Decode is set by the budget, with the four prompt lengths almost on one curve. The persistent cache is identical across paths at every prompt length. Prompts beyond 4,096 tokens are timed for cost only.
Replacement
Cap
Δ CE
Static path
6
− 0.05273 to − 0.05053
11–21
− 0.00037 to +0.00060
Static split
6
− 0.05273 to − 0.05053
11–21
+0.00005 to +0.00063
Mean totals
21
+0.00013 to +0.00014
Shuffled totals
21
+0.00166 to +0.00167
Appendix
Table 8: Allocation controls on two 200-update budget-conditioned planner continuations (seeds 42 and 43), needed because the original planner is poorly calibrated at cap 6. Each value is the range of replacement minus native-routing cross-entropy (CE); negative favors the replacement. At cap 6, static path and static split improve over poorly calibrated native routing. A cap bounds each token’s total calls.
Figure 4: Mechanism analysis on 256 test sequences. (a) The allocation-relevant negative log-likelihood (NLL) gain from 11 to 21 calls by difficulty decile (1: easiest); positive is improvement, and difficulty uses mean NLL over both paths, avoiding selection on one arm’s noise. (b) Relative residual-state updates at visits 1–4 of decoder layers 2–6, labeled D2–D6 in the panel, on the 21-call path.
Ctx.
RD − native
Static − native
4K
+3.9 [+0.6, +7.2]
+0.8 [+0.5, +1.3]
8K
+1.8 [ − 2.2, +5.4]
+1.3 [+0.9, +1.8]
16K
+1.9 [ − 1.8, +5.4]
+1.4 [+0.9, +1.8]
32K
+2.4 [ − 1.2, +5.8]
+1.3 [+0.9, +1.7]
64K
+6.4 [+2.8, +9.8]
+1.4 [+1.0, +1.8]
Appendix
Table 9: Paired negative log-likelihood (NLL) differences in 10−3 nats per token with whole-book 95% confidence intervals; negative favors the first model in each difference. RD denotes Random-Depth; Static and native denote the planner with its static path and with native routing, respectively. Point estimates favor native at every length: static minus native excludes zero at every length, while RD minus native excludes zero only at 4K and 64K.
Length
Single
Planner native
Planner static
RD
4K
61.67
68.33
68.33
68.33
8K
63.33
68.33
71.67
68.33
16K
66.67
70.00
70.00
70.00
32K
61.67
70.00
70.00
71.67
64K
53.33
63.33
65.00
58.33
Overall
61.33
68.00
69.00
67.33
Appendix
Table 10: Three-task needle-in-a-haystack (NIAH) macro accuracy (%, higher is better) by context length in tokens and overall (300 questions per model). Single denotes the single-pass model, and RD denotes Random-Depth. All arms use the same answer-format fine-tuning. Random-Depth and the planner with its static path use the static 21-call path, whereas the planner with native routing uses its learned per-token policy. The table reports point estimates; the overall paired interval for RD minus the single-pass model is [+1.33,+10.33] percentage points.
Looped language models deepen computation without adding parameters, yet caching full-context key-value (KV) states across layers and rounds severely bounds serving batch size. We design a looped causal encoder-decoder (CED) that achieves recurrent depth with zero round-wise KV growth, addressing where and how to loop an asymmetric stack in which only sparse encoder layers generate global KV. We find that recurrence must be placed in the encoder to iteratively refine global KV representations rather than re-reading static decoder projections, which both drives language-modeling gains and preserves long-context scaling. Furthermore, training the encoder loop under stochastic recurrence depth induces a fixed-point equilibrium where every round losslessly shares the last round's KV cache. Compounding cross-layer and cross-round KV sharing, the resulting Looped CED matches an unlooped CED's minimal memory footprint and recovers 86-93% of a looped Transformer's gain with 1/6 of its global KV. To test serving at production scale, we loop the 552B-parameter DeepSeek-V4.1-Flash for 8 rounds on one 4xGB200 node: with per-layer and per-round KV the loop cannot fit a single 128K-token request, whereas Looped CED holds 36.8x less KV per token, fits 38 such requests (as many as the unlooped model), and serves 3.6x higher throughput on 8K-token prompts.
Jiaqi Leng, Minshen Zhang, Zhaofeng Sun +3
University of California, San Diego · Tsinghua University · Nanjing University
Looped Transformers apply the same layers several times per token, adding compute to improve quality without more parameters. Each recursion, however, writes its own key-value cache, so memory still grows with compute. Inference-time techniques can shrink this cache at a cost in quality. We pretrain looped language models to share memory: only the first recursion writes a cache, and later recursions read it while keeping a short window of their own. Surprisingly, we find that sharing memory does not cost quality and instead improves it. At 150M-1B parameters, our Looped Prediction Transformer (LPT) and its hybrid variant set a new quality-memory frontier for looped models: with five recursions, the hybrid lowers validation perplexity on FineWeb-Edu by 1.12-1.82 relative to a same-size standard Transformer while using 76-79% less context memory. Through an extensive analysis, we investigate why memory sharing helps. Shared and local memory develop different representations, and later recursions attend mostly to the shared memory, which also acts as a gradient highway to the first recursion.
Looped Transformers increase computation per token by repeatedly applying a shared backbone, offering a way to improve language modeling and downstream task performance at a fixed backbone parameter count. However, each loop depends on the preceding loop, so increasing recurrent depth also increases decoding latency. Existing approaches include Parallel Loop Transformers, which reduce latency by overlapping loop computation across token positions, but yield rapidly diminishing marginal accuracy gains at larger depths. To address this challenge, we introduce LoopFold, which folds computation at multiple recurrent depths into parallel batches for the same token prefix. A lightweight predictor estimates inputs to later loops so they can start before earlier loops finish. A corrector then refines the outputs of these later loops using newly available results from earlier loops. All components are trained jointly using only the final next-token prediction loss. In experiments with 1.4B models, deeper LoopFold configurations achieve progressively higher average zero-shot accuracy across eight tasks, rising from 53.41% to 56.38%. Vanilla looped models and PLT show limited gains over the same depth range. Compared with a nine-loop vanilla model, LoopFold uses five sequential stages and additional parallel computation to reduce decoding latency by 46%.
Feichen Song, Boyi Zeng, Yiqin Hao +7
Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute