Looped Transformers reduce parameter count by repeatedly applying shared blocks, but parameter sharing alone does not determine how the results of repeated computation are reused. We take the depth-memory view of residual connections and analyze which representation is stored after each block application and how later reads weight it. A full block output contains both its input and its new update, so weighting that output also reweights the history already carried by the input. Reading individual updates instead, with the shared stack's input kept on a separate path, gives separate controls over input strength, total update weight, and allocation across updates. The Loopback Transformer implements these controls: geometric decay sets the total update weight and supplies a recency prior, and attention learns how to allocate that weight. On ClimbMix, Loopback reaches lower validation loss than standard Transformer baselines with 40-75% of their Transformer parameters (excluding token embeddings and the output head) at reference scales from 191M to 3B, with training tokens and executed depth matched. It also has higher mean accuracy over thirteen downstream tasks at 1.4B and 3B. At 191M, 374M, and 778M, it has the lowest validation loss and the highest thirteen-task mean among the standard Transformer and seven looped residual designs.
Figures & tables
Figure 1: Loopback versus the standard Transformer from 191M to 3B. Each pair uses the same token budget and executed depth at the 20-TPP setting (tokens per reference-model parameter). (a) Validation loss and (c) thirteen-task mean accuracy against the standard Transformer’s estimated training FLOPs; (b) the same losses against Transformer parameters, where percentages give Loopback’s share of the standard Transformer’s. Lines interpolate the points.
Figure 2: Same shared blocks, three residual pathways. Repeated colors denote shared blocks emitting block updates. (a) Addition. (b) Rs aggregates the input and updates computed so far using explicit history or a fixed-size recurrent memory. (c) Multiple streams with input read, transport, write, and final readout.
Common interface: Ms+1=MsWs+vsbs⊤,hs+1=Ms+1rs.
Method
Transport Ws
Write bs
Read rs
Addition ( He et al., 2016 ) (Standard, Plain Loop)
1
1
1
Explicit-history attention
History banks start from M0=h0 and store each new value separately. At block granularity, the value is δs ; enew selects its new storage location.
AttnRes ( Team et al., 2026 )
[I,0]
enew
πs+1
MHAR ( Luo et al., 2026b )
[I,0]
enew
headwise πs+1(h)
Table 1: Write, transport, and read operations for residual and attention memory, with Ms∈Rd×ms , applied per head where needed. Feature construction, output gates and normalization, feedthrough paths, and outer residual blends are not shown. The rows describe memory operations, not complete published architectures.
Figure 3: Loopback separates input, decay, and allocation. (a) Shared blocks with causal history reads. (b) Iteration-end decay sets the prior and total weight. (c) Attention redistributes that total over block updates, keeping the input coefficient at one.
Table 2: Residual adapters at matched tokens and depth. Validation loss, zero-shot accuracy (%), and WikiText perplexity at 191M (20 layers), 374M (32 layers), and 778M (48 layers) ( Table 12 ); every prompt starts with the beginning-of-sequence token, as in training ( Section D.2 ). Arrows mark lower-is-better metrics; bold marks the best result per row. Percentages compare Transformer parameters with Standard, excluding embeddings and the vocabulary head and counting shared weights once.
Figure 4: Training loss of the standard Transformer and Loopback. 100-step moving averages against tokens seen; insets show the last quarter of training; Δ is the final training-loss difference, Loopback minus the standard Transformer, over the last 100 steps. The five panels use 20, 24, 24, 25, and 24 executed layers, respectively. Loopback stores 40–75% of the standard baseline’s Transformer parameters.
Figure 5: General benchmarks at 1.4B and 3B. Loopback uses 68% and 75% of Standard’s Transformer parameters while attaining lower validation loss and higher thirteen-task mean accuracy. Bars use the left parameter axis; connected points use the right metric axis. Training tokens and executed depth are matched within each scale, with one run per arm at 20 TPP.
1.4B
3B
Reference-solution loss ↓
Standard
Loopback
Δ
Standard
Loopback
Δ
GSM8K
1.275
1.227
-0.048
1.143
1.086
-0.057
MATH-500
1.421
1.391
-0.030
1.289
1.259
-0.030
HumanEval
1.065
1.035
-0.030
0.906
0.868
-0.038
MBPP
1.282
1.235
-0.047
1.104
1.078
-0.026
Four-set mean
1.261
1.222
-0.039
1.110
1.073
-0.037
Table 3: Mathematical and code evaluation at 1.4B and 3B. Reference-solution loss is teacher-forced NLL (nats per solution token), not generation or code-execution accuracy. Word problems use mean five-shot numeric-answer accuracy over six sets. Δ is Loopback minus Standard (nats for loss; percentage points for accuracy). Training tokens and depth are matched, with one run per arm. Shading marks Loopback; bold scores are row-best, and green differences favor Loopback.
Figure 6: Robustness to additional visits without retraining. (a–c) Validation loss (nats/token), trained at 20, 32, and 25 layers; (d) thirteen-task mean accuracy, trained at 24 layers. Asterisks mark trained visit counts; the dashed line shows the fixed-depth Standard reference at 3B.
(a) Decay and attention at two scales
191M (Standard: 2.6234)
374M (Standard: 2.4198)
Decay
Fixed
Learned
Fixed
Learned
Off ( λ=1 )
2.6802
2.6621
2.4415
2.4308
On ( λ=1/4 )
2.6220
2.6187 ( − 0.0047)
2.3999
2.3971 ( − 0.0227)
(b) Weight initialization: 191M, 20 layers
Model
Transformer params (M)
N(0,0.022)
GPT-2
PyTorch
Table 4: Decay, attention, and weight initialization. Validation loss ( ↓ ) at 20 layers (191M) and 24 layers (374M); shading marks Loopback and green gives its difference from Standard. (a) Fixed allocation uses zero queries. (b–c) Three weight-initialization schemes with learning rates held fixed; bold marks the lowest loss in each initialization column. Parameter counts exclude embeddings and the vocabulary head.
Figure 7: Depth at fixed parameters. ClimbMix validation loss after 20 TPP as each scale trades width for depth at approximately fixed total parameters and fixed training tokens. Looped rules make two visits through a shared stack that stores 56–58% of the standard baseline’s Transformer parameters.
Figure 8: Depth-memory readers. ClimbMix validation-loss reduction over Plain Loop at (a) 191M with 20 layers and (b) 374M with 24 layers for the update interface (solid) and the state interface (hatched) of each reader; the standard Transformer and Loopback are references.
Figure 9: Compute and efficiency at 374M, 24 layers. Loopback and AttnRes run fused read kernels. (a) Per-GPU training throughput on eight H100s; bar labels give hours for the 20-TPP budget. (b) Validation loss versus forward FLOPs per token in excess of the standard baseline’s.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Scale
Width
Blocks
Heads
SwiGLU
Loopback
Seq./step
Micro × acc.
Steps
Tokens
LR ( ×10−3 )
191M
768
20
12
2,048
[2L→4L(×4)→2L]
128
16×1
7,280
3.82B
1 / 3
374M
1,024
24
16
2,816
[2L→10L(×2)→2L]
256
8×4
7,120
7.47B
2 / 2
778M
1,536
24
24
4,096
[2L→8L(×2)→6L]
512
8×8
7,440
15.60B
1 / 1.5
1.4B
2,048
25
32
5,632
[3L→8L(×2)→6L]
960
8×15
7,200
28.31B
1 / 1.5
3B
3,072
24
48
8,192
[6L→6L(×2)→6L]
1,920
4×60
7,600
59.77B
1 / 1
Appendix
Table 5: Configuration at each scale. Blocks and heads apply to Standard and to every looped arm; head width is 64 throughout. Sequences per optimizer step are the per-GPU micro-batch times gradient-accumulation steps, on 8 ranks, at 4,096 tokens per sequence. Peak learning rates are Standard / looped arms.
ClimbMix ( Diao et al., 2026 ) ; training shards 0–239
Input construction
Online tokenization; concatenated documents packed into fixed 4,096-token sequences
Global batch
128 sequences; 8 GPU ranks ×16 sequences per rank
Gradient accumulation
1; 524,288 input tokens per optimizer step
Training budget
7,280 steps; 3,816,816,640 input tokens
Optimizer
AdamW; (β1,β2)=(0.9,0.95) ; ϵ=10−15
Appendix
Table 7: Training recipe at the 20-TPP endpoint. Only peak learning rates differ across arms.
Peak learning rate
5×10−4
7×10−4
10−3
2×10−3
ClimbMix loss
2.6666
2.6420
2.6234
2.6245
Appendix
Table 8: 191M Standard learning-rate comparison. ClimbMix validation loss after 20 TPP on the same machine and training recipe. Bold marks the lowest loss.
1.4B, 25 layers, 28.31B tokens
3B, 24 layers, 59.77B tokens
Standard
Loopback
Standard
Loopback
Topology
–
[3L→8L(×2)→6L]
–
[6L→6L(×2)→6L]
Transformer params (M)
1,284.61
873.57
2,718.06
2,038.58
Relative to Standard
100.0%
68.0%
100.0%
75.0%
ClimbMix loss ↓
2.1283
2.1073
2.0005
1.9849
PIQA
73.88
74.48
77.04
75.90
Appendix
Table 9: Standard and Loopback at 1.4B and 3B. Results after 20 TPP, one run per arm. Task scores are zero-shot accuracies (%) with beginning-of-sequence prompts; bold marks the better result within each scale. Parameter shares compare Transformer parameters, excluding embeddings and the vocabulary head.
Validation loss ↓
Model
seed 42
seed 1337
191M, 20 layers
Standard
2.6234
2.6144
Plain Loop
2.6802
2.6815
mHC
2.6250
2.6292
HC
2.6449
2.6427
Appendix
Table 10: Validation-loss variation at 191M and 374M. ClimbMix loss (nats, lower is better) after 20 TPP. Each seed sets initialization and data order; difference rows compare models within a seed. Seed 42 is the run reported in the main text. Bold marks the best model in each column within each scale; shading marks Loopback.
Transformer (M)
Layers
Width
Heads
Looped topology
Standard total (M)
Standard
Loopback
Share
374M, 7.47B tokens at 20 TPP (7,120 steps of 1,048,576 tokens)
16
1216
19
[1L→7L(×2)→1L]
366.75
288.92
162.54
56.3%
24
1024
16
[2L→10L(×2)→2L]
373.87
308.33
179.88
58.3%
32
896
14
[2L→14L(×2)→2L]
380.36
323.02
181.72
56.3%
48
768
12
[3L→21L(×2)→3L]
388.97
339.81
191.18
56.3%
Appendix
Table 12: Depth-sweep configurations and validation loss at 20 TPP. Training tokens are fixed; Standard’s total parameters stay within 5% of its 24-layer reference. (a) Configurations: heads are 64-dimensional and looped models make two visits. Transformer counts exclude embeddings and the vocabulary head; shared weights are counted once. Other looped arms are within 0.7% of Loopback’s Transformer count. (b) ClimbMix validation loss: bold marks the lowest loss per row; shading marks Loopback.
Hyper-connection family
Attention family
Reference solutions ↓
Standard
Plain Loop
mHC
HC
iHC
Hyperloop
MHAR
AttnRes
Loopback
191M, 20 layers, [2L→4L(×4)→2L] (3.82B tokens)
GSM8K
1.822
1.865
1.795
1.801
1.826
1.833
1.826
1.797
1.768
MATH-500
1.967
2.028
1.961
1.999
2.023
2.024
1.994
1.962
1.972
HumanEval
1.827
1.935
1.861
1.900
1.924
1.905
1.897
1.895
1.842
MBPP
2.021
2.118
2.028
2.042
2.081
2.074
2.076
2.064
2.018
Appendix
Table 13: Reference solutions at matched tokens. Teacher-forced loss (nats per token, lower is better) of the reference solutions of GSM8K, MATH-500, HumanEval and MBPP given the problem, for all nine designs at 191M and 374M. Every prompt starts with the beginning-of-sequence token. Paired bootstrap standard errors of the differences between rules are at most 0.003 on the mean. Bold marks the lowest loss per row.
Task
2 visits (trained)
3 visits
Δ (pp)
PIQA
75.90
75.30
-0.60
HellaSwag
63.72
65.11
+1.39
WinoGrande
58.48
58.96
+0.48
OpenBookQA
41.80
42.80
+1.00
CommonsenseQA
21.87
22.36
+0.49
BoolQ
64.95
65.60
+0.65
Appendix
Table 14: 3B Loopback with one extra visit. Zero-shot scores (%) at two visits (trained) and three visits, without retraining. Δ is the change in percentage points computed from the displayed scores; bold marks the higher score in each row.
Model
Transformer params (M)
Extra FLOPs
Train tok/s/GPU
Peak GiB/GPU
Prefill tok/s
191M, [2L→4L(×4)→2L] , 3.82B tokens, 16 sequences per GPU
Standard
141.59
–
222k
38
885k
Plain Loop
56.64
+0.00%
226k
38
886k
mHC
56.93
+0.15%
169k
42
594k
HC
56.65
+0.07%
186k
42
671k
iHC
57.04
+0.17%
116k
67
370k
Appendix
Table 15: Compute and efficiency of the residual rules. Parameters are Transformer-only; extra forward FLOPs are relative to Standard. Percentages are rounded to two decimal places, with the positive 374M MHAR overhead rounded up to 0.01% . Training throughput and peak memory are per GPU. Prefill uses one H100.
KV cache per
Weights + KV (GiB)
Scale
Model
Total params (M)
Weights (GiB)
sequence (MiB)
1 sequence
8 sequences
191M, 20 layers
Standard
190.74
0.36
240
0.59
2.23
Loopback
105.80
0.20
240
0.43
2.07
374M, 24 layers
Standard
373.87
0.70
384
1.07
3.70
Loopback
245.42
0.46
384
0.83
3.46
778M, 24 layers
Standard
777.86
1.45
576
2.01
5.95
Appendix
Table 16: Persistent inference memory of Standard and Loopback. Analytical bf16 storage for all weights, including embeddings and the vocabulary head, plus the KV cache at 4,096 tokens. Totals exclude temporary activations and workspaces; they are not measured peaks. Shading marks Loopback.