Loopback Transformer: Towards a Unified View of Residual Looped Transformers
Organizations: University of Wisconsin–Madison · University of California, Santa Barbara · California Institute of Technology · Microsoft
Abstract
Looped Transformers reduce parameter count by repeatedly applying shared blocks, but parameter sharing alone does not determine how the results of repeated computation are reused. We take the depth-memory view of residual connections and analyze which representation is stored after each block application and how later reads weight it. A full block output contains both its input and its new update, so weighting that output also reweights the history already carried by the input. Reading individual updates instead, with the shared stack's input kept on a separate path, gives separate controls over input strength, total update weight, and allocation across updates. The Loopback Transformer implements these controls: geometric decay sets the total update weight and supplies a recency prior, and attention learns how to allocate that weight. On ClimbMix, Loopback reaches lower validation loss than standard Transformer baselines with 40-75% of their Transformer parameters (excluding token embeddings and the output head) at reference scales from 191M to 3B, with training tokens and executed depth matched. It also has higher mean accuracy over thirteen downstream tasks at 1.4B and 3B. At 191M, 374M, and 778M, it has the lowest validation loss and the highest thirteen-task mean among the standard Transformer and seven looped residual designs.
Figures & tables
| Common interface: | |||
| Method | Transport | Write | Read |
| Addition ( He et al., 2016 ) (Standard, Plain Loop) | |||
| Explicit-history attention | |||
| History banks start from and store each new value separately. At block granularity, the value is ; selects its new storage location. | |||
| AttnRes ( Team et al., 2026 ) | |||
| MHAR ( Luo et al., 2026b ) | headwise | ||
| Hyper-connection family | Attention family | ||||||||
| Benchmark | Standard | Plain Loop | mHC | HC | iHC | Hyperloop | MHAR | AttnRes | Loopback |
| 191M, 20 layers, , 40.0% of Standard, 3.82B tokens (20 TPP) | |||||||||
| Transformer params (M) | 141.59 | 56.64 | 56.93 | 56.65 | 57.04 | 56.79 | 56.64 | 56.66 | 56.65 |
| ClimbMix loss | 2.6234 | 2.6802 | 2.6250 | 2.6449 | 2.6564 | 2.6489 | 2.6564 | 2.6404 | 2.6187 |
| PIQA | 65.51 | 64.96 | 65.89 | 64.74 | 65.34 | 65.07 | 65.13 | 64.47 | 65.78 |
| HellaSwag | 33.26 | 32.14 | 33.79 | 33.18 | 32.74 | 33.04 | 33.12 | 33.07 | 33.83 |
| 1.4B | 3B | |||||
|---|---|---|---|---|---|---|
| Reference-solution loss | Standard | Loopback | Standard | Loopback | ||
| GSM8K | 1.275 | 1.227 | -0.048 | 1.143 | 1.086 | -0.057 |
| MATH-500 | 1.421 | 1.391 | -0.030 | 1.289 | 1.259 | -0.030 |
| HumanEval | 1.065 | 1.035 | -0.030 | 0.906 | 0.868 | -0.038 |
| MBPP | 1.282 | 1.235 | -0.047 | 1.104 | 1.078 | -0.026 |
| Four-set mean | 1.261 | 1.222 | -0.039 | 1.110 | 1.073 | -0.037 |
| (a) Decay and attention at two scales | ||||
| 191M (Standard: 2.6234) | 374M (Standard: 2.4198) | |||
| Decay | Fixed | Learned | Fixed | Learned |
| Off ( ) | 2.6802 | 2.6621 | 2.4415 | 2.4308 |
| On ( ) | 2.6220 | 2.6187 ( 0.0047) | 2.3999 | 2.3971 ( 0.0227) |
| (b) Weight initialization: 191M, 20 layers | ||||
| Model | Transformer params (M) | GPT-2 | PyTorch | |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Scale | Width | Blocks | Heads | SwiGLU | Loopback | Seq./step | Micro acc. | Steps | Tokens | LR ( ) |
|---|---|---|---|---|---|---|---|---|---|---|
| 191M | 768 | 20 | 12 | 2,048 | 128 | 7,280 | 3.82B | 1 / 3 | ||
| 374M | 1,024 | 24 | 16 | 2,816 | 256 | 7,120 | 7.47B | 2 / 2 | ||
| 778M | 1,536 | 24 | 24 | 4,096 | 512 | 7,440 | 15.60B | 1 / 1.5 | ||
| 1.4B | 2,048 | 25 | 32 | 5,632 | 960 | 7,200 | 28.31B | 1 / 1.5 | ||
| 3B | 3,072 | 24 | 48 | 8,192 | 1,920 | 7,600 | 59.77B | 1 / 1 |
| Setting | Value |
|---|---|
| Standard structure | Twenty independent blocks |
| Looped structure | ; eight distinct blocks, twenty applications |
| Attention | Full-sequence causal attention; 12 heads of width 64; no GQA |
| Hidden / feed-forward width | 768 / 2,048; SwiGLU feed-forward layers |
| Normalization | Pre-RMSNorm; |
| Position encoding | RoPE; |
| Setting | Value |
|---|---|
| Training data | ClimbMix ( Diao et al., 2026 ) ; training shards 0–239 |
| Input construction | Online tokenization; concatenated documents packed into fixed 4,096-token sequences |
| Global batch | 128 sequences; GPU ranks sequences per rank |
| Gradient accumulation | 1; 524,288 input tokens per optimizer step |
| Training budget | 7,280 steps; 3,816,816,640 input tokens |
| Optimizer | AdamW; ; |
| Peak learning rate | ||||
|---|---|---|---|---|
| ClimbMix loss | 2.6666 | 2.6420 | 2.6234 | 2.6245 |
| 1.4B, 25 layers, 28.31B tokens | 3B, 24 layers, 59.77B tokens | |||
| Standard | Loopback | Standard | Loopback | |
| Topology | – | – | ||
| Transformer params (M) | 1,284.61 | 873.57 | 2,718.06 | 2,038.58 |
| Relative to Standard | 100.0% | 68.0% | 100.0% | 75.0% |
| ClimbMix loss | 2.1283 | 2.1073 | 2.0005 | 1.9849 |
| PIQA | 73.88 | 74.48 | 77.04 | 75.90 |
| Validation loss | ||
| Model | seed 42 | seed 1337 |
| 191M, 20 layers | ||
| Standard | 2.6234 | 2.6144 |
| Plain Loop | 2.6802 | 2.6815 |
| mHC | 2.6250 | 2.6292 |
| HC | 2.6449 | 2.6427 |
| Transformer (M) | |||||||
| Layers | Width | Heads | Looped topology | Standard total (M) | Standard | Loopback | Share |
| 374M, 7.47B tokens at 20 TPP (7,120 steps of 1,048,576 tokens) | |||||||
| 16 | 1216 | 19 | 366.75 | 288.92 | 162.54 | 56.3% | |
| 24 | 1024 | 16 | 373.87 | 308.33 | 179.88 | 58.3% | |
| 32 | 896 | 14 | 380.36 | 323.02 | 181.72 | 56.3% | |
| 48 | 768 | 12 | 388.97 | 339.81 | 191.18 | 56.3% | |
| Hyper-connection family | Attention family | ||||||||
| Reference solutions | Standard | Plain Loop | mHC | HC | iHC | Hyperloop | MHAR | AttnRes | Loopback |
| 191M, 20 layers, (3.82B tokens) | |||||||||
| GSM8K | 1.822 | 1.865 | 1.795 | 1.801 | 1.826 | 1.833 | 1.826 | 1.797 | 1.768 |
| MATH-500 | 1.967 | 2.028 | 1.961 | 1.999 | 2.023 | 2.024 | 1.994 | 1.962 | 1.972 |
| HumanEval | 1.827 | 1.935 | 1.861 | 1.900 | 1.924 | 1.905 | 1.897 | 1.895 | 1.842 |
| MBPP | 2.021 | 2.118 | 2.028 | 2.042 | 2.081 | 2.074 | 2.076 | 2.064 | 2.018 |
| Task | 2 visits (trained) | 3 visits | (pp) |
|---|---|---|---|
| PIQA | 75.90 | 75.30 | -0.60 |
| HellaSwag | 63.72 | 65.11 | +1.39 |
| WinoGrande | 58.48 | 58.96 | +0.48 |
| OpenBookQA | 41.80 | 42.80 | +1.00 |
| CommonsenseQA | 21.87 | 22.36 | +0.49 |
| BoolQ | 64.95 | 65.60 | +0.65 |
| Model | Transformer params (M) | Extra FLOPs | Train tok/s/GPU | Peak GiB/GPU | Prefill tok/s |
| 191M, , 3.82B tokens, 16 sequences per GPU | |||||
| Standard | 141.59 | – | 222k | 38 | 885k |
| Plain Loop | 56.64 | +0.00% | 226k | 38 | 886k |
| mHC | 56.93 | +0.15% | 169k | 42 | 594k |
| HC | 56.65 | +0.07% | 186k | 42 | 671k |
| iHC | 57.04 | +0.17% | 116k | 67 | 370k |
| KV cache per | Weights + KV (GiB) | |||||
|---|---|---|---|---|---|---|
| Scale | Model | Total params (M) | Weights (GiB) | sequence (MiB) | 1 sequence | 8 sequences |
| 191M, 20 layers | Standard | 190.74 | 0.36 | 240 | 0.59 | 2.23 |
| Loopback | 105.80 | 0.20 | 240 | 0.43 | 2.07 | |
| 374M, 24 layers | Standard | 373.87 | 0.70 | 384 | 1.07 | 3.70 |
| Loopback | 245.42 | 0.46 | 384 | 0.83 | 3.46 | |
| 778M, 24 layers | Standard | 777.86 | 1.45 | 576 | 2.01 | 5.95 |
Transformer
Transformers are neural architectures that use attention to model relationships among sequence elements in language, vision, and scientific data.
See all 579 papers