Looped Transformers increase effective depth by repeatedly applying shared layers, offering a way to scale computation without increasing parameter count. However, full-stack looping places context construction and iterative state refinement within the same recurrent computation, without explicitly distinguishing their roles. We introduce the Asymmetric Looped Transformer (AsymLoop), which separates single-pass context encoding from recurrent state refinement. A single-pass causal encoder produces layer-specific Global key-value (KV) caches that remain fixed across iterations, while a weight-tied recurrent decoder generates Local KV from its evolving state. Joint attention over both sources under a single softmax allows each update to integrate persistent contextual information with the current intermediate state. For training, we apply a heavy-tail-guided layerwise learning-rate method to the encoder and decoder. Experiments on FineWeb-Edu across four model scales, from 60M to 1B, show consistently lower validation perplexity and higher average downstream accuracy than Full Loop under matched effective depth and training token budgets. At 1B, AsymLoop improves average zero-shot accuracy across seven commonsense reasoning benchmarks from 49.58% to 50.85%. Further experiments extend these benefits to longer training budgets, an OLMo-style backbone, and sparse mixture-of-experts models. These results support asymmetric context encoding and state refinement as an effective design for recurrent language models.
Figures & tables
Figure 1: AsymLoop architecture and Global–Local attention. (a) Dense executes each layer once. (b) Full Loop repeats the entire stack with shared parameters. (c) AsymLoop runs a causal encoder once and a shared-parameter decoder for K iterations. The encoder output provides layer-specific Global KV that remains fixed across iterations. (d) Each decoder layer jointly attends to fixed Global KV and Local KV computed from its current state, using a single softmax with causal masking for both sources.
Figure 2: Validation and zero-shot performance across model sizes. Full Loop ( K=2 ) and AsymLoop ( K=3 ) have the same effective depth. (a, b) Validation cross-entropy versus model parameter count and training token budget, respectively. (c) Average zero-shot accuracy across seven tasks.
Scale
Model
OBQA
WinoGrande
ARC-c
ARC-e
HellaSwag
SIQA
PIQA
Avg.
Zero-shot
60M
Dense
17.80
50.83
17.41
45.33
27.39
34.54
59.03
36.05
Full Loop
16.80
50.28
20.22
46.51
27.14
37.00
58.71
36.66
AsymLoop
17.80
50.59
18.17
47.31
27.72
36.49
59.41
36.79
135M
Dense
18.80
48.54
21.42
52.90
29.06
37.41
61.53
38.52
Full Loop
19.80
51.85
22.87
51.52
29.09
38.69
62.19
39.43
Table 1: Zero- and five-shot commonsense reasoning accuracy ( ↑ , % ). Full Loop ( K=2 ) and AsymLoop ( K=3 ) have the same effective depth. Avg. is the mean across seven tasks. Darker/lighter blue mark the best/second-best results at each model size and shot setting.
Model
OBQA
WinoGrande
ARC-c
ARC-e
HellaSwag
SIQA
PIQA
Avg.
Zero-shot
Dense
25.40
54.70
31.06
66.33
37.69
39.61
68.44
46.18
Full Loop ( K=2 )
27.20
53.43
31.91
66.20
39.04
41.40
68.55
46.82
AsymLoop ( K=3 , ALRS)
26.80
55.80
32.68
68.35
38.54
40.02
69.04
47.32
Five-shot
Dense
21.20
54.54
30.72
67.00
37.21
41.20
68.77
45.81
Table 2: Zero- and five-shot downstream accuracy ( ↑ , % ) of 350M models trained on 35B tokens. Avg. is the mean across seven tasks. Darker/lighter blue mark the best/second-best results within each shot setting.
Scale
Loop / Total Layers
K
Effective Depth
Architecture
Budget
PPL
60M
0 / 6
1
6
Dense
1.68B
23.72
3 / 6
1
6
AsymLoop (Ours)
1.68B
21.79
6 / 6
2
12
Full Loop
1.68B
23.03
3 / 6
3
12
AsymLoop (Ours)
1.68B
21.45
6 / 6
3
18
Full Loop
1.68B
22.77
3 / 6
5
18
AsymLoop (Ours)
1.68B
21.30
Table 3: Effect of recurrent depth. Validation perplexity ( ↓ ) of Dense, Full Loop, and AsymLoop across model sizes at matched effective depths. Darker/lighter blue mark lower/higher perplexity within each depth-matched pair.
Model Variant
PPL ↓ ( K=3 )
PPL ↓ ( K=5 )
(a) Architectural components
AsymLoop (ALRS)
23.751
23.580
AsymLoop (uniform LR)
24.727
24.619
w/o QKNorm
24.984
24.663
w/o Global KV
25.566
25.751
w/o Local KV
25.656
25.465
Table 4: Ablations at the 60M scale. Validation perplexity ( ↓ ). (a) Architectural components and (b) learning-rate ablations on AsymLoop ( K=3,5 ). (c) Uniform LR versus ALRS on Full Loop ( K=2 ). Darker/lighter blue mark the best/second-best results within each panel of (a, b).
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Configuration
60M
135M
350M
1B
Dense parameters
91.8M
134.3M
368.2M
1.34B
AsymLoop parameters
95.4M
141.4M
393.4M
1.44B
Layers L ( E+D )
6 (3+3)
12 (6+6)
24 (12+12)
24 (12+12)
dmodel
768
768
1024
2048
dff
2048
2048
2736
5461
Heads
6
12
16
32
Appendix
Table 5: Model architectures and pre-training hyperparameters. Backbone configurations follow He et al. (2026) . Parameter counts include the untied input embeddings and output head.
Figure 3: Learning-rate dynamics under ALRS at 135M and 350M. Left: Full training trajectories. Middle: The first 20% of training. Right: Decoder-to-encoder mean learning-rate ratios. Vertical dotted lines mark the end of allocation updates.
Figure 4: Normalized stable rank under uniform learning rates and ALRS. Results for 135M and 350M AsymLoop models ( K=3 ). For each matrix W , we compute ∥W∥F2/[∥W∥22min(shape(W))] and average equally within each module group.
Model Variant
CE ↓ ( K=1 )
CE ↓ ( K=3 )
CE ↓ ( K=5 )
AsymLoop
3.238735
3.207915
3.203515
w/o Global KV
3.260346
3.241257
3.248483
Global KV gain G(K)
0.021611
0.033343
0.044968
Gain beyond K=1
0.000000
0.011732
0.023357
Appendix
Table 6: Global KV and recurrent depth. Validation CE ( ↓ ) with and without Global KV. G(K) is the CE reduction from Global KV; the final row reports the additional gain G(K)−G(1) .
Model Variant
PPL ↓
AsymLoop
24.727
w/o Global KV
25.566
w/ additive injection
25.116
Full Loop
25.053
Appendix
Table 7: Context-access variants and a Full Loop baseline. Validation perplexity ( ↓ ). Full Loop uses K=2 ; all other variants use K=3 . Darker/lighter blue mark the best/second-best results.
Reference
K
Effective depth
CE
PPL
Dense, Uniform
1
6
3.271820
26.359274
Dense, layerwise LR
1
6
3.209192
24.759083
Full Loop, Uniform
2
12
3.241455
25.570904
Full Loop, Uniform
3
18
3.229346
25.263120
AsymLoop w/o QKNorm, Uniform
3
12
3.218239
24.984094
AsymLoop w/o QKNorm, Uniform
5
18
3.205305
24.663015
Appendix
Table 8: Additional validation results. Cross-entropy and perplexity ( ↓ ) under a common evaluation protocol. Darker/lighter blue mark the best/second-best results in each metric column.
Forward GFLOPs/token
Additional cost (%)
Model size
Full Loop
AsymLoop
Global KV proj.
Global attn.
Total
60M
0.256967
0.292356
2.75
11.02
13.77
135M
0.464585
0.535364
3.05
12.19
15.23
350M
1.476657
1.677984
3.41
10.23
13.63
1B
5.365891
5.869208
3.75
5.63
9.38
Appendix
Table 9: Analytical forward FLOPs at matched effective depth. Full Loop uses R=2 and AsymLoop uses K=3 , with N=1024 and V=32128 . Costs are reported per input position; overhead is relative to Full Loop.
Figure 5: Training compute–performance trade-offs across model scales and recurrent depths. Average zero-shot accuracy over seven downstream tasks versus analytical training FLOPs. The horizontal axis uses a logarithmic scale in EFLOPs ( 1018 FLOPs). Panels pair AsymLoop with the baseline of matched effective depth: (a) Dense and AsymLoop K=1 ( L ), (b) Full Loop K=2 and AsymLoop K=3 ( 2L ), and (c) Full Loop K=3 and AsymLoop K=5 ( 3L ). Purple, yellow, and orange denote AsymLoop, Full Loop, and Dense, respectively.
Model size
Model
FFN
Params
s/step
Peak mem.
Validation CE
60M
Full Loop K=2
2048
91.83M
0.232
20.2 GiB
3.0434
+ FLOPs matching
2656
100.23M
0.230
21.6 GiB
3.0139
AsymLoop K=3
2048
95.37M
0.216
21.7 GiB
2.9684
135M
Full Loop K=2
2048
134.30M
0.286
30.9 GiB
2.81550±0.00221
+ FLOPs matching
2656
151.11M
0.296
33.6 GiB
2.78983±0.00197
AsymLoop K=3
2048
141.38M
0.367
33.8 GiB
2.74947±0.00299
Appendix
Table 10: Validation cross-entropy and training cost with a widened Full Loop baseline. The widened baseline matches AsymLoop’s analytical per-step training FLOPs. The 60M results are single runs; the 135M results are means ± sample standard deviations over three training seeds. Step time and peak memory are measured on eight H100 GPUs. Darker/lighter blue marks the best/second-best cross-entropy at each model scale.
Measured GFLOPs/step
Widened Full Loop
Model size
AsymLoop
Full Loop (wide)
FFN Width
Difference vs. AsymLoop
60M
1134.878
1108.907
2656
−2.29%
135M
2118.157
2092.198
2656
−1.23%
Appendix
Table 11: Measured matrix FLOPs per training step. Counts include forward, backward, and activation recomputation, with sequence length 1024 and batch size 1.
Figure 6: Downstream performance during pre-training at 1B. Zero-shot accuracy on seven benchmarks and their average. Purple and yellow denote AsymLoop and Full Loop, respectively.
Model
Depth
CE ↓
PPL ↓
Dense
8
3.508664
33.403615
Full Loop ( K=2 )
16
3.501389
33.161466
AsymLoop ( K=3 )
16
3.341073
28.249416
Appendix
Table 12: Validation performance on an OLMo-style backbone. Cross-entropy and perplexity ( ↓ ). Depth denotes effective depth. Darker/lighter blue marks the best/second-best results.
Model
GSM8K Accuracy (%) ↑
Relative gain over Dense (%) ↑
Dense
8.94
–
Full Loop ( K=2 )
10.24
+14.54
AsymLoop ( K=3 )
11.75
+31.43
Appendix
Table 13: GSM8K accuracy of 1B models after three epochs of full-parameter supervised fine-tuning. The best result is highlighted in blue.
Model
Effective depth
CE ↓
PPL ↓
60M configuration: 1.6B training tokens
Dense + MoE
6
2.9856
19.798
Full Loop K2 + MoE
12
2.9363
18.845
AsymLoop K3 + MoE
12
2.8986
18.149
135M configuration: 3B training tokens
Dense + MoE
12
2.7489
15.625
Appendix
Table 14: Validation performance of sparse MoE models. Cross-entropy and perplexity ( ↓ ) with eight experts and top-2 routing. Darker/lighter blue marks the best/second-best results at each model scale.
Looped Transformers offer a parameter-efficient approach to language modeling by repeatedly applying a shared backbone. Vanilla looped Transformers use each backbone output directly as the input to the next iteration. These outputs may contain information useful several iterations later, but retaining it across the loop depends entirely on the single hidden state. We introduce StateLoop, which maintains one additional persistent state per token to retain information across iterations. At each iteration, a learned gate controls how much of the previous state is retained and how much information from the backbone output is incorporated. The updated state is projected back and added to the backbone output for subsequent iterations and final prediction. This design reduces Pile perplexity and improves average accuracy across eight zero-shot tasks over Vanilla Looped. Its perplexity advantage over Vanilla Looped grows with trained loop depth, consistent with persistent state becoming increasingly useful as recurrent computation deepens.
Feichen Song, Boyi Zeng, He Li +7
Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute
Looped Transformers increase computation per token by repeatedly applying a shared backbone, offering a way to improve language modeling and downstream task performance at a fixed backbone parameter count. However, each loop depends on the preceding loop, so increasing recurrent depth also increases decoding latency. Existing approaches include Parallel Loop Transformers, which reduce latency by overlapping loop computation across token positions, but yield rapidly diminishing marginal accuracy gains at larger depths. To address this challenge, we introduce LoopFold, which folds computation at multiple recurrent depths into parallel batches for the same token prefix. A lightweight predictor estimates inputs to later loops so they can start before earlier loops finish. A corrector then refines the outputs of these later loops using newly available results from earlier loops. All components are trained jointly using only the final next-token prediction loss. In experiments with 1.4B models, deeper LoopFold configurations achieve progressively higher average zero-shot accuracy across eight tasks, rising from 53.41% to 56.38%. Vanilla looped models and PLT show limited gains over the same depth range. Compared with a nine-loop vanilla model, LoopFold uses five sequential stages and additional parallel computation to reduce decoding latency by 46%.
Feichen Song, Boyi Zeng, Yiqin Hao +7
Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute
Looped Transformers achieve parameter efficiency by repeatedly executing a shared block across recurrent loops. Each loop yields an intermediate representation decodable for the same next token, yet standard decoding discards earlier states. Because earlier loops embody less computation, recurrence inherently supplies aligned weak-and-strong prediction pairs without auxiliary models or external training. We introduce LoopCD, a training-free contrastive decoding framework that guides token selection by contrasting the final prediction with an earlier recurrent pass, operating either in logit space with one extra output pass (LoopCD-Logits) or in hidden-state space with zero output overhead (LoopCD-Hidden). Across four looped Transformer families, LoopCD delivers substantial, consistent gains at full recurrent depth: LoopCD-Logits raises Ouro-2.6B-Thinking's AIME 2024 pass@1 from 61.88% to 73.33%, while LoopCD-Hidden lifts Huginn's HumanEval pass@1 from 22.56% to 31.71%. Crucially, these performance gains enable halving the number of recurrent loops while still matching or exceeding full-depth unguided baselines, reducing forward FLOPs by 22.5% to 48.2%. By transforming intermediate recurrent states into effective guidance signals, LoopCD achieves superior decoding quality while substantially reducing inference compute.