Looped Mixture-of-Experts (MoE) models combine parameter reuse with sparse computation to improve performance under matched budgets. Prior work demonstrates that different loop visits can play different roles even when they share parameters. To probe the contributions of different loop visits, we learn separate residual coefficients in a model that applies a shared block twice, observing a smaller scale in the first visit. Fixed coefficient experiments show that reducing the first visit's scale preserves or improves performance, whereas the same reductions to the second hurt it. This asymmetry motivates us to explore asymmetric compute allocation. We propose LatentLoop, which compresses the first visit's sequence, retains full resolution in the second, and reallocates saved computation to increase the hidden size. We further learn how strongly the first visit updates the full sequence, and share MoE weights across visits while using separate attention parameters to accommodate their different sequence resolutions. Across 15 configurations spanning 1.3B--7B parameters, with the largest budget exceeding 1021 FLOPs, LatentLoop consistently lowers validation NLL. At a fitted NLL of 1.988, predicted compute savings are 19.0% and 7.9% relative to vanilla MoE and the looped baseline. It also improves average scores on nine benchmarks at all sizes in 0-shot and 5-shot settings. These results suggest asymmetric compute allocation is a promising direction for looped MoE models.
Figures & tables
Figure 1: The four steps of budget matching. We adjust attention projection dimensions, model width, the number of stored experts, and training steps in this order.
Budget
SMELT
This work
Backbone parameters
1.0 %
0.6 %
KV cache
3.6 %
0.8 %
FLOPs per token
3.9 %
0.9 %
Table 1: Maximum absolute budget deviations.
Figure 2: Residual scale asymmetry at 1.3B and 3B. (a) Learned coefficients for the two visits. (b) Validation NLL with fixed coefficients; rows and columns correspond to the first and second visits. Colors show differences from the same size (1,1) baseline using scores rounded to three decimals.
Figure 3: LatentLoop Architecture. LatentLoop compresses the first visit’s sequence and retains full resolution in the second, sharing MoE weights with independent attention blocks.
Shot
Params
Token
Model
LD-O
LD-S
HS
PIQA
WG
ARC-E
ARC-C
SciQ
MMLU
Avg
MoE Base
31.44
29.30
48.73
70.46
53.91
69.40
35.41
90.40
32.16
51.24
1.3B / 0.2B
40B
MoE Loop
31.15
28.72
49.61
69.91
56.04
67.42
36.26
89.60
31.86
51.18
LatentLoop
32.99
30.20
50.36
69.80
56.35
67.38
37.12
91.10
31.93
51.91
MoE Base
40.89
36.17
57.89
73.18
57.93
73.40
42.49
93.10
35.12
56.69
5-shot
3B / 0.5B
70B
MoE Loop
39.72
37.26
59.48
74.70
59.19
72.90
43.69
92.90
36.09
57.33
LatentLoop
43.86
41.74
60.57
74.65
57.62
74.41
45.14
92.80
35.83
58.51
Table 2: Downstream benchmark scores (%). Params denote approximate total/active sizes; token budgets are FLOPs matching reference labels. LD-O/LD-S denote LAMBADA OpenAI/Standard, HS is HellaSwag, WG is WinoGrande, and MMLU uses continuation scoring.
Figure 4: Fitted compute-optimal NLL. Solid segments lie within the convex hull of observed (logF,logD) points; dashed segments involve extrapolation.
Strategy
NLL ↓
Baselines
MoE Base
2.0426
MoE Loop
2.0293
Initial first visit reductions
Residual width
2.0361
Expert intermediate width
2.0364
Table 3: Compute reduction strategies.
Table 8
Figure 5: Learned residual coefficients with three and four visits, for 1.3B and 3B.
Figure 6: Learned scaled decoder coefficients at 3B/40B. (a) Channel means during training, with 25th–75th percentile bands. (b) Final channel distributions.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Layers
d
de
dh
E
Pb
Pall
1.3B
MoE Base
12/12
1024
512
128
64
1.240232
1.305768
MoE Loop
12/18
864
432
86
91
1.241561
1.296857
LatentLoop
12/18
944
472
102
75
1.238587
1.299003
3B
MoE Base
16/16
1408
704
176
64
3.125857
3.215969
Appendix
Table 6: Actual architecture configurations. The Layers column gives stored/executed depth; d , de , and dh are model width, expert intermediate width, and attention head dimension. E is the stored expert count per layer. Parameter counts are in billions.
Model
KV (MiB)
Fstep
ΔPb
ΔPall
Δ KV
ΔFstep
1.3B
MoE Base
48.00
1.676
+0.00
+0.00
+0.00
+0.00
MoE Loop
48.38
1.685
+0.11
-0.68
+0.78
+0.54
LatentLoop
47.81
1.672
-0.13
-0.52
-0.39
-0.25
3B
MoE Base
88.00
3.767
+0.00
+0.00
+0.00
+0.00
Appendix
Table 7: Single-step resources and deviations from MoE Base at the same tier. KV is BF16 K/V tensor storage per 4096-token sequence; Fstep is in PFLOPs. All Δ columns are signed percentage differences computed before rounding.
Model
Updates
Token
ΔD
C
ΔC
1.3B
MoE Base
38,146
39.999B
+0.00
0.6392
+0.00
MoE Loop
37,941
39.784B
-0.54
0.6392
+12.47
LatentLoop
38,240
40.098B
+0.25
0.6392
-0.98
3B
MoE Base
66,757
70.000B
+0.00
2.5146
+0.00
Appendix
Table 8: Actual training budgets at the largest endpoint of each tier. Token counts are in billions and C is in units of 1020 FLOPs. Signed deviations from the same tier’s MoE Base are computed before rounding: ΔD is in percent and ΔC in 10−4% .
Hyperparameter
Value
Training precision
BF16 mixed precision
Context length
4096
Global batch size
256 sequences
Prediction tokens per update
1,048,576
Optimizer
AdamW ( Loshchilov and Hutter, 2017 )
(β1,β2)
(0.9,0.95)
Appendix
Table 9: Common training hyperparameters. Warmup is applied once per trajectory started from scratch. The variable p∈[0,1] denotes progress through the annealing phase.
Size
Token budgets (B)
Points
1.3B
5, 10, 20, 40
4
3B
5, 10, 20, 40, 70
5
7B
5, 10, 20, 40, 70, 140
6
Appendix
Table 10: Reference token budgets.
Benchmark
Harness task
Metric
LAMBADA OpenAI
lambada_openai
acc
LAMBADA Standard
lambada_standard
acc
HellaSwag
hellaswag
acc_norm
PIQA
piqa
acc_norm
WinoGrande
winogrande
acc
ARC-Easy
arc_easy
acc_norm
Appendix
Table 11: Downstream task identifiers and scoring metrics. Both shot settings use the same metric. acc_norm is the harness length-normalized accuracy.
Figure 7: GPU time normalized to MoE Base at each model size. GPU time is elapsed training time multiplied by GPU count.
Table 18Table 19
Shot
Params
Token
Model
LD-O
LD-S
HS
PIQA
WG
ARC-E
ARC-C
SciQ
MMLU
Avg
5-shot
1.4B / 1.4B
250B
SpiralFormer-L
46.32
45.29
51.66
70.51
59.51
60.61
31.91
93.20
30.35
54.37
7B / 1B
140B
LatentLoop
49.21
45.92
68.44
76.33
63.93
79.55
51.71
96.20
39.35
63.40
0-shot
1.4B / 1.4B
250B
SpiralFormer-L
52.86
46.69
50.88
69.64
56.43
50.59
28.75
80.70
29.24
51.75
7B / 1B
140B
LatentLoop
54.92
45.22
66.91
75.68
62.59
74.20
46.50
90.40
37.69
61.57
Appendix
Table 16: System comparison on downstream benchmarks (%). Params denote approximate total/active parameter counts. Benchmark abbreviations follow Table 2 ; Avg is the unweighted mean of the nine scores.
Params
Tokens
MoE Base
MoE Loop
LatentLoop
1.3B
5B
2.4093
2.4105
2.3979
10B
2.3026
2.3004
2.2863
20B
2.2201
2.2161
2.2022
40B
2.1572
2.1519
2.1393
3B
5B
2.3157
2.3077
2.2971
10B
2.2010
2.1910
2.1805
Appendix
Table 17: Validation NLL ( ↓ ) at all scaling endpoints. Parameter tiers denote approximate total parameter counts. The lowest NLL at each matched budget is bold.
Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.
Mixture-of-Experts (MoE) and looped architectures scale models along two orthogonal axes, namely parameter capacity and effective depth. However, mainstream looped architectures rely on dense backbones that couple parameter count with per-token FLOPs, which makes it impossible to isolate the effect of iterative computation under matched budgets. To this end, we present LoopMoE, a looped MoE language model that integrates sparse routing with iterative weight-shared computation through two designs. The first is IterAdaLN, which resolves weight-sharing symmetry via a modulation signal jointly conditioned on the iteration index and the per-token hidden state. The second is a capacity-balancing strategy that recovers the attention-to-FFN active parameter ratio of well-tuned non-looped references. Together, these designs enable the first strictly controlled, head-to-head evaluation of a looped MoE against a Vanilla MoE under identical total parameters, per-token FLOPs, and active sublayer ratios. At the 3B scale, LoopMoE outperforms the Vanilla MoE on 8 of 9 downstream benchmarks with an average improvement exceeding 1 point. At the 9B scale, LoopMoE continues to outperform the matched Vanilla MoE, indicating that the architectural gain persists at larger scale. Our work establishes a controlled synthesis of sparsity and recurrence, and suggests a promising direction for looped language models.
Wenkai Chen, Tianshu Li, Wenyong Huang +3
Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Co.,Ltd.
Looped transformers and Mixture-of-Experts (MoE) offer complementary routes to efficient scaling: recurrence increases computational depth at fixed parameters, while MoE sparsity expands total capacity at fixed active compute. Yet existing scaling laws model recurrence or sparsity in isolation. In this work, we introduce Loop Scaling Laws, the first scaling law to jointly model recurrence and sparsity alongside model size and data. At its core is a bounded, sparsity-conditional recurrence mapping that characterizes the effective-parameter gain from looping and how sparsity raises this gain. The laws predict the held-out loss of looped models more accurately than prior alternatives, and recover the standard dense and MoE scaling laws as special cases. Beyond prediction, the fitted laws provide a principled foundation for designing looped MoE models under compute and memory constraints. Downstream evaluations further demonstrate the complementary benefits of the two axes: sparsity delivers ~3x active-parameter efficiency, recurrence yields ~2x total-parameter efficiency on reasoning, and joint scaling further advances the performance frontier. As a practical extension, we show these gains hold at trillion-token scale: at matched training compute, a looped MoE with law-derived recurrence matches a ~2x larger non-looped MoE on the reasoning benchmarks, while enabling test-time scaling through recurrence.
Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries. However, looped models do not scale as favorably as standard transformers with unique layers. We compare standard and Mixture-of-Experts (MoE) transformers, with and without looping, and find two main results. First, we find Looped-MoE models scale better than the standard baseline while dense looped models do not. We trace this to routing divergence between loops: in Looped-MoE models, different experts are activated on each pass through the same shared layers, recovering expressivity without additional parameters. Our second finding is that looped models have better compute-quality trade-offs with early exits than standard models. Because each loop ends with the same layers that produce the final output, loop boundaries are superior exit points, as confirmed by earlier output convergence at these points. In sum, we provide a clear direction for scaling looped models: a Looped-MoE model with early exits can not only beat standard transformers at scale, but also enable significant memory and inference savings with minimal degradation in quality.
Ryan Lee, Jacob Biloki, Edward J. Hu +1
1USC Information Sciences Institute · 2Netflix · 3Independent Researcher