Scaling Laws for Looped Mixture of Experts

Yanbei Chen; Anirudh Goyal; Raghuraman Krishnamoorthi

cs.LGSep 30, 2026

Scaling Laws for Looped Mixture of Experts

Authors: Yanbei Chen, Anirudh Goyal, Raghuraman Krishnamoorthi

Organizations: Meta AI

Abstract

Looped transformers and Mixture-of-Experts (MoE) offer complementary routes to efficient scaling: recurrence increases computational depth at fixed parameters, while MoE sparsity expands total capacity at fixed active compute. Yet existing scaling laws model recurrence or sparsity in isolation. In this work, we introduce Loop Scaling Laws, the first scaling law to jointly model recurrence and sparsity alongside model size and data. At its core is a bounded, sparsity-conditional recurrence mapping that characterizes the effective-parameter gain from looping and how sparsity raises this gain. The laws predict the held-out loss of looped models more accurately than prior alternatives, and recover the standard dense and MoE scaling laws as special cases. Beyond prediction, the fitted laws provide a principled foundation for designing looped MoE models under compute and memory constraints. Downstream evaluations further demonstrate the complementary benefits of the two axes: sparsity delivers ~3x active-parameter efficiency, recurrence yields ~2x total-parameter efficiency on reasoning, and joint scaling further advances the performance frontier. As a practical extension, we show these gains hold at trillion-token scale: at matched training compute, a looped MoE with law-derived recurrence matches a ~2x larger non-looped MoE on the reasoning benchmarks, while enabling test-time scaling through recurrence.

Figures & tables

Figure 1: (a) Dense looped transformers reuse the same FFNs, whereas looped MoE models can route tokens to different experts across recurrent passes. (b) At fixed active parameters and training tokens, increasing recurrence yields diminishing returns, with the dense model ( $E{=}1$ ) saturating earlier than MoE models ( $E{>}1$ ). (c) Our Loop Scaling Laws model a bounded effective-parameter gain conditioned on recurrence $R$ and sparsity (varied by expert count $E$ ), unlike prior linear and power-law mappings that assume unbounded gains.

		Scaling axes					Optimality
LLM Category	Scaling law	$N/F$	$D$	$R$	$S/E$	Recurrence mapping	Compute	Memory
Dense	Chinchilla ( 2022 )	✓	✓	$\times$	$\times$	–	✓	$\times$
MoE	Unified routed ( 2022 )	✓	$\times$	$\times$	✓	–	$\times$	$\times$
MoE	Joint MoE ( 2025 )	✓	✓	$\times$	✓	–	✓	✓
Looped	Parcae ( 2026 /04 )	✓	✓	✓	$\times$	linear; unbounded †	✓	$\times$
Looped	Iso-Depth ( 2026 /05 )	✓	✓	✓	$\times$	power law; unbounded	✓	$\times$

Table 1: Comparison of scaling laws for training LLMs.

\checkmark

indicates variables explicitly modeled by each law.

N/F

denotes model size

N

/ training FLOPs

F

S/E

denotes MoE sparsity

S

(varied by expert expansion

E

Figure 2: (a) Illustration of effective-parameter gain under linear, power-law, bounded recurrence mappings. (b) Predictive loss by different recurrence mappings: each mapping is fitted on

R\leq 8

and applied to predict the held-out-

R

up to

R=16

. (c) Formula on recurrence mappings and their effective-parameter gains (top) and their evaluation on held-out recurrence

R

, model

N

, and data

D

with RMSE scores (bottom).

Figure 3: (a) Illustration of effective-parameter gain with sparsity-conditional recurrence mapping. (b) Predictive loss under different MoE recurrence mappings fitted on runs with

R\leq 8

; markers denote observations used for fitting. (c) Formula on MoE recurrence mappings and their effective-parameter gains (top), and their evaluation on held-out recurrence

R

, expert count

E

, model size

N

, and data

D

with RMSE scores (bottom).

Figure 4: (a,b) Effective-parameter multiplier over recurrence at

N_{\text{act}}{=}1.0

B, and its asymptote over expert count. (c,d) Recurrence IsoFLOP profiles for dense (

E{=}1

) and MoE (

E{=}8

) models; markers denote minima.

Figure 5: Predicted compute-optimal recurrence $R^{\star}$ of dense ( $E{=}1$ ) and MoE ( $E{=}8$ ) models. (a) Recurrence with compute-optimal loss across training-compute budgets. (b) IsoFLOP profiles over recurrence at $\bar{F}_{\text{train}}{=}5{\times}10^{21}$ FLOPs; stars mark $R^{\star}$ . (c) $R^{\star}$ across compute and active model size for $E{=}8$ (top) and $E{=}1$ (bottom).

Figure 6: Predicted memory-optimal expert count

E^{\star}

and joint optimum

(N_{\text{act}}^{\star},E^{\star},R^{\star})

\bar{F}_{\text{train}}{=}5{\times}10^{21}

FLOPs. (a)

E^{\star}

across bf16 weight-memory budgets at

R{=}4

(top) and

R{=}1

(bottom). (b)

E^{\star}

across INT4 weight memory and recurrence. (c) Predicted joint optimum

(N_{\text{act}}^{\star},E^{\star},R^{\star})

across INT4 weight memory.

Figure 7: Downstream scaling of sparsity and recurrence. (a,b) Scaling

E

R{=}1

; (c,d) Scaling

R

E{=}8

, with the dense baseline

E{=}1,R{=}1

. (a,c) report Overall performance, and (b,d) report Reasoning performance; dashed lines mark compute-matched comparisons.

Model	$\mathbf{R}$	$\displaystyle\frac{F_{\text{inf}}^{\text{0.3B}}(R)}{F_{\text{inf}}^{\text{0.6B}}(1)}$	Reasoning		Science			Commonsense Reasoning				Reading		Knowledge
Model	$\mathbf{R}$		BBH 3	GSM8K 8	ARC-C 25	ARC-E	OBQA	HS	PIQA	SIQA	Wino	BoolQ	DROP 3	MMLU 5	NQ 5	TQA 5	Overall ( $\Delta$ )
A0.6B-2.9B MoE	1	$1.0\times$	29.8	42.9	50.0	66.9	40.8	68.1	77.2	51.6	64.4	70.1	39.7	46.9	13.8	39.8	50.2
A0.3B-1.3B LoopMoE	1	$0.4\times$	18.6	10.5	36.1	57.3	33.8	52.7	70.7	43.8	54.6	58.1	23.0	29.2	5.7	18.2	36.6
	2	$0.8\times$	26.7	30.0	42.2	60.3	38.2	59.9	72.5	46.4	58.5	62.8	33.7	37.6	8.3	24.1	43.0 $\bm{(+6.4)}$
	3	$1.1\times$	30.1	41.0	44.9	60.9	39.6	62.4	73.2	50.2	61.3	65.9	38.4	41.6	8.9	25.6	46.0 $\bm{(+9.4)}$
	4	$1.5\times$	31.5	41.5	46.5	63.9	39.0	62.9	73.9	51.1	60.6	64.4	39.3	42.2	9.8	27.1	46.7 $\bm{(+10.1)}$

Table 2: Downstream results of A0.6B-2.9B MoE and A0.3B-1.3B LoopMoE across test-time recurrence

R

, trained at matched compute FLOPs

{\approx}1.5{\times}10^{22}

. Relative inference compute is normalized to the non-looped baseline.

\Delta

reports gains over LoopMoE at

R{=}1

. Evaluation protocols are provided in Appendix D.1 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

$E$	$R{=}1$	$R{=}2$	$R{=}4$	$R{=}8$
1	$1.0000\pm 0.0000$	$1.0000\pm 0.0000$	$1.0000\pm 0.0000$	$1.0000\pm 0.0000$
2	$1.0000\pm 0.0000$	$1.1212\pm 0.0692$	$1.2165\pm 0.0465$	$1.5088\pm 0.0529$
4	$1.0000\pm 0.0000$	$1.1917\pm 0.0974$	$1.2973\pm 0.0703$	$1.5300\pm 0.0785$
8	$1.0000\pm 0.0000$	$1.2099\pm 0.0868$	$1.4159\pm 0.0901$	$1.7035\pm 0.0861$

Appendix

Table B.1: Expert-path diversity

\Psi

across varying recurrence

R

and expert count

E

(mean

\pm

std).

	Architecture dimensions					$N_{\text{total}}(E)$ (B)
$N_{\text{act}}$ (B)	$d_{m}$	$n_{h}$	$n_{\ell}$	$n_{\ell,\text{loop}}$	$N_{\text{loop}}$ (B)	$E{=}1$	$E{=}2$	$E{=}4$	$E{=}8$	$E{=}16$
0.3	768	12	20	16	0.1	0.3	0.5	0.8	1.3	2.5
0.6	1024	16	26	22	0.3	0.6	0.9	1.6	2.9	5.5
1.0	1280	20	32	28	0.7	1.0	1.6	2.9	5.4	10.5
1.6	1536	24	38	34	1.2	1.6	2.7	4.8	9.1	17.7
2.4	1792	28	44	40	1.8	2.4	4.1	7.5	14.3	27.8

Appendix

Table C.1: Model scaling ladder over five model scales. Parameter counts are in billions (B).

N_{\text{act}}

and

N_{\text{total}}

include embedding parameters, whereas

N_{\text{loop}}

includes only parameters in the recurrent block.

Base scaling law coefficients					MoE coefficients						Recurrence coefficients			Fit
$A$	$\alpha$	$B$	$\beta$	$c$	$\delta$	$\gamma$	$\omega$	$\zeta$	$E_{\text{start}}$	$E_{\max}$	$\kappa_{1}$	$\kappa_{2}$	$\theta$	RMSE
0.8085	$-0.1951$	3.4291	$-0.7548$	1.3555	$-0.1935$	$-0.0137$	$-0.2056$	0.0881	1.3174	57.1201	0.3682	1.4037	0.3286	0.0037

Appendix

Table C.2: Fitted coefficients of MoE Loop Scaling Law in Eq. 9 . RMSE: root-mean-square error of the fit.

Coefficient	Estimate	90% CI
$\kappa_{1}$	0.3682	$[0.3658,\,0.3723]$
$\kappa_{2}$	1.4037	$[1.3896,\,1.4243]$
$\theta$	0.3286	$[0.3183,\,0.3350]$

Appendix

Table C.3: Bootstrap results on the MoE loop scaling law. (a) Coefficients fitted on full-data or the 90% sampled subsets with intervals spanning the 5th–95th percentiles over 100 refit runs. (b) Derived joint optimum under the compute and memory budgets (with INT4 weights) used in Figure 6 (c) on the 100 refit runs.

Category	Benchmark	Task name	$n$ -shot	Metric
Reasoning	BBH	leaderboard_bbh	3	acc_norm
	GSM8K	gsm8k_cot	8	exact_match,flexible-extract
Science	ARC-C	arc_challenge	25	acc_norm
	ARC-E	arc_easy	0	acc_norm
	OBQA	openbookqa	0	acc_norm
Commonsense Reasoning	HellaSwag	hellaswag	0	acc_norm

Appendix

Table D.1: Evaluation setups of 14 foundational benchmarks. Task names and metrics follow lm-eval .

Explore similar work

May 9, 2026cs.LG

Sparse Layers are Critical to Scaling Looped Language Models

Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries. However, looped models do not scale as favorably as standard transformers with unique layers. We compare standard and Mixture-of-Experts (MoE) transformers, with and without looping, and find two main results. First, we find Looped-MoE models scale better than the standard baseline while dense looped models do not. We trace this to routing divergence between loops: in Looped-MoE models, different experts are activated on each pass through the same shared layers, recovering expressivity without additional parameters. Our second finding is that looped models have better compute-quality trade-offs with early exits than standard models. Because each loop ends with the same layers that produce the final output, loop boundaries are superior exit points, as confirmed by earlier output convergence at these points. In sum, we provide a clear direction for scaling looped models: a Looped-MoE model with early exits can not only beat standard transformers at scale, but also enable significant memory and inference savings with minimal degradation in quality.

Ryan Lee, Jacob Biloki, Edward J. Hu +1

1USC Information Sciences Institute · 2Netflix · 3Independent Researcher

Transformer Architectures Layer-Wise Localized Lora-Moe Mixture-Of-Experts Peak Memory

Ask PDF

Jun 3, 2026cs.LG

LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling

Mixture-of-Experts (MoE) and looped architectures scale models along two orthogonal axes, namely parameter capacity and effective depth. However, mainstream looped architectures rely on dense backbones that couple parameter count with per-token FLOPs, which makes it impossible to isolate the effect of iterative computation under matched budgets. To this end, we present LoopMoE, a looped MoE language model that integrates sparse routing with iterative weight-shared computation through two designs. The first is IterAdaLN, which resolves weight-sharing symmetry via a modulation signal jointly conditioned on the iteration index and the per-token hidden state. The second is a capacity-balancing strategy that recovers the attention-to-FFN active parameter ratio of well-tuned non-looped references. Together, these designs enable the first strictly controlled, head-to-head evaluation of a looped MoE against a Vanilla MoE under identical total parameters, per-token FLOPs, and active sublayer ratios. At the 3B scale, LoopMoE outperforms the Vanilla MoE on 8 of 9 downstream benchmarks with an average improvement exceeding 1 point. At the 9B scale, LoopMoE continues to outperform the matched Vanilla MoE, indicating that the architectural gain persists at larger scale. Our work establishes a controlled synthesis of sparsity and recurrence, and suggests a promising direction for looped language models.

Wenkai Chen, Tianshu Li, Wenyong Huang +3

Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Co.,Ltd.

Mixture-Of-Experts Large Language Models Mixture-Of-Experts Localized Lora-Moe Iterative

Ask PDF

Oct 1, 2026cs.LG

Looping Beyond Twice: A Scalable Recipe for Looped Mixture-of-Experts

Looped Transformers introduce recurrent depth as a new scaling axis for LLMs: by repeatedly applying shared Transformer blocks, they increase effective depth without increasing parameter count. However, the benefits of looping remain unclear for large MoE LLMs under FLOPs-matched comparisons. The main reason is that the gains from additional iterations diminish quickly and can even turn into degradation, so the extra FLOPs spent on looping yield little substantial improvement. Consequently, prior work typically settles on two loops. We identify two main obstacles to scaling looped MoE. First, looping inherits and amplifies the curse of depth: hidden-state variance grows with each iteration as residual updates accumulate, which destabilizes deep recurrence and causes representations to drift. Second, looped MoE suffers from expert selection collapse: routers repeatedly select the same experts across loops, so extra iterations add computation without adding computational diversity. Guided by this diagnosis, we propose LOOM, built on a single principle: each loop should contribute new computation while keeping the recurrent state stable. LOOM stabilizes recurrence by scaling residual updates to bound variance growth and re-injecting the input embedding at every loop, and diversifies it through per-loop routers that engage different experts and a Looping Residual that carries earlier outputs forward. Experiments across 100M-1.7B models show stable scaling to 9-12 loops. Under near-iso-FLOP, the 700M model performs best at 5 loops, reducing perplexity from 18.36 to 16.54 and improving average zero-shot accuracy from 38.84% to 39.53% over the non-looped baseline. Without FLOP matching, the 1.7B model trained on 60B tokens peaks at 9 loops, reducing perplexity from 9.62 to 7.77 and improving average zero-shot accuracy from 42.4% to 47.7%. Code is available https://github.com/hed-ucas/LOOM.

Di He, Pengxiang Li, Da Chang +3

Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Peng Cheng Laboratory · University of Chinese Academy of Sciences +5

Mixture-Of-Experts Large Language Models Mixture-Of-Experts Transformer Architectures Hidden States Sequential Scaling Recipe

Ask PDF