Generalization and Scaling Laws for Mixture-of-Experts Transformers

Mansour Zoubeirou a Mayaki

cs.LGApr 10, 2026

Generalization and Scaling Laws for Mixture-of-Experts Transformers

Authors: Mansour Zoubeirou a Mayaki

Organizations: Univ Lyon, Lyon 2, LIRIS UMR 5205 F-69676, Lyon, France

Abstract

We develop a theory of generalization and scaling for Mixture-of-Experts (MoE) Transformers that cleanly separates active per-input capacity from routing combinatorics. By conditioning on fixed routing patterns and union-bounding across them, we derive a sup-norm covering-number bound whose metric entropy scales with the active parameter budget and incurs a MoE-specific routing overhead. Combined with a standard ERM analysis for squared loss, this yields a generalization bound under a $d$ -dimensional manifold data model and $C^β$ targets, showing that approximation and estimation trade off as in dense networks once active parameters are accounted for appropriately. We further prove a constructive approximation theorem for MoE architectures, showing that, under the approximation construction, error can decrease either by scaling active capacity or by increasing the number of experts, depending on the dominant bottleneck. From these results we derive neural scaling laws for model size, data size, and compute-optimal tradeoffs. Overall, our results provide a transparent statistical reference point for reasoning about MoE scaling, clarifying which behaviors are certified by worst-case theory and which must arise from data-dependent routing structure or optimization dynamics.

Figures & tables

Figure 1: Routing ablation across expert pool size $M$ and active experts $k$ . (a) In the moderate regime ( $M/k\leq 8$ ), validation loss increases with the routing complexity term $k\log(eM/k)$ , consistent with its interpretation as a worst-case routing overhead. (b) Over the full range, performance improves again for sufficiently large $M/k$ , suggesting gains from expert specialization not captured by the worst-case analysis.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Values Adopted for Scaling Analysis
Model (number of parameters)	Tinystories		Wikitext-103		OpenWebText
Model (number of parameters)	Mean (CI)	Median	Mean (CI)	Median	Mean (CI)	Median
gpt2 (117 million)	$22.9\pm 2.9$	$23.1$	$31.6\pm 2.9$	$32.4$	$46.7\pm 6.2$	$49.8$
gpt2-medium (345 million)	$21.4\pm 1.1$	$21.9$	$31.6\pm 1.7$	$32.9$	$43.7\pm 2.6$	$45.0$
gpt2-large (774 million)	$22.1\pm 0.9$	$22.4$	$31.5\pm 1.6$	$32.1$	$44.7\pm 2.6$	$47.7$
gpt2-xl (1558 million)	$19.7\pm 0.6$	$19.6$	$30.3\pm 1.0$	$31.1$	$43.0\pm 1.7$	$43.9$

Appendix

Table 1: Intrinsic Dimension (ID) Summary across Models and Datasets. The ID estimates remain remarkably stable across different model sizes for each dataset. For the subsequent scaling analysis (Figure 5 and 4 ), we chose the median ID values for each dataset.

Dataset	Chosen $d$	Best-Fit $\beta$	Theoretical Exponents			Empirical Exponents
Dataset	Chosen $d$	Best-Fit $\beta$	$\boldsymbol{\alpha_{N}=\frac{2\beta}{d}}$	$\boldsymbol{\alpha_{D}=\frac{2\beta}{2\beta+d}}$	Consistency $\frac{\alpha_{N}}{1+\alpha_{N}}$	$\boldsymbol{\widehat{\alpha}_{N}}$	$\boldsymbol{\widehat{\alpha}_{D}}$
WikiText-103	$32$	$1.0$	$0.063$	$0.059$	$0.059$	$\mathbf{0.060}$	$\mathbf{0.058}$
OpenWebText	$45$	$1.0$	$0.044$	$0.043$	$0.042$	$\mathbf{0.045}$	$\mathbf{0.043}$
OpenWebText	$45$	$1.5$	$0.067$	$0.062$	$0.063$	$\mathbf{0.068}$	$\mathbf{0.062}$
TinyStories	$23$	$1.0$	$0.087$	$0.080$	$0.080$	$\mathbf{0.085}$	$\mathbf{0.081}$

Appendix

Table 2: Theoretical Exponents derived from the Chosen

d

and

\beta

Panel of Figure 2 — Figure 2: Intrinsic Dimension (ID) Evolution Across Model Depth. The plots show the Maximum Likelihood Estimation (MLE) of the Intrinsic Dimension, $d$ , calculated for the representations at each layer of the four GPT-2 models (varying size). The ID remains stable across different model sizes for a given dataset but varies significantly across data corpora. (a) OpenWebText (OWT): Shows the highest intrinsic dimension, reflecting its diverse and complex nature. (b) TinyStories: Exhibits the lowest intrinsic dimension, consistent with its synthetic, simple structure. (c) WikiText-103 (WT-103): Has an intermediate intrinsic dimension.

Panel of Figure 3 — Figure 3: Intrinsic Dimension Scaling Law: $\bar{d}$ versus Number of Parameters ( $N$ ). The plots show the relationship between the Average Intrinsic Dimension ( $\bar{d}$ ) (log-log scale) and the Number of Parameters ( $N$ ) across the four GPT-2 model sizes for each dataset. This relationship is modeled as a power law, $\bar{d}\sim N^{\beta}$ , where the slope of the linear fit determines the ID Scaling Exponent $\beta$ . The results suggest that $\bar{d}$ exhibits weak or negligible scaling with model size across all three corpora.

$L_{T}$	$d_{ff}$	$N_{\mathrm{act}}$	$d_{\mathrm{model}}$	$n_{\mathrm{heads}}$	$M$	$k$
2	256	3,148,800	512	4	8	2
2	384	3,673,600	512	4	8	2
2	512	4,198,400	512	4	8	2
2	768	5,248,000	512	4	8	2
2	1024	6,297,600	512	4	8	2
4	256	6,297,600	512	4	8	2

Appendix

Table 3: Model configurations used in the scaling experiments for each data set . Each row corresponds to a distinct Transformer-MoE model used in the model-scaling sweep. The effective active parameter count

N_{\mathrm{act}}

varies with the number of layers and FFN width, while

d_{\mathrm{model}}=512

n_{\mathrm{heads}}=4

M=8

k=2

, and sequence length

\ell=256

are held fixed.

$L_{T}$	$d_{\mathrm{ff}}$	$N_{\mathrm{act}}$	$d_{\mathrm{model}}$	$n_{\mathrm{heads}}$	$M$	$k$	$D$
4	512	8,396,800	512	4	8	2	$5\times 10^{4}$
4	512	8,396,800	512	4	8	2	$1\times 10^{5}$
4	512	8,396,800	512	4	8	2	$2\times 10^{5}$
4	512	8,396,800	512	4	8	2	$4\times 10^{5}$
4	512	8,396,800	512	4	8	2	$8\times 10^{5}$

Appendix

Table 4: Model configuration used in the data-scaling experiments. The architecture is held fixed while the training token budget

D

is varied. Unless stated otherwise, the configuration corresponds to a representative mid-scale Transformer–MoE model with fixed active parameter budget

N_{\mathrm{act}}

Panel of Figure 4 — Figure 4: Data-scaling results (fixed $d$ , varying $\beta$ ). Empirical estimates of the data-scaling law (loss versus data size $D$ ) across datasets, compared with theoretical lines obtained by fixing the dataset-specific intrinsic dimension $d$ and varying $\beta$ . The empirical fits are most consistent with $\beta=1.5$ for OpenWebText and $\beta=1.0$ for TinyStories and WikiText-103.

Panel of Figure 5 — Figure 5: Model-scaling results (fixed $d$ , varying $\beta$ ). Empirical estimates of the model-scaling law (loss versus active parameter count $N_{\mathrm{act}}$ ) across datasets. For each plot, the dataset-specific intrinsic dimension $d$ is fixed and theoretical lines are shown for varying $\beta$ , with theoretical exponent $\alpha_{N}=2\beta/d$ . The empirical fits are most consistent with $\beta=0.5$ across all three datasets.

$E$	$\mu(E)$	$\nu(E)$	$\alpha_{N}$	$\alpha_{D}$	$\alpha_{C}$	$\alpha_{D}^{\text{pred}}$	$\alpha_{C}^{\text{pred}}$	$r_{D}$	$r_{C}$	$\mathrm{rel\_err}_{D}$	$\mathrm{rel\_err}_{C}$
1	-0.1817	-0.1965	0.1817	0.1965	0.0944	0.1537	0.0832	0.0427	0.0111	0.2779	0.1335
2	-0.1780	-0.2065	0.1780	0.2065	0.0955	0.1511	0.0817	0.0553	0.0138	0.3666	0.1697
4	-0.1731	-0.2192	0.1731	0.2192	0.0967	0.1475	0.0796	0.0716	0.0170	0.4855	0.2142
8	-0.1676	-0.2338	0.1676	0.2338	0.0976	0.1435	0.0773	0.0902	0.0203	0.6287	0.2625
16	-0.1617	-0.2494	0.1617	0.2494	0.0980	0.1391	0.0748	0.1102	0.0232	0.7917	0.3114
32	-0.1557	-0.2652	0.1557	0.2652	0.0981	0.1347	0.0722	0.1304	0.0258	0.9684	0.3582

Appendix

Table 5: Per-

E

identity checks.We set

\alpha_{N}(E)=-\mu(E)

and

\alpha_{D}(E)=-\nu(E)

. The compute-optimal exponent is

\alpha_{C}(E)=\frac{\alpha_{N}(E)\alpha_{D}(E)}{\alpha_{N}(E)+\alpha_{D}(E)}

. Predicted exponents use

\alpha_{D}^{\text{pred}}=\alpha_{N}/(1+\alpha_{N})

and

\alpha_{C}^{\text{pred}}=\alpha_{N}/(2+\alpha_{N})

. Residuals are

r_{D}=\alpha_{D}-\alpha_{D}^{\text{pred}}

and

r_{C}=\alpha_{C}-\alpha_{C}^{\text{pred}}

; relative errors are

r/\text{pred}

Panel of Figure 6 — Figure 6: Empirical amplification patterns across experts.

Panel of Figure 7 — Figure 7: Cross-budget compute-optimal allocations. (a) Cross-budget slopes for each $E$ . (b) Tokens-to-parameters drift across experts (per budget).

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 454 papers

Sep 30, 2026

Scaling Laws for Looped Mixture of Experts

Looped transformers and Mixture-of-Experts (MoE) offer complementary routes to efficient scaling: recurrence increases computational depth at fixed parameters, while MoE sparsity expands total capacity at fixed active compute. Yet existing scaling laws model recurrence or sparsity in isolation. In this work, we introduce Loop Scaling Laws, the first scaling law to jointly model recurrence and sparsity alongside model size and data. At its core is a bounded, sparsity-conditional recurrence mapping that characterizes the effective-parameter gain from looping and how sparsity raises this gain. The laws predict the held-out loss of looped models more accurately than prior alternatives, and recover the standard dense and MoE scaling laws as special cases. Beyond prediction, the fitted laws provide a principled foundation for designing looped MoE models under compute and memory constraints. Downstream evaluations further demonstrate the complementary benefits of the two axes: sparsity delivers ~3x active-parameter efficiency, recurrence yields ~2x total-parameter efficiency on reasoning, and joint scaling further advances the performance frontier. As a practical extension, we show these gains hold at trillion-token scale: at matched training compute, a looped MoE with law-derived recurrence matches a ~2x larger non-looped MoE on the reasoning benchmarks, while enabling test-time scaling through recurrence.

Yanbei Chen, Anirudh Goyal, Raghuraman Krishnamoorthi

Meta AI

Mixture-of-Experts Language Models Language Model Scaling Laws Inference-Time Scaling Recurrent Transformers AI Scaling Laws

Ask PDF

Oct 5, 2026

Distributionally Robust Mixture-of-Experts Training

Mixture-of-Experts (MoE) transformers scale capacity by activating only a few experts per token, but this sparsity creates a hidden reliability problem: when routing is imperfect, load-balanced models may send tokens to experts that are insufficiently trained for the assigned inputs. We propose Distributionally Robust MoE Training (DRMoET), a drop-in objective that treats layer-wise experts as endogenous robustness groups and optimizes high-loss routing outcomes rather than merely equalizing traffic. DRMoET updates a per-layer expert distribution by an entropy-regularized softmax rule on EMA-smoothed, activation-weighted expert losses, strengthening plausible non-top routing paths while preserving standard MoE computation. Under the FLAME-MoE recipe at 746M-total and 10.3B-total scales, DRMoET improves downstream averages over both standard FLAME-MoE and auxiliary-loss-free balancing. At 10.3B total parameters and 67B training tokens, DRMoET improves the seven-task average from 0.6625 to 0.6767, while the auxiliary-loss-free baseline achieves 0.6431. Mechanistic analyses show lower expert-loss variance with nearly unchanged mean loss, 4.3% lower excess loss under forced mid-

k

misrouting, and improved domain-expert specialization. These results position routing robustness-not only utilization balance-as a practical objective for reliable sparse MoE scaling. Project page and code are available at: https://drmoet.github.io/.

Xin Teng, Muxiao Li, Hongyi Wen

New York University Center for Data Science, NYU Shanghai

Mixture-of-Experts Language Models Sparse Mixture-of-Experts Distributionally Robust Optimization Mixture-of-Experts Models

Ask PDF

Sep 3, 2026

Towards a Statistical Understanding of Mixture-of-Experts

Mixture-of-experts (MoE) architectures increase model capacity by combining a collection of expert predictors through input-dependent routing, while often activating only a small subset of experts for each input. Despite their growing importance in modern large-scale models, the statistical roles of their design choices, especially routing, sparse activation, and shared experts, remain only partially understood, as existing theory has largely focused on parametric or correctly specified MoE models. In this paper, we view MoE as a form of localized aggregation and show how this localization reshapes the approximation-estimation-computation tradeoff. We derive oracle risk bounds for learning dense and sparse routing with evolving experts, separating approximation, expert-learning, and router-estimation errors, and characterize how sparse Top-K routing can retain the benefits of localized aggregation while controlling per-input computation. We also interpret gating through the geometry of input space, relating routing performance to regions of local expert advantage, and show how shared experts, as adopted in architectures such as DeepSeekMoE, can extract common predictive structure so that routed experts focus on residual local variation. Together, these results provide a unified statistical framework for understanding MoE through input-dependent expert aggregation, in which expert specialization and computational tradeoffs are governed by local predictive structure.

Siyuan He, Bokai Yang, Jie Hu +2

Qiuzhen College, Tsinghua University, Beijing, China. · KLATASDS-MOE, School of Statistics, East China Normal University, Shanghai, China. · Yau Mathematical Sciences Center, Tsinghua University, Beijing, China. +1

Mixture of Experts Sparse Mixture-of-Experts Mixture-of-Experts Models Statistical Learning Theory

Ask PDF

Abstract

Mixture-of-Experts Models

Explore similar work

Scaling Laws for Looped Mixture of Experts

Distributionally Robust Mixture-of-Experts Training

Towards a Statistical Understanding of Mixture-of-Experts