Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization

Weilin Wan; Jingtao Han; Debing Zhang; Weizhong Zhang; Cheng Jin

cs.LGMar 23, 2026

Holistic Scaling Laws for Optimal Mixture-of-Experts Architecture Optimization

Authors: Weilin Wan, Jingtao Han, Debing Zhang, Weizhong Zhang, Cheng Jin

Organizations: College of Computer Science and Artificial Intelligence, Fudan University · Hi Lab, Xiaohongshu Inc. · Project Numina · School of Data Science, Fudan University

Abstract

Scaling laws govern macroscopic resource allocation for LLMs, yet precise architectural configurations for Mixture-of-Experts (MoE) models remain guided by heuristics. Existing MoE scaling studies either incorporate MoE-specific variables into scaling formulas, causing fitted coefficients to grow rapidly without proportional experimental support, or fix all non-MoE factors, implicitly assuming global architecture does not influence local MoE scaling. We propose a reusable framework for holistic MoE architectural optimization. We first show that MoE architectures decouple FLOPs per token ( $M$ ), active parameters ( $N_a$ ), and total parameters ( $N$ ), which serve as distinct proxies for computation, weight access, and parameter storage. We therefore establish a joint constraint triad of $M$ , active parameters ( $N_a$ ), and total parameters ( $N$ ) for rigorous MoE characterization. To tame the resulting $O(n^{16})$ search space, we employ mathematical decoupling: structural constraints and a rank-preserving property of the hidden dimension factorize the optimization into an $O(n^3) + O(n^2)$ two-phase search. Through extensive validation across 670+ MoE models spanning $10^{18}$ to $3 \times 10^{20}$ FLOPs, we derive globally applicable scaling laws mapping any compute budget to its optimal architecture. A key finding is that the near-optimal configuration band widens with scale, providing a quantitative basis for trade-offs between scaling law recommendations and engineering constraints. Our work delivers actionable blueprints for optimal MoE design under arbitrary compute budgets.

Figures & tables

Overall Resource Metrics
Symbol	Description
$M$	Total floating-point operations (FLOPs) per token during the forward pass
$N_{a}$	Active parameter count per token, excluding embeddings
$N$	Total parameter count of the model, excluding embeddings
$C$	Total compute budget allocated for model training
Structural Parameters

Table 1: Summary of core mathematical notations and architectural variables.

Figure 1: Overview of the dimension reduction pipeline. Starting from the full 16-dimensional MoE architectural space, we systematically reduce the search complexity to two sequential phases of

\mathcal{O}(n^{3})

and

\mathcal{O}(n^{2})

through classical scaling laws, algebraic constraints, engineering fixations, and the rank-preserving property of

d

Panel of Figure 2 — Figure 2: Results visualized across different parameter spaces. (a) depicts the loss landscape in $(N_{a},N)$ for a specific compute budget ( $C=10^{20}$ , $M=3.2681$ GFLOPs); additional visualizations for different compute scales are available in Appendix C . (b) transforms this into the $(M/N_{a},N/N_{a})$ space, highlighting a black profile curve within the optimal loss region. (c) presents normalized loss profiles along $N/N_{a}$ for various compute scales, demonstrating the consistent flatness of the loss landscape near its optimal value.

Panel of Figure 3 — Figure 3: Analysis of $M/N_{a}$ optimization. The shaded grey regions in both subfigures represent "near-optimal bands," indicating parameter combinations where the loss is within 0.1% of the minimum for that scale, thereby implying robustness to slight deviations from the exact optimum. Specifically, (a) illustrates the loss data points and fitted curves across various compute scales, showing the relationship between loss and $M/N_{a}$ . (b) depicts the derived scaling laws for the optimal $M/N_{a}$ as a function of the compute budget $C$ , demonstrating how the preferred $M/N_{a}$ changes with increasing computational resources while remaining robust within these near-optimal bounds.

Panel of Figure 4 — Figure 4: Analysis of $d$ optimization. The shaded grey regions in both subfigures represent "near-optimal bands," indicating parameter combinations where the loss is within 0.1% of the minimum for that scale, thereby implying robustness to slight deviations from the exact optimum. Specifically, (a) illustrates the loss data points and fitted curves across various compute scales, showing the relationship between loss and $d$ . (b) depicts the derived scaling laws for the optimal $d$ as a function of the compute budget $C$ , demonstrating how the preferred $d$ changes with increasing computational resources while remaining robust within these near-optimal bounds.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Visualization of the mathematical feasible region for the model design space on the $(N_{a},N)$ plane with $M=5.9390$ GFLOPs. (a) shows the minimum available hidden dimension $d$ for each $(N_{a},N)$ point, (b) shows the maximum hidden dimension $d$ , and (c) illustrates the range width of feasible $d$ values.

Figure 6: Relationship between the median proxy (median feasible

d

proxy) and the true minimum loss under a fixed budget. The figure displays a scatter plot of the median proxy against the true minimum loss, along with a least-squares linear fit.

Metric	Pearson $r$	Spearman $\rho$	Kendall $\tau$	$R^{2}$
Value	0.9793	0.9758	0.9111	0.9590
$p$ -value	$7.89\times 10^{-7}$	$1.47\times 10^{-6}$	$2.98\times 10^{-5}$	–

Appendix

Table 2: Statistical summary of the relationship between the median proxy and true minimum loss.

$C$ (FLOPs)	$10^{18}$	$3\times 10^{18}$	$10^{19}$	$3\times 10^{19}$	$10^{20}$	$3\times 10^{20}$
Pearson $r$	0.9603	0.9908	0.9533	0.9845	0.9928	0.9883

Appendix

Table 3: Cross-scale median-proxy validation from the additional analysis.

Panel of Figure 7 — Figure 7: Additional results visualized in $(N_{a},N)$ space for different compute scales.

Panel of Figure 8 — Figure 8: Additional results visualized in $(M/N_{a},N/N_{a})$ space for different compute scales.

Panel of Figure 9 — Figure 9: Scaling curves and extrapolation for different MoE FFN width ratios. Solid lines represent fits within the experimental range ( $10^{18}$ to $3\times 10^{20}$ FLOPs); dashed lines show extrapolation to $10^{22}$ FLOPs. The vertical dotted line marks the boundary of the experimental range.

Panel of Figure 10 — Figure 10: Predicted loss vs. MoE FFN width ratio at each experimental compute scale, comparing E=0 (gray dashed) and E $\neq$ 0 (red solid) model predictions. The loss variation across different $(K+1)\cdot d_{m}/d$ values is consistently below $0.5\%$ .

Panel of Figure 11 — Figure 11: Predicted loss vs. MoE FFN width ratio at extrapolated compute scales, comparing E=0 (gray dashed) and E $\neq$ 0 (red solid) model predictions. $C=3\times 10^{20}$ (panel a) overlaps with the experimental range to bridge the two regimes.

$M$ (GFLOPs)	Attention share	$N$ (B)	Loss
3.02	1.9%	5.28	2.125
3.03	0.5%	9.90	2.102
3.01	0.2%	10.02	2.226

Appendix

Table 4: Configurations with nearly identical per-token compute at the

C=10^{20}

experimental scale. Similar

M

does not imply similar parameter storage.

$C$ (EFLOPs)	$M$ (GFLOPs)	$D$ (Billion Tokens)
1	$0.2672$	$3.7420$
3	$0.4856$	$6.1775$
10	$0.9345$	$10.7004$
30	$1.6983$	$17.6649$
100	$3.2681$	$30.5985$
300	$5.9390$	$50.5138$

Appendix

Table 5: Six compute-optimal

(C,M,D)

combinations

Overall Resource Metrics
Parameter	Different Compute Scales
$C$	$1e18$	$3e18$	$1e19$	$3e19$	$1e20$	$3e20$
$M$ (GFLOPs)	0.2672	0.4856	0.9345	1.6983	3.2681	5.9390
Structural Parameters
$d_{qkv}$	64	64	64	128	128	128
$N_{q}$	4	8	8	8	8	16

Appendix

Table 6: Common hyperparameters and architectural configurations across different compute scales. Note: The actual

M

values in our experiments might slightly deviate from theoretically derived optimal values, as model dimensions were constrained to be multiples of

2^{3}

2^{5}

for hardware efficiency. We ensured this deviation remained within 5% to maintain experimental reliability.

Panel of Figure 12 — Figure 12: Scaling laws for learning rate and batch size. This figure illustrates the scaling behavior of optimal hyperparameters with respect to computational budget (FLOPs). Both plots share a common x-axis, $\log_{10}(\text{FLOPs})$ . The blue dots represent experimental optimal settings, while the dashed blue and red lines define the lower and upper bounds of the robust linear fitted band (shaded gray area), respectively. (a) Scaling of learning rate: Y-axis shows $\log_{10}(\text{learning rate})$ . (b) Scaling of batch size: Y-axis shows $\log_{2}(\text{Batch Token Size})$ . These derived scaling laws inform the hyperparameter choices in our experiments.

Table 19

Table 20

Table 21

Table 22

Table 23

Table 24

Table 25

Table 26

$C$	$d$	$L_{d}$	$d_{d}$	$L_{m}$	$d_{m}$	$L$
$3e20$	1248	8	3744	8	1832	16
	1312	7	3936	9	1520	16
	1376	6	4128	10	1304	16
	1440	5	4320	11	1152	16
	1472	5	4416	11	1088	16
	1536	4	4608	12	984	16

Appendix

Table 23: Specific architectural parameters (

d,L_{d},d_{d},L_{m},d_{m},L

) in Sec 6 .

$C$ (FLOPs)	$M^{\mathrm{opt}}$ (GFLOPs)	$N_{a}^{\mathrm{opt}}$ (B)	$N$ (B)	FP16 storage (GB)
$10^{18}$	0.2672	0.025	0.49	1.0
$3\times 10^{18}$	0.4856	0.049	0.95	1.9
$10^{19}$	0.9345	0.100	2.08	4.2
$3\times 10^{19}$	1.6983	0.192	3.99	8.0
$10^{20}$	3.2681	0.388	8.25	16.5
$3\times 10^{20}$	5.9390	0.732	16.16	32.3

Appendix

Table 24: Representative pipeline outputs and non-embedding parameter-storage estimates. The flat loss landscape permits a broad near-optimal region; these fitted outputs are not the realized training configurations.

Mixture-of-Experts Models

Mixture-of-experts models use input-dependent routing to activate specialized subnetworks, increasing model capacity without running every parameter for each input.

See all 454 papers

May 12, 2026

Slicing and Dicing: Configuring Optimal Mixtures of Experts

Mixture-of-Experts (MoE) architectures have become standard in large language models, yet many of their core design choices - expert count, granularity, shared experts, load balancing, token dropping - have only been studied one or two at a time over narrow configuration ranges. It remains an open question whether these choices can be optimized independently, without considering interactions. We present the first systematic study of over 2,000 pretraining runs spanning models up to 6.6B total parameters, in which we exhaustively vary total experts, expert dimension, heterogeneous expert sizing within a single layer, shared expert size and load-balancing mechanisms. We find that at every active-parameter scale that we study, performance consistently improves with total MoE parameters even at extreme active expert parameter ratios like 128.Further, the optimal expert size is nearly invariant to total parameter count and depends only on active parameter count. Third, we see that other choices like shared experts, heterogeneous experts and load-balancing settings have small effects relative to expert count and granularity, although dropless routing yields a consistent gain. Overall, our results suggest a simpler recipe: focus on expert count and granularity, other choices have minimal effect on final quality.

Margaret Li, Sneha Kudugunta, Danielle Rothermel +1

Paul G Allen School of Computer Science University of Washington Seattle, USA · Courant School of Data Science New York University New York, USA

Mixture-of-Experts Language Models Mixture of Experts Sparse Mixture-of-Experts AI Scaling Laws

Ask PDF

Mar 11, 2026

Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design

This paper presents a novel extension of neural scaling laws to Mixture-of-Experts (MoE) models, focusing on the optimal allocation of compute between expert and attention sub-layers. As MoE architectures have emerged as an efficient method for scaling model capacity without proportionally increasing computation, determining the optimal expert-attention compute ratio becomes critical. We define the ratio

r

as the fraction of total FLOPs per token dedicated to the expert layers versus the attention layers, and explore how this ratio interacts with the overall compute budget and model sparsity. Through extensive experiments with GPT-style MoE Transformers, we empirically find that the optimal ratio

r^*

follows a power-law relationship with total compute and varies with sparsity. Our analysis leads to an explicit formula for

r^*

, enabling precise control over the expert-attention compute allocation. We generalize the Chinchilla scaling law by incorporating this architectural parameter, providing a new framework for tuning MoE models beyond size and data. Our findings offer practical guidelines for designing efficient MoE models, optimizing performance while respecting fixed compute budgets.

Junzhuo Li, Peijie Jiang, Changxin Tian +3

The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology · Ant Group

Mixture-of-Experts Models AI Scaling Laws Neural Scaling Laws

Ask PDF

Aug 11, 2026

Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts

In large-scale pretraining, the algorithm, architecture, and systems decisions are conventionally made in disconnected stages. A scaling law stage selects an architecture and training recipe, optimizing loss under compute constraints, and a separate systems stage then optimizes the implementation for hardware efficiency. In this work, we develop MOSAIC, which formulates model architecture and systems co-design as an optimization problem. MOSAIC couples a predictive scaling law with a calibrated performance model that estimates Model FLOPs Utilization (MFU), communication cost, memory footprint, and the best parallel layout. We instantiate the framework for sparse Mixture-of-Experts (MoE) language models, where expert count, routing sparsity, and other MoE layer dimensions affect both the loss and systems efficiency. We fit a scaling law on sparse MoE models trained on text data, whose scaling dimensions include the sparsity factor, which is the fraction of model parameters inactive per token in a forward pass. The scaling law sweeps in our work span active parameters from

104

million to

2.7

billion and total model sizes reaching

79

billion parameters. We show that, within the calibrated sparsity range, an efficiency-agnostic model-FLOPs budget admits no interior optimal sparsity. The fitted loss decreases monotonically with sparser models and the compute optimum lies at the upper boundary of the data support. An optimal sparsity in MoE models instead emerges under the cluster's systems constraints, as captured by MOSAIC. Our results argue for a shift towards unified architecture and systems co-design for frontier language model training.

Soumajyoti Sarkar, Yuxin Tang, Sheng Zha

Amazon AGI Foundations

Mixture-of-Experts Language Models Language Model Scaling Laws Efficient Language Model Training Hardware-Software Co-Design

Ask PDF

Abstract

Mixture-of-Experts Models

Explore similar work

Slicing and Dicing: Configuring Optimal Mixtures of Experts

Optimal Expert-Attention Allocation in Mixture-of-Experts: A Scalable Law for Dynamic Model Design

Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts