Mixture-of-Experts (MoE) has emerged as a promising paradigm for efficiently scaling large language models without a proportional increase in computational cost. However, the standard training strategy of Top-K router prevents MoE models from realizing their full potential for elastic inference. When the number of activated experts is altered at inference time, these models exhibit precipitous performance degradation. In this work, we introduce Matryoshka MoE (M-MoE), a training framework that instills a coarse-to-fine structure directly into the expert ensemble. By systematically varying the number of activated experts during training, M-MoE compels the model to learn a meaningful ranking: top-ranked experts collaborate to provide essential, coarse-grained capabilities, while subsequent experts add progressively finer-grained detail. We explore this principle at multiple granularities, identifying a layer-wise randomization strategy as the most effective. Our experiments demonstrate that a single M-MoE model achieves remarkable elasticity, with its performance at various expert counts closely matching that of an entire suite of specialist models, but at only a fraction of the total training cost. This flexibility not only unlocks elastic inference but also enables optimizing performance by allocating different computational budgets to different model layers. Our work paves the way for more practical and adaptable deployments of large-scale MoE models.
Figures & tables
Figure 1: MMLU score of DeepSeek-V2-Lite, Qwen3-30B-A3B-Base, and RedNote-Dots.LLM1.Base under varying numbers of activated experts.
Training Method
Inf. k
MMLU
ARC-C
BoolQ
HellaS.
LogiQA
OBQA
WinoG.
Avg
Specialist Baselines (Top-k)
Top-k (k=1)
1*
52.01
52.22
67.43
70.38
29.95
40.00
67.64
54.23
Top-k (k=2)
1
35.54
32.59
65.08
49.09
30.11
32.60
61.17
43.74
2*
52.16
52.30
70.73
71.47
31.49
42.00
69.93
55.73
Top-k (k=4)
1
41.50
40.02
63.46
59.28
28.42
33.60
62.19
46.92
4*
53.43
54.44
69.11
72.41
29.95
43.40
69.93
56.10
Table 1: Comprehensive performance comparison of specialist Top-k models and elastic M-MoE models. For each specialist model, performance at its native expert count is marked with an asterisk (*). Task abbreviations are: ARC-C (ARC Challenge), OBQA (OpenBookQA), WinoG. (Winogrande). For stable analysis, we bold and underline the best and second-best scores for each inference k value in the MMLU and Avg columns.
Training Method
Inf. k
MMLU
ARC-C
BoolQ
HellaS.
LogiQA
OBQA
WinoG.
Avg
Specialist Baselines (Top-k)
Top-k (k=1)
1*
27.91
35.75
57.13
52.41
27.04
35.20
51.30
40.96
Top-k (k=2)
1
24.74
26.19
39.14
26.42
24.12
25.40
50.43
30.92
2*
30.40
35.92
57.34
55.21
28.11
35.80
54.70
42.55
Top-k (k=4)
1
23.50
33.45
48.90
34.21
25.35
29.00
52.17
35.23
4*
30.96
38.31
54.98
56.53
28.42
35.00
54.14
42.62
Table 2: From-scratch pre-training results after 80B tokens. Best scores for each inference setting are in bold .
Activation Pattern
Avg. k
MMLU
ARC-C
BoolQ
HellaS.
LogiQA
OBQA
WinoG.
Avg
Baseline
[2, 2, 2, 2]
2.0
52.71
53.50
71.44
72.43
31.49
42.40
68.82
56.11
Increasing Capacity
[3, 3, 2, 2]
2.5
53.35
53.33
71.01
72.92
32.26
43.00
68.98
56.41
[2, 3, 3, 2]
2.5
53.13
53.84
71.44
72.46
30.88
42.40
67.80
55.99
[2, 2, 3, 3]
2.5
52.90
53.75
71.74
72.25
30.88
41.80
68.98
56.04
Table 3: Performance of the M-MoE-layer model under various layer-wise inference strategies.
Figure 2: Heatmaps illustrating the router’s expert ranking consistency for the Top-k ( k=6 ) model (top) and our M-MoE-Layer model (bottom). A bright color signifies a high correlation, indicating a strong nested, Matryoshka-like ranking structure.
Figure 3: Comparison of MODS for the Top-k and our model. Lower MODS indicates greater expert specialization.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Inf. k
MMLU
ARC-C
BoolQ
HellaS.
LogiQA
OBQA
WinoG.
Avg
1
51.67
51.96
71.07
69.92
31.64
40.20
68.27
54.96 (+0.35)
2
53.01
52.30
71.62
72.15
31.34
41.40
69.14
55.85 (-0.26)
4
53.89
55.83
71.55
73.27
30.57
44.20
68.59
56.84 (+0.15)
6
53.71
55.58
72.84
71.78
30.41
43.60
69.30
56.75 (+0.03)
Appendix
Table 4: Performance of the budget-constrained M-MoE-layer model (Avg. k=4.5). The final column shows the average score and, in parentheses, its difference from the unconstrained M-MoE-layer model in Table 1 .
Inf. k
MMLU
ARC-C
BoolQ
HellaS.
LogiQA
OBQA
WinoG.
Avg
1
52.63
51.62
70.31
70.41
31.34
41.60
67.72
55.09
2
53.89
54.18
71.22
72.85
31.03
43.60
68.51
56.47
4
55.03
54.52
72.20
72.62
31.18
44.80
70.24
57.23
6
54.59
54.78
72.91
72.05
31.34
44.40
70.32
57.20
Appendix
Table 5: Performance of the M-MoE-layer model after 208B tokens of continual pre-training. The final column shows the average score.
Figure 4: MMLU score of the M-MoE-layer model evaluated at different inference expert counts (k=1, 2, 4, 6) throughout continual pre-training. The x-axis represents training steps from the start of M-MoE training.
Mixture-of-Experts language models use input-dependent routing to activate a small subset of expert networks, increasing capacity without proportional compute.
Large Language Models (LLMs) increasingly adopt Mixture-of-Experts (MoE) architectures to scale efficiently under stringent resource constraints. However, MoE's sparse activation causes severe expert load imbalance, where a few experts become stragglers while others remain underutilized, leading to inflated inference latency and cost. Existing solutions assume static, serverful model deployments, limiting expert elasticity and often incurring costly expert swapping or degraded output quality. We present MoEless, an efficient serverless MoE serving framework that mitigates expert load imbalance via elastic expert execution. MoEless leverages lightweight, layer-aware predictors to estimate incoming expert load distributions and proactively identify stragglers. We design optimized scaling and placement strategies to improve function locality, GPU utilization, and cross-expert load balance. MoEless is prototyped on top of Megatron-LM and deployed on an eight-GPU testbed. Experiments with open-source MoE models and real-world workloads show that MoEless reduces inference latency by 43% and inference cost by 84% compared to state-of-the-art solutions.
Large-scale expert parallelism (EP) has become the standard deployment configuration for Mixture-of-Experts (MoE) models in production. Meanwhile, the highly dynamic nature of LLM workloads requires providers to continuously adjust serving capacity to match fluctuating demand. Scaling MoE serving cannot be efficiently achieved by simply adding or removing serving instances; instead, it often requires resizing the EP group within an existing instance, which we refer to as intra-instance scaling.Existing inference engines primarily rely on instance-level scaling, which destroys and reconstructs entire serving instances, incurring substantial scaling overhead. We present FastEEP (Fast Elastic Expert Parallel), a fast intra-instance scaling system for MoE serving based on the principle of Partial Reuse. FastEEP reuses existing runtime state across worker execution, GPU memory, and communication by separating reusable execution structures from the resources activated by the current EP configuration. This separation enables coordinated resource reclamation and reconfiguration while preserving costly-to-reconstruct state, allowing fast EP scaling without full instance teardown and reconstruction. Our evaluation shows that FastEEP achieves 4.5x--11.7x faster scaling than the existing vLLM-based approach across different models, enabling efficient, fine-grained scaling of MoE serving.
Yunfei Feng, Jingkai He, Shiqi Liu +3
Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University
Mixture-of-Experts (MoE) models enable model scaling while maintaining low inference-time compute by activating only a subset of experts per token. However, conventional routing relies on a fixed top-k selection, forcing the model to spend the same compute regardless of how many experts are relevant. We introduce elbow-based routing, a training-free inference-time modification that dynamically adjusts the number of experts on a per-token basis. Our method examines the sorted router probability distribution and identifies an elbow point that separates high- and low-probability experts. We find that most router distributions exhibit clear inflection points suitable for this strategy, and we show both theoretically and empirically that elbow-based routing preserves expert load balance. Experiments on a state-of-the-art MoE model demonstrate an average latency reduction of 5.3% while maintaining accuracy across six benchmarks.
Robin Pan, Raymond Liu, Daniel Fang +2
John A. Paulson School of Engineering and Applied Sciences, Harvard University · Department of Mathematics, Harvard University