Sparsely-activated Mixture-of-Experts (MoE) Transformers universally fix the same number of routed experts across all layers, a convention that ignores the well-documented heterogeneity in layer-wise redundancy. We demonstrate that this uniformity is systematically suboptimal and propose MAPLE, a plug-and-play framework that reallocates the routed-expert budget heterogeneously across layers of any pretrained MoE LLM, without modifying weights or requiring retraining. Our core contribution is a closed-form sensitivity-guided allocation: we probe each layer's response to variation in expert count, quantify sensitivity using three measures, and derive an analytically optimal budget assignment that directs capacity towards sensitive layers and absorbs reductions in redundant layers. This closed-form solution is further refined by a sensitivity-constrained genetic search that uses layer-wise sensitivity as a prior to guide exploration, yielding faster convergence and superior allocation quality. On four MoE models spanning different scales and architectures, MAPLE outperforms uniform and pruning-based baselines under a 75% routed-expert budget. Notably, on DeepSeek-MoE-16B, MAPLE uses only 75% of the experts yet surpasses the original 100% expert-uniform baseline on ARC-E, ARC-C, and BoolQ, improving accuracy from 65.09 to 71.40, 48.49 to 51.50, and 80.03 to 82.38, respectively. These accuracy gains translate into measured deployment efficiency: implementing MAPLE in SGLang reduces single-GPU end-to-end serving latency by 32.2% and improves throughput by 47.4%. These results show that well-designed heterogeneous allocation can be more effective than simply activating more experts, establishing it as a principled and practical axis for improving MoE efficiency.
Figures & tables
Figure 1: Main idea and key result of MAPLE . (a) Overview of MAPLE : uniform expert allocation is replaced by sensitivity-guided heterogeneous allocation under the same budget. (b) Less compute, better performance: with only 75% routed experts, MAPLE surpasses the full-expert uniform baseline on ARC-Easy (ARC-E), ARC-Challenge (ARC-C), and BoolQ reasoning benchmarks.
DeepSeek-MoE-16B
DeepSeek-V2-Lite
Method
ARC-C
ARC-E
BoolQ
PIQA
RTE
Avg.
ARC-C
ARC-E
BoolQ
PIQA
RTE
Avg.
Baseline
48.49
65.09
80.03
80.09
84.84
71.71
44.82
64.74
67.40
79.82
53.43
62.04
SparseGPT
46.15
62.11
79.69
79.33
81.22
69.70
44.48
64.91
67.34
78.98
57.76
62.69
Wanda
45.81
62.98
79.72
79.27
82.67
70.09
44.15
64.91
67.31
78.67
54.51
61.91
RIA
46.48
63.16
79.72
79.49
80.87
69.94
45.15
64.04
65.08
78.99
49.82
60.62
EPP
47.49
66.14
79.60
77.86
83.03
70.82
42.47
62.46
67.74
78.40
49.82
60.18
Table 1: Main results under the 75% routed-expert budget on four MoE architectures and five reasoning benchmarks. Baseline denotes the original full-budget uniform routed-expert configuration. Bold indicates the best result in each model–task block.
GSM8K
BBH
Configuration
Acc.
Total time
Acc.
Total time
Full uniform (100%)
58.53
492.26 s
43.43
5,776.97 s
MAPLE (75%)
57.92
231.48 s
44.33
5,434.15 s
Δ
−0.61
−53.0%
+0.90
−6.3%
Table 2: GSM8K and BBH accuracy and total evaluation wall-clock time at a 75% budget, DeepSeek-MoE-16B, directly calibrated on each benchmark’s chain-of-thought accuracy. GSM8K uses the complete 1,319-example test set (0-shot); BBH uses the complete 5,911-example set (3-shot CoT).
Table 3: MAPLE ’s efficiency profile. (a) One-time offline calibration cost per model, summed across five tasks, on a single RTX A6000. (b) Measured SGLang 0.4.6 serving time on DeepSeek-MoE-16B at a 75% budget, averaged over five runs, ARC-Easy workload. (c) Measured vLLM serving throughput and latency on DeepSeek-MoE-16B at a 75% budget, ARC-Easy workload. PP: pipeline parallelism; EP: expert parallelism.
Acc.
Weight
Output
Best Gen.
Best Acc.
Δ
1
–
84.84
–
2
✓
2
85.55
+0.71
3
✓
✓
4
85.20
+0.36
4
✓
✓
6
84.12
-0.72
5
✓
✓
10
83.75
-1.08
6
✓
✓
✓
9
85.55
+0.71
Table 4: Ablation of sensitivity metrics (Acc.: accuracy range, Weight: weight variance, Output: output deviation). Best Gen. is the generation at which the best accuracy is reached.
Figure 2: Performance of different sensitivity metrics in MAPLE under the 75% expert budget across five tasks. Bold bar outlines indicate the best result within each task.
Task
Closed-form
+Refinement
Δ
Seed Std.
ARC-C
48.49
51.50
+3.01
±1.16
ARC-E
64.39
71.40
+7.01
±0.90
BoolQ
80.70
82.39
+1.68
±0.45
PIQA
80.03
80.85
+0.82
±0.21
RTE
83.39
85.55
+2.16
±0.75
Table 5: Effect of refinement and seed stability. Closed-form denotes the initial allocation, +Refinement applies the proposed refinement step, and Seed Std. reports the standard deviation of +Refinement accuracy across five independent refinement seeds.
Figure 3: (a) Task-wise standardised sensitivity distributions under the 75% expert budget. (b) Normalised convergence trends across tasks, where Remaining Gap denotes the performance difference from the best-found solution.
Calibration signal
ARC-C
ARC-E
BoolQ
PIQA
RTE
Avg.
Reusable (joint 5-task)
49.83
68.95
82.05
78.62
83.39
72.57
Cross-task transfer (avg. of 20 pairs)
43.65
65.09
78.83
78.81
79.33
69.14
C4-NLL
44.15
65.26
78.99
79.43
80.14
69.59
FineWeb-Edu-NLL
44.48
62.11
76.45
79.76
81.59
68.88
ARC-C (source)
51.50
64.56
79.24
78.40
79.06
70.55
ARC-E (source)
44.82
71.40
78.75
78.67
76.90
70.11
Table 6: Robustness of MAPLE ’s calibration signal on DeepSeek-MoE-16B at a 75% budget. Reusable, Cross-task transfer, C4-NLL, and FineWeb-Edu-NLL are alternative calibration signals; C4-NLL/FineWeb-Edu-NLL replace task accuracy with generic-corpus perplexity. The remaining rows give the full cross-task transfer matrix underlying the Cross-task transfer average above.
Budget
Wanda
SparseGPT
MAPLE
50%
41.14 / 72.92
39.46 / 72.20
46.49 / 79.78
75%
45.81 / 82.67
46.15 / 81.22
51.50 / 85.55
100% (uniform baseline: 48.49 / 84.84)
–
–
51.84 / 88.09
Table 7: ARC-C / RTE accuracy under 50%, 75%, and 100% expert budgets on DeepSeek-MoE-16B. Wanda and SparseGPT are omitted at 100% since their unpruned configuration is identical to the uniform baseline.
Mixture-of-Experts (MoE) has become a dominant architecture for scaling large language models due to their sparse activation mechanism. However, the substantial number of expert activations creates a critical latency bottleneck during inference, especially in resource-constrained deployment scenarios. Existing approaches that reduce expert activations potentially lead to severe model performance degradation. In this work, we introduce the concept of activation budget as a constraint on the number of expert activations and propose Alloc-MoE, a unified framework that optimizes budget allocation coordinately at both the layer and token levels to minimize performance degradation. At the layer level, we introduce Alloc-L, which leverages sensitivity profiling and dynamic programming to determine the optimal allocation of expert activations across layers. At the token level, we propose Alloc-T, which dynamically redistributes activations based on routing scores, optimizing budget allocation without increasing latency. Extensive experiments across multiple MoE models demonstrate that Alloc-MoE maintains model performance under a constrained activation budget. Especially, Alloc-MoE achieves 1.15× prefill and 1.34× decode speedups on DeepSeek-V2-Lite at half of the original budget.
Baihui Liu, Kaiyuan Tian, Wei Wang +3
National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology
Standard LoRA fine-tuning of Mixture-of-Experts (MoE) models applies adapters to every expert, yet our profiling shows that per-layer expert routing is highly skewed: a small subset of experts handles most tokens in each layer, while many others are rarely activated ("cold"). We propose MoE-Sieve, a simple routing-guided framework for LoRA fine-tuning, and pair it with a systematic profiling study of expert routing across architectures and tasks. The method is simple: profile routing counts on a small calibration set, select the top-k most-routed experts per layer, and apply LoRA only to those experts. Across two architecturally distinct MoE models and three diverse tasks, tuning only the top 25% routed experts per layer remains competitive with full LoRA, with mean differences within +/-1 percentage point across all conditions. This reduces LoRA trainable parameters by 70-73%, adapter checkpoint size by 71-73%, and wall-clock training time by up to 50%. We also observe a non-monotonic relationship between expert count and seed-to-seed variance, consistent with the hypothesis that adapting cold experts can introduce gradient noise without improving accuracy. Further ablations show that random expert selection at matched budget is about 2.5 percentage points worse, indicating that the routing signal matters, while greedy per-layer budget optimization does not improve over uniform top-k.
Most Mixture-of-Experts (MoE) transformers preassign each expert to one layer for the whole of training. We study whether a brief global-access phase can learn a compatible expert-to-layer allocation that is then executed privately. UniPool-Lock gives every layer its own router over one global pool, balances aggregate pool usage, and scores experts with a scale-stable NormRouter. After the short ownership-learning phase, it assigns each expert to one layer, locks the disjoint allocation, and continues training as a layer-private MoE. With the same expert-FFN budget and routed expert FLOPs, this recipe lowers held-out loss relative to vanilla MoE by 0.024-0.037 across five dense-equivalent scales from 182M to 1.5B, including -0.0247 at 1.5B after 60B tokens. The ownership-learning phase lasts 2K steps, about 3.3% of training, and post-lock step time is within -0.7% to +2.8% of vanilla MoE in our throughput measurements. Controls locate the gain in full-pool training and in the allocation it produces: a random disjoint allocation fixed at initialization, trained with the same router and losses, matches vanilla, whereas locking the allocation learned during the full-pool phase recovers nearly all of the persistent full-pool gain, and substituting a random allocation at the lock forfeits about half of it. Keeping full-pool access throughout training (UniPool-Full) also improves over vanilla MoE at four scales and outperforms it with only 66.7% (182M) to 50% (469M and 650M) of its expert parameters.
Minbin Huang, Han Shi, Chuanyang Zheng +5
The Chinese University of Hong Kong · Huawei Technologies · The University of Hong Kong