Sparsely-activated Mixture-of-Experts (MoE) Transformers universally fix the same number of routed experts across all layers, a convention that ignores the well-documented heterogeneity in layer-wise redundancy. We demonstrate that this uniformity is systematically suboptimal and propose MAPLE, a plug-and-play framework that reallocates the routed-expert budget heterogeneously across layers of any pretrained MoE LLM, without modifying weights or requiring retraining. Our core contribution is a closed-form sensitivity-guided allocation: we probe each layer's response to variation in expert count, quantify sensitivity using three measures, and derive an analytically optimal budget assignment that directs capacity towards sensitive layers and absorbs reductions in redundant layers. This closed-form solution is further refined by a sensitivity-constrained genetic search that uses layer-wise sensitivity as a prior to guide exploration, yielding faster convergence and superior allocation quality. On four MoE models spanning different scales and architectures, MAPLE outperforms uniform and pruning-based baselines under a 75% routed-expert budget. Notably, on DeepSeek-MoE-16B, MAPLE uses only 75% of the experts yet surpasses the original 100% expert-uniform baseline on ARC-E, ARC-C, and BoolQ, improving accuracy from 65.09 to 71.40, 48.49 to 51.50, and 80.03 to 82.38, respectively. These accuracy gains translate into measured deployment efficiency: implementing MAPLE in SGLang reduces single-GPU end-to-end serving latency by 32.2% and improves throughput by 47.4%. These results show that well-designed heterogeneous allocation can be more effective than simply activating more experts, establishing it as a principled and practical axis for improving MoE efficiency.
Figures & tables
Figure 1: Main idea and key result of MAPLE . (a) Overview of MAPLE : uniform expert allocation is replaced by sensitivity-guided heterogeneous allocation under the same budget. (b) Less compute, better performance: with only 75% routed experts, MAPLE surpasses the full-expert uniform baseline on ARC-Easy (ARC-E), ARC-Challenge (ARC-C), and BoolQ reasoning benchmarks.
DeepSeek-MoE-16B
DeepSeek-V2-Lite
Method
ARC-C
ARC-E
BoolQ
PIQA
RTE
Avg.
ARC-C
ARC-E
BoolQ
PIQA
RTE
Avg.
Baseline
48.49
65.09
80.03
80.09
84.84
71.71
44.82
64.74
67.40
79.82
53.43
62.04
SparseGPT
46.15
62.11
79.69
79.33
81.22
69.70
44.48
64.91
67.34
78.98
57.76
62.69
Wanda
45.81
62.98
79.72
79.27
82.67
70.09
44.15
64.91
67.31
78.67
54.51
61.91
RIA
46.48
63.16
79.72
79.49
80.87
69.94
45.15
64.04
65.08
78.99
49.82
60.62
EPP
47.49
66.14
79.60
77.86
83.03
70.82
42.47
62.46
67.74
78.40
49.82
60.18
Table 1: Main results under the 75% routed-expert budget on four MoE architectures and five reasoning benchmarks. Baseline denotes the original full-budget uniform routed-expert configuration. Bold indicates the best result in each model–task block.
GSM8K
BBH
Configuration
Acc.
Total time
Acc.
Total time
Full uniform (100%)
58.53
492.26 s
43.43
5,776.97 s
MAPLE (75%)
57.92
231.48 s
44.33
5,434.15 s
Δ
−0.61
−53.0%
+0.90
−6.3%
Table 2: GSM8K and BBH accuracy and total evaluation wall-clock time at a 75% budget, DeepSeek-MoE-16B, directly calibrated on each benchmark’s chain-of-thought accuracy. GSM8K uses the complete 1,319-example test set (0-shot); BBH uses the complete 5,911-example set (3-shot CoT).
Table 3: MAPLE ’s efficiency profile. (a) One-time offline calibration cost per model, summed across five tasks, on a single RTX A6000. (b) Measured SGLang 0.4.6 serving time on DeepSeek-MoE-16B at a 75% budget, averaged over five runs, ARC-Easy workload. (c) Measured vLLM serving throughput and latency on DeepSeek-MoE-16B at a 75% budget, ARC-Easy workload. PP: pipeline parallelism; EP: expert parallelism.
Acc.
Weight
Output
Best Gen.
Best Acc.
Δ
1
–
84.84
–
2
✓
2
85.55
+0.71
3
✓
✓
4
85.20
+0.36
4
✓
✓
6
84.12
-0.72
5
✓
✓
10
83.75
-1.08
6
✓
✓
✓
9
85.55
+0.71
Table 4: Ablation of sensitivity metrics (Acc.: accuracy range, Weight: weight variance, Output: output deviation). Best Gen. is the generation at which the best accuracy is reached.
Figure 2: Performance of different sensitivity metrics in MAPLE under the 75% expert budget across five tasks. Bold bar outlines indicate the best result within each task.
Task
Closed-form
+Refinement
Δ
Seed Std.
ARC-C
48.49
51.50
+3.01
±1.16
ARC-E
64.39
71.40
+7.01
±0.90
BoolQ
80.70
82.39
+1.68
±0.45
PIQA
80.03
80.85
+0.82
±0.21
RTE
83.39
85.55
+2.16
±0.75
Table 5: Effect of refinement and seed stability. Closed-form denotes the initial allocation, +Refinement applies the proposed refinement step, and Seed Std. reports the standard deviation of +Refinement accuracy across five independent refinement seeds.
Figure 3: (a) Task-wise standardised sensitivity distributions under the 75% expert budget. (b) Normalised convergence trends across tasks, where Remaining Gap denotes the performance difference from the best-found solution.
Calibration signal
ARC-C
ARC-E
BoolQ
PIQA
RTE
Avg.
Reusable (joint 5-task)
49.83
68.95
82.05
78.62
83.39
72.57
Cross-task transfer (avg. of 20 pairs)
43.65
65.09
78.83
78.81
79.33
69.14
C4-NLL
44.15
65.26
78.99
79.43
80.14
69.59
FineWeb-Edu-NLL
44.48
62.11
76.45
79.76
81.59
68.88
ARC-C (source)
51.50
64.56
79.24
78.40
79.06
70.55
ARC-E (source)
44.82
71.40
78.75
78.67
76.90
70.11
Table 6: Robustness of MAPLE ’s calibration signal on DeepSeek-MoE-16B at a 75% budget. Reusable, Cross-task transfer, C4-NLL, and FineWeb-Edu-NLL are alternative calibration signals; C4-NLL/FineWeb-Edu-NLL replace task accuracy with generic-corpus perplexity. The remaining rows give the full cross-task transfer matrix underlying the Cross-task transfer average above.
Budget
Wanda
SparseGPT
MAPLE
50%
41.14 / 72.92
39.46 / 72.20
46.49 / 79.78
75%
45.81 / 82.67
46.15 / 81.22
51.50 / 85.55
100% (uniform baseline: 48.49 / 84.84)
–
–
51.84 / 88.09
Table 7: ARC-C / RTE accuracy under 50%, 75%, and 100% expert budgets on DeepSeek-MoE-16B. Wanda and SparseGPT are omitted at 100% since their unpruned configuration is identical to the uniform baseline.