Expert pruning reduces the memory and serving cost of Mixture-of-Experts (MoE) models by removing low-importance experts identified by the router, assuming router probabilities provide a reliable importance signal. We observe that this assumption breaks down under over-dispersed routing, a regime associated with aggressive load-balancing during training, in which tokens are distributed nearly uniformly across experts and importance signals collapse. In this regime, perplexity does not predict downstream task accuracy: on gpt-oss-20B, the lowest-perplexity pruning configuration yields the worst mathematical reasoning, while the highest-perplexity configuration preserves it. This does not occur under standard routing (e.g., Mixtral-8x7B-Instruct), where perplexity and accuracy degrade together. Pruning under over-dispersed routing also exposes a capability trade-off in which no single scoring metric dominates: activation-aware scoring preserves mathematical reasoning but severely degrades knowledge-intensive science (an 18-point gap on GPQA), whereas frequency-based scoring exhibits the reverse. We propose Minimax Expert Score Allocation (MESA), a domain-aware method that iteratively boosts importance scores for experts serving whichever domain is currently worst-affected, minimizing worst-case domain degradation rather than average accuracy. At 25% expert pruning MESA achieves the smallest worst-case degradation across domains, outperforming activation-aware baselines on 7 of 11 benchmarks at a correspondingly reduced memory footprint, and it generalizes to gpt-oss-120B, Gemma-4-26B-A4B, and OLMoE-1B-7B. Our results indicate that over-dispersed routing is a qualitatively distinct pruning regime in which standard assumptions fail, and that recognizing it is a prerequisite for principled expert pruning of load-balanced MoE models.
Figures & tables
Figure 1: Per-layer routing entropy (normalized to % of maximum) for gpt-oss-20b and Mixtral-8x7B-Instruct. gpt-oss maintains > 82% across all layers (over-dispersed), while Mixtral drops to 64–68% in later layers as specialization emerges. This gap explains why global importance scores are informative for Mixtral but noisy for gpt-oss.
Method
WikiText-2 PPL ( ↓ )
GSM8K (%) ( ↑ )
Baseline (unpruned)
126.02
88.86
MESA (ours)
107.14
92.87
REAP
145.04
89.31
RM-Uniform
83.29
83.24
Random
40.95
78.54
Table 1: WikiText-2 PPL vs. GSM8K accuracy across pruning configurations at r=0.25 . PPL and task accuracy are uncorrelated in over-dispersed models: random pruning achieves the lowest PPL (40.95) but near-worst task accuracy, while MESA improves GSM8K above the unpruned baseline despite moderate PPL.
Figure 2: PPL vs task accuracy across three benchmarks at r=0.25 . Each point is one pruning method. The expected negative correlation (lower PPL = better accuracy) does not hold: ρ is positive on GSM8K and MMLU and near zero on GPQA, confirming PPL is uninformative or misleading for method comparison in over-dispersed models.
Math Reasoning
Knowledge & Commonsense
Multi-domain & Instruction
Method
AIME’25
GSM8K
GPQA
ARC-C
ARC-E
OBQA
PIQA
MMLU
MMLU-Pro
WinoGr.
IFEval
Baseline
76.67 ± 3.27
88.86 ± 0.38
64.47 ± 1.46
95.30 ± 0.27
98.32 ± 0.15
92.99 ± 0.45
83.34 ± 0.36
84.01 ± 0.10
72.51 ± 0.22
76.78 ± 1.04
91.67 ± 0.67
MESA (ours)
63.33 ± 6.06
92.87 ± 0.45
65.48 ± 4.2
92.66 ± 0.8
97.60 ± 0.3 ∗
90.38 ± 1.2
80.73 ± 1.5
79.51 ± 0.6
67.25 ± 0.9
69.59 ± 2.1
89.92 ± 1.1
REAP
73.33 ± 9.7
89.31 ± 1.8
47.72 ± 5.6
91.63 ± 1.0
95.03 ± 0.7
92.18 ± 1.0
74.90 ± 2.0
77.36 ± 0.8
64.27 ± 1.2
78.44 ± 2.4
90.88 ± 0.9
RM-Uni.
46.67 ± 8.5
83.24 ± 1.5
56.85 ± 5.1
94.36 ± 0.6
97.56 ± 0.4
92.18 ± 0.9
74.85 ± 1.8
80.31 ± 0.5
66.32 ± 0.8
69.27 ± 2.3
86.07 ± 1.4
Random
33.33 ± 7.2
78.54 ± 2.3
49.75 ± 4.8
90.01 ± 1.2
96.46 ± 0.5
86.97 ± 1.5
78.61 ± 1.6
74.76 ± 0.9
56.06 ± 1.3
72.91 ± 2.8
89.68 ± 1.2
Table 2: Task accuracy (%) at r=0.25 (25% expert pruning, 32→24 experts per layer), 5-seed evaluation with 95% CIs. Best pruned result per task in bold . MESA achieves the most balanced profile, with the best or near-best accuracy on the majority of benchmarks and the smallest worst-case degradation.
Benchmark
MESA (ours)
REAP
EvoESAP
RM-Uniform
Random
AIME’25
−13.34
−3.34
−20.00
−30.00
−43.34
GSM8K
+4.01
+0.45
−10.54
−5.62
−10.32
ARC-C
−2.64
−3.67
−0.34
−0.94
−5.29
ARC-E
−0.72
−3.29
−0.68
−0.76
−1.86
GPQA Diamond
+1.01
−16.75
−20.31
−7.62
−14.72
OpenBookQA
−2.61
−0.81
−3.81
−0.81
−6.02
Table 3: Per-benchmark accuracy change ( Δ , percentage points) relative to the unpruned baseline, gpt-oss-20B at r=0.25 , single-seed evaluation across all 11 benchmarks. Best (least-negative or most-positive) Δ per row in bold ; ties bolded.
Method
WikiText-2 PPL
Baseline (8E)
4.78
REAP (6E)
6.00
Random (6E)
8.31
RM-Uniform (6E)
14.35
MESA (6E)
16.43
Table 4: Cross-architecture validation on Mixtral-8x7B-Instruct ( λaux≈0.001 , r=0.25 , 8→6 experts). Unlike gpt-oss, Mixtral shows PPL increases with pruning (standard behavior). REAP achieves the best PPL (6.00), while MESA, designed for over-dispersed routing, produces the worst (16.43).
Method
ARC-C
ARC-E
GPQA
GSM8K
MMLU
IFEval
PIQA
Baseline
85.29 ± 0.24
94.51 ± 0.15
31.07 ± 1.21
31.42 ± 1.25
68.54 ± 0.16
66.58 ± 1.34
85.60 ± 0.34
REAP
82.19 ± 0.30
92.29 ± 0.15
28.93 ± 2.61
35.00 ± 0.97
64.29 ± 0.19
65.23 ± 0.73
83.58 ± 0.26
EvoESAP
73.63 ± 0.32
86.14 ± 0.30
31.68 ± 1.59
40.56 ± 0.66
56.65 ± 0.27
65.62 ± 0.23
77.21 ± 0.38
Random
70.13 ± 0.50
80.61 ± 0.29
19.70 ± 1.36
20.71 ± 0.84
52.56 ± 0.11
57.98 ± 1.25
74.95 ± 0.46
MESA
51.97 ± 1.20
67.15 ± 1.13
22.34 ± 5.74
3.49 ± 0.48
38.49 ± 0.34
49.44 ± 1.51
58.42 ± 0.71
RM-Uniform
49.77 ± 1.05
60.23 ± 0.60
37.66 ± 0.22
2.96 ± 0.38
24.60 ± 2.36
45.95 ± 1.68
57.04 ± 1.14
Table 5: Cross-architecture task accuracy (%) on Mixtral-8x7B-Instruct at r=0.25 ( 8→6 experts), 5-seed means ± std. Best pruned result per task in bold . REAP retains 98% of baseline average accuracy while MESA retains 63%.
Task (cloze acc_norm)
Unpruned
MESA (prefill)
MESA (gen)
REAP (gen)
RM (gen)
OpenBookQA
45.2
39.8
35.8
38.4
35.8
PIQA
80.6
77.6
74.2
71.9
72.2
WinoGrande
72.0
67.7
59.0
67.6
59.0
ARC-Challenge
55.2
50.0
44.0
40.6
45.8
ARC-Easy
82.3
77.4
72.2
64.0
73.2
HellaSwag
76.0
69.2
58.5
64.9
57.7
Table 6: Generality to a third, independently-trained over-dispersed MoE family: OLMoE-1B-7B-0924 (a fully-open base model; 16 layers, 64 experts, top-8; native routing entropy Hˉ=0.925⋅log2E ). Loglikelihood/cloze accuracy (%) at r=0.25 (25% expert pruning, 64→48 experts per layer), single seed. Best pruned result per task in bold . On these six commonsense tasks (cloze/acc_norm), MESA calibrated in the matching prefill regime leads every pruned method and beats REAP on all six, retaining 92.8% of unpruned commonsense accuracy.
Task
Unpruned
MESA r=0.25
MESA r=0.375
MESA r=0.50
OpenBookQA
45.2
39.8
35.6
31.2
PIQA
80.6
77.6
72.4
68.1
WinoGrande
72.0
67.7
58.6
53.8
ARC-Challenge
55.2
50.0
43.5
34.5
ARC-Easy
82.3
77.4
73.4
53.7
HellaSwag
76.0
69.2
59.6
46.2
Table 7: OLMoE-1B-7B prefill-MESA ratio sweep (single seed): commonsense accuracy degrades gracefully with no cliff. Eval-matched (prefill) calibration buys ∼ one extra ratio step of headroom: prefill MESA at r=0.375 (57.2) matches every gen-route method at r=0.25 ( 57.3 – 57.9 ; Table 6 ).
Math Reasoning
Knowledge & Commonsense
Multi-domain & Instruction
Method
AIME’25
GSM8K
GPQA
ARC-C
ARC-E
OBQA
PIQA
MMLU
MMLU-Pro
WinoGr.
IFEval
Unpruned
75.56
96.41
71.21
96.62
98.78
94.47
90.82
86.58
82.05
80.27
89.81
MESA (ours)
72.22
96.21
67.85
95.53
98.30
92.13
88.03
83.73
79.40
77.56
88.98
REAP
35.56
95.60
53.54
95.82
98.22
90.67
86.85
78.94
67.90
76.95
88.70
RM-Uni.
50.00
95.12
46.63
91.52
96.55
88.13
81.27
78.26
69.56
73.24
86.67
Table 8: Task accuracy (%) on Gemma-4-26B-A4B-it at r=0.25 (128 routed experts, top-8; 128→96 experts per layer), temperature 0.6 , single measurement per cell. Best pruned result per task in bold . MESA matches or exceeds both REAP and RM-Uniform on every one of the 11 benchmarks, with the largest gap on GPQA-Diamond, the hardest task in the suite ( +14.3 pp over REAP, +21.2 pp over RM-Uniform ).
Method
ΔR (%)
ΔK (%)
Character
MESA (ours)
−4.67
−1.68
Balanced
REAP
−1.45
−3.48
Reasoning-preserving
EvoESAP †
−15.27
−0.51
Knowledge-preserving
RM-Uniform
−17.81
−0.85
Knowledge-preserving
Random
−26.83
−3.58
Severe collapse
Table 9: Reasoning/Knowledge degradation at r=0.25 (single-seed results). ΔR : mean accuracy change on AIME’25 and GSM8K; ΔK : mean change on ARC-C and ARC-E. Each method occupies a distinct position on the capability trade-off; MESA achieves the most balanced trade-off with a net reasoning improvement on GSM8K offsetting its AIME’25 drop.
Figure 3: Capability trade-off: reasoning vs knowledge preservation at r=0.25 . Each method occupies a distinct position; MESA is closest to the origin (balanced), while REAP favors reasoning preservation.
Method
Scoring compute
Allocation compute
Extra fwd. passes beyond harvest
MESA (ours)
Static per-domain L×E router-mass matrix, already produced by the calibration harvest
CPU-only iterative minimax reweighting over the L×E matrix ( ∼ 15–20 steps)
0
REAP [ 12 ]
Per-expert activation-norm scoring requires a dedicated forward pass over its calibration set (24,576 samples at 16,384 tokens)
Global budget allocation over per-expert activation-norm scores
Full calibration-set forward pass (activation-norm scoring)
RM
Router-mass matrix only, identical byproduct of the same harvest MESA consumes
Uniform per-layer threshold on aggregate router mass
0
Table 10: Qualitative compute characterization of expert-scoring and allocation for MESA (ours) vs. REAP and Router-Mass (RM). MESA’s allocation step (Algorithm 1 ) operates entirely on a small, pre-computed L×E router-mass matrix (e.g., 24×32 for gpt-oss-20B) rather than requiring additional model forward passes, making it structurally cheaper than activation-based scoring methods such as REAP. No wall-clock timings are claimed here; see § 5.2 for the verified ∼ 49 × calibration-sample-count reduction relative to REAP.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
(a) Samples per domain (fixed ℓ=2048 )
Samples/domain
Jaccard J
Δ experts
Character
5
0.655
40
Knowledge-biased
10
0.829
18
Moderate
20
1.000
0
✓Reference
50
0.979
2
✓Near-reference
100
0.901
10
Domain-fair-biased
Appendix
Table 11: Calibration sensitivity of MESA expert selection. Jaccard similarity measures agreement with the reference configuration ( s=20 , ℓ=2048 ) on which experts to prune ( 24×8=192 binary decisions at r=0.25 ). Different calibration budgets explore different regions of the capability Pareto frontier: lower calibration favors knowledge preservation while higher calibration emphasizes domain fairness.
(a) Iterations T (fixed η0=0.15 )
T
Worst-domain loss
Jaccard vs. T=80
Best step
1
0.0524
76.1%
0
3
0.0509
81.1%
2
5
0.0503
86.4%
4
10
0.0492
92.0%
8
15
0.0491
95.9%
13
Appendix
Table 12: MESA hyperparameter sensitivity at r=0.25 . (a) Increasing iterations T monotonically improves worst-domain loss and selection agreement (Jaccard vs. T=80 reference). Convergence is effectively reached by T=15 . (b) The boost rate η0 controls convergence speed vs. stability; values in [0.05,0.25] achieve near-identical solutions with low oscillation.
Figure 4: MESA convergence for different pruning ratios r . The algorithm converges within 6–20 iterations for r≤0.375 . Higher ratios produce higher loss plateaus but similar convergence dynamics, suggesting the difficulty is in the objective landscape, not the optimizer.
Figure 5: Per-domain routing mass across all 24 layers and 32 experts in gpt-oss-20b. Each cell shows the fraction of routing probability mass that domain d assigns to expert e at layer l . The narrow colorbar range (0.028–0.038 vs. the uniform expectation of 1/32≈0.031 ) confirms over-dispersion: no single expert dominates for any domain. However, subtle per-domain structure is visible; certain experts receive slightly elevated mass for specific domains (e.g., experts 8–12 in layers 16–20 for Code). These weak but consistent signals are exactly what MESA’s domain-conditional scoring exploits: individually negligible, they become collectively meaningful when the minimax objective accumulates evidence across all calibration tokens.
Figure 6: Jaccard overlap between domains’ top-8 most-routed experts per layer, averaged across all 24 layers. High off-diagonal values (0.41–0.47) confirm that domains share nearly half their preferred experts due to over-dispersed routing. The slight asymmetries (e.g., Math–Science 0.47 vs. Code–Instruction 0.41) reflect residual domain structure that domain-fair scoring can exploit.
Figure 7: Per-task accuracy change from baseline at r=0.25 for all evaluated pruning methods. The heatmap reveals structured degradation patterns: reasoning tasks (left columns) show large variance across methods (AIME25 spans −3.3 to −43.3 %), while knowledge tasks (right columns) show uniformly mild degradation ( <5 % for most methods). MESA achieves the lightest overall shading (smallest absolute degradation across all tasks), while REAP shows a distinctive pattern of severe GPQA loss ( −16.8 %) alongside strong reasoning preservation.
Figure 8: Routing entropy determines importance signal quality. As routing entropy increases toward uniformity, Kendall- τ between global importance ranking and worst-domain ranking decreases sharply. The 85% band (dashed) is a practical diagnostic, not a sharp physical boundary, and separates reliable (Mixtral, 74.3%) from degraded (gpt-oss, 90.8%) regimes. Simulated on 24-layer, 32-expert matrices with 6 calibration domains; real data points marked with stars.
Figure 9: Knowledge degradation on GPQA-Diamond. Same model (gpt-oss-20b), same graduate-level electrostatics problem. Left: the unpruned model correctly identifies that system energy is half the interaction energy for a grounded conductor. Right: REAP-pruned model sets up the method of images identically but omits the factor of 1/2 , a specific physical concept lost during pruning. This pattern, correct technique but missing conceptual knowledge, explains REAP’s 17.8% GPQA gap relative to MESA: the pruned model retains procedural skills but loses domain-specific physical reasoning stored in the removed experts.
Task
Unpruned
MESA (ours)
REAP
RM-Uni.
AIME’25
86.67 ± 3.33
90.00 ± 3.33
76.67 ± 4.71
70.00 ± 5.44
GSM8K
96.51 ± 0.38
95.98 ± 0.45
93.18 ± 0.62
89.54 ± 0.88
GPQA-Diamond
74.24 ± 1.21
65.66 ± 1.82
55.56 ± 1.95
48.99 ± 2.15
ARC-Challenge
96.84 ± 0.42
95.56 ± 0.51
92.40 ± 0.68
88.15 ± 0.82
ARC-Easy
98.82 ± 0.25
98.23 ± 0.31
96.50 ± 0.45
93.80 ± 0.60
Appendix
Table 13: Gemma-4-26B-A4B-it r=0.25 head-to-head at temperature 0 (deterministic decoding), mean ± std over 3 seeds, on the 5 tasks with temperature- 0 replication data. Best pruned result per task in bold ; non-overlapping intervals on GPQA-Diamond. This is an independent replication of the main-text temperature- 0.6 result (Table 8 ) under deterministic decoding, disclosed alongside it rather than in place of it. Additional benchmarks (MMLU, MMLU-Pro, IFEval, PIQA, OpenBookQA, WinoGrande) are pending temperature- 0 replication.
Math Reasoning
Knowledge & Commonsense
Multi-domain & Instruction
Method
AIME’25
GSM8K
GPQA
ARC-C
ARC-E
OBQA
PIQA
MMLU
MMLU-Pro
WinoGr.
IFEval
Unpruned
74.00
95.63
70.36
96.38
98.67
93.91
84.93
87.76
77.78
81.03
86.50
MESA (ours)
72.22
95.58
58.46
92.38
95.88
93.56
77.95
83.01
64.20
77.27
87.15
REAP
48.89
96.26
51.85
95.14
98.18
93.33
77.93
82.05
68.96
81.16
84.50
RM-Uni.
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Appendix
Table 14: Task accuracy (%) on gpt-oss-120B at r=0.5 (aggressive expert pruning, 128→64 experts per layer), single measurement per cell. Best pruned result per task in bold . RM-Uniform collapses to 0.00 on every task at this ratio, a genuine failure of the method at r=0.5 rather than a missing measurement. On the worst domain (GPQA-Diamond, the hardest task in the suite), MESA leads REAP by 6.6 pp; MESA also leads on AIME, IFEval, MMLU, OpenBookQA, and PIQA. REAP edges ahead on GSM8K, ARC-Challenge, ARC-Easy, MMLU-Pro, and WinoGrande. We report this honestly as a mixed outcome at this aggressive ratio: MESA’s worst-domain protection holds, but it does not dominate every benchmark under such heavy pruning.
Micro-expert pruning reduces the capacity of mixture-of-experts (MoE) models by removing coupled channels within routed experts. However, finer pruning granularity alone does not guarantee the preservation of downstream capabilities, as similar language-modeling perplexity can accompany substantially different task performance. We revisit micro-expert mask construction as two distinct decisions: allocating retained capacity across experts and selecting channels within that capacity. We propose UC-RAMP, a two-stage post-training pruning framework that explicitly separates these decisions. CAPE uses cross-domain calibration evidence to redistribute expert widths under an exact layer-wise budget, while RAMP refines channel composition at fixed widths through one-out, one-in exchanges guided by a channel-interaction surrogate. Across three MoE models at 25% and 50% routed-expert channel pruning, UC-RAMP achieves the highest ten-task mean accuracy among evaluated pruning methods in five of six settings. At 50% pruning, it outperforms the strongest evaluated baseline by 8.19, 3.96, and 6.14 percentage points across the three models, respectively. In a single-A100 deployment at 25% pruning, UC-RAMP reduces measured time to first token by 19.7--23.9% and time per output token by 20.2--22.3% relative to the unpruned model. The code is available at https://github.com/xzktx003/UC_RAMP.
Zukang Xu, Xinpei Gao, Xing Hu +2
Houmo AI · University of Science and Technology of China · Southeast University
Mixture-of-Experts (MoE) language models reduce per-token computation through sparse expert activation, yet deployment still requires storing the full expert pool, making one-shot expert pruning a practical approach for reducing memory usage. Although effective, existing criteria are largely heuristic, and no single criterion is universally optimal. Thus, establishing a principle for selecting pruning criteria suited to different deployment objectives remains an important yet largely underexplored problem in one-shot expert pruning. To this end, we introduce a unified formulation for one-shot MoE expert pruning organized around three factors: routing frequency, gate weighting, and activation strength. The formulation yields a criteria selection principle: task-agnostic pruning should favor routed-token-averaged, gate-free activation-based criteria, whereas task-specific pruning can benefit from retaining routing-frequency and gate-weight information. Beyond this principle, the formulation also provides a systematic view of existing heuristic criteria and gives rise to two new task-agnostic criteria, Mean Activation Norm (MAN) and Mean Squared Activation Norm (MSAN). Across four representative MoE models and 16 diverse benchmarks, MAN and MSAN are consistently strong in the task-agnostic setting, obtain the top-two average ranks, and improve average performance by up to 8.8 points over the strongest baseline.
Zongfang Liu, Jinghui Zhang, Zijian Ma +2
1Zhejiang University · 2Westlake University · 3Mohamed bin Zayed University of Artificial Intelligence +1
Mixture-of-Experts (MoE) architectures scale model capacity efficiently, yet their substantial memory footprint poses a major barrier to deployment. Expert pruning reduces this footprint, but most methods focus on which experts to remove and prune every layer equally. We propose SPARE (Sparsity Allocation via Routing Entropy), a training-free, selector-agnostic method that decides how many experts to remove from each layer using routing entropy, the entropy of tokens' normalized Top-K gate weights on calibration data. When a token's gate weight is spread evenly, removing one active expert discards little of it, so SPARE prunes layers with more diffuse routing more aggressively. A perturbation bound under random pruning formalizes this intuition, and controlled interventions with a fixed selector link concentrated routing to larger output changes. With the same selector and budget, SPARE improves average accuracy over uniform allocation on Mixtral, Qwen3, and DeepSeek models, especially at high pruning ratios. At 50% expert pruning, it raises the eight-task average of Qwen3-30B-A3B from 57.2% to 63.8%, and the math, code, and science average of DeepSeek-R1 from 80.1 to 82.8.