Organizations: Zhejiang University, Hangzhou, Zhejiang, China · Westlake University, Hangzhou, Zhejiang, China · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
Sparse Mixture-of-Experts (SMoE) language models achieve strong capability at low per-token compute, yet deployment remains constrained by memory footprint and throughput because the full expert pool must still be stored and served. Post-training expert pruning reduces this cost, but most methods focus on which experts to prune within each layer and default to a uniform layer-wise sparsity allocation, even though the layer-wise allocation can strongly affect performance. We decouple pruning into within-layer expert ranking and across-layer budget allocation, and introduce Expected Speculative Acceptance Proxy (ESAP), a speculative-decoding-inspired, teacher-forced metric that measures how well a pruned model matches the full model without costly autoregressive decoding. ESAP is bounded and stable, enabling cheap comparison of many candidates. Building on ESAP, we propose EvoESAP, an evolutionary search framework that finds an improved non-uniform layer-wise sparsity allocation under a fixed global budget while holding the within-layer pruning order fixed, making it a plug-and-play method for criteria such as Frequency, EAN, SEER, and REAP. Across 7B--30B SMoE LLMs at 25% and 50% sparsity, EvoESAP consistently discovers non-uniform allocations that improve open-ended generation (up to +19.6% on MATH-500 at 50% sparsity) while preserving competitive multiple-choice accuracy compared with uniform pruning at the same sparsity. Code is available at https://github.com/ZongfangLiu/EvoESAP.
Figures & tables
Figure 1. Layer-wise density schedules and performance for OLMoE-1B-7B-0125-Instruct at 25% global sparsity (density = 1−sparsity ). Each panel shows the per-layer remaining expert density under a fixed global pruning budget, using REAP to rank experts in each layer (computed from 1,024 calibration samples from evol-codealpaca-v1 ). Uniform prunes the same fraction of experts in every layer. Frequency-based ranks experts globally by routing frequency, counts how many experts from each layer fall in the tail 25%, and uses those counts to set layer-wise sparsity. Searched finds a non-uniform sparsity schedule with EvoESAP under the same budget. Numbers above panels report average performance on Code/Math/MC benchmarks, with deltas relative to uniform. The results imply that given the same pruning metric, non-uniform allocation has the potential to better preserve the model’s capabilities for SMoE expert pruning; however, finding an effective non-uniform allocation is non-trivial, and a poor allocation can harm overall performance. Three panels compare uniform, frequency-based, and EvoESAP-searched layer-wise expert density schedules for OLMoE at 25 percent global sparsity, together with average coding, math, and multiple-choice results.
Figure 2. Overview of EvoESAP. (a) Evolutionary search with budget-preserving level-switch mutation. Histograms visualize the layer-wise density distribution of each candidate model (density =1− sparsity) induced by an allocation r (experts removed per layer) under a fixed global budget B . Offspring are generated from the top m survivors by a level-switch that transfers Δ units of pruning budget between two layers (gray: decrease; yellow: increase), keeping ∑ℓrℓ=B unchanged. (b) ESAP as per-sample fitness. Under teacher forcing, ESAP scores a sample by the full-vocabulary overlap between the target next-token distribution (dark gray, p(⋅∣x) ) and the candidate/draft distribution (blue, q(⋅∣x) ), averaged over answer-token positions. The ESAP score is defined in Equation 13 , and higher values are better. The left panel shows evolutionary search over layer-wise sparsity allocations under a fixed pruning budget. The right panel illustrates ESAP as overlap between target and candidate next-token distributions under teacher forcing.
Coding
Creative Writing
Math
MC
Model
Sparsity
Method
Eval+
LiveCode
Avg
WildBench
GSM8K
MATH-500
Avg
MC Avg
OLMoE
Full
0.341
0.033
0.187
0.444
0.682
0.222
0.452
0.653
25%
EAN
Uni
0.343
0.022
0.183
0.269
0.585
0.190
0.387
0.551
ESAP
0.312
0.027
0.170
0.258
0.576
0.232
0.404
0.543
SEER
Uni
0.339
0.027
0.183
0.253
0.577
0.204
0.390
0.545
ESAP
0.306
0.022
0.164
0.254
0.601
0.248
0.424
0.539
Table 1. Benchmark results for OLMoE-1B-7B-0125-Instruct and ERNIE-4.5-21B-A3B-PT on coding, WildBench, math, and MC benchmarks. OLMoE is evaluated at 25% global sparsity, and ERNIE at 25% and 50%. Uni uses uniform layer-wise sparsity, while ESAP uses the non-uniform allocation found by EvoESAP. Bold indicates the better result within each pair.
Coding
Creative Writing
Math
MC
Model
Sparsity
Method
Eval+
LiveCode
Avg
WildBench
GSM8K
MATH-500
Avg
MC Avg
DeepSeek
Full
0.549
0.104
0.327
0.418
0.610
0.298
0.454
0.678
25%
EAN
Uni
0.459
0.071
0.265
0.259
0.531
0.224
0.378
0.571
ESAP
0.417
0.071
0.244
0.301
0.572
0.224
0.398
0.588
SEER
Uni
0.404
0.099
0.252
0.260
0.553
0.210
0.382
0.586
ESAP
0.460
0.115
0.288
0.290
0.559
0.226
0.393
0.594
Table 2. Benchmark results for DeepSeek-V2-Lite-Chat and Qwen3-30B-A3B-Instruct-2507 on coding, WildBench, math, and multiple-choice benchmarks. DeepSeek is evaluated at 25% global sparsity, while Qwen3 is evaluated at 25% and 50%. Uni uses uniform layer-wise sparsity, while ESAP uses the non-uniform allocation found by EvoESAP.
Model
GPU
Generations
Time (h)
Memory (GB)
OLMoE-7B
1 × L40S
50
5.8
9.89 / 12.9
DeepSeek-16B
1 × L40S
30
6.7
22.57 / 29.27
ERNIE-21B
2 × L40S
20
5.0
31.17 / 40.66
Qwen3-30B
2 × L40S
10
5.2
43.42 / 56.92
Table 3. Search cost and model-loading GPU memory at 25% global sparsity. We report the GPU configuration, wall-clock search time, and the bfloat16 GPU memory required to load the pruned and full models. Pruning reduces model-loading memory by roughly 23%–24% across models.
Method
GPU
Time (h)
Code Avg
WildBench
MC Avg
DiEP
2 × L40S
11.25
0.114
0.208
0.495
Ours
1 × L40S
1.92
0.178
0.275
0.565
Table 4. Comparison with DiEP on OLMoE-7B at 25% global sparsity. Since DiEP jointly optimizes within-layer pruning order and layer-wise sparsity allocation, we include both calibration (0.75h) and search time (1.17h) in EvoESAP’s reported runtime to ensure a fair comparison.
Coding
Creative Writing
Math
MC
Ablation
Value
Eval+
LiveCode
Code Avg
WildBench
GSM8K
MATH-500
Math Avg
MC Avg
Reference
REAP + Uni
0.314
0.005
0.160
0.292
0.596
0.200
0.398
0.579
Fitness
KL
0.331
0.016
0.174
0.285
0.595
0.216
0.405
0.582
NLL
0.334
0.016
0.175
0.295
0.619
0.230
0.424
0.576
SAP
0.339
0.005
0.172
0.289
0.622
0.218
0.420
0.584
ESAP (Ours)
0.344
0.033
0.189
0.279
0.636
0.216
0.426
0.581
Table 5. Ablations of fitness choice and search hyperparameters at 25% global sparsity on OLMoE. We include the uniform REAP baseline as a reference. Blue cells mark the default configuration corresponding to the OLMoE 25% REAP+ ESAP result in Table 1 . Relative to the uniform REAP baseline, all searched settings improve both Code Avg and Math Avg. Among the searched fitness functions, ESAP gives the strongest Code Avg and Math Avg.
Fitness
GPU
Time (h)
Code Avg
WildBench
MC Avg
SPEC-DEC
2 × L40S
29.49
0.171
0.269
0.565
ESAP
1 × L40S
1.64
0.173
0.256
0.557
Table 6. Comparison of true speculative-decoding acceptance (SPEC-DEC) and the proposed ESAP. ESAP significantly reduces the search time from 29.49 hours to 1.64 hours.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
τ←min(U{1,…,τmax},U{1,…,τmax}).
Appendix
Algorithm 1 Evolutionary Search for Non-uniform Layer-wise Sparsity Allocation
Coding
Creative Writing
Math
MC
Model
Sparsity
Method
Eval+
LiveCode
Avg
WildBench
GSM8K
MATH-500
Avg
MC Avg
ERNIE
Full
0.867
0.247
0.557
0.479
0.829
0.780
0.804
0.721
25%
EAN
Uni
0.248
0.049
0.148
0.412
0.670
0.366
0.518
0.685
ESAP
0.354
0.093
0.223
0.433
0.752
0.438
0.595
0.669
SEER
Uni
0.249
0.055
0.152
0.406
0.738
0.374
0.556
0.660
ESAP
0.367
0.060
0.213
0.419
0.737
0.378
0.557
0.671
Appendix
Table 7. MC and open-ended generation benchmark results for ERNIE-4.5-21B-A3B-PT under 25% global sparsity. Pruning is calibrated on C4. ESAP is the non-uniform allocation searched on evol-codealpaca-v1 with the same pruning metric.
Figure 3. Layer-wise density distributions (density =1−sparsity ) of the searched non-uniform allocations across different pruning metrics. Two vertically stacked subfigures summarize searched layer-wise density schedules across pruning metrics. The 25 percent panel uses a placeholder if the local figure asset is unavailable, and the 50 percent panel shows the available searched density visualization.
Coding
Model
Compression
Method
HumanEval
HumanEval +
MBPP
MBPP +
Eval+
LiveCodeBench
Code Avg
OLMoE-1B-7B- 0125-Instruct
Baseline
0.354
0.323
0.373
0.312
0.341
0.033
0.187
25%
EAN
Uniform
0.366
0.341
0.360
0.307
0.343
0.022
0.183
Searched
0.317
0.293
0.347
0.291
0.312
0.027
0.170
SEER
Uniform
0.360
0.341
0.349
0.307
0.339
0.027
0.183
Searched
0.311
0.274
0.347
0.294
0.306
0.022
0.164
Appendix
Table 8. Full results on coding benchmarks. Eval+ is the average of HumanEval, HumanEval+, MBPP and MBPP+.
WildBench + Math
Model
Compression
Method
WildBench
GSM8K
MATH-500
Math Avg
OLMoE-1B-7B- 0125-Instruct
Baseline
0.444
0.682
0.222
0.452
25%
EAN
Uniform
0.269
0.585
0.190
0.387
Searched
0.258
0.576
0.232
0.404
SEER
Uniform
0.253
0.577
0.204
0.390
Searched
0.254
0.601
0.248
0.424
Appendix
Table 9. Full results on WildBench and math benchmarks.
Multiple Choice
Model
Compression
Method
MMLU
ARC-C
ARC-E
HellaSwag
WinoGrande
BoolQ
OpenBookQA
RTE
MC Avg
OLMoE-1B-7B- 0125-Instruct
Baseline
0.534
0.490
0.758
0.808
0.684
0.766
0.470
0.711
0.653
25%
EAN
Uniform
0.452
0.358
0.558
0.630
0.602
0.727
0.336
0.747
0.551
Searched
0.454
0.349
0.573
0.628
0.597
0.716
0.326
0.700
0.543
SEER
Uniform
0.457
0.352
0.566
0.620
0.599
0.723
0.340
0.708
0.545
Searched
0.447
0.338
0.573
0.621
0.601
0.715
0.330
0.690
0.539
Appendix
Table 10. Full results on multiple-choice benchmarks.
Mixture-of-Experts (MoE) architectures scale model capacity efficiently, yet their substantial memory footprint poses a major barrier to deployment. Expert pruning reduces this footprint, but most methods focus on which experts to remove and prune every layer equally. We propose SPARE (Sparsity Allocation via Routing Entropy), a training-free, selector-agnostic method that decides how many experts to remove from each layer using routing entropy, the entropy of tokens' normalized Top-K gate weights on calibration data. When a token's gate weight is spread evenly, removing one active expert discards little of it, so SPARE prunes layers with more diffuse routing more aggressively. A perturbation bound under random pruning formalizes this intuition, and controlled interventions with a fixed selector link concentrated routing to larger output changes. With the same selector and budget, SPARE improves average accuracy over uniform allocation on Mixtral, Qwen3, and DeepSeek models, especially at high pruning ratios. At 50% expert pruning, it raises the eight-task average of Qwen3-30B-A3B from 57.2% to 63.8%, and the math, code, and science average of DeepSeek-R1 from 80.1 to 82.8.
Empirical scaling laws for language models have encouraged the development of ever-larger LLMs, despite their growing computational and memory costs. Sparse Mixture-of-Experts (MoEs) offer a promising alternative by activating only a subset of experts per forward pass, improving efficiency without sacrificing performance. However, the large number of expert parameters still leads to substantial memory consumption. Existing pruning methods typically allocate budgets uniformly across layers, overlooking the heterogeneous redundancy that arises in sparse MoEs. We propose GRAPE (Global Redundancy-Aware Pruning of Experts, a global pruning strategy that dynamically allocates pruning budgets based on cross-layer redundancy. Experiments on Mixtral-8x7B, Mixtral-8x22B, DeepSeek-MoE, Qwen-MoE, and GPT-OSS show that, under the same pruning budget, GRAPE consistently achieves the best average performance. On the three main models reported in the paper, it improves average accuracy over the strongest local baseline by 1.40% on average across pruning settings, with gains of up to 2.45%.
Zeliang Zhang, Nikhil Ghosh, Jiani Liu +2
University of Rochester · Flatiron Institute · University of California, Berkeley +1
Sparsely-activated Mixture-of-Experts (MoE) language models achieve remarkable inference efficiency by activating only a small fraction of parameters per token, yet their full expert banks reside in memory at all times, creating a prohibitive deployment bottleneck. Existing structured pruning methods, largely designed for dense transformers, assess expert importance using locally derived heuristics that are blind to the interdependent nature of MoE routing. We introduce MAESTRO (Markov-chain Approximated Expert Sparsification via Transition-based ROuting), a structured pruning framework designed for MoE architectures that models autoregressive expert activation trajectories as Ergodic Markov chains whose stationary distributions encode cross-layer dependencies, yielding a globally aware importance heuristic. Evaluated across five diverse domains including Safety, Bias, and Ethics, MAESTRO outperforms state-of-the-art baselines by up to 10.61% in average performance retention under a strict 50% compression regime, while exhibiting substantially lower cross-task variance, indicating that global, routing-congruent pruning produces models that generalize more consistently across heterogeneous tasks.