EvoESAP: Non-Uniform Expert Pruning for Sparse MoE

Zongfang Liu; Shengkun Tang; Boyang Sun; Zhiqiang Shen; Xin Yuan

cs.LGMar 6, 2026

EvoESAP: Non-Uniform Expert Pruning for Sparse MoE

Authors: Zongfang Liu, Shengkun Tang, Boyang Sun, Zhiqiang Shen, Xin Yuan

Organizations: Zhejiang University, Hangzhou, Zhejiang, China · Westlake University, Hangzhou, Zhejiang, China · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates

Abstract

Sparse Mixture-of-Experts (SMoE) language models achieve strong capability at low per-token compute, yet deployment remains constrained by memory footprint and throughput because the full expert pool must still be stored and served. Post-training expert pruning reduces this cost, but most methods focus on which experts to prune within each layer and default to a uniform layer-wise sparsity allocation, even though the layer-wise allocation can strongly affect performance. We decouple pruning into within-layer expert ranking and across-layer budget allocation, and introduce Expected Speculative Acceptance Proxy (ESAP), a speculative-decoding-inspired, teacher-forced metric that measures how well a pruned model matches the full model without costly autoregressive decoding. ESAP is bounded and stable, enabling cheap comparison of many candidates. Building on ESAP, we propose EvoESAP, an evolutionary search framework that finds an improved non-uniform layer-wise sparsity allocation under a fixed global budget while holding the within-layer pruning order fixed, making it a plug-and-play method for criteria such as Frequency, EAN, SEER, and REAP. Across 7B--30B SMoE LLMs at 25% and 50% sparsity, EvoESAP consistently discovers non-uniform allocations that improve open-ended generation (up to +19.6% on MATH-500 at 50% sparsity) while preserving competitive multiple-choice accuracy compared with uniform pruning at the same sparsity. Code is available at https://github.com/ZongfangLiu/EvoESAP.

Figures & tables

Figure 1. Layer-wise density schedules and performance for OLMoE-1B-7B-0125-Instruct at 25% global sparsity (density =

1-\text{sparsity}

). Each panel shows the per-layer remaining expert density under a fixed global pruning budget, using REAP to rank experts in each layer (computed from 1,024 calibration samples from evol-codealpaca-v1 ). Uniform prunes the same fraction of experts in every layer. Frequency-based ranks experts globally by routing frequency, counts how many experts from each layer fall in the tail 25%, and uses those counts to set layer-wise sparsity. Searched finds a non-uniform sparsity schedule with EvoESAP under the same budget. Numbers above panels report average performance on Code/Math/MC benchmarks, with deltas relative to uniform. The results imply that given the same pruning metric, non-uniform allocation has the potential to better preserve the model’s capabilities for SMoE expert pruning; however, finding an effective non-uniform allocation is non-trivial, and a poor allocation can harm overall performance. Three panels compare uniform, frequency-based, and EvoESAP-searched layer-wise expert density schedules for OLMoE at 25 percent global sparsity, together with average coding, math, and multiple-choice results.

Panel of Figure 2 — Figure 2. Overview of EvoESAP. (a) Evolutionary search with budget-preserving level-switch mutation. Histograms visualize the layer-wise density distribution of each candidate model (density $=1-\,$ sparsity) induced by an allocation $\mathbf{r}$ (experts removed per layer) under a fixed global budget $B$ . Offspring are generated from the top $m$ survivors by a level-switch that transfers $\Delta$ units of pruning budget between two layers (gray: decrease; yellow: increase), keeping $\sum_{\ell}r_{\ell}=B$ unchanged. (b) ESAP as per-sample fitness. Under teacher forcing, ESAP scores a sample by the full-vocabulary overlap between the target next-token distribution (dark gray, $p(\cdot\mid x)$ ) and the candidate/draft distribution (blue, $q(\cdot\mid x)$ ), averaged over answer-token positions. The ESAP score is defined in Equation 13 , and higher values are better. The left panel shows evolutionary search over layer-wise sparsity allocations under a fixed pruning budget. The right panel illustrates ESAP as overlap between target and candidate next-token distributions under teacher forcing.

				Coding			Creative Writing	Math			MC
Model	Sparsity	Method		Eval+	LiveCode	Avg	WildBench	GSM8K	MATH-500	Avg	MC Avg
OLMoE	Full			0.341	0.033	0.187	0.444	0.682	0.222	0.452	0.653
	25%	EAN	Uni	0.343	0.022	0.183	0.269	0.585	0.190	0.387	0.551
		EAN	ESAP	0.312	0.027	0.170	0.258	0.576	0.232	0.404	0.543
		SEER	Uni	0.339	0.027	0.183	0.253	0.577	0.204	0.390	0.545
		SEER	ESAP	0.306	0.022	0.164	0.254	0.601	0.248	0.424	0.539

Table 1. Benchmark results for OLMoE-1B-7B-0125-Instruct and ERNIE-4.5-21B-A3B-PT on coding, WildBench, math, and MC benchmarks. OLMoE is evaluated at 25% global sparsity, and ERNIE at 25% and 50%. Uni uses uniform layer-wise sparsity, while ESAP uses the non-uniform allocation found by EvoESAP. Bold indicates the better result within each pair.

				Coding			Creative Writing	Math			MC
Model	Sparsity	Method		Eval+	LiveCode	Avg	WildBench	GSM8K	MATH-500	Avg	MC Avg
DeepSeek	Full			0.549	0.104	0.327	0.418	0.610	0.298	0.454	0.678
	25%	EAN	Uni	0.459	0.071	0.265	0.259	0.531	0.224	0.378	0.571
		EAN	ESAP	0.417	0.071	0.244	0.301	0.572	0.224	0.398	0.588
		SEER	Uni	0.404	0.099	0.252	0.260	0.553	0.210	0.382	0.586
		SEER	ESAP	0.460	0.115	0.288	0.290	0.559	0.226	0.393	0.594

Table 2. Benchmark results for DeepSeek-V2-Lite-Chat and Qwen3-30B-A3B-Instruct-2507 on coding, WildBench, math, and multiple-choice benchmarks. DeepSeek is evaluated at 25% global sparsity, while Qwen3 is evaluated at 25% and 50%. Uni uses uniform layer-wise sparsity, while ESAP uses the non-uniform allocation found by EvoESAP.

Model	GPU	Generations	Time (h)	Memory (GB)
OLMoE-7B	1 $\times$ L40S	50	$5.8$	9.89 / 12.9
DeepSeek-16B	1 $\times$ L40S	30	$6.7$	22.57 / 29.27
ERNIE-21B	2 $\times$ L40S	20	$5.0$	31.17 / 40.66
Qwen3-30B	2 $\times$ L40S	10	$5.2$	43.42 / 56.92

Table 3. Search cost and model-loading GPU memory at 25% global sparsity. We report the GPU configuration, wall-clock search time, and the bfloat16 GPU memory required to load the pruned and full models. Pruning reduces model-loading memory by roughly 23%–24% across models.

Method	GPU	Time (h)	Code Avg	WildBench	MC Avg
DiEP	2 $\times$ L40S	11.25	0.114	0.208	0.495
Ours	1 $\times$ L40S	1.92	0.178	0.275	0.565

Table 4. Comparison with DiEP on OLMoE-7B at 25% global sparsity. Since DiEP jointly optimizes within-layer pruning order and layer-wise sparsity allocation, we include both calibration (0.75h) and search time (1.17h) in EvoESAP’s reported runtime to ensure a fair comparison.

		Coding			Creative Writing	Math			MC
Ablation	Value	Eval+	LiveCode	Code Avg	WildBench	GSM8K	MATH-500	Math Avg	MC Avg
Reference	REAP + Uni	0.314	0.005	0.160	0.292	0.596	0.200	0.398	0.579
Fitness	KL	0.331	0.016	0.174	0.285	0.595	0.216	0.405	0.582
	NLL	0.334	0.016	0.175	0.295	0.619	0.230	0.424	0.576
	SAP	0.339	0.005	0.172	0.289	0.622	0.218	0.420	0.584
	ESAP (Ours)	0.344	0.033	0.189	0.279	0.636	0.216	0.426	0.581

Table 5. Ablations of fitness choice and search hyperparameters at 25% global sparsity on OLMoE. We include the uniform REAP baseline as a reference. Blue cells mark the default configuration corresponding to the OLMoE 25% REAP+ ESAP result in Table 1 . Relative to the uniform REAP baseline, all searched settings improve both Code Avg and Math Avg. Among the searched fitness functions, ESAP gives the strongest Code Avg and Math Avg.

Fitness	GPU	Time (h)	Code Avg	WildBench	MC Avg
SPEC-DEC	2 $\times$ L40S	29.49	0.171	0.269	0.565
ESAP	1 $\times$ L40S	1.64	0.173	0.256	0.557

Table 6. Comparison of true speculative-decoding acceptance (SPEC-DEC) and the proposed ESAP. ESAP significantly reduces the search time from 29.49 hours to 1.64 hours.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

\tau\leftarrow\min\big(U\{1,\dots,\tau_{\max}\},\;U\{1,\dots,\tau_{\max}\}\big).

Appendix

Algorithm 1 Evolutionary Search for Non-uniform Layer-wise Sparsity Allocation

				Coding			Creative Writing	Math			MC
Model	Sparsity	Method		Eval+	LiveCode	Avg	WildBench	GSM8K	MATH-500	Avg	MC Avg
ERNIE	Full			0.867	0.247	0.557	0.479	0.829	0.780	0.804	0.721
	25%	EAN	Uni	0.248	0.049	0.148	0.412	0.670	0.366	0.518	0.685
		EAN	ESAP	0.354	0.093	0.223	0.433	0.752	0.438	0.595	0.669
		SEER	Uni	0.249	0.055	0.152	0.406	0.738	0.374	0.556	0.660
		SEER	ESAP	0.367	0.060	0.213	0.419	0.737	0.378	0.557	0.671

Appendix

Table 7. MC and open-ended generation benchmark results for ERNIE-4.5-21B-A3B-PT under 25% global sparsity. Pruning is calibrated on C4. ESAP is the non-uniform allocation searched on evol-codealpaca-v1 with the same pruning metric.

Panel of Figure 3 — Figure 3. Layer-wise density distributions (density $=1-\text{sparsity}$ ) of the searched non-uniform allocations across different pruning metrics. Two vertically stacked subfigures summarize searched layer-wise density schedules across pruning metrics. The 25 percent panel uses a placeholder if the local figure asset is unavailable, and the 50 percent panel shows the available searched density visualization.

				Coding
Model	Compression	Method		HumanEval	HumanEval $+$	MBPP	MBPP $+$	Eval+	LiveCodeBench	Code Avg
OLMoE-1B-7B- 0125-Instruct	Baseline			0.354	0.323	0.373	0.312	0.341	0.033	0.187
	25%	EAN	Uniform	0.366	0.341	0.360	0.307	0.343	0.022	0.183
		EAN	Searched	0.317	0.293	0.347	0.291	0.312	0.027	0.170
		SEER	Uniform	0.360	0.341	0.349	0.307	0.339	0.027	0.183
		SEER	Searched	0.311	0.274	0.347	0.294	0.306	0.022	0.164

Appendix

Table 8. Full results on coding benchmarks. Eval+ is the average of HumanEval, HumanEval+, MBPP and MBPP+.

				WildBench + Math
Model	Compression	Method		WildBench	GSM8K	MATH-500	Math Avg
OLMoE-1B-7B- 0125-Instruct	Baseline			0.444	0.682	0.222	0.452
	25%	EAN	Uniform	0.269	0.585	0.190	0.387
		EAN	Searched	0.258	0.576	0.232	0.404
		SEER	Uniform	0.253	0.577	0.204	0.390
		SEER	Searched	0.254	0.601	0.248	0.424

Appendix

Table 9. Full results on WildBench and math benchmarks.

				Multiple Choice
Model	Compression	Method		MMLU	ARC-C	ARC-E	HellaSwag	WinoGrande	BoolQ	OpenBookQA	RTE	MC Avg
OLMoE-1B-7B- 0125-Instruct	Baseline			0.534	0.490	0.758	0.808	0.684	0.766	0.470	0.711	0.653
	25%	EAN	Uniform	0.452	0.358	0.558	0.630	0.602	0.727	0.336	0.747	0.551
		EAN	Searched	0.454	0.349	0.573	0.628	0.597	0.716	0.326	0.700	0.543
		SEER	Uniform	0.457	0.352	0.566	0.620	0.599	0.723	0.340	0.708	0.545
		SEER	Searched	0.447	0.338	0.573	0.621	0.601	0.715	0.330	0.690	0.539

Appendix

Table 10. Full results on multiple-choice benchmarks.

Mixture-of-Experts Pruning

Mixture-of-Experts pruning removes or trims experts to reduce model memory and computation while preserving useful capabilities.

See all 59 papers

SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs
Oct 21, 2026Yifan Yang, Hanling Yi, Dongquan Yang +4Mixture-of-Experts Pruning Sparse Mixture-of-Experts
Intellifusion Inc.
Does a Global Perspective Help Prune Sparse MoEs Elegantly?
Apr 8, 2026Zeliang Zhang, Nikhil Ghosh, Jiani Liu +2Mixture-of-Experts Pruning Neural Network Pruning
University of Rochester · Flatiron Institute · University of California, Berkeley +1
It Takes a MAESTRO To Prune Bad Experts
Jul 9, 2026Palaash Goel, Ayush Maheshwari, Tanmoy ChakrabortyMixture-of-Experts Pruning LLM Pruning
Indian Institute of Technology Delhi, India · NVIDIA, India

Oct 21, 2026J/K move · Enter open · S save

SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs

Yifan Yang, Hanling Yi, Dongquan Yang, Feng Lin+3

Intellifusion Inc.

Mixture-of-Experts Pruning Sparse Mixture-of-Experts Mixture-of-Experts Models Mixture-of-Experts Inference Expert Routing

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) architectures scale model capacity efficiently, yet their substantial memory footprint poses a major barrier to deployment. Expert pruning reduces this footprint, but most methods focus on which experts to remove and prune every layer equally. We propose SPARE (Sparsity Allocation via Routing Entropy), a training-free, selector-agnostic method that decides how many experts to remove from each layer using routing entropy, the entropy of tokens' normalized Top-K gate weights on calibration data. When a token's gate weight is spread evenly, removing one active expert discards little of it, so SPARE prunes layers with more diffuse routing more aggressively. A perturbation bound under random pruning formalizes this intuition, and controlled interventions with a fixed selector link concentrated routing to larger output changes. With the same selector and budget, SPARE improves average accuracy over uniform allocation on Mixtral, Qwen3, and DeepSeek models, especially at high pruning ratios. At 50% expert pruning, it raises the eight-task average of Qwen3-30B-A3B from 57.2% to 63.8%, and the math, code, and science average of DeepSeek-R1 from 80.1 to 82.8.

Abstract

Mixture-of-Experts Pruning

Explore similar work

SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs

Does a Global Perspective Help Prune Sparse MoEs Elegantly?

It Takes a MAESTRO To Prune Bad Experts