SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs

Yifan Yang; Hanling Yi; Dongquan Yang; Feng Lin; Rong Xiao; Aijun Li; Xiaotian Yu

cs.LGOct 21, 2026

SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs

Authors: Yifan Yang, Hanling Yi, Dongquan Yang, Feng Lin, Rong Xiao, Aijun Li, Xiaotian Yu

Organizations: Intellifusion Inc.

Abstract

Mixture-of-Experts (MoE) architectures scale model capacity efficiently, yet their substantial memory footprint poses a major barrier to deployment. Expert pruning reduces this footprint, but most methods focus on which experts to remove and prune every layer equally. We propose SPARE (Sparsity Allocation via Routing Entropy), a training-free, selector-agnostic method that decides how many experts to remove from each layer using routing entropy, the entropy of tokens' normalized Top-K gate weights on calibration data. When a token's gate weight is spread evenly, removing one active expert discards little of it, so SPARE prunes layers with more diffuse routing more aggressively. A perturbation bound under random pruning formalizes this intuition, and controlled interventions with a fixed selector link concentrated routing to larger output changes. With the same selector and budget, SPARE improves average accuracy over uniform allocation on Mixtral, Qwen3, and DeepSeek models, especially at high pruning ratios. At 50% expert pruning, it raises the eight-task average of Qwen3-30B-A3B from 57.2% to 63.8%, and the math, code, and science average of DeepSeek-R1 from 80.1 to 82.8.

Figures & tables

Figure 1: Visualization of routing heterogeneity and domain sensitivity on Qwen3-30B-A3B. Left: gating distributions for two representative tokens at selected layers. These single-token examples illustrate diffuse and concentrated routing; quantitative analyses use layer-aggregated statistics. Right: expert selection frequencies on C4 and CodeAlpaca. The selected expert identities can differ substantially across domains, which makes pruning directly from single-domain expert frequencies brittle. SPARE instead uses layer-wise routing entropy, an aggregated concentration statistic, to guide inter-layer sparsity allocation.

Model	PR	Inter.	Intra.	ARC-c	ARC-e	BoolQ	Hella	MMLU	OBQA	RTE	WinG	Avg. $\uparrow$
Mixtral-8x7B -Instruct-v0.1	0%	Baseline		65.0	84.2	88.7	86.1	69.1	49.6	72.2	74.0	73.6
	25%	Uni.	Random	59.6	81.4	86.9	82.9	62.7	46.8	71.1	75.6	70.9
		Uni.	REAP	60.6	80.7	87.5	83.5	63.3	48.6	71.9	70.9	70.9
		SPARE	Random	60.1	81.9	87.1	83.2	63.1	47.5	71.9	75.8	71.3 ( $\uparrow$ 0.4)
		SPARE	REAP	61.8	81.5	87.9	84.0	64.5	49.2	71.9	72.1	71.6 ( $\uparrow$ 0.7 )
	50%	Uni.	Random	51.4	73.3	81.5	75.7	52.1	41.0	70.2	69.5	64.3

Table 1: Benchmark accuracy (%) and unweighted eight-task averages under expert pruning. PR denotes the global pruning ratio. Inter. : uniform (Uni.) or SPARE budget allocation. Intra. : Random, REAP, or EASY-EP expert selection. Bold indicates the best score within each model/PR setting; gains in Avg. compare SPARE with uniform allocation using the same selector.

Model	PR	Inter.	LCB	Math	GPQA	Avg. $\uparrow$
DS-R1	–	Baseline	84.5	96.0	69.7	83.4
	50%	Uni.	79.0	94.7	66.7	80.1
	50%	SPARE	82.0	94.3	72.2	82.8 ( $\uparrow$ 2.7)
Qwen3	–	Baseline	92.5	96.0	77.5	88.7
	50%	Uni.	89.0	95.7	74.4	86.4
	50%	SPARE	93.0	95.7	75.1	87.9 ( $\uparrow$ 1.5)

Table 2: Large-scale MoEs at 50% global expert pruning (PR), using EASY-EP for both allocation rules. DS-R1: DeepSeek-R1; Qwen3: Qwen3-235B-A22B. LCB/Math/GPQA: LiveCodeBench/MATH-500/GPQA-Diamond. Avg. is the unweighted three-score mean; gains compare with uniform allocation.

Domain	High	Middle	Low
WikiText2	0.721	0.641	0.726
MATH-500	0.673	0.736	0.728
HumanEval	0.726	0.725	0.763

Table 3: Normalized output perturbation on Mixtral-8

\times

7B under the same REAP selection procedure. Columns denote layer entropy groups.

Model	PR	Mean-domain KL	Worst-domain KL
DeepSeek-V2-Lite	25%	+0.435	+0.450
DeepSeek-V2-Lite	50%	+0.340	+0.272
Qwen3-30B-A3B	25%	+0.288	+0.257
Qwen3-30B-A3B	50%	+0.314	+0.329

Table 4: Pearson correlation between routing concentration

C_{\ell}=1-\bar{H}_{\ell}/\log K

and pruning-induced KL under single-layer REAP pruning, computed over

n=26

(DeepSeek-V2-Lite) and

n=48

(Qwen3-30B-A3B) MoE layers. PR is the pruning ratio of the intervened layer; KL is truncated to the teacher’s top-64 tokens.

Panel of Figure 2 — Figure 2: Layer-wise allocation profiles on Qwen3-30B-A3B with EASY-EP at 50% global expert pruning. Left: varying $p$ with $\beta=0$ changes the contrast between more- and less-pruned layers. Right: varying $\beta\in[0,0.6]$ with $p=2.0$ produces similar profiles.

Model	Calibration analysis	Pearson $r$	Spearman $\rho$
DeepSeek-R1	C4 vs. MATH-500	0.988	0.962
DeepSeek-R1	C4 vs. LiveCodeBench	0.991	0.974
DeepSeek-R1	C4 vs. GPQA-Diamond	0.979	0.958

Table 5: Cross-domain correlations of layer-average routing entropy on DeepSeek-R1, using C4 as the reference calibration domain.

	$p$						$\beta$
Param	0.8	1.0	1.5	2.0	2.5	Param	0.0	0.2	0.4	0.6
Avg. $\uparrow$	61.9	61.5	62.8	63.2	58.5	Avg. $\uparrow$	63.2	63.1	63.1	63.8

Table 6: Hyper-parameter ablation. Qwen3-30B-A3B with EASY-EP at 50% global pruning. Unweighted eight-task mean accuracy (%); the

p

sweep fixes

\beta=0

, and the

\beta

sweep fixes

p=2.0

. Bold marks the best score within each sweep.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Routed Experts	Top- $K$	Shared	Layers
DeepSeek-R1 (671B)	256	8	Yes (1)	61
Qwen3-30B-A3B	128	8	No	48
Qwen3-235B-A22B	128	8	No	94
GLM-5	256	8	Yes (1)	78

Appendix

Table 7: Architectural configurations of fine-grained MoEs. We compare the routed expert count, routing top-

K

, shared expert design, and layer count across the evaluated MoE models.

Batch size	1	16	32	64
Uniform	8.61	172.32	181.54	162.31
SPARE	8.51	172.57	182.56	162.24

Appendix

Table 8: Throughput under different batch sizes.

Figure 3: Average benchmark accuracy versus global expert-pruning ratio on Qwen3-30B-A3B.

Mixture-of-Experts Pruning

Mixture-of-Experts pruning removes or trims experts to reduce model memory and computation while preserving useful capabilities.

See all 59 papers

It Takes a MAESTRO To Prune Bad Experts
Jul 9, 2026Palaash Goel, Ayush Maheshwari, Tanmoy ChakrabortyMixture-of-Experts Pruning LLM Pruning
Indian Institute of Technology Delhi, India · NVIDIA, India
MoRA: MoE Pruning via Router Bias Learning and Expert Approximation
Sep 30, 2026Yushuai Sun, Zikun Zhou, Lin Gao +2Mixture-of-Experts Pruning LLM Compression
Harbin Institute of Technology, Shenzhen
Does a Global Perspective Help Prune Sparse MoEs Elegantly?
Apr 8, 2026Zeliang Zhang, Nikhil Ghosh, Jiani Liu +2Mixture-of-Experts Pruning Neural Network Pruning
University of Rochester · Flatiron Institute · University of California, Berkeley +1

Jul 9, 2026J/K move · Enter open · S save

It Takes a MAESTRO To Prune Bad Experts

Palaash Goel, Ayush Maheshwari, Tanmoy Chakraborty

Indian Institute of Technology Delhi, India · NVIDIA, India

Mixture-of-Experts Pruning LLM Pruning Mixture-of-Experts Language Models LLM Compression Activation Sparsity Structured Sparsity+1

Open paper AskPDF arXiv

Sparsely-activated Mixture-of-Experts (MoE) language models achieve remarkable inference efficiency by activating only a small fraction of parameters per token, yet their full expert banks reside in memory at all times, creating a prohibitive deployment bottleneck. Existing structured pruning methods, largely designed for dense transformers, assess expert importance using locally derived heuristics that are blind to the interdependent nature of MoE routing. We introduce MAESTRO (Markov-chain Approximated Expert Sparsification via Transition-based ROuting), a structured pruning framework designed for MoE architectures that models autoregressive expert activation trajectories as Ergodic Markov chains whose stationary distributions encode cross-layer dependencies, yielding a globally aware importance heuristic. Evaluated across five diverse domains including Safety, Bias, and Ethics, MAESTRO outperforms state-of-the-art baselines by up to 10.61% in average performance retention under a strict 50% compression regime, while exhibiting substantially lower cross-task variance, indicating that global, routing-congruent pruning produces models that generalize more consistently across heterogeneous tasks.

Abstract

Mixture-of-Experts Pruning

Explore similar work

It Takes a MAESTRO To Prune Bad Experts

MoRA: MoE Pruning via Router Bias Learning and Expert Approximation

Does a Global Perspective Help Prune Sparse MoEs Elegantly?