Experts Exhibit Structured Redundancy: Structure-Preserving Expert Pruning for Mixture-of-Experts Large Language Models

Tian Wang; Yongding Tao; Yihong Dong; Xiaoming Wu; Ying Guo; Ge Li

cs.AIOct 30, 2026

Experts Exhibit Structured Redundancy: Structure-Preserving Expert Pruning for Mixture-of-Experts Large Language Models

Authors: Tian Wang, Yongding Tao, Yihong Dong, Xiaoming Wu, Ying Guo, Ge Li

Organizations: Peking University · Key Laboratory of High Confidence Software Technologies, Peking University, Ministry of Education · Shanghai Jiaotong University · Qilu University of Technology ·

Abstract

Sparse MoE mechanism significantly improves the trade-off between model capacity and inference efficiency. However, practical deployment still requires storing all experts, resulting in substantial memory overhead. We show that expert outputs in MoE layers exhibit structured redundancy. Across different models and task categories, the stable rank and entropy effective rank of expert outputs are consistently far below the nominal number of experts, while expert output distributions also exhibit fewer effective modes. Motivated by these observations, we reformulate expert pruning as a structure-preserving expert subset selection problem and efficiently optimize it via a greedy strategy, rather than only independently scoring and ranking individual experts or exhaustively searching over expert combinations. We propose two expert pruning methods under the unified structure-preserving framework: Projection Energy Expert Pruning (PEEP), which preserves the functional subspace by maximizing weighted projection energy, and Distributional Coverage Expert Pruning (DCEP), which preserves the distributional geometry by minimizing weighted coverage error. Both methods incorporate router-derived weights as priors, enabling them to jointly preserve the global structural relationships among experts while accounting for the local importance of individual experts. Experiments on DeepSeek-V2-Lite and Qwen3-30B-A3B demonstrate that, at the expert retention ratios of 75% and 50%, our methods outperform the state-of-the-art baselines on the majority of benchmarks and consistently achieve the best average performance across both general tasks and mathematical reasoning tasks.

Figures & tables

Figure 1: Ranks of expert representations and numbers of effective modes of expert output distributions across MoE layers. Left: DeepSeek-V2-Lite with 64 experts per layer. Right: Qwen3-30B-A3B with 128 experts per layer. Across all MoE layers, the entropy effective rank, stable rank, and number of effective modes remain well below the nominal number of experts, reaching at most 63%–65% of the total number of experts and thereby revealing substantial structured redundancy in expert outputs.

Figure 2: Overview of structure-preserving MoE expert pruning. Probing: collect expert outputs and router statistics on calibration dataset. Selection: PEEP maximizes projection energy; DCEP minimizes distributional coverage error. Evaluation: prune experts, restrict routing to the retained expert set and evaluate the pruned MoE LLMs on downstream tasks.

DeepSeek-V2-Lite
$r$	Method	General					Mathematical Reasoning
$r$	Method	HellaSwag	ARC-Challenge	MMLU	C-Eval	Average	GSM8K	MATH500	AIME24	AIME25	Average
100%	Unpruned	77.7	46.0	55.5	57.8	59.3	40.8	16.2	–	–	28.5
75%	Random	60.1	35.5	36.3	35.7	41.9	9.9	4.8	–	–	7.4
	Frequency	74.3	40.5	36.2	29.6	45.2	36.5	13.2	–	–	24.9
	EASY-EP	65.5	37.5	35.5	35.1	43.4	25.7	6.4	–	–	16.1

Table 1: Performance comparison of PEEP and DCEP with baseline expert pruning methods on DeepSeek-V2-Lite and Qwen3-30B-A3B across general and mathematical reasoning benchmarks at expert retention ratios of

r{=}75\%

and

r{=}50\%

. The best and second-best results among pruned methods under each model and expert retention ratio are highlighted in bold and underlined, respectively. “–” indicates that the model fails to complete the corresponding task.

Router Signals			DeepSeek-V2-Lite				Qwen3-30B-A3B
$freq_{i}^{(\ell)}$	$\hat{g}_{i}^{(\ell)}$	$\bar{g}_{i}^{(\ell)}$	75%		50%		75%		50%
$freq_{i}^{(\ell)}$	$\hat{g}_{i}^{(\ell)}$	$\bar{g}_{i}^{(\ell)}$	PEEP	DCEP	PEEP	DCEP	PEEP	DCEP	PEEP	DCEP
$\times$	$\times$	$\times$	43.6	42.6	29.8	29.4	70.0	70.0	46.1	51.0
$\checkmark$	$\checkmark$	$\checkmark$	42.2	43.9	30.0	30.4	75.5	75.0	57.7	58.3
$\checkmark$	$\times$	$\times$	42.9	43.1	30.5	29.4	75.0	74.0	57.1	58.8
$\times$	$\checkmark$	$\times$	43.2	43.0	24.9	29.4	70.2	72.4	55.4	54.8

Table 2: Ablation study of router-derived weights on DeepSeek-V2-Lite and Qwen3-30B-A3B. Columns

freq_{i}^{(\ell)}

\hat{g}_{i}^{(\ell)}

, and

\bar{g}_{i}^{(\ell)}

indicate whether activation frequency, mean conditional routing score, and mean routing score are included in the multiplicative weight construction. A

\checkmark

indicates that the corresponding router signal is included, whereas a

\times

indicates that it is excluded. For each model, performance is averaged over all evaluated tasks. The best and second-best results in each column are highlighted in bold and underlined, respectively.

Figure 3: Left : Performance comparison among different pruning methods with different calibration dataset sizes on DeepSeek-V2-Lite model and GSM8K task at expert retention ratio of

r{=}75\%

. Middle : Performance comparison among different pruning methods with different calibration dataset sizes on Qwen3-30B-A3B model and MMLU task at expert retention ratio of

r{=}50\%

. Right : Performance comparison among different pruning methods with different expert retention ratios on Qwen3-30B-A3B model and HellaSwag task.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

$r$	Router Signals			General					Mathematical Reasoning
$r$	$freq_{i}^{(\ell)}$	$\hat{g}_{i}^{(\ell)}$	$\bar{g}_{i}^{(\ell)}$	HellaSwag	ARC-Challenge	MMLU	C-Eval	Average	GSM8K	MATH500	AIME24	AIME25	Average
DeepSeek-V2-Lite
75%	$\times$	$\times$	$\times$	73.9	41.6	47.2	46.3	52.2	37.0	15.8	–	–	26.4
	$\checkmark$	$\checkmark$	$\checkmark$	75.2	41.0	45.2	41.9	50.8	36.8	13.0	–	–	24.9
	$\checkmark$	$\times$	$\times$	76.1	43.3	45.2	39.5	51.0	39.9	13.2	–	–	26.5
	$\times$	$\checkmark$	$\times$	75.3	44.8	45.5	46.6	53.0	36.9	10.2	–	–	23.6

Appendix

Table 3: Ablation study of router-derived weights for PEEP on DeepSeek-V2-Lite and Qwen3-30B-A3B. Columns

freq_{i}^{(\ell)}

\hat{g}_{i}^{(\ell)}

, and

\bar{g}_{i}^{(\ell)}

indicate whether activation frequency, mean conditional routing score, and mean routing score are included in the multiplicative weight construction. “

\checkmark

” indicates that the corresponding router signal is included in the weights, whereas “

\times

” indicates that it is not included. The best and second-best results among pruned methods under each model and expert retention ratio are highlighted in bold and underlined, respectively. “–” indicates that the model fails to complete the corresponding task.

$r$	Router Signals			General					Mathematical Reasoning
$r$	$freq_{i}^{(\ell)}$	$\hat{g}_{i}^{(\ell)}$	$\bar{g}_{i}^{(\ell)}$	HellaSwag	ARC-Challenge	MMLU	C-Eval	Average	GSM8K	MATH500	AIME24	AIME25	Average
DeepSeek-V2-Lite
75%	$\times$	$\times$	$\times$	75.4	40.4	44.6	43.7	51.0	38.5	12.8	–	–	25.7
	$\checkmark$	$\checkmark$	$\checkmark$	75.5	43.1	47.1	47.8	53.4	36.8	13.0	–	–	24.9
	$\checkmark$	$\times$	$\times$	76.1	41.8	44.3	42.7	51.2	40.8	12.6	–	–	26.7
	$\times$	$\checkmark$	$\times$	75.0	44.1	46.2	44.7	52.5	37.4	10.6	–	–	24.0

Appendix

Table 4: Ablation study of router-derived weights for DCEP on DeepSeek-V2-Lite and Qwen3-30B-A3B. Columns

freq_{i}^{(\ell)}

\hat{g}_{i}^{(\ell)}

, and

\bar{g}_{i}^{(\ell)}

indicate whether activation frequency, mean conditional routing score, and mean routing score are included in the multiplicative weight construction. “

\checkmark

” indicates that the corresponding router signal is included in the weights, whereas “

\times

Figure 4: Performance comparison among different pruning methods with different calibration dataset sizes at expert retention ratio of

r{=}75\%

. Results are averaged over five runs with calibration datasets independently sampled using different random seeds. The shaded regions represent

\pm

one standard deviation. Top : DeepSeek-V2-Lite. Bottom : Qwen3-30B-A3B. From left to right : GSM8K, HellaSwag, MMLU, ARC-Challenge, and C-Eval.

Figure 5: Performance comparison among different pruning methods with different calibration dataset sizes at expert retention ratio of

r{=}50\%

. Results are averaged over five runs with calibration datasets independently sampled using different random seeds. The shaded regions represent

\pm

one standard deviation. Top : DeepSeek-V2-Lite. Bottom : Qwen3-30B-A3B. From left to right : GSM8K, HellaSwag, MMLU, ARC-Challenge, and C-Eval.

Figure 6: Performance comparison among different pruning methods with different expert retention ratios. Top : DeepSeek-V2-Lite. Bottom : Qwen3-30B-A3B. From left to right : GSM8K, HellaSwag, MMLU, ARC-Challenge, and C-Eval.

Figure 7: Numbers of effective modes of expert output distributions across MoE layers under different values of scaling factor

c

. Left: DeepSeek-V2-Lite with 64 experts per layer. Right: Qwen3-30B-A3B with 128 experts per layer. Across all MoE layers and different values of scaling factor

c

, the number of effective modes remains well below the nominal number of experts.

Mixture-of-Experts Pruning

Mixture-of-Experts pruning removes or trims experts to reduce model memory and computation while preserving useful capabilities.

See all 61 papers

Oct 19, 2026

Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models

Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters. We study post-training expert pruning using task-domain calibration data as a subset-selection problem: given a pretrained MoE layer, select a fixed number of experts that best reconstruct its original output. Exact subset search is combinatorial and becomes intractable for modern MoE models with many experts per layer. We develop OptEP, a scalable continuous relaxation that directly optimizes differentiable expert-selection variables, followed by a lightweight reconstruction-based update of the retained expert weights. Across multiple MoE architectures and tasks spanning mathematical reasoning and code generation, our approach improves over the strongest existing expert-pruning baselines, with absolute accuracy gains of up to 15.05% on Qwen3-30B-A3B, 14.55% on Qwen3-235B-A22B, 11.06% on GLM-4.5-Air, and 16.20% on Mixtral-8x7B. The improvements are particularly pronounced under aggressive pruning, demonstrating that reconstruction-driven expert selection and weight adaptation can preserve substantially more of the original model's performance while reducing the number of retained experts.

Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin +7

LinkedIn · MIT · Google

Mixture-of-Experts Pruning Mixture-of-Experts Language Models LLM Compression Neural Network Pruning

Ask PDF

Sep 30, 2026

MoRA: MoE Pruning via Router Bias Learning and Expert Approximation

Mixture-of-Experts (MoE) models enable parameter scaling with limited per-token computation by activating only a small subset of experts for each token, but deploying them still requires loading the complete expert pool into memory. Structured expert pruning can effectively reduce the memory usage by removing experts. However, existing pruning methods either use expert ranking criteria that are not well aligned with model performance or rely on effective expert subset searching that is computationally expensive. Moreover, these methods typically overlook the routing-behavior redundancy among the retained experts. In this paper, we propose MoE Pruning via Router Bias Learning and Expert Approximation (MoRA), a framework for structured MoE expert pruning. We introduce a learnable router bias for each expert and optimize these biases by minimizing the language-modeling loss and a routing-diversity regularizer. The learned router biases sharpen the routing probability distributions to identify experts critical to model performance while encouraging the selection of experts with diverse routing preferences. In addition, we introduce an expert approximation mechanism as a post-pruning enhancement. It leverages the remaining experts to approximate the outputs of pruned experts by affine transformation, further improving the performance of the pruned model. We evaluate MoRA on Qwen3-30B-A3B, DeepSeek-V2-Lite, and Moonlight-16B-A3B, removing 25% and 50% of the routed experts in each MoE layer. Extensive experiments on nine zero-shot benchmarks show that MoRA outperforms state-of-the-art pruning algorithms. Our code will be released.

Yushuai Sun, Zikun Zhou, Lin Gao +2

Harbin Institute of Technology, Shenzhen

Mixture-of-Experts Pruning LLM Compression Expert Routing Mixture-of-Experts Models

Ask PDF

Jul 9, 2026

It Takes a MAESTRO To Prune Bad Experts

Sparsely-activated Mixture-of-Experts (MoE) language models achieve remarkable inference efficiency by activating only a small fraction of parameters per token, yet their full expert banks reside in memory at all times, creating a prohibitive deployment bottleneck. Existing structured pruning methods, largely designed for dense transformers, assess expert importance using locally derived heuristics that are blind to the interdependent nature of MoE routing. We introduce MAESTRO (Markov-chain Approximated Expert Sparsification via Transition-based ROuting), a structured pruning framework designed for MoE architectures that models autoregressive expert activation trajectories as Ergodic Markov chains whose stationary distributions encode cross-layer dependencies, yielding a globally aware importance heuristic. Evaluated across five diverse domains including Safety, Bias, and Ethics, MAESTRO outperforms state-of-the-art baselines by up to 10.61% in average performance retention under a strict 50% compression regime, while exhibiting substantially lower cross-task variance, indicating that global, routing-congruent pruning produces models that generalize more consistently across heterogeneous tasks.

Palaash Goel, Ayush Maheshwari, Tanmoy Chakraborty

Indian Institute of Technology Delhi, India · NVIDIA, India

Mixture-of-Experts Pruning LLM Pruning Mixture-of-Experts Language Models LLM Compression Activation Sparsity Structured Sparsity+1

Ask PDF

Abstract

Mixture-of-Experts Pruning

Explore similar work

Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models

MoRA: MoE Pruning via Router Bias Learning and Expert Approximation

It Takes a MAESTRO To Prune Bad Experts