Experts Exhibit Structured Redundancy: Structure-Preserving Expert Pruning for Mixture-of-Experts Large Language Models

Tian Wang; Yongding Tao; Yihong Dong; Xiaoming Wu; Ying Guo; Ge Li

cs.AIOct 30, 2026

Experts Exhibit Structured Redundancy: Structure-Preserving Expert Pruning for Mixture-of-Experts Large Language Models

Authors: Tian Wang, Yongding Tao, Yihong Dong, Xiaoming Wu, Ying Guo, Ge Li

Organizations: Peking University · Key Laboratory of High Confidence Software Technologies, Peking University, Ministry of Education · Shanghai Jiaotong University · Qilu University of Technology ·

Abstract

Sparse MoE mechanism significantly improves the trade-off between model capacity and inference efficiency. However, practical deployment still requires storing all experts, resulting in substantial memory overhead. We show that expert outputs in MoE layers exhibit structured redundancy. Across different models and task categories, the stable rank and entropy effective rank of expert outputs are consistently far below the nominal number of experts, while expert output distributions also exhibit fewer effective modes. Motivated by these observations, we reformulate expert pruning as a structure-preserving expert subset selection problem and efficiently optimize it via a greedy strategy, rather than only independently scoring and ranking individual experts or exhaustively searching over expert combinations. We propose two expert pruning methods under the unified structure-preserving framework: Projection Energy Expert Pruning (PEEP), which preserves the functional subspace by maximizing weighted projection energy, and Distributional Coverage Expert Pruning (DCEP), which preserves the distributional geometry by minimizing weighted coverage error. Both methods incorporate router-derived weights as priors, enabling them to jointly preserve the global structural relationships among experts while accounting for the local importance of individual experts. Experiments on DeepSeek-V2-Lite and Qwen3-30B-A3B demonstrate that, at the expert retention ratios of 75% and 50%, our methods outperform the state-of-the-art baselines on the majority of benchmarks and consistently achieve the best average performance across both general tasks and mathematical reasoning tasks.

Figures & tables

Figure 1: Ranks of expert representations and numbers of effective modes of expert output distributions across MoE layers. Left: DeepSeek-V2-Lite with 64 experts per layer. Right: Qwen3-30B-A3B with 128 experts per layer. Across all MoE layers, the entropy effective rank, stable rank, and number of effective modes remain well below the nominal number of experts, reaching at most 63%–65% of the total number of experts and thereby revealing substantial structured redundancy in expert outputs.

Figure 2: Overview of structure-preserving MoE expert pruning. Probing: collect expert outputs and router statistics on calibration dataset. Selection: PEEP maximizes projection energy; DCEP minimizes distributional coverage error. Evaluation: prune experts, restrict routing to the retained expert set and evaluate the pruned MoE LLMs on downstream tasks.

DeepSeek-V2-Lite
$r$	Method	General					Mathematical Reasoning
$r$	Method	HellaSwag	ARC-Challenge	MMLU	C-Eval	Average	GSM8K	MATH500	AIME24	AIME25	Average
100%	Unpruned	77.7	46.0	55.5	57.8	59.3	40.8	16.2	–	–	28.5
75%	Random	60.1	35.5	36.3	35.7	41.9	9.9	4.8	–	–	7.4
	Frequency	74.3	40.5	36.2	29.6	45.2	36.5	13.2	–	–	24.9
	EASY-EP	65.5	37.5	35.5	35.1	43.4	25.7	6.4	–	–	16.1

Table 1: Performance comparison of PEEP and DCEP with baseline expert pruning methods on DeepSeek-V2-Lite and Qwen3-30B-A3B across general and mathematical reasoning benchmarks at expert retention ratios of

r{=}75\%

and

r{=}50\%

. The best and second-best results among pruned methods under each model and expert retention ratio are highlighted in bold and underlined, respectively. “–” indicates that the model fails to complete the corresponding task.

Router Signals			DeepSeek-V2-Lite				Qwen3-30B-A3B
$freq_{i}^{(\ell)}$	$\hat{g}_{i}^{(\ell)}$	$\bar{g}_{i}^{(\ell)}$	75%		50%		75%		50%
$freq_{i}^{(\ell)}$	$\hat{g}_{i}^{(\ell)}$	$\bar{g}_{i}^{(\ell)}$	PEEP	DCEP	PEEP	DCEP	PEEP	DCEP	PEEP	DCEP
$\times$	$\times$	$\times$	43.6	42.6	29.8	29.4	70.0	70.0	46.1	51.0
$\checkmark$	$\checkmark$	$\checkmark$	42.2	43.9	30.0	30.4	75.5	75.0	57.7	58.3
$\checkmark$	$\times$	$\times$	42.9	43.1	30.5	29.4	75.0	74.0	57.1	58.8
$\times$	$\checkmark$	$\times$	43.2	43.0	24.9	29.4	70.2	72.4	55.4	54.8

Table 2: Ablation study of router-derived weights on DeepSeek-V2-Lite and Qwen3-30B-A3B. Columns

freq_{i}^{(\ell)}

\hat{g}_{i}^{(\ell)}

, and

\bar{g}_{i}^{(\ell)}

indicate whether activation frequency, mean conditional routing score, and mean routing score are included in the multiplicative weight construction. A

\checkmark

indicates that the corresponding router signal is included, whereas a

\times

indicates that it is excluded. For each model, performance is averaged over all evaluated tasks. The best and second-best results in each column are highlighted in bold and underlined, respectively.

Figure 3: Left : Performance comparison among different pruning methods with different calibration dataset sizes on DeepSeek-V2-Lite model and GSM8K task at expert retention ratio of

r{=}75\%

. Middle : Performance comparison among different pruning methods with different calibration dataset sizes on Qwen3-30B-A3B model and MMLU task at expert retention ratio of

r{=}50\%

. Right : Performance comparison among different pruning methods with different expert retention ratios on Qwen3-30B-A3B model and HellaSwag task.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

$r$	Router Signals			General					Mathematical Reasoning
$r$	$freq_{i}^{(\ell)}$	$\hat{g}_{i}^{(\ell)}$	$\bar{g}_{i}^{(\ell)}$	HellaSwag	ARC-Challenge	MMLU	C-Eval	Average	GSM8K	MATH500	AIME24	AIME25	Average
DeepSeek-V2-Lite
75%	$\times$	$\times$	$\times$	73.9	41.6	47.2	46.3	52.2	37.0	15.8	–	–	26.4
	$\checkmark$	$\checkmark$	$\checkmark$	75.2	41.0	45.2	41.9	50.8	36.8	13.0	–	–	24.9
	$\checkmark$	$\times$	$\times$	76.1	43.3	45.2	39.5	51.0	39.9	13.2	–	–	26.5
	$\times$	$\checkmark$	$\times$	75.3	44.8	45.5	46.6	53.0	36.9	10.2	–	–	23.6

Appendix

Table 3: Ablation study of router-derived weights for PEEP on DeepSeek-V2-Lite and Qwen3-30B-A3B. Columns

freq_{i}^{(\ell)}

\hat{g}_{i}^{(\ell)}

, and

\bar{g}_{i}^{(\ell)}

indicate whether activation frequency, mean conditional routing score, and mean routing score are included in the multiplicative weight construction. “

\checkmark

” indicates that the corresponding router signal is included in the weights, whereas “

\times

” indicates that it is not included. The best and second-best results among pruned methods under each model and expert retention ratio are highlighted in bold and underlined, respectively. “–” indicates that the model fails to complete the corresponding task.

$r$	Router Signals			General					Mathematical Reasoning
$r$	$freq_{i}^{(\ell)}$	$\hat{g}_{i}^{(\ell)}$	$\bar{g}_{i}^{(\ell)}$	HellaSwag	ARC-Challenge	MMLU	C-Eval	Average	GSM8K	MATH500	AIME24	AIME25	Average
DeepSeek-V2-Lite
75%	$\times$	$\times$	$\times$	75.4	40.4	44.6	43.7	51.0	38.5	12.8	–	–	25.7
	$\checkmark$	$\checkmark$	$\checkmark$	75.5	43.1	47.1	47.8	53.4	36.8	13.0	–	–	24.9
	$\checkmark$	$\times$	$\times$	76.1	41.8	44.3	42.7	51.2	40.8	12.6	–	–	26.7
	$\times$	$\checkmark$	$\times$	75.0	44.1	46.2	44.7	52.5	37.4	10.6	–	–	24.0

Appendix

Table 4: Ablation study of router-derived weights for DCEP on DeepSeek-V2-Lite and Qwen3-30B-A3B. Columns

freq_{i}^{(\ell)}

\hat{g}_{i}^{(\ell)}

, and

\bar{g}_{i}^{(\ell)}

indicate whether activation frequency, mean conditional routing score, and mean routing score are included in the multiplicative weight construction. “

\checkmark

” indicates that the corresponding router signal is included in the weights, whereas “

\times

Figure 4: Performance comparison among different pruning methods with different calibration dataset sizes at expert retention ratio of

r{=}75\%

. Results are averaged over five runs with calibration datasets independently sampled using different random seeds. The shaded regions represent

\pm

one standard deviation. Top : DeepSeek-V2-Lite. Bottom : Qwen3-30B-A3B. From left to right : GSM8K, HellaSwag, MMLU, ARC-Challenge, and C-Eval.

Figure 5: Performance comparison among different pruning methods with different calibration dataset sizes at expert retention ratio of

r{=}50\%

. Results are averaged over five runs with calibration datasets independently sampled using different random seeds. The shaded regions represent

\pm

one standard deviation. Top : DeepSeek-V2-Lite. Bottom : Qwen3-30B-A3B. From left to right : GSM8K, HellaSwag, MMLU, ARC-Challenge, and C-Eval.

Figure 6: Performance comparison among different pruning methods with different expert retention ratios. Top : DeepSeek-V2-Lite. Bottom : Qwen3-30B-A3B. From left to right : GSM8K, HellaSwag, MMLU, ARC-Challenge, and C-Eval.

Figure 7: Numbers of effective modes of expert output distributions across MoE layers under different values of scaling factor

c

. Left: DeepSeek-V2-Lite with 64 experts per layer. Right: Qwen3-30B-A3B with 128 experts per layer. Across all MoE layers and different values of scaling factor

c

, the number of effective modes remains well below the nominal number of experts.

Mixture-of-Experts Pruning

Mixture-of-Experts pruning removes or trims experts to reduce model memory and computation while preserving useful capabilities.

See all 61 papers

Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
Oct 19, 2026Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin +7Mixture-of-Experts Pruning Mixture-of-Experts Language Models
LinkedIn · MIT · Google
MoRA: MoE Pruning via Router Bias Learning and Expert Approximation
Sep 30, 2026Yushuai Sun, Zikun Zhou, Lin Gao +2Mixture-of-Experts Pruning LLM Compression
Harbin Institute of Technology, Shenzhen
It Takes a MAESTRO To Prune Bad Experts
Jul 9, 2026Palaash Goel, Ayush Maheshwari, Tanmoy ChakrabortyMixture-of-Experts Pruning LLM Pruning
Indian Institute of Technology Delhi, India · NVIDIA, India

Oct 19, 2026J/K move · Enter open · S save

Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models

Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin, Jincheng Cao+6

LinkedIn · MIT · Google

Mixture-of-Experts Pruning Mixture-of-Experts Language Models LLM Compression Neural Network Pruning

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters. We study post-training expert pruning using task-domain calibration data as a subset-selection problem: given a pretrained MoE layer, select a fixed number of experts that best reconstruct its original output. Exact subset search is combinatorial and becomes intractable for modern MoE models with many experts per layer. We develop OptEP, a scalable continuous relaxation that directly optimizes differentiable expert-selection variables, followed by a lightweight reconstruction-based update of the retained expert weights. Across multiple MoE architectures and tasks spanning mathematical reasoning and code generation, our approach improves over the strongest existing expert-pruning baselines, with absolute accuracy gains of up to 15.05% on Qwen3-30B-A3B, 14.55% on Qwen3-235B-A22B, 11.06% on GLM-4.5-Air, and 16.20% on Mixtral-8x7B. The improvements are particularly pronounced under aggressive pruning, demonstrating that reconstruction-driven expert selection and weight adaptation can preserve substantially more of the original model's performance while reducing the number of retained experts.

Abstract

Mixture-of-Experts Pruning

Explore similar work

Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models

MoRA: MoE Pruning via Router Bias Learning and Expert Approximation

It Takes a MAESTRO To Prune Bad Experts