AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning

Zongfang Liu; Guangyi Chen; Shengkun Tang; Yifan Shen; Huan Wang; Xin Yuan

cs.LGMar 19, 2026

AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning

Authors: Zongfang Liu, Guangyi Chen, Shengkun Tang, Yifan Shen, Huan Wang, Xin Yuan

Organizations: Zhejiang University · Westlake University · Carnegie Mellon University · Mohamed bin Zayed University of Artificial Intelligence

Abstract

Mixture-of-Experts (MoE) language models increase parameter capacity without proportional per-token computation, yet deployment still requires storing the full expert pool, making expert pruning important for reducing memory and serving overhead. Existing task-agnostic expert-pruning methods are typically calibration-dependent: they estimate expert importance from routing or activation statistics on a calibration set, making pruning decisions sensitive to calibration-data variation while introducing substantial preprocessing cost. We propose AIMER (Absolute mean over root mean square IMportance for Expert Ranking), a simple calibration-free criterion that identifies more distinct experts by capturing the concentration pattern of expert weights, making it well suited for task-agnostic expert pruning. Across 7B to 47B MoE language models with distinct architectures and 16 diverse benchmarks, AIMER consistently delivers stronger capability balance across diverse tasks than existing calibration-free methods. Surprisingly, AIMER also achieves better balance than strong calibration-based expert-pruning baselines calibrated on the widely used task-agnostic C4 corpus, while requiring only 0.22--2.06 seconds to score all experts.

Figures & tables

Figure 1: Sensitivity of REAP Lasby et al. (2025) to calibration set size on Qwen3-30B at 50% pruning ratio. We fix the calibration corpus to C4 Allen Institute for AI (2024) and vary only the size of the calibration set from 0.5M to 2.1M tokens. The horizontal axis reports calibration tokens in millions (M = million tokens), and the vertical axis reports performance change relative to the 0.5M-token setting (pp = percentage points). Half of the benchmarks show significant variation. Some benchmarks improve while others degrade, showing that performance is highly sensitive to calibration set size even with the same corpus.

Figure 2: Change in expert distinctiveness after pruning. For each model and pruning ratio, we use 4096 C4 tokens to compute pairwise CKA similarities among expert outputs within each layer. Each cell reports the layer-averaged relative decrease in mean CKA after pruning, $(\mathrm{CKA}_{\mathrm{all}}-\mathrm{CKA}_{\mathrm{kept}})/\mathrm{CKA}_{\mathrm{all}}$ . Positive values indicate that the retained experts are less similar to one another than the original expert pool, i.e., more functionally distinct; negative values indicate increased redundancy among retained experts. AIMER (Ours) yields positive distinctiveness gains in all shown settings and is strongest or near-strongest in most of them.

Method	Data	Act.	Route
Frequency	$\checkmark$	$\times$	$\checkmark$
SEER	$\checkmark$	$\times$	$\checkmark$
EAN	$\checkmark$	$\checkmark$	$\times$
REAP	$\checkmark$	$\checkmark$	$\checkmark$
AIMER (Ours)	$\times$	$\times$	$\times$

Table 1: Comparison of statistics requirements for AIMER and calibration-based baselines. Data = calibration set, Act. = expert activations, and Route = router weights. AIMER ranks experts from pretrained weights alone, so none of these extra signals are required. Red marks required; green marks not required.

			Coding			Creative Writing	Math			MC	Rank
Model	Ratio	Method	Eval+	LiveCode	Code Avg	WildBench	GSM8K	MATH-500	Math Avg	MC Avg	Rank Avg
OLMoE	0%	Full	0.341	0.033	0.187	0.444	0.682	0.222	0.452	0.653	N/A
	25%	Frequency	0.000	0.000	0.000	0.127	0.033	0.024	0.028	0.560	6.67
		SEER	0.000	0.000	0.000	0.141	0.037	0.012	0.025	0.564	6.00
		EAN	0.000	0.000	0.000	0.205	0.115	0.006	0.061	0.577	5.50
		REAP	0.014	0.000	0.007	0.260	0.202	0.056	0.129	0.588	2.92

Table 2: Expert-pruning results on OLMoE-1B-7B-0125-Instruct (OLMoE) and ERNIE-4.5-21B-A3B-PT (ERNIE). AIMER is compared with calibration-based baselines calibrated on C4 and with calibration-free baselines. It achieves the best rank average in both settings, indicating the strongest overall capability balance.

			Coding			Creative Writing	Math			MC	Rank
Model	Ratio	Method	Eval+	LiveCode	Code Avg	WildBench	GSM8K	MATH-500	Math Avg	MC Avg	Rank Avg
DeepSeek	0%	Full	0.549	0.104	0.327	0.418	0.610	0.298	0.454	0.678	N/A
	25%	Frequency	0.000	0.000	0.000	0.291	0.023	0.012	0.017	0.602	6.17
		SEER	0.000	0.000	0.000	0.155	0.034	0.016	0.025	0.602	6.33
		EAN	0.000	0.000	0.000	0.307	0.290	0.018	0.154	0.625	4.00
		REAP	0.001	0.000	0.001	0.152	0.242	0.046	0.144	0.640	3.92

Table 3: Expert-pruning results on DeepSeek-V2-Lite-Chat (DeepSeek), Qwen3-30B-A3B-Instruct-2507 (Qwen3), and Mixtral-8x7B-Instruct-v0.1 (Mixtral). AIMER achieves the best rank average on DeepSeek and Qwen3 and ties for the best rank average on Mixtral, indicating a consistently strong overall capability balance across coding, creative writing, math, and multiple-choice evaluation.

Figure 3: Layer-wise Magnitude and AIMER score profiles across three MoE models. Columns show OLMoE, DeepSeek,ERNIE, and Qwen3. The top row uses Magnitude, and the bottom row uses AIMER. Within each layer, experts are ranked by the corresponding score, and scores are min-max rescaled to $[0,1]$ . The $x$ -axis reports within-layer expert rank, the $y$ -axis reports layer index, and color indicates the rescaled score. Compared with Magnitude, AIMER yields a more separable distribution over experts, making the differences more distinguishable.

Model	Calibration Time		Peak Calibration Memory		Loading Memory (GB) After / Before
Model	REAP	AIMER (Ours)	REAP	AIMER (Ours)	Loading Memory (GB) After / Before
OLMoE-7B	0.75 h	0.22 s	15.51 GB	13.00 GB	6.89 / 12.89
DeepSeek-16B	1.04 h	0.51 s	34.44 GB	29.42 GB	15.86 / 29.27
ERNIE-21B	1.37 h	0.51 s	44.72 GB	40.85 GB	21.67 / 40.66
Qwen3-30B	2.96 h	1.27 s	63.07 GB	57.01 GB	29.93 / 56.92
Mixtral-47B	1.61 h	2.06 s	93.18 GB	92.27 GB	44.99 / 86.99

Table 4: Efficiency comparison between AIMER (Ours) and REAP. All calibration measurements are collected on two NVIDIA L40S 48GB GPUs, except Mixtral, which uses four GPUs. Calibration time and peak calibration memory measure the expert-scoring stage before structural pruning. Loading memory is reported as after / before pruning at 50% pruning ratio. Overall, AIMER incurs substantially lower resource consumption than calibration-based REAP during expert scoring.

Figure 4: Radar plot of Qwen3-30B performance across all benchmarks at 50% pruning ratio. The dashed outline denotes the dense model, and each colored trace corresponds to one pruning method. Higher values indicate better task performance on the corresponding benchmark. Among the pruned models, AIMER encloses the largest area and has a capability profile closer to that of the full model. Additional radar plots for the other settings are provided in Appendix E .

Model	Total params	Experts per layer	Routing sparsity	Shared experts	Dense/MoE layout
OLMoE-1B-7B-0125-Instruct	7B	64	top-8	No	0 dense, 16 MoE layers
DeepSeek-V2-Lite-Chat	16B	64 routed	top-6	Yes (2 shared)	1 dense, 26 MoE layers
ERNIE-4.5-21B-A3B-PT	21B	64 routed	top-6	Yes (2 shared)	1 dense, 27 MoE layers
Qwen3-30B-A3B-Instruct-2507	30B	128	top-8	No	0 dense, 48 MoE layers
Mixtral-8x7B-Instruct-v0.1	47B	8	top-2	No	0 dense, 32 MoE layers

Table 5: Architectural diversity of the evaluated MoE models.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

			WildBench	Math
Model	Ratio	Method	WildBench	GSM8K	MATH-500	Math Avg
OLMoE	0%	Full	0.444	0.682	0.222	0.452
	25%	Frequency	0.127	0.033	0.024	0.028
		SEER	0.141	0.037	0.012	0.025
		EAN	0.205	0.115	0.006	0.061
		REAP	0.260	0.202	0.056	0.129

Appendix

Table 6: Full benchmark comparison on WildBench and math benchmarks.

			Code
Model	Ratio	Method	HumanEval	HumanEval $+$	MBPP	MBPP $+$	Eval+	LiveCode	Code Avg
OLMoE	0%	Full	0.354	0.323	0.373	0.312	0.341	0.033	0.187
	25%	Frequency	0.000	0.000	0.000	0.000	0.000	0.000	0.000
		SEER	0.000	0.000	0.000	0.000	0.000	0.000	0.000
		EAN	0.000	0.000	0.000	0.000	0.000	0.000	0.000
		REAP	0.012	0.012	0.016	0.016	0.014	0.000	0.007

Appendix

Table 7: Full benchmark comparison on coding benchmarks.

			Multiple Choice
Model	Ratio	Method	MMLU	ARC-C	ARC-E	HellaSwag	BoolQ	OpenBookQA	RTE	WinoGrande	MC Avg
OLMoE	0%	Full	0.534	0.490	0.758	0.808	0.766	0.470	0.711	0.684	0.653
	25%	Frequency	0.325	0.382	0.582	0.727	0.691	0.416	0.690	0.665	0.560
		SEER	0.320	0.388	0.590	0.733	0.694	0.414	0.711	0.661	0.564
		EAN	0.347	0.432	0.627	0.748	0.734	0.430	0.632	0.665	0.577
		REAP	0.391	0.462	0.685	0.766	0.727	0.448	0.581	0.642	0.588

Appendix

Table 8: Full benchmark comparison on multiple-choice benchmarks.

Model	Ratio	Calibration Data	Method	Code Avg	WildBench	MC Avg
ERNIE	25%	C4	REAP	0.126	0.405	0.685
		CodeAlpaca	REAP	0.534	0.352	0.665
		CodeAlpaca:C4 = 1:1	REAP	0.520	0.410	0.692
		CodeAlpaca:C4 = 1:3	REAP	0.501	0.426	0.692
		N/A	AIMER (Ours)	0.455	0.402	0.681
	50%	C4	REAP	0.012	0.199	0.592

Appendix

Table 9: Effect of calibration data choice. We report summary performance using C4, CodeAlpaca, and CodeAlpaca–C4 mixtures. All REAP settings use approximately 4.2M calibration tokens; CodeAlpaca denotes Evol-CodeAlpaca-v1.

Figure 5: Radar plot of DeepSeek performance across all benchmarks at 25% pruning.

Figure 6: Radar plots of ERNIE-21B performance across all benchmarks. The left and right panels show 25% and 50% pruning ratios, respectively.

Figure 7: Radar plots of Mixtral performance across all benchmarks. The left and right panels show 25% and 50% pruning ratios, respectively.

Figure 8: Radar plots of Qwen3-30B and OLMoE-7B performance across all benchmarks at 25% pruning. The left and right panels show Qwen3-30B and OLMoE-7B, respectively.

Mixture-of-Experts Pruning

Mixture-of-Experts pruning removes or trims experts to reduce model memory and computation while preserving useful capabilities.

See all 59 papers

How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle
Jun 14, 2026Zongfang Liu, Jinghui Zhang, Zijian Ma +2Mixture-of-Experts Pruning LLM Pruning
1Zhejiang University · 2Westlake University · 3Mohamed bin Zayed University of Artificial Intelligence +1
Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models
Jul 2, 2026Yongqin Zeng, Sicheng Pan, Jiale Wang +4Mixture-of-Experts Pruning LLM Pruning
It Takes a MAESTRO To Prune Bad Experts
Jul 9, 2026Palaash Goel, Ayush Maheshwari, Tanmoy ChakrabortyMixture-of-Experts Pruning LLM Pruning
Indian Institute of Technology Delhi, India · NVIDIA, India

Jun 14, 2026J/K move · Enter open · S save

How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle

Zongfang Liu, Jinghui Zhang, Zijian Ma, Guangyi Chen+1

1Zhejiang University · 2Westlake University · 3Mohamed bin Zayed University of Artificial Intelligence+1

Mixture-of-Experts Pruning LLM Pruning Mixture-of-Experts Language Models Mixture-of-Experts Models

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) language models reduce per-token computation through sparse expert activation, yet deployment still requires storing the full expert pool, making one-shot expert pruning a practical approach for reducing memory usage. Although effective, existing criteria are largely heuristic, and no single criterion is universally optimal. Thus, establishing a principle for selecting pruning criteria suited to different deployment objectives remains an important yet largely underexplored problem in one-shot expert pruning. To this end, we introduce a unified formulation for one-shot MoE expert pruning organized around three factors: routing frequency, gate weighting, and activation strength. The formulation yields a criteria selection principle: task-agnostic pruning should favor routed-token-averaged, gate-free activation-based criteria, whereas task-specific pruning can benefit from retaining routing-frequency and gate-weight information. Beyond this principle, the formulation also provides a systematic view of existing heuristic criteria and gives rise to two new task-agnostic criteria, Mean Activation Norm (MAN) and Mean Squared Activation Norm (MSAN). Across four representative MoE models and 16 diverse benchmarks, MAN and MSAN are consistently strong in the task-agnostic setting, obtain the top-two average ranks, and improve average performance by up to 8.8 points over the strongest baseline.

Abstract

Mixture-of-Experts Pruning

Explore similar work

How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle

Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models

It Takes a MAESTRO To Prune Bad Experts