Towards Robust Calibration Dataset for Domain-General Mixture of Experts Pruning

Luis Frentzen Salim; Chia-Chun Wu; Tran Van Nhiem; Lun-Wei Ku; Yung-Hui Li

cs.CLOct 11, 2026

Towards Robust Calibration Dataset for Domain-General Mixture of Experts Pruning

Authors: Luis Frentzen Salim, Chia-Chun Wu, Tran Van Nhiem, Lun-Wei Ku, Yung-Hui Li

Organizations: Institute of Information Science, Academia Sinica · National Taiwan University of Science and Technology · AI Research Center, Hon Hai Research Institute

Abstract

Post-training expert pruning is a promising deployment technique for Mixture-of-Experts (MoE) models. The calibration set is an integral part of the pruning process, however, its correlation with evaluation performance is largely misunderstood and unintuitive. In this work, we present an empirical study of the robustness of calibration perturbations, specifically in MoE models. Showing how calibration size, language, and data distribution affect layer-wise expert pruning for gpt-oss-20b. Our findings reveal that (1) domain-general pruning quality receives minimal benefit from scaling calibration data beyond small sample sizes, (2) cross-lingual calibration transfers effectively, with English-only calibration maintaining strong performance across seven languages of varying typologies and scripts, and (3) a diverse calibration data resembling the pretraining distribution can yield comparable or superior performance to test-distribution data. Ultimately, we provide practical insights as a recipe for better understanding calibration for more effective domain-general expert pruning.

Figures & tables

Figure 1: Evaluation performance across different calibration set sizes shows that scaling up calibration doesn’t necessarily yield better post-prune performance. We fixed the number of retained experts to be

24

Figure 2: Evaluation performance ablation of different numbers of retained experts.

Figure 3: Evaluation performance across different languages with the language-specific MMLU-ProX calibration (left) and English C4 calibration (right) datasets. Each line represents one of 7 typologically diverse languages calibrated with English-only data.

Model	Calibration	MathQA	MMLU	Avg.
baseline	N/A	0.365	0.563	0.464
pruned	MATH	0.408	0.585	0.497
	C4	0.3889	0.617	0.503
	Private	0.405	0.611	0.508

Table 1: MathQA and MMLU performance after pruning with different calibration sets.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Category	Labels
Math	gsm8k, abstract_algebra, college_mathematics, elementary_mathematics, high_school_mathematics, high_school_statistics
STEM	anatomy, astronomy, college_biology, college_chemistry, college_computer_science, college_physics, computer_security, conceptual_physics, electrical_engineering, high_school_biology, high_school_chemistry, high_school_computer_science, high_school_physics, machine_learning
Humanities	humanities, formal_logic, high_school_european_history, high_school_us_history, high_school_world_history, international_law, jurisprudence, logical_fallacies, moral_disputes, moral_scenarios, philosophy, prehistory, professional_law, world_religions
Social Sciences	social_sciences, econometrics, high_school_geography, high_school_government_and_politics, high_school_macroeconomics, high_school_microeconomics, high_school_psychology, human_sexuality, professional_psychology, public_relations, security_studies, sociology, us_foreign_policy
Other	other, business_ethics, clinical_knowledge, college_medicine, global_facts, human_aging, management, marketing, medical_genetics, miscellaneous, nutrition, professional_accounting, professional_medicine, virology

Appendix

Table 2: Grouping of MMLU subtasks used in visualizations for brevity, with GSM8K as an addition in the Math category.

Mixture-of-Experts Pruning

Mixture-of-Experts pruning removes or trims experts to reduce model memory and computation while preserving useful capabilities.

See all 59 papers

Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models
Oct 19, 2026Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin +7Mixture-of-Experts Pruning Mixture-of-Experts Language Models
LinkedIn · MIT · Google
Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models
Jul 2, 2026Yongqin Zeng, Sicheng Pan, Jiale Wang +4Mixture-of-Experts Pruning LLM Pruning
AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning
Mar 19, 2026Zongfang Liu, Guangyi Chen, Shengkun Tang +3Mixture-of-Experts Pruning Mixture-of-Experts Language Models
Zhejiang University · Westlake University · Carnegie Mellon University +1

Oct 19, 2026J/K move · Enter open · S save

Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models

Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin, Jincheng Cao+6

LinkedIn · MIT · Google

Mixture-of-Experts Pruning Mixture-of-Experts Language Models LLM Compression Neural Network Pruning

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models reduce computation through sparse expert activation, but deployment still requires storing a large number of expert parameters. We study post-training expert pruning using task-domain calibration data as a subset-selection problem: given a pretrained MoE layer, select a fixed number of experts that best reconstruct its original output. Exact subset search is combinatorial and becomes intractable for modern MoE models with many experts per layer. We develop OptEP, a scalable continuous relaxation that directly optimizes differentiable expert-selection variables, followed by a lightweight reconstruction-based update of the retained expert weights. Across multiple MoE architectures and tasks spanning mathematical reasoning and code generation, our approach improves over the strongest existing expert-pruning baselines, with absolute accuracy gains of up to 15.05% on Qwen3-30B-A3B, 14.55% on Qwen3-235B-A22B, 11.06% on GLM-4.5-Air, and 16.20% on Mixtral-8x7B. The improvements are particularly pronounced under aggressive pruning, demonstrating that reconstruction-driven expert selection and weight adaptation can preserve substantially more of the original model's performance while reducing the number of retained experts.

Abstract

Mixture-of-Experts Pruning

Explore similar work

Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models

Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models

AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning