Mixture-of-Experts Pruning

Overview

Mixture-of-Experts pruning studies how to shrink models made of specialized subnetworks while retaining their accuracy and efficiency advantages. Methods rank experts using signals such as routing frequency, activation strength, weight or gradient sensitivity, and estimated deletion error; others account for expert interactions or allocate pruning budgets across layers. Approaches range from training-free expert removal and neuron-level trimming to merging, distillation, and dynamic selection at inference, with evaluation spanning language, reasoning, coding, and domain-specific tasks. A central challenge is that importance estimates and safe pruning levels vary by model, task, and deployment setting, so gains in memory or speed must be weighed against capability loss and changes in generation behavior.

Latest papers

CardsList

68 papers14 papers in the last four weeks, level with the four weeks before. 0.1% of all new papers.

  1. SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training

    May 9, 2026Shengkun Tang, Zekun Wang, Bo Zheng +7Mixture-of-Experts PruningLLM Pruning

  2. Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning

    Apr 29, 2026Weihang Li, Jianchun Liu, Hongli XuMixture-of-Experts PruningLLM Fine-Tuning

  3. MoEITS: A Green AI approach for simplifying MoE-LLMs

    Apr 12, 2026Luis Balderas, Miguel Lastra, José M. BenítezLLM CompressionMixture-of-Experts Pruning

  4. Does a Global Perspective Help Prune Sparse MoEs Elegantly?

    Apr 8, 2026Zeliang Zhang, Nikhil Ghosh, Jiani Liu +2Mixture-of-Experts PruningNeural Network Pruning

  5. REAM: Merging Improves Pruning of Experts in LLMs

    Apr 6, 2026Saurav Jha, Maryam Hashemzadeh, Ali Saheb Pasand +3LLM CompressionMixture-of-Experts Pruning

  6. AIMER: Calibration-Free Task-Agnostic MoE Expert Pruning

    Mar 19, 2026Zongfang Liu, Guangyi Chen, Yifan Shen +3Mixture-of-Experts PruningMixture-of-Experts Language Models

  7. EvoESAP: Non-Uniform Expert Pruning for Sparse MoE

    Mar 6, 2026Zongfang Liu, Shengkun Tang, Boyang Sun +2Mixture-of-Experts PruningLLM Pruning

  8. ExpertWeaver: Unlocking the Inherent MoE in Dense LLMs with GLU Activation Patterns

    Feb 17, 2026Ziyu Zhao, Tong Zhu, Zhi Zhang +6Mixture-of-Experts Language ModelsMixture-of-Experts Pruning