Mixture-of-Experts Pruning

Follow this topic Ask about this topic

Momentum

11 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 35

All topics

Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization
Sep 30, 2026Zheng Lin, Shaoke Fang, Yuxin Zhang +6Submodular Optimization Mixture-of-Experts Pruning
Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg, Luxembourg · Department of Computer Science, Peking University, China · Institute of Space Internet, Fudan University, China +4
MoRA: MoE Pruning via Router Bias Learning and Expert Approximation
Sep 30, 2026Yushuai Sun, Zikun Zhou, Lin Gao +2Mixture-of-Experts Pruning LLM Compression
Harbin Institute of Technology, Shenzhen
DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference
Sep 30, 2026Xudong Tan, Peng Ye, Ming Xie +4Mixture-of-Experts Pruning Efficient Multimodal Inference
Fudan University · The Chinese University of Hong Kong
DIET: Deletion-response Expert Trimming for Video Diffusion Transformers
Sep 29, 2026Jiachang Zhang, Teng Hu, Bohao Feng +4Mixture-of-Experts Pruning Video Diffusion Models
Xi’an Jiao Tong University, China · Shanghai Jiao Tong University, China · Alibaba Token Hub, Alibaba Group, China +1
SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving
Sep 28, 2026Gunho Park, Kyoungho Jeun, Juntaek Oh +3Mixture-of-Experts Pruning Disaggregated LLM Serving
a2sys · KAIST
EAT: Expert Account Tracker for Efficient MoE Inference
Sep 27, 2026Yuexian Li, Yifei Yang, Zouying Cao +1Mixture-of-Experts Pruning Mixture-of-Experts Language Models
Paris Elite Institute of Technology, Shanghai Jiao Tong University · AGI Institute, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University +1
RAZOR: Pruning Replaceable Experts in LLMs
Sep 24, 2026Mingyang Song, Mao ZhengMixture-of-Experts Pruning LLM Pruning
Foundation Model Department, Tencent, China
You Only Need 2/3 of the Chosen Experts: An Empirical Study of Dynamic Expert Pruning in Fine-Grained MoE LLMs
Sep 22, 2026Yuanteng Chen, Qiwei Lai, Chen Tianqi +7Mixture-of-Experts Pruning Mixture-of-Experts Language Models
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy +1
Higher-order pruning of experts in mixture-of-experts language models
Sep 16, 2026Alex M. Tseng, Prannay Kaul, Luca Zancato +2Mixture-of-Experts Pruning Mixture-of-Experts Language Models
AI Fundamental Research AWS Agentic AI
ESTS at WMT26: Routing-Informed Expert Pruning for Model Compression
Sep 14, 2026Liu O. Martin, Lucas Bandarkar, Nanyun PengModel Compression Mixture-of-Experts Pruning
University of California, Los Angeles
OMP-MoE: Efficient Expert Pruning for Mixture-of-Experts LLMs via Orthogonal Matching Pursuit
Sep 9, 2026Dezhi Li, Lujun Li, Qiyuan Zhu +4Mixture-of-Experts Pruning Mixture-of-Experts Language Models
The Hong Kong University of Science and Technology
PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Composition
Sep 1, 2026Ziyan Gan, Fangxin Liu, Chenyang Guan +10Mixture-of-Experts Pruning Mixture-of-Experts Inference
1Shanghai Jiao Tong University, China · 2Alibaba Group, China
Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
Aug 8, 2026Ali Janati, Kaoutar El Maghraoui, Chengke Zou +2Mixture-of-Experts Pruning Mixture-of-Experts Models
Data Science Institute, Columbia University, New York, NY, USA · Department of Computer Science, Columbia University, New York, NY, USA
When Compression Scores Cannot Decide: Information Boundaries for Group-Robust LLM Pruning
Aug 3, 2026Andrew ZhangModel Selection Mixture-of-Experts Pruning
KTH Royal Institute of Technology
Half the Experts, All the Code: One-Shot Domain Pruning of Mixture-of-Experts LLMs for Coding
Jul 18, 2026Anik JhaMixture-of-Experts Pruning Mixture-of-Experts Language Models
1Independent Researcher
UMoE:Unlocking Every Expert in Domain-Specific Training
Jul 13, 2026Xuefeng Li, Pengfei LiuMixture-of-Experts Pruning Mixture-of-Experts Language Models
1SII · 2SJTU · 3GAIR
It Takes a MAESTRO To Prune Bad Experts
Jul 9, 2026Palaash Goel, Ayush Maheshwari, Tanmoy ChakrabortyMixture-of-Experts Pruning LLM Pruning
Indian Institute of Technology Delhi, India · NVIDIA, India
Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs
Jul 5, 2026Akhiad Bercovich, Talor Abramovich, Daniel Afrimi +67Mixture-of-Experts Pruning LLM Pruning
EPnG: Adaptive Expert Prune-and-Grow for Parameter-Efficient MoE Fine-tuning
Jul 2, 2026Ahin Lee, Sehyun Yun, Taesik GongMixture-of-Experts Pruning Mixture-of-Experts Language Models
Ulsan National Institute of Science & Technology (UNIST)
Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models
Jul 2, 2026Yongqin Zeng, Sicheng Pan, Jiale Wang +4Mixture-of-Experts Pruning LLM Pruning
On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domain
Jul 1, 2026Atsuki Yamaguchi, Szymon Palucha, Léo Bijar +2Mixture-of-Experts Pruning LLM Reliability
University of Sheffield, United Kingdom · AstraZeneca · University of Exeter, United Kingdom +1
FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
Jun 26, 2026Fan Mo, Yuxuan Han, Geng Zhang +2Mixture-of-Experts Pruning LLM Pruning
National University of Singapore
Attribution-Guided and Coverage-Maximized Pruning for Structural MoE Compression
Jun 16, 2026Yifu Ding, Jiacheng Wang, Ge Yang +4Mixture-of-Experts Pruning LLM Pruning
State Key Laboratory of Complex & Critical Software Environment, Beihang University · School of Computer Science and Engineering, Beihang University · Nanyang Technological University +1
How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle
Jun 14, 2026Zongfang Liu, Jinghui Zhang, Zijian Ma +2Mixture-of-Experts Pruning LLM Pruning
1Zhejiang University · 2Westlake University · 3Mohamed bin Zayed University of Artificial Intelligence +1
From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Models
Jun 9, 2026Leonard Engmann, Christian Medeiros Adriano, Holger GieseMixture-of-Experts Pruning LLM Auditing
Hasso Plattner Institute, University of Potsdam, Germany.
Less is MoE: Trimming Experts in Domain-Specialist Language Models
Jun 4, 2026Haoze He, Xinkai Zou, Xuan Jiang +4Mixture-of-Experts Pruning LLM Pruning
Carnegie Mellon University · UCSD · MIT
SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs
Jun 3, 2026Yuhao ZhangMixture-of-Experts Pruning LLM Pruning
Beihang University
TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts
Jun 3, 2026Jiangyang He, Shaolin Zhu, Deyi XiongMixture-of-Experts Pruning LLM Pruning
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China
PrunePath: Towards Highly Structured Sparse Language Models
May 27, 2026Zhexuan Gu, Zixun Fu, Yancheng YuanMixture-of-Experts Pruning LLM Pruning
Department of Applied Mathematics, The Hong Kong Polytechnic University
Pruning and Distilling Mixture-of-Experts into Dense Language Models
May 27, 2026Junhyuck Kim, Jihun Yun, Haechan Kim +3Mixture-of-Experts Pruning LLM Pruning
1KRAFTON · 2KAIST

Sep 30, 2026, cs.LGJ/K move · Enter open · S save

Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization

Zheng Lin, Shaoke Fang, Yuxin Zhang, Jinfeng Xu+5

Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg, Luxembourg · Department of Computer Science, Peking University, China · Institute of Space Internet, Fudan University, China+4

Submodular Optimization Mixture-of-Experts Pruning Mixture of Experts Mixture-of-Experts Models

Open paper AskPDF arXiv

While Mixture-of-Experts (MoE) models effectively scale model capacity through sparse activation, their deployment is often bottlenecked by prohibitive memory requirements. Extracting a compact subset of experts presents a promising solution. However, existing expert selection heuristics predominantly rely on Top-k ranking, which isolates the evaluation of individual experts and ignores the intricate inter-expert dependencies introduced by the MoE gating network. In this paper, we propose DS-MoE, a theoretically grounded framework that redefines expert selection via difference-of-submodular (DS) optimization. By analyzing the second-order Taylor expansion of the loss degradation, we reveal functional duality within expert combinations: redundancy (where experts encode overlapping representations) and synergy (where experts provide complementary error cancellation). To navigate this duality, we mathematically decouple redundancy reduction from synergy maximization by formulating the selection objective as a DS function. Furthermore, we devise a tailored majorization-minimization (MM) algorithm with provable monotonicity guarantees to efficiently identify the optimal expert subset. Extensive experiments demonstrate that DS-MoE effectively preserves indispensable expert combinations, achieving superior performance compared to the state-of-the-art baselines.

Momentum

Related topics

Latest papers 35