Mixture-of-Experts Pruning

Follow this topic Ask about this topic

Momentum

11 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 35

All topics

Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts
May 27, 2026Liu O. Martin, Lucas Bandarkar, Nanyun PengMixture-of-Experts Pruning LLM Pruning
University of California, Los Angeles
Dense2MoE: Pushing the Pareto Frontier of On-Device LLMs via Unified Pruning and Upcycling
May 26, 2026Fengfa Li, Hongjin Ji, Yifeng Ding +2Mixture-of-Experts Pruning LLM Pruning
Li Auto · The Chinese University of Hong Kong, Shenzhen
HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts
May 13, 2026Tao Zhong, Dongzhe Zheng, Christine Allen-BlanchetteMixture-of-Experts Pruning Hodge Decomposition
Princeton University
SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
May 9, 2026Shengkun Tang, Zekun Wang, Bo Zheng +7Mixture-of-Experts Pruning LLM Pruning
Qwen Team, Alibaba Inc. · MBZUAI · KAUST
Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning
Apr 29, 2026Weihang Li, Jianchun Liu, Hongli XuMixture-of-Experts Pruning LLM Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China, China · Suzhou Institute for Advanced Research, University of Science and Technology of China, China

May 27, 2026, cs.CLJ/K move · Enter open · S save

Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts

Liu O. Martin, Lucas Bandarkar, Nanyun Peng

University of California, Los Angeles

Mixture-of-Experts Pruning LLM Pruning Mixture-of-Experts Language Models LLM Compression Mixture-of-Experts Models Machine Translation

Open paper AskPDF arXiv

Modern large language models (LLMs) achieve state-of-the-art machine translation performance, but they do so as broad generalists largely trained for many tasks and capabilities unrelated to translation. Thus, they are heavily overparameterized for this task, resulting in excessive memory and compute requirements. In this paper, we present a method for aggressively pruning experts from modern mixture-of-experts LLMs while incurring negligible degradation in translation quality. Our approach exploits expert specialization and the separability of multilingual capabilities in LLMs to identify experts irrelevant to translation. And because of the modular nature of MoEs, these can be easily pruned without any training. Without retraining, we are able to prune half of all experts with negligible degradation and 70% with only minor losses. With a very short SFT, we prune 75% of experts while recovering baseline performance, and in some settings remove nearly 90% while maintaining reasonable translation quality. Overall, our results show that translation requires only a fraction of the LLM, enabling substantial compression of the MoE blocks that contain over 90% of parameters.

Momentum

Related topics

Latest papers 35

Extracting Small Translation Specialists from LLMs by Aggressively Pruning Experts

Dense2MoE: Pushing the Pareto Frontier of On-Device LLMs via Unified Pruning and Upcycling

HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts

SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training

Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning