Mixture-of-Experts Language Models

Follow this topic Ask about this topic

Momentum

18 papers in the last four weeks, up 38% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 147

All topics

Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models
Jul 2, 2026Yongqin Zeng, Sicheng Pan, Jiale Wang +4Mixture-of-Experts Pruning LLM Pruning
CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield
Jun 30, 2026Dohyeon Kwon, Youngjin ParkMixture-of-Experts Language Models LLM Compression
AXOps Team, TeamSparta Inc., Seoul, South Korea
Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs
Jun 30, 2026Seyed Alireza Molavi, Zhan Su, Yan Hu +3Mixture-of-Experts Language Models Low-Rank Adaptation
1Halmstad University, Halmstad, Sweden · 2The Chinese University of Hong Kong, Shenzhen, China
FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models
Jun 26, 2026Fan Mo, Yuxuan Han, Geng Zhang +2Mixture-of-Experts Pruning LLM Pruning
National University of Singapore
CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation
Jun 23, 2026Zhuoren Ye, Tianyu Wo, Dinghao Xue +4Mixture-of-Experts Language Models LLM Serving
School of Software, Beihang University
Does the Same Token Mean the Same State? MoE Routing as Signal for Reasoning Control
Jun 22, 2026Kang Chen, Minshen Yu, Junjie Nian +3Mixture-of-Experts Language Models LLM Routing
Fudan University · Shanghai Innovation Institute
SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
Jun 16, 2026Mikołaj Zasada, Łukasz Struski, Jacek Tabor +1Mixture-of-Experts Language Models Sparse Mixture-of-Experts
AGH University of Krakow, Poland · Faculty of Mathematics and Computer Science, Jagiellonian University, Poland · Jagiellonian University, Poland +1
Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models
Jun 15, 2026Martin JaggiMixture-of-Experts Language Models LLM Compression
EPFL
How to Score Experts for One-Shot MoE Expert Pruning: A Unified Formulation and Selection Principle
Jun 14, 2026Zongfang Liu, Jinghui Zhang, Zijian Ma +2Mixture-of-Experts Pruning LLM Pruning
1Zhejiang University · 2Westlake University · 3Mohamed bin Zayed University of Artificial Intelligence +1
Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning
Jun 12, 2026NVIDIA, :, Aaron Blakeman +571Mixture-of-Experts Language Models Long-Context Language Model Inference
Allan
Decoupled Mixture-of-Experts for Parametric Knowledge Injection
Jun 12, 2026Baoqing Yue, Weihang Su, Qingyao Ai +5Mixture-of-Experts Language Models Knowledge Augmentation for Language Models
Department of Computer Science and Technology, Tsinghua University
Redesign Mixture-of-Experts Routers with Manifold Power Iteration
Jun 10, 2026Songhao Wu, Ang Lv, Ruobing Xie +1Mixture-of-Experts Language Models Expert Routing
Gaoling School of Artificial Intelligence, Renmin University of China · Large Language Model Department, Tencent
LoMC: Localized Multidirectional Correction for Refusal Suppression in Routed Foundation Models
Jun 10, 2026Yan Hong, Kedong Xiu, Wei Li +6Mixture-of-Experts Language Models LLM Refusal Behavior
Ant Group · Zhejiang University · Shanghai Jiao Tong University
Entropy-Aware Domain-Routed Mixture-of-Experts Speech-LLM Framework: A Case Study of Multi-Domain Child-Adult ASR
Jun 9, 2026Mohan Shi, Kaiyuan Zhang, Zilai Wang +3Mixture-of-Experts Language Models Automatic Speech Recognition
University of California, Los Angeles, USA
PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning
Jun 9, 2026Xinyue Peng, Yi Qian, Jiaojiao Lin +2Mixture-of-Experts Language Models Expert Routing
Intel Corporation, China · Zhejiang University, China
Routing-Aware Expert Calibration for Machine Unlearning in Mixture-of-Experts Language Models
Jun 9, 2026Jingyi Xie, Yijun Lin, Yinjiang Xiong +2Mixture-of-Experts Language Models Machine Unlearning
Renmin University of China · Lightstandard · Zhejiang University +1
Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning
Jun 5, 2026Fatema Siddika, Md Anwar Hossen, Tanwi Mallick +1Continual Learning for LLMs Mixture-of-Experts Language Models
Iowa State University, Ames, USA · Argonne National Laboratory, USA
Sparsely gated tiny linear experts
Jun 5, 2026Simon SchugTransformer FFNs Mixture-of-Experts Language Models
Department of Computer Science Princeton University
Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling
Jun 5, 2026Rohan ShravanMixture-of-Experts Language Models Efficient Language Model Training
The School of AI Bengaluru, India
AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization
Jun 3, 2026Wanqi Yang, Yuexiao Ma, Alexander Conzelmann +4Mixed-Precision Quantization Mixture-of-Experts Language Models
Max Planck Institute for Intelligent Systems · ELLIS Institute Tübingen · Tübingen AI Center +5
TENP: Trapezoidal Expert Neuron Pruning For Mixture-of-Experts
Jun 3, 2026Jiangyang He, Shaolin Zhu, Deyi XiongMixture-of-Experts Pruning LLM Pruning
TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China
LoopMoE: Unifying Iterative Computation with Mixture-of-Experts for Language Modeling
Jun 3, 2026Wenkai Chen, Tianshu Li, Wenyong Huang +3Mixture-of-Experts Language Models Language Modeling
Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Co.,Ltd.
Expert-Aware Refusal Steering
Jun 2, 2026Anna C. Marbut, Daniel R. Olson, Travis J. WheelerMixture-of-Experts Language Models Language Model Steering
Department of Interdisciplinary Studies University of Montana Missoula, MT, USA · Department of Pharmacy Practice & Science University of Arizona Tucson, AZ, USA · European Bioinformatics Institute European Molecular Biology Laboratory Wellcome Genome Campus, Hinxton, United Kingdom
Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models
Jun 2, 2026Yuetian Lu, Ali Modarressi, Yihong Liu +1Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Center for Information and Language Processing (CIS), LMU Munich
When Meaning Travels: A Granular Lens on Hybrid-MoE's Role in Idiomatic Understanding for Language Models
Jun 1, 2026Sarmistha Das, Vaibhav Vishal, Shreyas Guha +3Mixture-of-Experts Language Models Low-Resource Language Processing
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India · School of Information Technology, King Mongkut’s Institute of Technology Ladkrabang, Thailand
DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
May 31, 2026Jiarui Feng, Hanqing Zeng, Karish Grover +11Mixture-of-Experts Language Models Sparse Mixture-of-Experts
Meta MRS · Washington University in St. Louis · Carnegie Mellon University +1
Confidence-Adaptive SwiGLU for Mixture-of-Experts
May 30, 2026Shaohua Li, Xiuchao Sui, Xiaobing Sun +4Mixture-of-Experts Language Models Mixture-of-Experts Models
Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore · Shanghai University of Engineering Science, China
MESA: Improving MoE Safety Alignment via Decentralized Expertise
May 30, 2026Yitong Sun, Yao Huang, Teng Li +5Mixture-of-Experts Language Models LLM Safety Alignment
Institute of Artificial Intelligence, State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, 100191, China · College of AI, Tsinghua University, Beijing, 100083, China · School of Computer Science, Fudan University, Shanghai, 200438, China +2
PithTrain: A Compact and Agent-Native MoE Training System
May 29, 2026Ruihang Lai, Hao Kang, Haozhan Tang +6Mixture-of-Experts Language Models Efficient Language Model Training
1Carnegie Mellon University · 3NVIDIA · 2Xlue
Mellum2 Technical Report
May 29, 2026Marko Kojic, Ivan Bondyrev, Aral de Moor +6Mixture-of-Experts Language Models Open-Weight Language Models
1JetBrains · 2Constructor University, Bremen, Germany

Jul 2, 2026, cs.AIJ/K move · Enter open · S save

Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models

Yongqin Zeng, Sicheng Pan, Jiale Wang, Hai-tao Zheng+3

Mixture-of-Experts Pruning LLM Pruning Mixture-of-Experts Language Models Sparse Mixture-of-Experts Neural Network Pruning

Open paper AskPDF arXiv

Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging. Existing expert-pruning methods typically rely on a single aggregated importance score, which can bias the retained set toward experts favored by dominant calibration patterns. We propose \textbf{Generic TB-Coverage}, a coverage-aware expert pruning method that uses only generic text corpora (WikiText2 and C4) for calibration. Instead of collapsing expert utility into one score, our method profiles per-expert utility separately on each corpus and enforces a fixed-budget coverage rule that preserves high-utility experts from each corpus before constructing the final pruning mask. Across Qwen1.5-MoE-A2.7B and DeepSeek-MoE-16B-Base at 25%, 50%, and 75% retention budgets, our method improves average accuracy on six common zero-shot benchmarks over random pruning, REAP, and ExpertSparsity, while also reducing perplexity degradation on WikiText2 and C4. The gains are largest under aggressive pruning (25% and 50% retain), suggesting that preserving cross-corpus expert coverage is an effective generic-data prior for MoE pruning. Our improvements hold with fixed pruning budgets and no downstream calibration data.

Momentum

Related topics

Latest papers 147