Mixture-of-Experts Language Models

Overview

Mixture-of-Experts language models divide parts of a transformer into expert networks and use a router to select experts for each token, so a model can have many parameters without activating them all at once. Research explores top-k and other routing rules, load-balancing methods, and training strategies such as pretraining sparse models or converting dense models into experts; evaluations commonly include language-model loss and reasoning, coding, and multilingual benchmarks. A large body of work also aims to make these models easier to fine-tune, compress, interpret, and serve through pruning, quantization, expert sharing, and routing-aware systems. A central tension is how to encourage useful expert specialization while keeping routing reliable and balanced, since communication and memory costs can limit the practical gains from sparse computation.

Latest papers

CardsList

248 papers23 papers in the last four weeks, down 39% on the four weeks before. 0.2% of all new papers.

  1. What MoE Routing Controls: Causal Interventions on Expert Trajectories

    Nov 17, 2026Mukund Athmaram, Manoj Rao, Liang MaMixture-of-Experts Language ModelsLanguage Model Steering

  2. GEAR2^2: Pruned Experts Live On through Gated Reuse of Activated Experts and a Missing-Mass Residual

    Nov 16, 2026Linliang Chen, Yan Zhong, Wang Kang +3Mixture-of-Experts PruningMixture-of-Experts Language Models

  3. From Facts to Personas: Interpretable Role Unlearning in LLMs via Mixture-of-Experts

    Nov 14, 2026Ruihong Zeng, Puning Yang, Jinghui Zhang +4LLM UnlearningLarge Language Model-Based Role-Play Simulation

  4. Understanding Mixture-of-Experts Language Models Beyond Expert Boundaries

    Nov 10, 2026Zhenhong Zhou, Mao Zheng, Mingyang Song +3Mixture-of-Experts Language ModelsMixture-of-Experts Expert Specialization

  5. Rules, Traces and Routing Statistics: Training and Calibration Data for Open Models on ARC-AGI-2

    Nov 4, 2026Nguyen Minh Thang, Trinh Thi PhuongARC-AGIFine-Tuning Language Models

  6. AdaFloor: Adaptive Floor Planning and Rank-Time Reuse for Long-Tail MoE RL Rollouts

    Nov 4, 2026Jia Wei, Ziyi Qiu, Yi Xu +1RL Post-TrainingReinforcement Learning

  7. SafeStart: Mitigating the Overlooked Safety Cost of Active Expert Reduction in MoE LLM Serving

    Nov 4, 2026Erfan Baghaei Potraghloo, Saeid Shokoufa, Seyedarmin Azizi +4LLM SafetyMixture-of-Experts Language Models

  8. Experts Exhibit Structured Redundancy: Structure-Preserving Expert Pruning for Mixture-of-Experts Large Language Models

    Oct 30, 2026Tian Wang, Yongding Tao, Yihong Dong +3Mixture-of-Experts PruningMixture-of-Experts Language Models

  9. Characterizing Recoverable Expert-Utilization Signatures in Fine-Tuned Mixture-of-Experts LLMs

    Oct 29, 2026Fei Wang, Baochun Li, Bo LiMixture-of-Experts Language ModelsMemorization in Language Models

  10. Muon Makes the First Mixture-of-Experts Layer Harder to Balance

    Oct 27, 2026Marcos HernanzMixture-of-Experts Language ModelsExpert Load Balancing

  11. DuplexTune: Coupled-State Replay for Memory-Efficient Mixture-of-Experts Adaptation

    Oct 26, 2026Paul Raj, Simon Caldwell, Daniel Whitmore +1Mixture-of-Experts Fine-TuningMemory-Efficient Fine-Tuning

  12. The Router Already Knows

    Oct 20, 2026Andrew Bistras, Marty JohnsonSparse AutoencodersMixture-of-Experts Language Models

  13. Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models

    Oct 19, 2026Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin +7Mixture-of-Experts PruningMixture-of-Experts Language Models

  14. An Empirical Study of Parameter-Efficient Fine-Tuning for Mixture-of-Experts LLMs

    Oct 19, 2026Minglei Li, Peng Ye, Haonan He +3Mixture-of-Experts Fine-TuningParameter-Efficient Fine-Tuning