Mixture-of-Experts Language Models

Ask about this topic

Overview

Mixture-of-Experts language models divide parts of a transformer into expert networks and use a router to select experts for each token, so a model can have many parameters without activating them all at once. Research explores top-k and other routing rules, load-balancing methods, and training strategies such as pretraining sparse models or converting dense models into experts; evaluations commonly include language-model loss and reasoning, coding, and multilingual benchmarks. A large body of work also aims to make these models easier to fine-tune, compress, interpret, and serve through pruning, quantization, expert sharing, and routing-aware systems. A central tension is how to encourage useful expert specialization while keeping routing reliable and balanced, since communication and memory costs can limit the practical gains from sparse computation.

Latest papers

CardsList

229 papers37 papers in the last four weeks, up 42% on the four weeks before. 0.2% of all new papers.

  1. Multi-View MoE: Letting Every Router Shape Expert Computation

    Oct 29, 2026Zelin Zhao, Tianchi Ma, Junyuan Shang +2Mixture-of-Experts Language ModelsMixture-of-Experts Models

    ERNIE Team, Baidu

  2. Nemotron 3.5 Super and Lightning: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Models for Agentic AI

    Oct 28, 2026NVIDIA, :, Aastha Jhunjhunwala +556LLM Agent TrainingMixture-of-Experts Language Models

  3. Muon Makes the First Mixture-of-Experts Layer Harder to Balance

    Oct 27, 2026Marcos HernanzMixture-of-Experts Language ModelsExpert Load Balancing

    Escuela Politécnica Superior, Universidad Autónoma de Madrid

  4. Think Twice: The Metis-1.6 Technical Report

    Oct 27, 2026Giulianno VollmerLLM TrainingEfficient Language Model Training

    Lernex Research

  5. Agent-Conditioned Routing: Diverse Reasoning Paths Within One Frozen Model

    Oct 27, 2026Ananth Kalyanasundaram, Ankush Kadu, Athira Gopal +1Reasoning DiversityMixture-of-Experts Language Models

    QpiAI, India

  6. DuplexTune: Coupled-State Replay for Memory-Efficient Mixture-of-Experts Adaptation

    Oct 26, 2026Paul Raj, Simon Caldwell, Daniel Whitmore +1Mixture-of-Experts Fine-TuningMemory-Efficient Fine-Tuning

    Texas A&M University-Commerce · Michigan State University

  7. Prefetch-Steered Routing: Deterministic Cache-Aware Expert Selection for Offloaded Mixture-of-Experts Decoding

    Oct 26, 2026Siming HuangMixture-of-Experts Language ModelsMixture-of-Experts Inference

    The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.

  8. HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget

    Oct 25, 2026Harish Santhanalakshmi Ganesan, Hariharan M. S., Prabhurajhan RMixture-of-Experts Language ModelsEfficient Language Model Training

  9. Kolibri: A Sovereign European Model on the Pareto Frontier

    Oct 22, 2026Aleph Alpha, :, Moritz Althaus +65Language ModelingLanguage Model Pretraining

  10. IQuest-Q1: Advancing Foundation Capabilities for Agentic CLI Systems

    Oct 22, 2026A. Jian Yang, An Huang, Bo Dan +174LLM Agent TrainingAI Coding Agents

  11. MoEZip: Routing-Aware KV Cache Compression for Sparse Mixture-of-Experts LLMs

    Oct 21, 2026Minsang Kim, Seungjun BaekKV-Cache CompressionLong-Context Language Model Inference

    Korea University · SK Telecom

  12. The Router Already Knows

    Oct 20, 2026Andrew Bistras, Marty JohnsonSparse AutoencodersMixture-of-Experts Language Models

    University of California, Los Angeles · Astrid Cybernetics

  13. Reconstruction-Optimized Expert Pruning for Mixture-of-Experts Language Models

    Oct 19, 2026Jelena Markovic-Voronov, Petros Prastakos, Kayhan Behdin +7Mixture-of-Experts PruningMixture-of-Experts Language Models

    LinkedIn · MIT · Google

  14. An Empirical Study of Parameter-Efficient Fine-Tuning for Mixture-of-Experts LLMs

    Oct 19, 2026Minglei Li, Peng Ye, Haonan He +3Mixture-of-Experts Fine-TuningParameter-Efficient Fine-Tuning

    Fudan University · Shanghai Artificial Intelligence Laboratory · The Chinese University of Hong Kong +3

  15. A Stronger Teacher Within: Strengthening MoE Models via On-Policy Bidirectional Distillation

    Oct 16, 2026Haoyu Chen, Jun Bai, Yang Liu +2Mixture-of-Experts Language ModelsLanguage Model Distillation

    National College for Excellent Engineers, Beihang University · Beijing Institute for General Artificial Intelligence, BIGAI · Meituan Inc. +1

  16. Explaining the Roles of Experts in Mixture-of-Experts LLMs for Personalization

    Oct 15, 2026Sekh Mainul Islam, Pepa Atanasova, Isabelle AugensteinMixture-of-Experts Expert SpecializationPersonalized Language Models

    University of Copenhagen

  17. Beyond Expert Rankings: Contrast Calibration for Mixed-Precision MoE Quantization

    Oct 14, 2026Renjie Xie, Juncheng Yang, Aoting Hu +5Mixed-Precision QuantizationMixture-of-Experts Quantization

    Nanjing University of Posts and Telecommunications · Tylogi AI Lab / TAIL · Anhui University of Technology +4

  18. Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation

    Oct 14, 2026Joel Dag, Nikit Srivastava, Hamada M. Zahera +1Multilingual Language ModelsMixture-of-Experts Language Models

    Data Science Group, Heinz Nixdorf Institute, Paderborn University, Germany

  19. Spectral-MoE: Heterogeneous Per-Layer Expert Utilization in Dense-to-MoE Conversion

    Oct 13, 2026Zachary Broveak, Christian Kubicka, Thierry TambeMixture-of-Experts Language ModelsLLM Inference Efficiency

    Stanford University

  20. SLEM: Spectrum-Aware Latent Expert Merging

    Oct 12, 2026Dohyeon Kim, Bedionita Soro, Bruno Andreis +1Language Model MergingMixture-of-Experts Language Models

    KAIST · University of Oxford · Slater Labs

  21. Towards Robust Calibration Dataset for Domain-General Mixture of Experts Pruning

    Oct 11, 2026Luis Frentzen Salim, Chia-Chun Wu, Tran Van Nhiem +2Mixture-of-Experts PruningNeural Network Pruning

    Institute of Information Science, Academia Sinica · National Taiwan University of Science and Technology · AI Research Center, Hon Hai Research Institute

  22. A Sovereign, Indic-First, Reasoning Model

    Oct 11, 2026Sutanshu Saksena Raj, Prasanjith Pasunuti, Parijat Chatterjee +4Language ModelingCross-Lingual Reasoning in Language Models

  23. Shaping Latent Mixture-of-Experts: From Expert Geometry to Layerwise Capacity Allocation

    Oct 10, 2026Yaoxiang Wang, Kai Zhang, Ruobing Xie +4Mixture of ExpertsMixture-of-Experts Models

    Xiamen University · Tencent Hunyuan · Soochow University

  24. Overcoming Data Imbalances with Expert Selection Biases

    Oct 9, 2026Ayush K. Tarun, Clara Meister, Antoine BosselutMixture-of-Experts Language ModelsMixture-of-Experts Models

    EPFL

  25. DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving

    Oct 9, 2026Mengqi Guo, Zhe Wei, Yuan Yuan +3Mixture-of-Experts QuantizationMixture-of-Experts Inference

    Huawei Technologies Ltd.

  26. RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing

    Oct 8, 2026Ilya Lasy, Nora Yinuo Cai, Kola AyonrindeMixture-of-Experts InterpretabilityMixture of Experts

    Faculty of Informatics, TU Wien · Independent · UK AI Security Institute

  27. Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts

    Oct 8, 2026Yunkai Chai, Tong Zhu, Xiaoye Qu +4Mixture-of-Experts Language ModelsSparse Mixture-of-Experts

    Shanghai AI Laboratory, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · Nanyang Technological University, Singapore

  28. When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetry

    Oct 7, 2026Yixin Tan, Jiayang Liu, Lu Sun +3Privacy Leakage in Language ModelsMembership Inference Attacks

    Institute of Science Tokyo · Nanyang Technological University · Tohoku University +2

  29. Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling

    Oct 7, 2026Radha Gulhane, Quentin Anthony, Beren MillidgeCommunication-Efficient Mixture-of-ExpertsMixture-of-Experts Language Models

    Zyphra

  30. Shared Low-rank Basis Factorization for Data-free Mixture-of-Experts Compression

    Oct 7, 2026Tianxiao Cao, Jiahe Shao, Yuning Qiu +3LLM CompressionMixture-of-Experts Language Models

    Kyoto University · The University of Tokyo · RIKEN AIP