Sparse Mixture-of-Experts

Ask about this topic

Overview

Sparse mixture-of-experts models use a router to send each token or input to only a few expert networks, aiming to combine large model capacity with manageable computation. Common approaches include top-k and threshold routing, hierarchical or shared experts, load-balancing objectives, and upcycling dense models into expert-based ones. Research spans language models as well as vision, speech, forecasting, and multimodal tasks, with work on training stability, pruning, quantization, and efficient inference. A central tension is that balancing expert use can prevent collapse but may weaken specialization, while high stored capacity does not always translate into better results or lower end-to-end cost.

Latest papers

CardsList

121 papers16 papers in the last four weeks, up 33% on the four weeks before. 0.1% of all new papers.

  1. EnergyCal: Taming Loss Spikes in Muon-Family Optimizers for Sparse MoE Training

    Oct 27, 2026Masato FujitakeMuon OptimizerSparse Mixture-of-Experts

    FA Research, Fast Accounting Co., Ltd.

  2. SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs

    Oct 21, 2026Yifan Yang, Hanling Yi, Dongquan Yang +4Mixture-of-Experts PruningSparse Mixture-of-Experts

    Intellifusion Inc.

  3. GaP-MoE : Explicit Expert Specialization in Mixture-of-Experts via Orthogonal Gradient Accumulation and Projection

    Oct 21, 2026Donghyun Jeong, Yoon-Yeong KimSparse Mixture-of-ExpertsMixture-of-Experts Models

    Department of Statistics and Data Science University of Seoul Republic of Korea · Department of Statistics University of Seoul Republic of Korea

  4. Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

    Oct 20, 2026Ermo Hua, Xiangyu Hong, Baiting Wu +11Sparse Mixture-of-ExpertsMixture-of-Experts Models

    Tsinghua University · Shanghai AI Laboratory

  5. Rethinking Routing Exploration in Modern Mixture-of-Experts

    Oct 15, 2026Bingcong Li, Hao Ma, Michael Muehlebach +2Mixture-of-Experts ModelsExpert Routing

    ETH Zurich · The University of Hong Kong · Max Planck Institute for Intelligent Systems +1

  6. How Many Ranks Does a Token Need? A Noise-Floor Squelch for Rank-One LoRA Experts

    Oct 13, 2026Mei Okonkwo, Owen Castellanos, Ingrid Solberg +1Low-Rank AdaptationAdaptive Rank Allocation

    University of Edinburgh · Delft University of Technology

  7. FORGE: Procedural Weight Generation for Sparse MoE Inference

    Oct 12, 2026Chanyoung Moon, Travis Pence, Daisuke Yamada +4Mixture-of-Experts InferenceLLM Inference Acceleration

    University of Wisconsin-Madison Madison, Wisconsin, USA · Korea University Seoul, Republic of Korea · NVIDIA Santa Clara, California, USA

  8. Sparsity Saves Compute, Not Samples: Oracle Inequalities for Dense and Sparse Mixture of Experts

    Oct 10, 2026Sagar Ghosh, Huy Nguyen, Alessandro RinaldoSparse Mixture-of-ExpertsMixture-of-Experts Models

    The University of Texas at Austin

  9. Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts

    Oct 8, 2026Yunkai Chai, Tong Zhu, Xiaoye Qu +4Mixture-of-Experts Language ModelsSparse Mixture-of-Experts

    Shanghai AI Laboratory, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · Nanyang Technological University, Singapore

  10. DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition

    Oct 8, 2026Yuxuan Lou, Kai Yang, Geng Zhang +2Mixture-of-Experts ModelsSparse Mixture-of-Experts

    School of Computing, National University of Singapore, Singapore · Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

  11. Distributionally Robust Mixture-of-Experts Training

    Oct 5, 2026Xin Teng, Muxiao Li, Hongyi WenMixture-of-Experts Language ModelsSparse Mixture-of-Experts

    New York University Center for Data Science, NYU Shanghai

  12. SEAL: Mixture-Closed Additive Reconstruction and Refinement-Aware Expert Routing for Efficient Speech Separation

    Oct 5, 2026Shao-Chun Hu, Zi-Xiang Lin, Jeih-Weih Hung +1Speech SeparationSparse Mixture-of-Experts

    Department of Electrical Engineering, National Chi Nan University · Graduate Institute of AI Interdisciplinary Applied Technology, National Taiwan Normal University

  13. Conditional Capacity and Routing in Mixture-of-Experts Particle Transformers

    Oct 2, 2026Kaushik Pendiyala, Haris Zia, Trevin Lee +8Mixture-of-Experts ModelsSparse Mixture-of-Experts

    University of California Davis, Davis, CA, USA · University of California San Diego, San Diego, CA, USA · University of Illinois at Chicago, Chicago, IL, USA +1

  14. RouteRec: Behavior-Guided Sparse Routing for Sequential Recommendation

    Sep 30, 2026Junyeong Song, Jaemin YooSequential RecommendationSparse Mixture-of-Experts

    Korea Advanced Institute of Science and Technology Daejeon, Republic of Korea · Seoul National University Seoul, Republic of Korea

  15. Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models

    Sep 29, 2026Yury Nahshan, Nati Daniel, Jacob Goldberger +1Mixture-of-Experts Language ModelsParameter-Free Mixture-of-Experts Routing

    Bar-Ilan University, Ramat-Gan, Israel · NVIDIA, Israel

  16. Routing in Gradient Space: Balanced Usage Is Not Expert Specialization

    Sep 29, 2026Yuchen Li, Mingyu Du, Zongqi Fan +2Sparse Mixture-of-ExpertsMixture-of-Experts Models

    University of Sydney · University of New South Wales

  17. How to Loop MoE: Flatten the Experts, Untie the Attention

    Sep 28, 2026Shouren Wang, Chuang Ma, Mohsen Hariri +6Mixture-of-Experts Language ModelsSparse Mixture-of-Experts

    Case Western Reserve University · Kyoto University · NII LLMC

  18. HERO-MoE: Historical Expert Routing with Scale-Preserving Fusion

    Sep 26, 2026Junxiang Qiu, Zhengsu Chen, Xinting Hu +6Mixture-of-Experts Language ModelsLLM Routing

    University of Science and Technology of China · Huawei Inc. · Southeast University

  19. OceanMoE: Structured Conditional Sparse Computation for Long-Horizon Multivariate Ocean Forecasting

    Sep 17, 2026Yishun Zhu, Jian WangMultivariate Time Series ForecastingLong-Term Time Series Forecasting

    Hangzhou Institute for Advanced Study, University of the Chinese Academy of Sciences Zhejiang, China · Computer Network Information Center, Chinese Academy of Sciences Beijing, China

  20. MoRE: Mixture of Reused Experts

    Sep 16, 2026Eric S. Qiu, Utku Umur Acikalin, Justin Lovelace +4Mixture-of-Experts Language ModelsSparse Mixture-of-Experts

    Department of Computer Science, Cornell University

  21. Beyond the Previous Layer: Residual Predictive Structure in Sparse MoE Routing

    Sep 15, 2026Hao Li, Yasuyuki Tahara, Yuichi SeiSparse Mixture-of-ExpertsExpert Routing

    Department of Informatics, Graduate School of Informatics and Engineering, The University of Electro-Communications, Tokyo, Japan

  22. ExpertHTR: Unified Handwritten Text Recognition with Multi-Task Learning and Sparse Mixture-of-Experts

    Sep 11, 2026Dang Hoai Nam, Nguyen Duy Hieu, Quang Huu Hieu +1Vision-Language ModelsSparse Mixture-of-Experts

    University of Information Technology, Ho Chi Minh City, Vietnam · Vietnam National University, Ho Chi Minh City, Vietnam · AJ Technologies, Nagoya, Japan +1

  23. Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

    Sep 10, 2026Atindra Jha, Margaret Li, Jure Leskovec +2Mixture of ExpertsSparse Mixture-of-Experts

    Stanford University · Paul G. Allen School of Computer Science, University of Washington

  24. Hyperparameter Scaling Laws Across MoE Sparsity

    Sep 8, 2026Changxin Tian, Kunlong Chen, Jia Liu +3Language Model Scaling LawsSparse Mixture-of-Experts

    Ling Team, Ant Group

  25. Distribution-Consistent Inference for Dynamic Sparse Mixture-of-Experts

    Sep 8, 2026Dohyeon Kim, Bedionita Soro, Sung Ju HwangParameter-Free Mixture-of-Experts RoutingSparse Mixture-of-Experts

    KAIST

  26. Towards a Statistical Understanding of Mixture-of-Experts

    Sep 3, 2026Siyuan He, Bokai Yang, Jie Hu +2Mixture of ExpertsSparse Mixture-of-Experts

    Qiuzhen College, Tsinghua University, Beijing, China. · KLATASDS-MOE, School of Statistics, East China Normal University, Shanghai, China. · Yau Mathematical Sciences Center, Tsinghua University, Beijing, China. +1

  27. Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts

    Sep 2, 2026Kirill Labzin, Stepan Kulibaba, Artem Dzhalilov +1Sparse Mixture-of-ExpertsExpert Routing

    Central University, Moscow · Innopolis University, Innopolis · Independent Researcher

  28. WiSDoM: Wireless Sparse Decision Transformer with Mixture-of-Experts for Multi-Task Mobile Network Optimization

    Aug 31, 2026Fatih Temiz, Shavbo Salehi, Melike Erol-KantarciSparse Mixture-of-ExpertsDecision Transformer

    School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada

  29. Improving Parameter Utilization by Sharing Neural Experts Across Layers in Transformers

    Aug 31, 2026Dian Jiao, Jiaxin Duan, Shuai Zhao +3Mixture-of-Experts Language ModelsMixture-of-Experts Models

    China Electronics Cloud Technology Co., Ltd.

  30. Task-State Adaptation with Prototype Memory for Multi-Task Dense Prediction

    Aug 28, 2026Yangyang Xu, Haobo Yuan, Yuzhu Wang +4Multi-Task LearningVision Foundation Model Adaptation

    Tsinghua University · University of California, Merced · Shanghai Jiao Tong University