Sparse Mixture-of-Experts

Ask about this topic

Overview

Sparse mixture-of-experts models use a router to send each token or input to only a few expert networks, aiming to combine large model capacity with manageable computation. Common approaches include top-k and threshold routing, hierarchical or shared experts, load-balancing objectives, and upcycling dense models into expert-based ones. Research spans language models as well as vision, speech, forecasting, and multimodal tasks, with work on training stability, pruning, quantization, and efficient inference. A central tension is that balancing expert use can prevent collapse but may weaken specialization, while high stored capacity does not always translate into better results or lower end-to-end cost.

Latest papers

CardsList

121 papers16 papers in the last four weeks, up 33% on the four weeks before. 0.1% of all new papers.

  1. Affinity Is Not Enough: Recovering the Free Energy Principle in Mixture-of-Experts

    May 1, 2026Man Yung WongSparse Mixture-of-ExpertsMixture-of-Experts Inference

    Russell · Independent Researcher

  2. The Thinking Pixel: Recursive Sparse Reasoning in Multimodal Diffusion Latents

    Apr 28, 2026Yuwei Sun, Yuxuan Yao, Hui Li +1Diffusion TransformerConditional Image Generation

    Shanghai Academy of AI for Science · Fudan University

  3. Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning

    Apr 24, 2026Haoze He, Xingyuan Ding, Xuan Jiang +5Supervised Fine-TuningFine-Tuning

    Carnegie Mellon University · MIT · UCSD +1

  4. Teacher-Guided Routing for Sparse Vision Mixture-of-Experts

    Apr 23, 2026Masahiro Kada, Ryota Yoshihashi, Satoshi Ikehata +2Adaptive Model RoutingSparse Mixture-of-Experts

    Institute of Science Tokyo · Tokyo, Japan · DENSO IT Laboratory

  5. Quadruped Parkour Learning: Sparsely Gated Mixture of Experts with Visual Input

    Apr 21, 2026Michael Ziegltrum, Jianhao Jiao, Tianhu Peng +2Robot Policy LearningSparse Mixture-of-Experts

    Robot Perception and Learning Lab, Intelligent Robotics, Department of Computer Science, University College London, Gower Street, WC1E 6BT, London, UK.

  6. Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentation

    Apr 15, 2026Svetlana Pavlitska, Haixi Fan, Konstantin Ditschuneit +1Sparse Mixture-of-ExpertsMixture-of-Experts Models

    FZI Research Center for Information Technology · Karlsruhe Institute of Technology (KIT)

  7. Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling

    Apr 15, 2026Sanghyeok Chu, Pyunghwan Ahn, Gwangmo Song +3Mixture-of-Experts Expert SpecializationMixture-of-Experts Models

    ECE, Seoul National University · LG AI Research · University of Michigan +1

  8. Plasticity-Enhanced Multi-Agent Mixture of Experts for Dynamic Objective Adaptation in UAVs-Assisted Emergency Communication Networks

    Apr 10, 2026Wen Qiu, Zhiqiang He, Wei Zhao +1UAV CommunicationReinforcement Learning

    Department of Information and Communication Engineering, Kitami Institute of Technology, Japan. · Graduate School of Informatics and Engineering, the University of Electro-Communications, Japan. · School of Computer Science and Technology, Anhui University of Technology, China. +1

  9. MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale

    Apr 8, 2026Tobias Falke, Nicolas Anastassacos, Samson Tan +6Mixture of ExpertsSparse Mixture-of-Experts

    Amazon AGI

  10. Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees

    Apr 7, 2026Mohammed Nowaz Rabbani Chowdhury, Kaoutar El Maghraoui, Hsinyu Tsai +4Mixture-of-Experts QuantizationNeural Network Quantization

    Rensselaer Polytechnic Institute · IBM Research

  11. Do Domain-specific Experts exist in MoE-based LLMs?

    Apr 7, 2026Giang Do, Hung Le, Truyen TranMixture-of-Experts Language ModelsMixture-of-Experts Models

    Applied Artificial Intelligence Intiative (A2I2), Deakin University

  12. HI-MoE: Hierarchical Instance-Conditioned Mixture-of-Experts for Object Detection

    Apr 6, 2026Vadim Vashkelis, Natalia TrukhinaObject DetectionMixture-of-Experts Models

  13. SARES-DEIM: Sparse Mixture-of-Experts Meets DETR for Clutter-Aware SAR Ship Detection

    Apr 5, 2026Fenghao Song, Shaojing Yang, Xi ZhouObject DetectionRemote Sensing Small Object Detection

    Yunnan Normal University, No. 768, Jucheng Avenue, Chenggong District, Kunming 650500, China.

  14. Adaptive Semantic Communication for Wireless Image Transmission Leveraging Mixture-of-Experts Mechanism

    Apr 3, 2026Haowen Wan, Qianqian YangAdaptive Model RoutingSparse Mixture-of-Experts

    Polytechnic Institute, Zhejiang University, Hangzhou, China · College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, China

  15. Distilled Large Language Model-Driven Dynamic Sparse Expert Activation Mechanism

    Mar 21, 2026Qinghui Chen, Zekai Zhang, Zaigui Zhang +5Industrial Anomaly DetectionIndustrial Inspection

    School of Control Science and Engineering, Shandong University, Jinan, China · Laoshan Laboratory, Qingdao, China · Jinan Inspur Data Technology Co., Ltd, Jinan, China +2

  16. Path-Constrained Mixture-of-Experts

    Mar 18, 2026Zijin Gu, Tatiana Likhomanenko, Vimal Thilak +2Mixture-of-Experts Language ModelsSparse Mixture-of-Experts

    Apple · †Google

  17. Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancing

    Mar 12, 2026Hanchi Sun, Yixin Liu, Yonghui Wu +1Expert RoutingMixture-of-Experts Models

    Computer Science and Engineering, Lehigh University, Bethlehem, PA, USA · MD-HOBI-BIOMED INFORMATICS, University of Florida, Gainesville, FL, USA

  18. Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers

    Mar 11, 2026Mynampati Sri Ranganadha AvinashMixture-of-Experts InterpretabilityMixture-of-Experts Language Models

    Independent Researcher | Asthra Labs

  19. FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach

    Mar 9, 2026Ning Liao, Xiaoxing Wang, Xiaohan Qin +1Sparse Mixture-of-ExpertsMixture-of-Experts Models

    Shanghai Jiao Tong University

  20. Mixture of experts architectures for machine learning interatomic potentials

    Mar 9, 2026Yuzhi Liu, Duo Zhang, Anyang Peng +3Machine Learning Interatomic PotentialsMixture-of-Experts Models

    AI for Science Institute, Beijing 100080, P. R. China. · DP Technology, Beijing 100080, P. R. China. · Academy for Advanced Interdisciplinary Studies, Peking University, Beijing 100871, P. R. China. +3

  21. Swimba: Switch Mamba Model Scales State Space Models

    Mar 6, 2026Zhixu Du, Krishna Teja Chitty-Venkata, Murali Emani +3State Space ModelsSelective SSMs

    Duke University · Red Hat, Inc. · Argonne National Laboratory

  22. EvoESAP: Non-Uniform Expert Pruning for Sparse MoE

    Mar 6, 2026Zongfang Liu, Shengkun Tang, Boyang Sun +2Mixture-of-Experts PruningLLM Pruning

    Zhejiang University, Hangzhou, Zhejiang, China · Westlake University, Hangzhou, Zhejiang, China · Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates

  23. Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guarantees

    Mar 3, 2026Mohammed Nowaz Rabbani Chowdhury, Hsinyu Tsai, Geoffrey W. Burr +3Mixture-of-Experts InferenceHeterogeneous Computing

    Rensselaer Polytechnic Institute · IBM Research

  24. DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks

    Mar 2, 2026Gökdeniz GülmezMixture-of-Experts ModelsExpert Routing

    Machine Learning Research, Stuttgart, Germany

  25. Excitation: Momentum For Experts

    Feb 25, 2026Sagi ShaierSparse Mixture-of-ExpertsMixture of Experts

    Aleph Alpha Research, Heidelberg, Germany.

  26. Modality-Guided Mixture of Structured Experts with Entropy-Triggered Routing for Multimodal Recommendation

    Feb 24, 2026Ji Dai, Quan Fang, DeSheng CaiSparse Mixture-of-ExpertsRecommender Systems

    Beijing University of Posts and Telecommunications, China · Beijing University of Posts and Telecommunications, Beijing, China · Tianjin University of Technology, China

  27. A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs

    Feb 23, 2026Zijie Liu, Jie Peng, Jinhao Duan +7Expert Load BalancingMixture of Experts

    University of North Carolina at Chapel Hill · University of Minnesota Twin Cities · North Carolina State University +2

  28. Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds

    Feb 19, 2026Ibne Farabi Shihab, Sanjeda Akter, Anuj SharmaExpert RoutingMixture-of-Experts Language Models

    Department of Computer Science, Iowa State University · Department of Civil, Construction and Environmental Engineering, Iowa State University

  29. Arcee Trinity Large Technical Report

    Feb 19, 2026Varun Singh, Lucas Krauss, Sami Jaghouar +23LLM TrainingMixture-of-Experts Language Models

    Arcee AI · Prime Intellect · DatologyAI

  30. Shape Distribution Matters: Shape-Specific Mixture-of-Experts for Amodal Segmentation Under Diverse Occlusions

    Aug 3, 2025Zhixuan Li, Yujia Liu, Chen Hui +3Amodal SegmentationOcclusion Reasoning

    College of Computing and Data Science, Nanyang Technological University, Singapore · School of Computer Science, Peking University, Beijing, China · National Key Laboratory for Multimedia Information Processing, Peking University, Beijing, China +3