Overview

Sparse mixture-of-experts models use a router to send each token or input to only a few expert networks, aiming to combine large model capacity with manageable computation. Common approaches include top-k and threshold routing, hierarchical or shared experts, load-balancing objectives, and upcycling dense models into expert-based ones. Research spans language models as well as vision, speech, forecasting, and multimodal tasks, with work on training stability, pruning, quantization, and efficient inference. A central tension is that balancing expert use can prevent collapse but may weaken specialization, while high stored capacity does not always translate into better results or lower end-to-end cost.

Latest papers

CardsList

128 papers11 papers in the last four weeks, down 21% on the four weeks before. 0.1% of all new papers.

  1. Server-Side Sparse Assignment for Federated Mixture-of-Experts

    Nov 4, 2026Xue Zheng, Aylin YenerSparse Mixture-of-ExpertsFederated Learning

  2. When Does Sparse Routing Help? The Geometry of Expert Separation in Mixture of Experts

    Nov 1, 2026Huy Nguyen, Dung Le, Alessandro Rinaldo +1Sparse Mixture-of-ExpertsMixture-of-Experts Models

  3. MoETok: Sparse Expert Routing for 1D Visual Tokenizers

    Oct 30, 2026Jiahao Guo, Bao Tang, Haoxiang Cao +6Visual TokenizationDiscrete Visual Tokenization

  4. Sparse Text Diffusion under Dual Budgets: Context Connectivity and Expert Activation

    Oct 30, 2026Alexandros Christoforos, Chadbourne Davis, Paul RajDiffusion Language ModelsLong-Context Modeling

  5. Anatomical Prior-Guided Routing for 3D Lung Tumor Segmentation

    Oct 29, 2026Uihyeon Maeng, Jonghun Kim, Hyunjin ParkMedical Image Segmentation3D Medical Image Segmentation

  6. Muon Makes the First Mixture-of-Experts Layer Harder to Balance

    Oct 27, 2026Marcos HernanzMixture-of-Experts Language ModelsExpert Load Balancing

  7. GaP-MoE : Explicit Expert Specialization in Mixture-of-Experts via Orthogonal Gradient Accumulation and Projection

    Oct 21, 2026Donghyun Jeong, Yoon-Yeong KimSparse Mixture-of-ExpertsMixture-of-Experts Models

  8. Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE

    Oct 20, 2026Ermo Hua, Xiangyu Hong, Baiting Wu +11Sparse Mixture-of-ExpertsMixture-of-Experts Models

  9. Rethinking Routing Exploration in Modern Mixture-of-Experts

    Oct 15, 2026Bingcong Li, Hao Ma, Michael Muehlebach +2Mixture-of-Experts ModelsExpert Routing

  10. How Many Ranks Does a Token Need? A Noise-Floor Squelch for Rank-One LoRA Experts

    Oct 13, 2026Mei Okonkwo, Owen Castellanos, Ingrid Solberg +1Low-Rank AdaptationAdaptive Rank Allocation

  11. FORGE: Procedural Weight Generation for Sparse MoE Inference

    Oct 12, 2026Chanyoung Moon, Travis Pence, Daisuke Yamada +4Mixture-of-Experts InferenceLLM Inference Acceleration

  12. Smoothing the Top-k Exposure Boundary for Sparse Mixture-of-Experts

    Oct 8, 2026Yunkai Chai, Tong Zhu, Xiaoye Qu +4Mixture-of-Experts Language ModelsSparse Mixture-of-Experts

  13. DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition

    Oct 8, 2026Yuxuan Lou, Kai Yang, Geng Zhang +2Mixture-of-Experts ModelsSparse Mixture-of-Experts

  14. Distributionally Robust Mixture-of-Experts Training

    Oct 5, 2026Xin Teng, Muxiao Li, Hongyi WenMixture-of-Experts Language ModelsSparse Mixture-of-Experts

  15. SEAL: Mixture-Closed Additive Reconstruction and Refinement-Aware Expert Routing for Efficient Speech Separation

    Oct 5, 2026Shao-Chun Hu, Zi-Xiang Lin, Jeih-Weih Hung +1Speech SeparationSparse Mixture-of-Experts

  16. Conditional Capacity and Routing in Mixture-of-Experts Particle Transformers

    Oct 2, 2026Kaushik Pendiyala, Haris Zia, Trevin Lee +8Mixture-of-Experts ModelsSparse Mixture-of-Experts

  17. RouteRec: Behavior-Guided Sparse Routing for Sequential Recommendation

    Sep 30, 2026Junyeong Song, Jaemin YooSequential RecommendationSparse Mixture-of-Experts

  18. Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models

    Sep 29, 2026Yury Nahshan, Nati Daniel, Jacob Goldberger +1Mixture-of-Experts Language ModelsParameter-Free Mixture-of-Experts Routing

  19. Routing in Gradient Space: Balanced Usage Is Not Expert Specialization

    Sep 29, 2026Yuchen Li, Mingyu Du, Zongqi Fan +2Sparse Mixture-of-ExpertsMixture-of-Experts Models

  20. How to Loop MoE: Flatten the Experts, Untie the Attention

    Sep 28, 2026Shouren Wang, Chuang Ma, Mohsen Hariri +6Mixture-of-Experts Language ModelsSparse Mixture-of-Experts

  21. HERO-MoE: Historical Expert Routing with Scale-Preserving Fusion

    Sep 26, 2026Junxiang Qiu, Zhengsu Chen, Xinting Hu +6Mixture-of-Experts Language ModelsLLM Routing

  22. OceanMoE: Structured Conditional Sparse Computation for Long-Horizon Multivariate Ocean Forecasting

    Sep 17, 2026Yishun Zhu, Jian WangMultivariate Time Series ForecastingLong-Term Time Series Forecasting

  23. MoRE: Mixture of Reused Experts

    Sep 16, 2026Eric S. Qiu, Utku Umur Acikalin, Justin Lovelace +4Mixture-of-Experts Language ModelsSparse Mixture-of-Experts

  24. Beyond the Previous Layer: Residual Predictive Structure in Sparse MoE Routing

    Sep 15, 2026Hao Li, Yasuyuki Tahara, Yuichi SeiSparse Mixture-of-ExpertsExpert Routing

  25. ExpertHTR: Unified Handwritten Text Recognition with Multi-Task Learning and Sparse Mixture-of-Experts

    Sep 11, 2026Dang Hoai Nam, Nguyen Duy Hieu, Quang Huu Hieu +1Vision-Language ModelsSparse Mixture-of-Experts

  26. Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

    Sep 10, 2026Atindra Jha, Margaret Li, Jure Leskovec +2Mixture of ExpertsSparse Mixture-of-Experts