Mixture-of-Experts Models

Follow this topic Ask about this topic

Momentum

41 papers in the last four weeks, up 116% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 297

All topics

DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition
Oct 8, 2026Yuxuan Lou, Kai Yang, Geng Zhang +2Mixture-of-Experts Models Sparse Mixture-of-Experts
School of Computing, National University of Singapore, Singapore · Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China
Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling
Oct 7, 2026Radha Gulhane, Quentin Anthony, Beren MillidgeMixture-of-Experts Language Models Communication-Efficient Distributed Training
Zyphra
Beyond the Leaderboard: Multi-Dimensional Evaluation of Dense and Mixture-of-Experts Models for Automated Program Repair
Oct 6, 2026Anvi Kalpesh Shah, Umamaheswara Sharma BSoftware Engineering Automated Program Repair
National Institute of Technology, Calicut
MASKerade: Token-Routed Mask Experts for Dense-to-MoE Upcycling
Oct 6, 2026Mingyuan Zhang, Yue Bai, Zhongruo Wang +5Expert Routing Sparse Neural Networks
Department of Electrical and Computer Engineering, Northeastern University · Khoury College of Computer Science, Northeastern University
CoRE: Learning Collaboration-Role Experts for Decentralized Collaborative Manipulation with One Policy
Oct 6, 2026Yanan Zhou, Zhaoyan Qian, Zihao Li +3Multi-Robot Systems Multi-Task Robotic Manipulation
School of Computer Science, The University of Sydney, Australia. · College of Computer Science and Technology, Zhejiang University of Technology, China.
Structuring MoE Expert Selection for Agentic Reinforcement Learning
Oct 5, 2026Bolian Li, Ting-Yao Hu, Cheng-Yu Hsieh +3Agentic RL Expert Routing
Apple · Purdue University
Distributionally Robust Mixture-of-Experts Training
Oct 5, 2026Xin Teng, Muxiao Li, Hongyi WenMixture-of-Experts Language Models Sparse Mixture-of-Experts
New York University Center for Data Science, NYU Shanghai
BRANCH-MoE: Balance-Aware Tree Routing for Large Embedding Models
Oct 5, 2026Gang Fu, Adel Javanmard, MohammadHossein Bateni +1Expert Load Balancing Mixture-of-Experts Models
Google Research · University of Southern California
CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training
Oct 5, 2026Jing Li, Jian Meng, Yingmeng Gao +8Expert Load Balancing Expert Routing
Tongji University · Cornell University · Harbin Institute of Technology, Shenzhen +1
RoMod: Temporal Routing Modulation via Mixture-of-Experts for Video Anomaly Detection
Oct 4, 2026Chao Huang, Pengfei Wei, Benfeng Wang +5Video Anomaly Detection Multimodal Foundation Models
Sun Yat-sen University · University of Macau
InstMoE: Adaptive Multimodal Routing with Specialized Experts
Oct 4, 2026Guimin Hu, Xiang He, Yingjian Li +3Cross-Modal Alignment Multimodal Sentiment Analysis
Guangdong University of Technology · University of Copenhagen · Pengcheng Laboratory +1
Conditional Capacity and Routing in Mixture-of-Experts Particle Transformers
Oct 2, 2026Kaushik Pendiyala, Haris Zia, Trevin Lee +8Mixture-of-Experts Models Sparse Mixture-of-Experts
University of California Davis, Davis, CA, USA · University of California San Diego, San Diego, CA, USA · University of Illinois at Chicago, Chicago, IL, USA +1
Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation
Oct 1, 2026Damiano Marsili, Raphi Kang, Aditya Mehta +2Mixture of Experts Mixture-of-Experts Models
California Institute of Technology
MoLE: Mixture of Latent Experts for Complementary Visual Reasoning
Oct 1, 2026Yingcheng Liu, Tianyi Jiang, Yujuan Ding +5Large Vision-Language Models Latent Visual Reasoning
Tongji University, China · Hong Kong Polytechnic University, Hong Kong · Alibaba Group, China +1
SpikeMoE: Brain-Inspired Competitive Routing for Flexible Spiking Mixture-of-Experts
Oct 1, 2026Xiaoli Liu, Yujie Liang, Jialin Li +1Spiking Neural Networks Expert Routing
School of Computer Science and Engineering, University of Electronic Science and Technology of China
From Task Mixtures to Specialized Experts
Sep 30, 2026Hojat Allah Salehi, Mehrdad Mahdavi, Andrew Arash Mahyari +1Multi-Task Learning Mixture-of-Experts Models
Florida International University, Miami, FL, USA · Security, Optimization, and Learning for InterDependent networks laboratory (solid lab), Miami, FL, USA · The Pennsylvania State University, University Park, PA, USA +1
Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization
Sep 30, 2026Zheng Lin, Shaoke Fang, Yuxin Zhang +6Submodular Optimization Mixture-of-Experts Pruning
Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg, Luxembourg · Department of Computer Science, Peking University, China · Institute of Space Internet, Fudan University, China +4
Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Experts
Sep 30, 2026Priya Nair, Lukas Brenner, Maya Lindqvist +5Mixture-of-Experts Language Models Parameter-Free Mixture-of-Experts Routing
University of California, Irvine · University of Washington
OpenTSLM TeeMoE: A Unified Time-Series Language Model for Forecasting, Contextual Prediction, and Reasoning
Sep 30, 2026Tony Chen, Timo Stoffregen, Maxwell Xu +8Temporal Reasoning in Language Models Time Series Forecasting
Columbia University, USA · Stanford University, USA · Aionic Labs, Switzerland +4
Raw-Routed Mixture of Adapters: A Causal Intervention for Routing Collapse in Time Series Foundation Models
Sep 30, 2026Hung Phan, Thuy T. Nguyen, Minh Ngoc Dinh +1Expert Routing Mixture-of-Experts Models
HAPMoE: Heterogeneity-Aware Automatic Parallelism Planning for Mixture-of-Experts Models Training
Sep 30, 2026Mengyuan Fan, Peizhuang Cong, Zixiao Huang +7Expert Parallelism Heterogeneous Computing
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · Infinigence AI · Tsinghua University
ElectrolyteFM: Unifying Electrolyte Property Prediction through Cross-Property Knowledge Learning
Sep 30, 2026Jiaxin Yu, Shuo Wang, Peng Wang +2Multi-Task Learning Materials Property Prediction
VoltaAI, Beijing, China · Renmin University of China, Beijing, China
MoRA: MoE Pruning via Router Bias Learning and Expert Approximation
Sep 30, 2026Yushuai Sun, Zikun Zhou, Lin Gao +2Mixture-of-Experts Pruning LLM Compression
Harbin Institute of Technology, Shenzhen
Breaking the Uniformity Trap: Scaling Video Diffusion Model via SplitMoE
Sep 29, 2026Yu Xu, Yuxin Zhang, Xiao Yang +7Video Diffusion Models Video Generation
University of Chinese Academy of Sciences · ByteDance · Canva Research +1
E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models
Sep 29, 2026Arseny Ivanov, Alexander Kolesov, Alexander Korotin +2Masked Diffusion Models Few-Step Diffusion Sampling
AXXX, Moscow, Russia · Applied AI Institute, Moscow, Russia
MoRE: Scaling mixture of experts with hardware-aware low-rank routing
Sep 28, 2026Honam Wong, Surbhi Goel, Enric Boix-AdseràParameter-Free Mixture-of-Experts Routing Low-Rank Matrix Decomposition
University of Pennsylvania · The Wharton School, University of Pennsylvania
TopoEP: Topology-Aware Load Balancing for Expert-Parallel MoE Training
Sep 28, 2026Jiacheng Zhu, Xie Zhao, Gongming Zhao +3Expert Parallelism Expert Load Balancing
University of Science and Technology of China
Role-Guided MOE for Encoder-Level Pathology Representation Learning in WSI Classification
Sep 28, 2026Xinyu Ma, Xing Yang, Hongtao Jin +4Pathology Foundation Models Mixture-of-Experts Models
Artificial Intelligence Research Institute, Shenzhen MSU-BIT University, Shenzhen 518172, China · Shenzhen People’s Hospital, Shenzhen 518020, China
A Persistent State for Auditable Mixture-of-Experts Routing
Sep 28, 2026Abdurrahman Javat, Allan KazakovLLM Auditing State Tracking
Department of Artificial Intelligence Bahçeşehir University Istanbul, Türkiye
X-MoD: Practical Scaling Laws for Sparse-Depth Routing Beyond Mixture-of-Depths
Sep 28, 2026Bowen Dong, Yilong Fan, Tengyu Pan +6Language Model Scaling Laws Mixture-of-Experts Models
Tsinghua University · Tianjin University

Oct 8, 2026, cs.AIJ/K move · Enter open · S save

DivMoE: Fine-Grained MoE Upcycling via Cross-Domain Expert Composition

Yuxuan Lou, Kai Yang, Geng Zhang, Yong Liu+1

School of Computing, National University of Singapore, Singapore · Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China

Mixture-of-Experts Models Sparse Mixture-of-Experts Expert Routing

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, with recent work demonstrating the benefits of fine-grained expert designs. Training such models from scratch is expensive, and sparse upcycling from pre-trained dense models is an attractive alternative. However, we identify a structural pathology of fine-grained upcycling: when fine-grained experts are derived from a single source model, naive routing collapses and downstream accuracy drops to near-random (e.g., on Qwen3-1.7B, Drop-Upcycling-fine-grained reaches only 23.2% average accuracy across 15 benchmarks, essentially matching from-scratch training at 22.2%, while the same method's coarse-grained variant reaches 50.2%). We propose DivMoE, the first framework achieving fine-grained MoE Upcycling with structurally-balanced routing. DivMoE introduces domain-specialized fine-grained expert initialization, deriving experts from dense models that have undergone domain-adaptive continual pre-training, and diversity-constrained routing, a hard structural constraint guaranteeing that each token activates experts from distinct domain groups. Across two base models and 15 benchmarks, DivMoE consistently outperforms six upcycling baselines (55.6% vs. 51.6% for the strongest baseline on Qwen3-1.7B) and strictly improves over the dense base model on every benchmark after Stage 2 continual pre-training -- closing the regression gap that has plagued prior fine-grained upcycling. After supervised fine-tuning on a public reasoning mixture, our 12B-parameter DivMoE model matches Moonlight-MoE (16B) at 64.5% average accuracy while outperforming a controlled NVIDIA-Upcycling baseline by 6.1 percentage points.

Momentum

Related topics

Latest papers 297