Mixture-of-Experts Models

Follow this topic Ask about this topic

Momentum

41 papers in the last four weeks, up 116% on the four weeks before. 0.4% of all new papers.

Jul 13Week of Sep 28

Latest papers 297

All topics

RoutePack: Expert Placement and Attention-Aware Data Packing for MoE Reinforcement Learning
Aug 12, 2026Yibo Shen, Xudong Han, Xiaowei Zhu +2RL for Language Models Expert Load Balancing
Ant Group
FM-LLM: A frequency-enhanced mixture-of-experts framework for adapting LLMs to time series forecasting
Aug 12, 2026Rentao Gu, Yihang Ding, Junjie Li +5Zero-Shot Time Series Forecasting Time Series Forecasting
Beijing University of Posts and Telecommunications, Beijing, 100876, China · China Telecom Research Institute, Beijing, 100032, China
CLEAR: Class-wise Expert Aggregation with Structured Sampling for Long-Tailed Classification
Aug 11, 2026Gawon LimLong-Tailed Classification Ensemble Learning
School of Information Sciences University of Illinois Urbana-Champaign
Mixture-of-Experts-based Entropy Model for Learned Image Compression
Aug 11, 2026Jonas Brenig, Radu TimofteLearned Image Compression Mixture-of-Experts Models
Computer Vision Lab, CAIDAS & IFI, University of Würzburg, Germany
MoE Proxy Models for Low-Cost Failure Reproduction and Diagnosis in LLM RL Post-Training
Aug 11, 2026Yikai Wang, Chuansai Zhou, Yuhang Zhou +10RL Post-Training Mixture-of-Experts Models
State Key Laboratory for Novel Software Technology, Nanjing University, China · Huawei, China
MammoMix: Leveraging Mixture of Experts for Robust Mammogram Breast Detection
Aug 11, 2026Dinh Tan Nguyen, Hoang Quan Dang, Chen Zhang +1Domain Generalization Mammography
School of Biomedical Engineering, University of Technology Sydney · School of Computer Science, University of Technology Sydney · School of Electrical and Data Engineering, University of Technology Sydney
Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation
Aug 11, 2026Gongli Zhang, Zhulin Liu, C. L. Philip ChenMixture of Experts Mixture-of-Experts Inference
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science and Engineering, South China University of Technology, Guangzhou 510006, China · Pazhou Lab, Guangzhou 510335, China · Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou 510641, China
SonicWeave: Chunk-Routed Mixture-of-Experts for Unified Audio Scene Generation
Aug 10, 2026Yunrui Cai, Xu Li, Yucheng Zhou +8Text-to-Audio Generation Mixture-of-Experts Models
Kling Team, Kuaishou Technology · The Chinese University of Hong Kong · Shenzhen International Graduate School, Tsinghua University
MixFormer: Linear Transformer with Mixture of Memory Experts
Aug 10, 2026Yu Guo, Lei DuanMemory-Augmented Neural Networks Transformer
School of Computer Science, Sichuan University Chengdu, China · School of Artificial Intelligence, Sichuan University Chengdu, China
Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts
Aug 9, 2026Zongfei LiMixture-of-Experts Language Models Sparse Mixture-of-Experts
AdapterMoE: A Two-Stage Hard-Routing Mixture-of-Experts Architecture for Multi-Crop Disease Recognition with Calibrated Rejection and Incremental Learning
Aug 9, 2026Pin-Hsun Huang, Shaou-Gang MiaouClass-Incremental Learning Plant Disease Classification
Department of Electronic Engineering, Chung Yuan Christian University, Taoyuan, Taiwan
The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism
Aug 9, 2026Jiguo LiExpert Parallelism Expert Load Balancing
This report was completed with the assistance of Codex.
UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models
Aug 9, 2026Lei Xin, Bin Gu, Peize Li +8Mixture of Experts Mixture-of-Experts Inference
Kuaishou Technology · Hohai University · Wuhan University +4
Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
Aug 8, 2026Ali Janati, Kaoutar El Maghraoui, Chengke Zou +2Mixture-of-Experts Pruning Mixture-of-Experts Models
Data Science Institute, Columbia University, New York, NY, USA · Department of Computer Science, Columbia University, New York, NY, USA
Policy-Masked Private Experts: Auditable and Reversible Capability Access Control in Sparse MoE Models
Aug 7, 2026Zhuoheng Huang, Mukesh SinghLLM Auditing Sparse Mixture-of-Experts
Independent Researcher
MobileWAM: Bridging World Action Models to Mobile Manipulation with Chain-of-Foresight
Aug 5, 2026Zehua Fan, Junjie He, Wenxuan Song +14World Models for Robotics Mobile Manipulation
Institute for AI Industry Research (AIR), Tsinghua University · 2Shanghai Jiao Tong University · 3The Hong Kong University of Science and Technology (Guangzhou) +8
Beyond Global Routing Aggregation: Phase-Aware Expert Merging for MoE Vision-Language Models
Aug 5, 2026Hongyu Zhang, Cheng Yan, Xiang Xia +1Vision-Language Models Mixture of Experts
University of Science and Technology of China, Hefei, China
MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training
Aug 5, 2026Masato FujitakeNeural Network Optimization Memory-Efficient Optimization
FA Research, Fast Accounting Co., Ltd.
SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specialization
Aug 4, 2026Boyao Wang, Zhihan LeiParameter-Free Mixture-of-Experts Routing Adaptive Model Routing
Machine Learning Department Carnegie Mellon University
MoEGen: Mixture-of-Experts for Instance-Adaptive LoRA Generation
Aug 4, 2026Yiming Zeng, Lei Lu, Zexin Li +10LLM Fine-Tuning Low-Rank Adaptation
University of Connecticut · Northeastern University · Nanyang Technological University +5
Context-Aware Mixture of Domain Experts for Bodily Expression of Emotion in the Wild
Aug 3, 2026Mohammad Mahdi Dehshibi, David MasipMultimodal Emotion Recognition Mixture-of-Experts Models
Department of Computer Science, University of the West of England, Bristol, U.K. · Department of Computer Science, Universitat Oberta de Catalunya, Barcelona, Catalonia, Spain.
STEAM: A Spatio-TEmporal Alignment Mixture-of-Experts Model with Hierarchical Pre-training for EEG Decoding
Aug 3, 2026Zhu Chen, Dingkun Liu, Yuheng Chen +1Brain-Computer Interfaces EEG Decoding
Ministry of Education Key Laboratory of Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China · Zhongguancun Academy, Beijing 100094, China
Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget
Aug 2, 2026Zhichao Xu, Xueguang Ma, Shengyao Zhuang +5Expert Parallelism Cross-Encoder Reranking
University of Utah · University of Waterloo · The University of Queensland +2
Beyond Routing Weights: Faithful Response-Level Interpretation of Mixture-of-Experts Reward Models via Contribution Contrast
Jul 31, 2026Yifan Wang, Jinyi Mu, Mayank Jobanputra +4Reward Modeling Neural Network Interpretability
Saarland University · University of California San Diego · Bielefeld University +2
Weight-Space Mixture-of-Experts for Implicit Neural Representation Classification
Jul 31, 2026Stanislaw Janik, Michal ByraImplicit Neural Representations Neural Network Interpretability
Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland · Samsung AI Center, Warsaw, Poland
MoPET: Parameter-Efficient Mixture-of-Experts for Unified Medical Image Classification
Jul 31, 2026Sebastian Doerrich, Daniel Würtinger, Francesco Di Salvo +2Medical Imaging Foundation Models Medical Image Classification
xAILab Bamberg, University of Bamberg, Bamberg, Germany
Heterogeneous Ranking in Industrial-Scale Recommender Systems: A Case Study
Jul 30, 2026Di Bai, Jintao Liu, Zhenwei Tang +3Learning to Rank Multi-Task Learning
Google LLC Mountain View, CA, USA · Google LLC Toronto, ON, Canada · Google LLC New York, NY, USA +1
TIER-MoE: Trust-Informed Expert Routing via Conditional Modality Risk for Multimodal Fusion in Biomedical Classification
Jul 29, 2026Yu Chang, Anzhe Cheng, Chenwei Wu +7Sparse Mixture-of-Experts Multimodal Classification
University of Southern California · University of Michigan · 3The University of British Columbia +1
Mixture-of-experts for handwriting trajectory reconstruction from IMU sensors
Jul 29, 2026Florent Imbert, Eric Anquetil, Yann Soullard +1Mixture-of-Experts Models
Data Fusion and Contrastive Alignment for Unconstrained IR Molecular Structure Elucidation
Jul 28, 2026Ethan J. Mick, Campbell A. Sweet, Matthias J. Young +1Mixture-of-Experts Models
Electrical Engineering and Computer Science University of Missouri Columbia, MO 65211 · Chemical and Biomedical Engineering University of Missouri Columbia, MO 65211

Aug 12, 2026, cs.DCJ/K move · Enter open · S save

RoutePack: Expert Placement and Attention-Aware Data Packing for MoE Reinforcement Learning

Yibo Shen, Xudong Han, Xiaowei Zhu, Gen Li+1

Ant Group

RL for Language Models Expert Load Balancing Mixture-of-Experts Language Models Mixture-of-Experts Models

Open paper AskPDF arXiv

Training Mixture-of-Experts (MoE) models for reinforcement learning (RL) couples two load-balancing problems: sequence composition determines dense attention work in each data-parallel microbatch, while token routing determines sparse expert work on expert-parallel ranks. Optimizing either alone can shift the bottleneck to the other. In MoE RL, rollout-time routing replay exposes every sample's sequence length and layer-wise expert demand before its training step. We present RoutePack, a hierarchical planner that coordinates state-consistent, layer-wise expert rerouting with joint attention- and expert-aware data packing over an optimizer-step window. RoutePack first places experts independently at each MoE layer using aggregate routing demand. It then packs samples into the smallest certified, or best-known feasible, number of token-capped execution rows and optimizes their DP layout with a projected EDP-shard-aware objective. The objective combines a window-normalized linear-quadratic attention proxy with per-layer physical EP-rank peaks and minimizes the accumulated cost of the slowest EDP shard. Parallel population annealing searches fixed-row feasible layouts while preserving sample coverage, capacity, nonempty cells, equal microbatch counts, and communicator topology. State-consistent materialization preserves logical top-k routing and existing MoE kernels without microbatch-level expert replication. Across Ling-3.0-Tiny and Ling-3.0-Flash, expert rerouting improves mean trainer-measured token throughput by 3.80% and 10.50%, while routing-aware packing adds another 4.86% and 3.98%, respectively. Overall, RoutePack improves throughput by 8.85% and 14.89% over the baseline.

Momentum

Related topics

Latest papers 297