Mixture-of-Experts Models

Follow this topic Ask about this topic

Recent momentum

-34%

43 papers in the last 28 days · 0.7% of indexed attention

Twelve weeks of publication activity for this topic as it is defined today.

Weekly history

Recent digests

What was published in this topic, kept on the site without email delivery.

Period ending 2026-09-21

14 new papers

A weekly snapshot of new work published in Mixture-of-Experts Models.

Period ending 2026-09-14

2 new papers

A weekly snapshot of new work published in Mixture-of-Experts Models.

Period ending 2026-09-07

12 new papers

A weekly snapshot of new work published in Mixture-of-Experts Models.

419 papers

Latest in Mixture-of-Experts Models

Browse all topics →

CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning
Jun 1, 2026Jun-Tao Tang, Zhen-Hao Xie, Yu-Cheng Shi +1Multimodal Instruction Tuning Instruction Tuning
ResMerge: Residual-based Spectral Merging of Large Language Models
Jun 1, 2026Yandu Sun, Zhiyan Hou, Haokai Ma +6Model Merging Mixture-of-Experts Models
Hierarchically Decoupled Mixture-of-Experts for Robust Traffic Sign Recognition in Complex Driving Scenarios
Jun 1, 2026Mingxiao Wang, Xiaozhen Qu, Bolin Gao +2Traffic Sign Recognition Autonomous Driving
Gate the Filter, Not the Message: Node-Channel Mixtures for Pre-Propagation GNNs
Jun 1, 2026Zichao Yue, Zhiru ZhangGraph Neural Networks Propagation
Deft Scheduling of Dynamic Cloud Workflows with Varying Deadlines via Mixture-of-Experts
May 31, 2026Ya Shen, Gang Chen, Hui Ma +1Scheduling Deadlines
DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts
May 31, 2026Jiarui Feng, Hanqing Zeng, Karish Grover +11Mixture-of-Experts Models
Learning Multi-Modal Trajectory Policies for Data-Efficient Robotic Manipulation
May 31, 2026Zijia Chen, Yuenan Hou, Xinhua Jiang +3Generalizable Robotic Manipulations Trajectory Prediction
Beyond Task-Agnostic: Task-Aware Grouping for Communication-Efficient Multi-Task MoE Inference
May 31, 2026Zhiyao Xu, Aoxue Liu, Zhanjie Ding +3Mixture-Of-Expert Inference Mixture-of-Experts Models
Confidence-Adaptive SwiGLU for Mixture-of-Experts
May 30, 2026Shaohua Li, Xiuchao Sui, Xiaobing Sun +4Mixture-of-Experts Models Sharpness-Aware Minimization
ViBE: Co-Optimizing Workload Skew and Hardware Variability for MoE Serving
May 30, 2026Seokjin Go, Marko Scrbak, Ephrem Wu +2Mixture-Of-Expert Inference Inference Workloads
MESA: Improving MoE Safety Alignment via Decentralized Expertise
May 30, 2026Yitong Sun, Yao Huang, Teng Li +5Safety Alignment Mixture-Of-Expert Language Models
PithTrain: A Compact and Agent-Native MoE Training System
May 29, 2026Ruihang Lai, Hao Kang, Haozhan Tang +6Mixture-Of-Expert Language Models Mixture-of-Experts Models
Mellum2 Technical Report
May 29, 2026Marko Kojic, Ivan Bondyrev, Aral de Moor +6Mixture-Of-Expert Language Models Mixture-of-Experts Models
Eigenvectors of Experts are Training-free Non-collapsing Routers
May 29, 2026Giang Do, Hung Le, Truyen TranMixture-of-Experts Models Experts
dMoE: dLLMs with Learnable Block Experts
May 29, 2026Sicheng Feng, Zigeng Chen, Gongfan Fang +2Mixture-of-Experts Models Diffusion Language Models
DAMEL: Dual-Axis Multi-Expert Learning for Class-Imbalanced Learning
May 28, 2026Hyuck Lee, Taemin Park, Heeyoung KimExtreme Class Imbalance Mixture-of-Experts Models
Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptation
May 28, 2026Aditi Khandelwal, Marius Mosbach, Verna Dankers +2Multilingual Language Models Mixture-of-Experts Models
ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compression
May 28, 2026Yilun Yao, Jiaming Pan, Elsie Dai +3Mixture-Of-Expert Language Models Mixture-of-Experts Models
A Minimal Bifurcation Model of Load Imbalance in a Softmax Mixture-of-Experts Router
May 27, 2026O. M. KiselevLoad Balancing Mixture-of-Experts Models
Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models
May 27, 2026Guanzhi Deng, Kuan Wu, Haibo Wang +3Mixture-Of-Expert Language Models Mixture-of-Experts Models
How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving
May 27, 2026Hanjiang Wu, Abhimanyu Rajeshkumar Bambhaniya, Sarbartha Banerjee +9Mixture-Of-Expert Inference Mixture-Of-Expert Language Models
VidPrism: Heterogeneous Mixture of Experts for Image-to-Video Transfer
May 27, 2026Rui Lin, Chuanming Wang, Huadong MaMixture-of-Experts Models Motion-Focused Representations
Pruning and Distilling Mixture-of-Experts into Dense Language Models
May 27, 2026Junhyuck Kim, Jihun Yun, Haechan Kim +3Mixture-Of-Expert Language Models Mixture-of-Experts Models
FPMoE: A Sparse Mixture-of-Experts Approach to Functional Code Generation
May 27, 2026Loc Pham, Lang Hong Nguyet Anh, Thanh Le-CongMixture-Of-Expert Language Models Code Generation
ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference
May 26, 2026Xiongwei Zhu, Xiaojian Liao, Tianyang Jiang +3Mixture-Of-Expert Inference LLM Inference Efficiency
Towards Generalization-Oriented Models for Vehicle Routing Problems with Mixture-of-Experts
May 26, 2026Changhao Miao, Yuntian Zhang, Tongyu Wu +2Vehicle Routing Problem Mixture-of-Experts Models
L2Rec: Towards Dual-View Understanding of LLMs for Personalized Recommendation
May 26, 2026Pingjun Pan, Tingting Zhou, Peiyao Lu +3Recommendation Personalization
Dense2MoE: Pushing the Pareto Frontier of On-Device LLMs via Unified Pruning and Upcycling
May 26, 2026Fengfa Li, Hongjin Ji, Yifeng Ding +2Mixture-Of-Expert Inference Mixture-of-Experts Models
VEN-VL: A Visual Ensemble MoE Framework for Effective and Efficient Multi-Modal Understanding
May 25, 2026Yinghao Wu, Zhuoyan Luo, Yiyao Yu +3Universal Multimodal Visual Tokens
RotMoLE: Enhancing Mixture of Low-Rank Experts through Rotational Gating Mechanism
May 25, 2026Mengyang Sun, Maochuan Dou, Tao Feng +5Mixture-Of-Expert Language Models Mixture-of-Experts Models
Cross-Stage Attention Multi-Expert Network for Radiologist-Inspired Breast Ultrasound Diagnosis
May 25, 2026Xinyang Zhai, Chong Yang, Ruizhi ZhangMammography Ultrasound
AME-TS: Anchored Mixture-of-Experts for Time Series Forecasting
May 24, 2026Rui Wang, Renhao Xue, Ray Razi +2Time Series Foundation Models Time Series
Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs
May 23, 2026Bo Li, Tianyu Dong, Shaolin Zhu +1Mixture-of-Experts Models Text Corpora
ConceptM $^3$ oE: Concept-Guided Multimodal Mixture of Experts for Interpretable Computational Pathology
May 23, 2026Xuan Wang, Zhongling Xu, Gopi Kannedhara +13Pathology Foundation Models Multimodal Clinical Data
Safety-Oriented Routing Analysis of Mixtral MoE Under Benign and Harmful Prompts
May 22, 2026Md Nurul Absar SiddikyMixture-Of-Expert Language Models Large Language Model Routing
Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models
May 22, 2026Hongwu Peng, Ohiremen Dibua, Yuanjun Xiong +3Mixture-of-Experts Models Large Language
Semantically Structured Mixture-of-Experts for Compositional Robotic Manipulation
May 22, 2026Chengyu Deng, Guanqi Chen, Yizhou Chen +4Generalizable Robotic Manipulations Diffusion Policies
CARE: Class-Adaptive Expert Consensus for Reliable Learning with Long-Tailed Noisy Labels
May 22, 2026Mengke Li, Haiquan Ling, Lihao Chen +3Noisy Labels Vision-Language Model Adaptation
Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey
May 22, 2026Liangwei Nathan Zheng, Wei Emma Zhang, Olaf Maennel +2Robust Multimodal Learning Mixture-of-Experts Models
SpikingMoE: SDPrompt-Guided Dynamic Expert Fusion in Spiking Neural Networks
May 22, 2026Yukai Yang, Chenxi Qin, Jungang Li +3Spiking Neural Networks Binary Spikes
GMENet: Generative Mixture of Experts Network for Multi-Center Glioma Diagnosis with Incomplete Imaging Sequences
May 22, 2026Pengfei Song, Fangjin Liu, Wenwen Zeng +5Glioma Magnetic Resonance Imaging
GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMs
May 21, 2026Jianing Deng, Song Wang, Dongwei Wang +4Mixed-Precision Quantization Mixture-Of-Expert Language Models
FAME: Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection
May 21, 2026Huanchi Wang, Zihang Huang, Yifang Tian +3Mixture-of-Experts Models
Dynamic Mixture of Latent Memories for Self-Evolving Agents
May 21, 2026Dianzhi Yu, Vireo Zhang, Hongru Wang +7Catastrophic Forgetting Self-Evolving Agents
PALS: Power-Aware LLM Serving for Mixture-of-Experts Models
May 20, 2026Can Hankendi, Rana Shahout, Minlan Yu +1LLM Inference Efficiency Single Graphics Processing Unit
FedCoE: Bridging Generalization and Personalization via Federated Coordinated Dual-level MoEs
May 20, 2026Penglin Dai, Fulian Li, Xincao Xu +3Federated Learning Fedavg
Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory
May 20, 2026Bole Ma, Jan Eitzinger, Harald Koestler +1Load Balancing Mixture-of-Experts Models
Task-Routed Mixture-of-Experts with Cognitive Appraisal for Implicit Sentiment Analysis
May 20, 2026Yaping Chai, Haoran Xie, Joe S. QinSentiment Analysis Mixture-of-Experts Models
HDMoE: A Hierarchical Decoupling-Fusion Mixture-of-Experts Framework for Multimodal Cancer Survival Prediction
May 20, 2026Huayi Wang, Haochao Ying, Yuyang Xu +5Multimodal Clinical Data Mixture-of-Experts Models
Dynamic TMoE: A Drift-Aware Dynamic Mixture of Experts Framework for Non-Stationary Time Series Forecasting
May 20, 2026Jiawen Zhu, Shuhan Liu, Di Weng +1Non-Stationarity Mixture-of-Experts Models
LER-YOLO: Reliability-Aware Expert Routing for Misaligned RGB-Infrared UAV Detection
May 20, 2026Liming Hou, Yueping Peng, Hexiang Hao +6Unmanned Aerial Vehicle Detection Infrared
Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts
May 20, 2026Gene Tangtartharakul, Katherine R. StorrsMixture-of-Experts Models Experts
GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
May 19, 2026Sourish Wawdhane, Avinash Kumar, Poulami DasMixture-Of-Expert Inference Mixture-of-Experts Models
A novel YOLO26-MoE optimized by an LLM agent for insulator fault detection considering UAV images
May 19, 2026João Pedro Matos-Carvalho, Laio Oriel Seman, Stefano Frizzo Stefenon +2Yolo26 Aerial Imagery
FPED: A Functional-Network Prior-Guided Mixture-of-Experts Framework for Interpretable Brain Decoding
May 19, 2026Yudan Ren, Pengcheng Shi, Zihan Ma +2Human Brain Networks Brain Magnetic Resonance Imaging
Post-Trained MoE Can Skip Half Experts via Self-Distillation
May 18, 2026Xingtai Lv, Li Sheng, Kaiyan Zhang +12Mixture-of-Experts Models Sparse Expert Activation
DBES: A Systematic Benchmark and Metric Suite for Evaluating Expert Specialization in Large-Scale MoEs
May 18, 2026Jing Wang, Hongxuan Lu, Jazze Young +2Mixture-of-Experts Models
A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM $Δ$ Integration into Upcycled MoE
May 18, 2026Hao Zhou, Tianhao Li, Zhijun Wang +6Multilingual Large Language Models Mixture-Of-Expert Language Models
CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
May 18, 2026Yang Liu, Toan Nguyen, Flora D. SalimEfficient Continual Learning Mixture-Of-Expert Language Models
CoX-MoE: Coalesced Expert Execution for High-Throughput MoE Inference with AMX-Enabled CPU-GPU Co-Execution
May 18, 2026Muyoung Son, Yi Chen, Seungjae Yoo +2Mixture-Of-Expert Inference Mixture-of-Experts Models

Jun 1, 2026 · cs.CLJ/K move · Enter open · S save

CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning

Jun-Tao Tang, Zhen-Hao Xie, Yu-Cheng Shi, Da-Wei Zhou

School of Artificial Intelligence, Nanjing University, China · State Key Laboratory of Novel Software Technology, Nanjing University, China

Multimodal Large Language Models (MLLMs) unify heterogeneous vision-language tasks under a shared generative framework via instruction tuning, yet real-world deployment demands continuous capability expansion, making Multimodal Continual Instruction Tuning (MCIT) essential. Existing methods either update all tasks with a shared parameter set or allocate dedicated modules for each new task. Shared updates force heterogeneous tasks to compete, causing forgetting of learned capabilities. Conversely, isolated expansion prevents interference but severely limits parameter efficiency over long task streams. To address this dilemma, we propose CRAM. Specifically, by isolating task-specific patterns into independent modules, CRAM mitigates catastrophic forgetting across tasks. To further boost parameter efficiency, we utilize adaptive-rank instantiation to identify the capability gap between existing expert capability and new task demands, and dynamically allocate only the necessary parameters. To ensure stable reuse among tasks, centroid-guided routing recognizes and activates existing experts' capabilities, while an orthogonality penalty confines new updates to task-specific directions, preventing re-learning general capability. Extensive experiments across diverse benchmarks consistently demonstrate its superiority over existing methods.

Multimodal Instruction Tuning Instruction Tuning Efficient Continual Learning Unseen-Task Adaptation Multi-Modal Large Language Models Mixture-of-Experts Models Instruction

Open paper Ask about it PDF

Recent momentum

Neighbouring directions

Recent digests

14 new papers

2 new papers

12 new papers

Latest in Mixture-of-Experts Models