Mixture-of-Experts Models

Follow this topic Ask about this topic

Mixture-of-experts models divide computation among expert networks and use a router to select which experts process each input, often activating only a small subset. Common approaches include top-k sparse routing, shared or hierarchical experts, and parameter-efficient expert adapters; researchers also study load balancing to prevent some experts from being overused or ignored. These models are used in language, vision, multimodal learning, forecasting, and robotics, while systems work targets the memory, communication, and latency costs of serving and training them. A central tension is that specialization can improve capacity and efficiency, but balanced routing does not guarantee useful expertise, and routing overhead or weaker performance under changed conditions can offset the gains.

Latest papers

456 papers75 papers in the last four weeks, up 97% on the four weeks before. 0.4% of all new papers.

Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data
Oct 27, 2026Geunhyoung Kim, Sung Ju HwangMixture-of-Experts Inference Efficient Language Model Inference
KAIST
Multi-task Disentangling Fusion Mamba-CNN for MODIS Spectral-Temporal Unmixing with Endmember Variability Modeling
Oct 27, 2026Motasem Alkayid, Yimin Zhu, Naser El-Sheimy +1Spectral Unmixing Mixture-of-Experts Models
Department of Geomatics Engineering, University of Calgary, Canada · Department of Geography, Faculty of Arts, The University of Jordan, Amman, Jordan
Learning What to Share: Expert Specialization for Time-Series Forecasting
Oct 24, 2026Bashar Khoury, Shadi Omari, Assaf SchusterMultivariate Time Series Forecasting Mixture-of-Experts Expert Specialization
Department of Computer Science, Technion – Israel Institute of Technology
FedSEAM: Federated Continual Learning of LLMs via Subspace Expert Agreement Merging
Oct 24, 2026Fatema Siddika, Abhijit Chunduru, Zilinghan Li +2Continual Learning for LLMs Federated Continual Learning
Iowa State University, Ames, USA · Argonne National Laboratory, USA · University of Massachusetts Amherst, Amherst, USA
Accordion: Elastic-Depth Inference via Fixed-Prefix Layer Looping
Oct 23, 2026Yudong Zhang, Wei Ding, Kaili Zheng +2Efficient Language Model Inference LLM Inference Acceleration
Tsinghua University · Baidu Inc.
EDWARD-NMT: Elastic Dimensions With Adaptive Routing and Dual-consistency for Neural Machine Translation
Oct 23, 2026Edward Nguyen, Thai Tran, Kiet Anh Hoang +4Machine Translation Mixture-of-Experts Models
UNEY, Switzerland
GS-MoE: Training-Free MoE Compression via Groupwise Subspace Sharing and Pruning
Oct 23, 2026Carson Cary, Yukai Jin, Yihao Huang +5Mixture-of-Experts Models Mixture-of-Experts Pruning
Florida State University · University of Pennsylvania · University of Notre Dame
Nested Sparsity for Scalable Pretraining and Continual Learning
Oct 22, 2026Tianzhe Chu, Druv Pai, Yi MaConditional Computation Sparse Attention
University of Hong Kong · UC Berkeley
Expandable Modality Experts for Medical Vision-Language Learning
Oct 21, 2026DM BAPPY, Eunsu Kim, Hanjoo Lee +2Medical VLMs Continual Learning for VLMs
Korea University · The Catholic University of Korea
SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs
Oct 21, 2026Yifan Yang, Hanling Yi, Dongquan Yang +4Mixture-of-Experts Pruning Sparse Mixture-of-Experts
Intellifusion Inc.
A Strong MoE Is All You Need for Process Supervision
Oct 21, 2026Guanzhi Deng, Zengyan Liu, Haibo Wang +5Process Supervision Process Reward Models
City University of Hong Kong · Carnegie Mellon University · University of Hong Kong
GaP-MoE : Explicit Expert Specialization in Mixture-of-Experts via Orthogonal Gradient Accumulation and Projection
Oct 21, 2026Donghyun Jeong, Yoon-Yeong KimSparse Mixture-of-Experts Mixture-of-Experts Models
Department of Statistics and Data Science University of Seoul Republic of Korea · Department of Statistics University of Seoul Republic of Korea
Meta Expert: Cross-Layer Expert Sharing for Storage- and Retrieval-Efficient MoE
Oct 20, 2026Ermo Hua, Xiangyu Hong, Baiting Wu +11Sparse Mixture-of-Experts Mixture-of-Experts Models
Tsinghua University · Shanghai AI Laboratory
Text-Anchored Modality Experts with Uncertainty-Guided Routing for Multimodal Sentiment Analysis
Oct 20, 2026Xiaohua Huang, Bingrui Lei, Xiuyan Shao +4Multimodal Sentiment Analysis Modality Reliance in Multimodal Models
the School of Communication and Information Engineering, Nanjing Institute of Technology, Nanjing 210096, China. · the School of Economics and Management, Southeast University, Nanjing 211196, China. · the School of Computer Science and Engineering, Macau University of Science and Technology, Macau SAR, 999078, China. +2
The Router Already Knows
Oct 20, 2026Andrew Bistras, Marty JohnsonSparse Autoencoders Mixture-of-Experts Language Models
University of California, Los Angeles · Astrid Cybernetics
Solving the Ho--Nguyen Conjecture for Gaussian Mixtures: Applications to Mixtures of Experts
Oct 19, 2026Yuyang Zhang, Qihuang ZhangGaussian Mixture Models Parameter Identifiability
McGill University
Map-Distribution Mixture-of-Experts for Multi-Agent Pathfinding
Oct 19, 2026Fumimaro Suzuki, Keisuke OkumuraMulti-Agent Path Finding Mixture-of-Experts Models
University of Tsukuba, Tsukuba, Japan · National Institute of Advanced Industrial Science and Technology (AIST), Tokyo, Japan
Functional Cross-layer Experts Routing for One-step Diffusion Distillation
Oct 19, 2026Zile Huang, Ser-nam LimDiffusion Model Distillation Diffusion Distillation
University of Central Florida
Fine-Grained Units, Flat Decisions: Hierarchical Micro-Expert Pruning for Mixture-of-Experts Models
Oct 19, 2026Zukang Xu, Xinpei Gao, Xing Hu +2Channel Pruning Mixture-of-Experts Pruning
Houmo AI · University of Science and Technology of China · Southeast University
COMET: Decoupled Distillation, Routing, and Capacity Control for Task-Agnostic Continual Vision--Language Learning
Oct 16, 2026Seungyoon Woo, Keonhee Park, Gunhee KimContinual Learning for VLMs VLM Distillation
Seoul National University
TWIN-MoE: Two-stage Intra-Expert Imputation Network for Incomplete Multimodal Learning
Oct 15, 2026Liang Nathan Zheng, Jiyuan Liu, Wenhao Liang +3Missing-Modality Learning Multimodal Learning
School of Computer Science and Information Technology, Adelaide University, Adelaide, SA 5000, Australia
Rethinking Routing Exploration in Modern Mixture-of-Experts
Oct 15, 2026Bingcong Li, Hao Ma, Michael Muehlebach +2Mixture-of-Experts Models Expert Routing
ETH Zurich · The University of Hong Kong · Max Planck Institute for Intelligent Systems +1
PW-Pruning: Phase-Aware Expert Selection for Speech-Centric Omni-Modal Models
Oct 15, 2026Junjie Cao, Fengping Tian, Wenjie Zhou +5Mixture-of-Experts Pruning Mixture-of-Experts Models
Alibaba, Hangzhou, China · University of Macau
Router Drift Control: Mitigating Off-Policy Degradation in Mixture-of-Experts Reinforcement Learning
Oct 15, 2026Liangyao Jin, Zhixin Wang, Liming Liu +8Reinforcement Learning Off-Policy RL
Peking University · Infrawaves Inc · Shanghai Innovation Institute +2
Learnable Hodge-Laplacian Driven Mixture of Experts for Steady-PDE Surrogates on Coarse Unstructured Meshes
Oct 14, 2026Stavros Nousias, Akis Nousias, Konstantinos GkrispanisPDE Surrogate Modeling Mixture-of-Experts Models
Technical University of Munich · K3Y Labs
OrionMoE-v1.0: A Mixture-of-Experts Foundation Model for Tabular In-Context Learning
Oct 14, 2026Mohamed Bouadi, Nassim Bouarour, Aditya Tanna +1Tabular Foundation Models Tabular ICL
Lexsi Labs
LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation
Oct 14, 2026Zefeng He, Yunkai Chai, Xiaoyang Chen +5Mixture-of-Experts Models Language Model Scaling
SJTU · Shanghai AI Lab · USTC +2
PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts
Oct 13, 2026Ruixiao Dong, Jianmin Bao, Bozhi Luan +6Diffusion-Based Image Generation Pixel-Space Diffusion Models
University of Science and Technology of China · KlingAI Research · Peking University
UniW(A)M: A Unified Mixture-of-Experts Framework for World and World-Action Models
Oct 13, 2026Jingbo Xu, Jianjie Fang, Zhuohang Li +9World Models for Robotics World Action Models
Tsinghua University · Zhongguancun Academy · East China Normal University +1
Regime-Aware Router Learning for Multi-Domain Medical Image Generation with Diffusion Experts
Oct 12, 2026Yaoshuai Ma, Wei Yao, Yongxiang Liu +5Expert Routing Diffusion-Based Image Generation
Peng Cheng Laboratory, Shenzhen, China · Southern University of Science and Technology, Shenzhen, China

Oct 27, 2026J/K move · Enter open · S save

Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data

Geunhyoung Kim, Sung Ju Hwang

KAIST

Mixture-of-Experts Inference Efficient Language Model Inference Mixture-of-Experts Models

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models are bottlenecked by loading routed-expert weights from memory during interactive decoding at small batch sizes. Because most intermediate channels in a routed expert contribute little to any single token, channel selectors predict which channels matter and read only their weights, typically keeping channels above a threshold calibrated on offline data. We show that channel selection needs no calibration data. First, a calibrated threshold does not fix how much a selector reads, because the number of channels passing it varies across inputs; Prox with thresholds calibrated for an 18% read budget actually reads only 12.9% on average across our benchmarks. Second, top-k keeps each token's k highest-scoring channels and needs no calibration data. At the calibrated budget, a selector's top-k version matches or beats its threshold version on the same score in 40 of 43 comparisons at 13% and 18% budgets, including all 23 on generation tasks. The top-k versions also avoid the repetition-loop collapse that calibrated thresholds cause on Gemma 4. Calibrating a threshold thus provides neither of the two things selectors use it for: a known read budget and a consistent accuracy gain at that budget. We furthermore introduce a reader built entirely from the model's weights, which screens channels with a ternary sketch and recovers the skipped ones with a low-rank map. Across three post-trained MoE models, our reader retains 98.1% of full-model accuracy at 17.8% of routed-expert reads, with Triton kernels running MoE blocks 1.60 to 1.90x faster than optimized vLLM at batch one and up to 2.69x at larger batches on the same three models.