Mixture-of-Experts Models

Follow this topic Ask about this topic

Mixture-of-experts models divide computation among expert networks and use a router to select which experts process each input, often activating only a small subset. Common approaches include top-k sparse routing, shared or hierarchical experts, and parameter-efficient expert adapters; researchers also study load balancing to prevent some experts from being overused or ignored. These models are used in language, vision, multimodal learning, forecasting, and robotics, while systems work targets the memory, communication, and latency costs of serving and training them. A central tension is that specialization can improve capacity and efficiency, but balanced routing does not guarantee useful expertise, and routing overhead or weaker performance under changed conditions can offset the gains.

Latest papers

462 papers75 papers in the last four weeks, up 97% on the four weeks before. 0.4% of all new papers.

Scene-Adaptive Continual Learning for CSI-based Human Activity Recognition with Mixture of Experts
May 7, 2026Wenhan Zheng, Yuyi Mao, Ivan Wang-Hei HoAdaptive Inference Human Activity Recognition
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong · School of Computer Science and Engineering, The Macau University of Science and Technology, Macau
E = T*H/(O+B): A Dimensionless Control Parameter for Mixture-of-Experts Ecology
May 7, 2026Qingjun ZhangExpert Load Balancing Mixture of Experts
School of Integrated Circuits, Wuxi Taihu University Department of Communication Engineering · Department of Communication Engineering
Expert Routing for Communication-Efficient MoE via Finite Expert Banks
May 6, 2026Mohammad Reza Deylam Salehi, Ali KhalesiMixture of Experts Information-Theoretic Generalization Bounds
IEEE Graduate Member · Nice, France · Institut Polytechnique des Sciences Avancées (IPSA) +1
Piper: Efficient Large-Scale MoE Training via Resource Modeling and Pipelined Hybrid Parallelism
May 6, 2026Sajal Dash, Feiyi WangCommunication-Efficient Distributed Training High-Performance Computing
Oak Ridge National Laboratory Oak Ridge, USA
Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts
May 6, 2026Klaus-Rudolf Kladny, Maximilian Mordig, Bernhard Schölkopf +1Mixture-of-Experts Inference Expert Routing
Max Planck Institute for Intelligent Systems, Tübingen · Tübingen AI Center · ETH Zurich +1
AxMoE: Characterizing the Impact of Approximate Multipliers on Mixture-of-Experts DNN Architectures
May 6, 2026Omkar B Shende, Marcello Traiola, Gayathri AnanthanarayananEfficient Neural Network Inference Edge Inference
Dept. of Computer Science and Engineering · Univ Rennes, CNRS, Inria, IRISA
YOTOnet: Zero-Shot Cross-Domain Fault Diagnosis via Domain-Conditioned Mixture of Experts
May 6, 2026Zesen Wang, Zihao Wu, Yue Hu +2Cross-Domain Transfer Learning Bearing Fault Diagnosis
School of Mechanical and Power Engineering, East China University of Science and Technology, Shanghai, China
GEM: Graph-Enhanced Mixture-of-Experts with ReAct Agents for Dialogue State Tracking
May 6, 2026Ziqi Zhu, Adithya Suresh, Tomal Deb +1Spoken Dialogue Systems State Tracking
Generative AI Innovation Center, Amazon Web Services
MILE: Mixture of Incremental LoRA Experts for Continual Semantic Segmentation across Domains and Modalities
May 5, 2026Shishir Muralidhara, Didier Stricker, René SchusterDomain-Incremental Learning Incremental Learning
German Research Center for Artificial Intelligence (DFKI), Trippstadter Str. 122, 67663 Kaiserslautern, Germany · University of Kaiserslautern-Landau (RPTU), Gottlieb-Daimler-Str. 47, 67663 Kaiserslautern, Germany
Ortho-Hydra: Orthogonalized Experts for DiT LoRA
May 5, 2026Seunghyun JiFine-Tuning Diffusion Transformer
RouteHijack: Routing-Aware Attack on Mixture-of-Experts LLMs
May 1, 2026Zhiyuan Xu, Joseph Gardiner, Sana Belguith +1Mixture-of-Experts Language Models Adversarial Attacks on LLMs
University of Bristol Bristol, United Kingdom
Prediction-powered Inference by Mixture of Experts
Apr 30, 2026Yanwu Gu, Linglong Kong, Dong XiaUncertainty Quantification Mixture-of-Experts Models
Department of Mathematics, Hong Kong University of Science and Technology · Department of Mathematical and Statistical Sciences, University of Alberta
Advancing multi-site emission control: A physics-informed transfer learning framework with mixture of experts for carbon-pollutant synergy
Apr 29, 2026Yuxuan Ying, Hanqing Yang, Kaige Wang +6Cross-Domain Transfer Learning Physics-Informed ML
College of Mechanical Engineering, Zhejiang University of Technology, Hangzhou, 310023, China. · Alibaba Group, Hangzhou, 311100, China. · School of Computer Science and Technology, Zhejiang Normal University, Hangzhou, 311231, China. +2
Adaptive and Fine-grained Module-wise Expert Pruning for Efficient LoRA-MoE Fine-Tuning
Apr 29, 2026Weihang Li, Jianchun Liu, Hongli XuMixture-of-Experts Pruning LLM Fine-Tuning
School of Computer Science and Technology, University of Science and Technology of China, China · Suzhou Institute for Advanced Research, University of Science and Technology of China, China
Mixture of Experts Framework in Machine Learning Interatomic Potentials for Atomistic Simulations
Apr 28, 2026Gabriel de Miranda Nascimento, Marc L. Descoteaux, Laura Zichi +9Machine Learning Interatomic Potentials Multifidelity Modeling
Department of Materials Science and Engineering, Massachusetts Institute of Technology, Cambridge, MA, USA · John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA · Robert Bosch LLC Research and Technology Center, Watertown, MA, USA
SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs
Apr 27, 2026Zi-Hao Bo, Yaqian Li, Anzhou Hou +6Vision-Language Models Efficient VLM Inference
Li Auto Inc.
Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
Apr 25, 2026Abhimanyu Bambhaniya, Geonhwa Jeong, Jason Park +6Expert Load Balancing Mixture-of-Experts Inference
1Georgia Institute of Technology, USA · 2Meta Platforms, Inc.
Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning
Apr 24, 2026Haoze He, Xingyuan Ding, Xuan Jiang +5Supervised Fine-Tuning Fine-Tuning
Carnegie Mellon University · MIT · UCSD +1
Towards Adaptive Continual Model Merging via Manifold-Aware Expert Evolution
Apr 24, 2026Haiyun Qiu, Xingyu Wu, Kay Chen TanAdaptive Model Routing Mixture-of-Experts Models
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University, Hong Kong, China
On Bayesian Softmax-Gated Mixture-of-Experts Models
Apr 22, 2026Nicola Bariletto, Huy Nguyen, Nhat Ho +1Bayesian Mixture Models Bayesian Inference
The University of Texas at Austin
Temporally Extended Mixture-of-Experts Models
Apr 22, 2026Zeyu Shen, Peter HendersonMixture-of-Experts Inference Memory-Efficient Inference
1Princeton University
Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
Apr 21, 2026Chaitanya Dwivedi, Binxuan Huang, Himanshu Gupta +3Efficient Language Model Training Language Model Scaling
1Amazon Stores Foundation AI · 3Anthropic · 2Carnegie Mellon University
SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning
Apr 21, 2026Boyan Shi, Wei Chen, Shuyuan Zhao +4Multi-Task Learning Low-Rank Adaptation
School of Computer Science and Technology, Beijing Jiaotong University, China · 2Guangxi Key Lab of Trusted Software, Guilin University of Electronic Technology, China · 3Beijing Key Lab of Traffic Data Mining and Embodied Intelligence, China +2
Multi-Domain Learning with Global Expert Mapping
Apr 20, 2026Pourya Shamsolmoali, Masoumeh Zareapoor, Huiyu Zhou +3Vision Foundation Model Adaptation Expert Routing
Department of Computer Science, University of York, UK · School of Computing and Mathematical Sciences, University of Leicester, UK · Centre for Vision Speech and Signal Processing, University of Surrey, UK +2
Domain-Specialized Object Detection via Model-Level Mixtures of Experts
Apr 20, 2026Svetlana Pavlitska, Malte Stüven, Beyza Keskin +1Object Detection Mixture-of-Experts Models
FZI Research Center for Information Technology · Karlsruhe Institute of Technology (KIT)
Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs
Apr 20, 2026Charles Ye, Bo Yuan, Lee SharkeyMixture-of-Experts Interpretability LLM Interpretability
Independent · Georgia Institute of Technology · Goodfire
IMA-MoE: An Interpretable Modality-Aware Mixture-of-Experts Framework for Characterizing the Neurobiological Signatures of Binge Eating Disorder
Apr 18, 2026Lin Zhao, Qiaohui Gao, Elizabeth Martin +5Multimodal Classification Mixture-of-Experts Models
Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, 07102, NJ, USA · College of Engineering, Northeastern University, Boston, 02115, MA, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, 30602, NY, 10029 +1
CoGR-MoE: Concept-Guided Expert Routing with Consistent Selection and Flexible Reasoning for Visual Question Answering
Apr 18, 2026Xiyin Zeng, Yi Lu, Hao WangVisual Question Answering Expert Routing
Hong Kong University of Science and Technology (Guangzhou)
Application of a Mixture of Experts-based Foundation Model to the GlueX DIRC Detector
Apr 17, 2026Cristiano Fanelli, James Giroux, Cole Granger +1High-Energy Physics Mixture-of-Experts Models
William & Mary, Department of Data Science, Williamsburg, VA 23185, USA · William & Mary, Department of Physics, Williamsburg, VA 23185, USA
FL-MHSM: Spatially-adaptive Fusion and Ensemble Learning for Flood-Landslide Multi-Hazard Susceptibility Mapping at Regional Scale
Apr 17, 2026Aswathi Mundayatt, Jaya Sreevalsan-NairLandslide Susceptibility Mapping Mixture-of-Experts Models
Graphics-Visualization-Computing Lab, International Institute of Information Technology Bangalore, Karnataka 560100, India.

May 7, 2026J/K move · Enter open · S save

Scene-Adaptive Continual Learning for CSI-based Human Activity Recognition with Mixture of Experts

Wenhan Zheng, Yuyi Mao, Ivan Wang-Hei Ho

Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong · School of Computer Science and Engineering, The Macau University of Science and Technology, Macau

Adaptive Inference Human Activity Recognition Continual Learning Cross-Domain Transfer Learning Mixture-of-Experts Models

Open paper AskPDF arXiv

Channel state information (CSI)-based human activity recognition (HAR) is vulnerable to performance degradation under domain shifts across varying physical environments. Continual learning (CL) offers a principled way to learn new domains sequentially while preserving past knowledge, but existing CL solutions for CSI-based HAR scale poorly with accumulating domains, rely on a large replay buffer, or incur linearly growing inference cost. In this letter, we propose Scene-Adaptive Mixture of Experts with Clustered Specialists (SAMoE-C), which formulates cross-domain CSI-based HAR as a mixture-of-experts system that enables scene-specific adaptation, via an attention-based semantic router that activates only selected experts for each input. Moreover, we develop a novel training protocol, which requires only a tiny replay buffer for stabilizing domain discrimination of the router. Experimental results on a four-scene CSI dataset demonstrate that SAMoE-C approaches the state-of-the-art accuracy, while maintaining a significantly lower inference cost. By jointly combining modular experts, selective activation with router and a lightweight training protocol, SAMoE-C enables scalable cross-domain CSI-based HAR deployment with low training overhead and high computational efficiency in real-world settings.