Mixture-of-Experts Inference

Follow this topic Ask about this topic

Mixture-of-experts inference studies how to select and execute only the experts needed for each token or input, aiming to reduce computation without losing model quality. Methods include adaptive routing and expert skipping, quantization and pruning, predictive prefetching and caching, speculative decoding, and expert-parallel scheduling across GPUs or heterogeneous devices. Systems are evaluated in settings such as prefill versus token-by-token decoding, long-context serving, distributed GPU clusters, and memory-limited edge devices, using measures like latency, throughput, memory traffic, and task accuracy. A central tension is that activating fewer parameters does not necessarily make inference faster: moving expert weights and coordinating irregular workloads can dominate the computation, especially when memory or network bandwidth is limited.

Latest papers

177 papers28 papers in the last four weeks, up 12% on the four weeks before. 0.2% of all new papers.

GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems
May 19, 2026Sourish Wawdhane, Avinash Kumar, Poulami DasExpert Load Balancing Mixture-of-Experts Inference
The University of Texas at Austin
FPED: A Functional-Network Prior-Guided Mixture-of-Experts Framework for Interpretable Brain Decoding
May 19, 2026Yudan Ren, Pengcheng Shi, Zihan Ma +2Neural Decoding Mixture-of-Experts Inference
School of Electronic Information (School of Artificial Intelligence), Northwest University, Xi’an 710127, China
Post-Trained MoE Can Skip Half Experts via Self-Distillation
May 18, 2026Xingtai Lv, Li Sheng, Kaiyan Zhang +12Mixture of Experts Communication-Efficient Mixture-of-Experts
Tsinghua University · Shanghai AI Lab · WeChat AI +1
CoX-MoE: Coalesced Expert Execution for High-Throughput MoE Inference with AMX-Enabled CPU-GPU Co-Execution
May 18, 2026Muyoung Son, Yi Chen, Seungjae Yoo +2Expert Offloading GPU Acceleration
KAIST · Daejeon, Republic of Korea
BEAM: Binary Expert Activation Masking for Dynamic Routing in MoE
May 14, 2026Juntong Wu, Jialiang Cheng, Qishen Yin +5Sparse Mixture-of-Experts Mixture-of-Experts Inference
Taobao & Tmall Group of Alibaba · Shenzhen Graduate School, Peking University
ROMER: Expert Replacement and Router Calibration for Robust MoE LLMs on Analog Compute-in-Memory Systems
May 12, 2026Wenyong Zhou, Yuannuo Feng, Yizhe Chen +6Compute-in-Memory Mixture-of-Experts Language Models
The Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong · The School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China
Fast MoE Inference via Predictive Prefetching and Expert Replication
May 12, 2026Ankit Jyothish, Ali Jannesari, Aishwarya Sarkar +1Expert Load Balancing GPU Acceleration
Iowa State University Ames, USA
TRACE: Temporal Routing with Autoregressive Cross-channel Experts for EEG Representation Learning
May 12, 2026Fan Ma, Qier An, Peng Chen +6Representation Learning Self-Supervised Pre-Training
Department of Biomedical Informatics and Data Science, Yale University
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
May 11, 2026Chenyang Song, Weilin Zhao, Xu Han +3Sparse Mixture-of-Experts Mixture-of-Experts Inference
Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University, Beijing, China
TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tiling
May 10, 2026Hongyaoxing Gu, Xinzhe Chen, Lijuan Hu +1Efficient Inference Mixture-of-Experts Inference
Institute of Software Chinese Academy of Sciences · University of Chinese Academy of Sciences
Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
May 9, 2026Jongseok Park, Sunga Kim, Zhenyu Gu +2Activation Sparsity Mixture-of-Experts Inference
University of California, Berkeley · Work done during an internship at Advanced Micro Devices, Inc. · Advanced Micro Devices, Inc.
EMO: Pretraining Mixture of Experts for Emergent Modularity
May 7, 2026Ryan Wang, Akshita Bhagia, Sewon MinLanguage Model Pretraining Mixture-of-Experts Language Models
University of California, Berkeley · Allen Institute for AI
Federation of Experts: Communication Efficient Distributed Inference for Large Language Models
May 7, 2026Muhammad Shahir Abdurrahman, Chun Deng, Azalia Mirhoseini +1Expert Parallelism Mixture-of-Experts Inference
Stanford University
Relay Buffer Independent Communication over Pooled HBM for Efficient MoE Inference on Ascend
May 7, 2026Tianlun Hu, Tiancheng Hu, Shengsheng Litang +8Communication-Efficient Mixture-of-Experts Mixture-of-Experts Inference
Huawei Technologies
VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading
May 7, 2026Cheng Xu, Xiaofeng Hou, Jiacheng Liu +1Expert Offloading Vision-Language Models
Shanghai Jiao Tong University Minhang Qu, Shanghai, China
Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts
May 6, 2026Klaus-Rudolf Kladny, Maximilian Mordig, Bernhard Schölkopf +1Mixture-of-Experts Inference Expert Routing
Max Planck Institute for Intelligent Systems, Tübingen · Tübingen AI Center · ETH Zurich +1
ZeRO-Prefill: Zero Redundancy Overheads in MoE Prefill Serving
May 3, 2026Zhaoyuan Su, Olatunji Ruwase, Karthik Ganesan +5Expert Parallelism LLM Serving
University of Virginia · Snowflake AI Research · Harvard University
Affinity Is Not Enough: Recovering the Free Energy Principle in Mixture-of-Experts
May 1, 2026Man Yung WongSparse Mixture-of-Experts Mixture-of-Experts Inference
Russell · Independent Researcher
SpaceMoE: Realizing Distributed Mixture-of-Experts Inference over Space Networks
May 1, 2026Zhanwei Wang, Huiling Yang, Min Sheng +2Mixture-of-Experts Language Models Mixture-of-Experts Inference
Department of Electrical and Computer Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China · State Key Laboratory of Integrated Service Networks, Institute of Information Science, Xidian University (XDU), Xi’an, Shaanxi, China · Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology (HKUST), Hong Kong SAR, China
Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding
May 1, 2026Lehan Pan, Ziyang Tao, Ruoyu Pang +3Mixture-of-Experts Inference Speculative Decoding
University of Science and Technology of China · Tianyijiaotong Technology Ltd., Suzhou, China
Rethinking Network Topologies for Cost-Effective Mixture-of-Experts LLM Serving
Apr 30, 2026Junsun Choi, Sam Son, Sunjin Choi +5LLM Serving Cost-Aware Inference
UC Berkeley · UC Berkeley & ICSI
FaaSMoE: A Serverless Framework for Multi-Tenant Mixture-of-Experts Serving
Apr 29, 2026Minghe Wang, Trever Schirmer, Mohammadreza Malekabbasi +1Mixture-of-Experts Language Models LLM Serving
TU Berlin · Berlin, Germany
RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts
Apr 28, 2026Vyom Sharma, Debajyoti DattaGPU Kernel Optimization GPU Acceleration
Hippocratic AI
Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
Apr 25, 2026Abhimanyu Bambhaniya, Geonhwa Jeong, Jason Park +6Expert Load Balancing Mixture-of-Experts Inference
1Georgia Institute of Technology, USA · 2Meta Platforms, Inc.
Mixture of Heterogeneous Grouped Experts for Language Modeling
Apr 25, 2026Zhicheng Ma, Xiang Liu, Zhaoxiang Liu +5Expert Load Balancing Mixture-of-Experts Language Models
1Data Science & Artificial Intelligence Research Institute, China Unicom · 2Unicom Data Intelligence, China Unicom
Temporally Extended Mixture-of-Experts Models
Apr 22, 2026Zeyu Shen, Peter HendersonMixture-of-Experts Inference Memory-Efficient Inference
1Princeton University
Efficient Mixture-of-Experts LLM Inference with Apple Silicon NPUs
Apr 20, 2026Afsara Benazir, Felix Xiaozhu LinAI Accelerator Inference Energy-Efficient ML
University of Virginia Charlottesville, Virginia, USA
MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
Apr 19, 2026Bo Li, Chuan Wu, Shaolin ZhuEfficient Multimodal Inference Expert Load Balancing
School of Software, Tsinghua University, Beijing, China · TJUNLP Lab, School of Computer Science and Technology, Tianjin University, China · School of New Media and Communication, Tianjin University, China
ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving
Apr 16, 2026Yuseon Choi, Jingu Lee, Jungjun Oh +5LLM Serving Mixture-of-Experts Inference
KAIST · Samsung Electronics · GIST
Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
Apr 15, 2026Ivan Ternovtsii, Yurii BilakMixture-of-Experts Language Models Expert Routing
Department of Software Systems, Uzhhorod National University Narodna sq. 3, Uzhhorod, Ukraine, 88000 · HengeBytes

May 19, 2026J/K move · Enter open · S save

GEM: GPU-Variability-Aware Expert to GPU Mapping for MoE Systems

Sourish Wawdhane, Avinash Kumar, Poulami Das

The University of Texas at Austin

Expert Load Balancing Mixture-of-Experts Inference Heterogeneous Computing Mixture-of-Experts Models

Open paper AskPDF arXiv

Mixture-of-Expert (MoE) models enable efficient inference by employing smaller experts and activating only a subset of them per token. MoE serving engines distribute experts across multiple GPUs and route tokens to appropriate GPUs at inference time based on experts activated. They process tokens in lock-step fashion, where tokens within a batch must finish processing before proceeding to the next layer. This synchronization barrier acts as a critical bottleneck because the performance of MoE models is limited by the straggler GPU that finishes last. Stragglers emerge when too many heavily used experts are placed on the same GPU or the slowest GPU. While prior works place experts that balance token loads across GPUs, they all overlook GPU variability and often place highly used experts on the slowest GPUs. We propose GEM, GPU-variability-aware Expert Mapping, a framework for GPU variability-aware expert to GPU mapping for MoE models. GEM exploits two insights. First, we must place experts such that each GPU receives non-uniform token loads based on their variability and they all finish processing a layer at about the same time. Our studies show that there are two types of experts: consistent that are used most of the time and temporal that are often used together for the remaining time. Our second insight is that we must place simultaneously used consistent and temporal experts on different GPUs and avoid placing them on slower GPUs to reduce slowdown. GEM gathers the variability profile of GPUs for each model and task and uses the token load distributions per task to map experts to GPUs. Our experiments show that GEM improves end-to-end latency by 7.9% on average and by up to 16.5% compared to the baseline.