Mixture-of-Experts Inference

Follow this topic Ask about this topic

Mixture-of-experts inference studies how to select and execute only the experts needed for each token or input, aiming to reduce computation without losing model quality. Methods include adaptive routing and expert skipping, quantization and pruning, predictive prefetching and caching, speculative decoding, and expert-parallel scheduling across GPUs or heterogeneous devices. Systems are evaluated in settings such as prefill versus token-by-token decoding, long-context serving, distributed GPU clusters, and memory-limited edge devices, using measures like latency, throughput, memory traffic, and task accuracy. A central tension is that activating fewer parameters does not necessarily make inference faster: moving expert weights and coordinating irregular workloads can dominate the computation, especially when memory or network bandwidth is limited.

Latest papers

177 papers28 papers in the last four weeks, up 12% on the four weeks before. 0.2% of all new papers.

Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations
Apr 15, 2026Wentao Hu, Yanbo Zhai, Xiaohui Hu +6LLM Hallucination Mitigation Mixture-of-Experts Inference
Xi’an Jiaotong University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd. · Beijing Foreign Studies University +1
SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding
Apr 11, 2026Jehyeon Bang, Eunyeong Cho, Ranggi Hwang +2Mixture-of-Experts Inference LLM Inference Acceleration
KAIST · UNIST
Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
Apr 9, 2026Baihui Liu, Kaiyuan Tian, Wei Wang +3Mixture-of-Experts Inference Mixture of Experts
National Key Laboratory of Parallel and Distributed Computing, College of Computer Science and Technology, National University of Defense Technology
MoBiE: Efficient Inference of Mixture of Binary Experts under Post-Training Quantization
Apr 8, 2026Zhixiong Zhao, Zukang Xu, Zhixuan Chen +1Mixture-of-Experts Inference Mixture-of-Experts Quantization
Houmo AI
FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
Apr 3, 2026Qingxiu Liu, Yongchao He, Runhan Jiang +4Expert Offloading GPU Acceleration
CUHK · Unaffiliated · SCITIX +1
DWDP: Distributed Weight Data Parallelism for High-Performance LLM Inference on NVL72
Apr 2, 2026Wanqian Li, Jintao Peng, Zongfei Jing +7LLM Inference Acceleration Distributed Inference
NVIDIA
Expert Streaming: Accelerating Low-Batch MoE Inference via Multi-chiplet Architecture and Dynamic Expert Trajectory Scheduling
Mar 29, 2026Songchen Ma, Hongyi Li, Weihao Zhang +8Mixture-of-Experts Inference Efficient Neural Network Inference
AI Chip Center for Emerging Smart Systems, Hong Kong SAR, China · The Hong Kong University of Science and Technology, Hong Kong SAR, China · Shanghai UniVista Industrial Software Group Co., Ltd., Shanghai, China
DyMoE: Dynamic Expert Orchestration with Mixed-Precision Quantization for Efficient MoE Inference on Edge
Mar 19, 2026Yuegui Huang, Zhiyuan Fang, Weiqi Luo +3Mixture-of-Experts Inference Mixture-of-Experts Quantization
Sun Yat-sen University, Guangzhou, China · Tencent, Shenzhen, China
Bridging Local and Global Knowledge: Cascaded Mixture-of-Experts Learning for Near-Shortest Path Routing
Mar 16, 2026Yung-Fu Chen, Anish AroraMixture-of-Experts Models Mixture-of-Experts Inference
The Ohio State University, Columbus, OH, USA
NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL
Mar 13, 2026Amos Goldman, Nimrod Boker, Maayan Sheraizin +15Communication-Efficient Mixture-of-Experts Expert Parallelism
NVIDIA Corporation
The $qs$ Inequality: Quantifying the Double Penalty of Mixture-of-Experts at Inference
Mar 9, 2026Vignesh Adhinarayanan, Nuwan JayasenaMixture-of-Experts Inference Long-Context Language Model Inference
AMD Research and Advanced Development (RAD).
LAR-MoE: Latent-Aligned Routing for Mixture of Experts in Robotic Imitation Learning
Mar 9, 2026Ariel Rodriguez, Chenpan Li, Lorenzo Mazza +5Robot Imitation Learning Agent Skill Learning
Department of Translational Surgical Oncology, NCT/UCC Dresden, a partnership between DKFZ, MF UKD, TUD, HZDR Dresden, Germany. · Cluster of Excellence-CeTI, TUD, Germany. · Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TUD, Germany.
FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach
Mar 9, 2026Ning Liao, Xiaoxing Wang, Xiaohan Qin +1Sparse Mixture-of-Experts Mixture-of-Experts Models
Shanghai Jiao Tong University
Speculating Experts Accelerates Inference for Mixture-of-Experts
Mar 9, 2026Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2Mixture-of-Experts Inference LLM Inference Acceleration
University of Maryland · TogetherAI
MoEless: Efficient MoE LLM Serving with Serverless Experts
Mar 6, 2026Hanfei Yu, Bei Ouyang, Shwai He +2LLM Inference Efficiency Expert Load Balancing
University of Maryland College Park
Robust Heterogeneous Analog-Digital Computing for Mixture-of-Experts Models with Theoretical Generalization Guarantees
Mar 3, 2026Mohammed Nowaz Rabbani Chowdhury, Hsinyu Tsai, Geoffrey W. Burr +3Mixture-of-Experts Inference Heterogeneous Computing
Rensselaer Polytechnic Institute · IBM Research
TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading
Mar 1, 2026Yudong Pan, Yintao He, Tianhua Han +7Mixture-of-Experts Inference LLM Inference Acceleration
SKLP, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
From Isolation to Integration: Building an Adaptive Expert Forest for Pre-Trained Model-based Class-Incremental Learning
Feb 24, 2026Ruiqi Liu, Boyu Diao, Hangda Liu +3Adapter-Based Continual Learning Class-Incremental Learning
Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs
Feb 23, 2026Zijie Liu, Jie Peng, Jinhao Duan +7Expert Load Balancing Mixture of Experts
University of North Carolina at Chapel Hill · University of Minnesota Twin Cities · North Carolina State University +2
MoE-Spec: Expert Budgeting for Efficient Speculative Decoding
Feb 17, 2026Bradley McDanel, Steven Li, Sruthikesh Surineni +1Speculative Decoding LLM Inference Acceleration
Franklin and Marshall College · Meta Reality Labs
MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios
Feb 12, 2026Shuhuai Li, Jianghao Lin, Dongdong Ge +1Mixture-of-Experts Inference Edge Inference
Shanghai University, Shanghai, China · Shanghai Jiao Tong University, Shanghai, China · Standford University, California, United States
OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale
Feb 5, 2026Jingze Shi, Zhangyang Peng, Yizhang Zhu +3Mixture-of-Experts Inference LLM Inference Acceleration
The Hong Kong University of Science and Technology (Guangzhou) · Beijing Academy of Artificial Intelligence
FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices
Jan 22, 2026Byeongju Kim, Jungwan Lee, Donghyeon Han +2Edge Inference Mixture-of-Experts Inference
Dynamic Expert Quantization for Scalable Mixture-of-Experts Inference
Nov 19, 2025Kexin Chu, Dawei Xiang, Zixu Shen +3Mixed-Precision Quantization GPU Acceleration
University of Connecticut · University of California, Santa Cruz · Independent Researcher
Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
Nov 18, 2025Wenfeng Wang, Xiaofeng Hou, Xinfeng Xia +5Mixture-of-Experts Inference LLM Inference Acceleration
Shanghai Jiao Tong University, Shanghai, China · Shanghai Jiao Tong University, China · Hong Kong University of Science and Technology, Hongkong, China
Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization
Sep 30, 2025Yaoxiang Wang, Qingguo Hu, Yucheng Ding +6Mixture-of-Experts Language Models Mixture-of-Experts Models
Xiamen University · Shanghai Jiao Tong University · University of Science and Technology of China +1
3D-MoE: Towards Spatial Intelligence with Mixture-of-Experts for 3D Reasoning and Action Generation
Jan 28, 2025Yueen Ma, Zenglin Xu, Irwin King3D Spatial Reasoning Mixture-of-Experts Inference
Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong SAR

Apr 15, 2026J/K move · Enter open · S save

Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations

Wentao Hu, Yanbo Zhai, Xiaohui Hu, Mingkuan Zhao+5

Xi’an Jiaotong University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd. · Beijing Foreign Studies University+1

LLM Hallucination Mitigation Mixture-of-Experts Inference LLM Routing Mixture-of-Experts Models

Open paper AskPDF arXiv

Sparse Mixture-of-Experts (MoE) models have achieved remarkable scalability, yet they remain vulnerable to hallucinations, particularly when processing long-tail knowledge. We identify that this fragility stems from static Top-

k

routing: routers tend to favor high-frequency patterns over rare factual associations. Consequently, specialist experts'' possessing critical long-tail knowledge are often assigned low gating scores and remain dormant'' -- under-prioritized for specific tokens despite their proven causal importance on other inputs. To address this, we propose Counterfactual Routing (CoR), a training-free inference framework designed to awaken these dormant experts. CoR integrates layer-wise perturbation analysis with the Counterfactual Expert Impact (CEI) metric to dynamically shift computational resources from syntax-dominant to knowledge-intensive layers while maintaining a constant total activation count, effectively retrieving causally decisive experts via virtual ablation. Extensive experiments on TruthfulQA, FACTOR, and TriviaQA demonstrate that CoR improves factual accuracy by 3.1% on average without increasing the inference budget, establishing a superior Pareto frontier compared to static scaling strategies.