Mixture-of-Experts Inference

Ask about this topic

Overview

Mixture-of-experts inference studies how to select and execute only the experts needed for each token or input, aiming to reduce computation without losing model quality. Methods include adaptive routing and expert skipping, quantization and pruning, predictive prefetching and caching, speculative decoding, and expert-parallel scheduling across GPUs or heterogeneous devices. Systems are evaluated in settings such as prefill versus token-by-token decoding, long-context serving, distributed GPU clusters, and memory-limited edge devices, using measures like latency, throughput, memory traffic, and task accuracy. A central tension is that activating fewer parameters does not necessarily make inference faster: moving expert weights and coordinating irregular workloads can dominate the computation, especially when memory or network bandwidth is limited.

Latest papers

CardsList

177 papers28 papers in the last four weeks, up 12% on the four weeks before. 0.2% of all new papers.

  1. CRISP: Co-designing Pruning and Scheduling for Efficient MoE Inference on Edge Servers

    Oct 27, 2026Congwei Zhang, Youming Tao, Yifei Zou +6Mixture-of-Experts InferenceMemory-Efficient Inference

    School of Electrical Engineering and Computer Science, TU Berlin, Berlin, Germany · School of Computer Science and Technology, Shandong University, Qingdao, China · Department of Data Science, City University of Hong Kong, Hong Kong, China

  2. Channel Selection for Mixture-of-Experts Decoding Needs No Calibration Data

    Oct 27, 2026Geunhyoung Kim, Sung Ju HwangMixture-of-Experts InferenceEfficient Language Model Inference

    KAIST

  3. Prefetch-Steered Routing: Deterministic Cache-Aware Expert Selection for Offloaded Mixture-of-Experts Decoding

    Oct 26, 2026Siming HuangMixture-of-Experts Language ModelsMixture-of-Experts Inference

    The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.

  4. Speculative Decoding for Mixture-of-Experts Models: Expert Bytes, Kernel Bandwidth and the Best Draft Length

    Oct 25, 2026Yuzheng LanSpeculative DecodingMixture-of-Experts Inference

    ReClan AI Studio, LLC

  5. COPE: Composable Performance Emulation for Mixture-of-Experts Offloading Policies

    Oct 24, 2026Hanming Zhang, Erhu FengExpert OffloadingMixture-of-Experts Inference

    Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University

  6. Low-Latency MoE Inference in Space Data Centers: A Wireless-Power-Transfer Perspective

    Oct 22, 2026Haotian Zheng, Zhanwei Wang, Kaibin HuangEnergy-Aware LLM InferenceLLM Inference

    Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China

  7. SPARE: Sparsity Allocation via Routing Entropy for Efficient MoEs

    Oct 21, 2026Yifan Yang, Hanling Yi, Dongquan Yang +4Mixture-of-Experts PruningSparse Mixture-of-Experts

    Intellifusion Inc.

  8. Don't Skip the Baseline: A Controlled Comparison of Dynamic Expert Skipping in MoEs

    Oct 20, 2026Jakob Steimle, Thomas Elsken, Daniel Ehrhardt +3Mixture-of-Experts InferenceMixture-of-Experts Pruning

    Technical University of Munich · Tensordyne · ETH Zurich +1

  9. TempoMoE: Identifying and Steering Fast-Slow Thinking Experts for Efficient Multimodal Reasoning

    Oct 19, 2026Xiyue Zhang, Keru Zhou, Ziyi Wang +2Mixture-of-Experts InferenceEfficient Multimodal Inference

    Zhejiang University · Tsinghua University · Wenzhou Business College

  10. 3D-MoE: Complementarity-Guided Hardware-Software Co-Design for MoE Decode on 3D-Stacked NMP

    Oct 19, 2026Yudong Pan, Lian Liu, Shixin Zhao +8Neural Network Hardware AccelerationMixture-of-Experts Inference

    SKLP, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · University of Edinburgh, Edinburgh, United Kingdom

  11. ScoutRoute: Efficient Mixture-of-Experts Routing Using Internal Language Model Signals

    Oct 16, 2026Rajdeep Pandhare, Delzaan BharuchaLLM RoutingMixture-of-Experts Inference

    Department of Artificial Intelligence and Data Science, Rajiv Gandhi Institute of Technology, Mumbai, Maharashtra, India

  12. Efficient CPU-GPU collaborative MoE inference via Adaptive Expert Assignment

    Oct 16, 2026Wenhui Lv, Yunming Liao, Hongli Xu +2Mixture-of-Experts InferenceLLM Inference Acceleration

  13. Where Prefill Expert Reduction Hurts MoE Models: A Few Prompt Tokens Carry Most of the Short-Answer F1 Loss

    Oct 15, 2026Zhiyuan Wang, Han Wang, Shuo ZhangInput Order Sensitivity in Language ModelsMixture-of-Experts Inference

    Independent Researcher

  14. DPA-MoE: A Runtime Precision Control Framework for MoE Inference

    Oct 15, 2026Jaeyun Kim, Hoonki Lee, Susie Shin +3Mixture-of-Experts InferenceAdaptive Inference

    Agency for Defense Development · Yonsei University

  15. FastEEP: Fast Intra-Instance Elastic Scaling for Expert-Parallel MoE Serving

    Oct 13, 2026Yunfei Feng, Jingkai He, Shiqi Liu +3Efficient InferenceMixture-of-Experts Inference

    Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University

  16. FORGE: Procedural Weight Generation for Sparse MoE Inference

    Oct 12, 2026Chanyoung Moon, Travis Pence, Daisuke Yamada +4Mixture-of-Experts InferenceLLM Inference Acceleration

    University of Wisconsin-Madison Madison, Wisconsin, USA · Korea University Seoul, Republic of Korea · NVIDIA Santa Clara, California, USA

  17. SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference

    Oct 11, 2026Hongliang Cao, Xiuyu Li, Siqi Wang +3LLM Inference AccelerationSpeculative Decoding

    Beihang University · Monash University · The Hong Kong University of Science and Technology

  18. Coarse-Graining Recursive Expert Computation: Continuation-Relative Interfaces and Failure Modes

    Oct 10, 2026Shihua Yu, Yanzhi Wang, Tianchi ChenEfficient Language Model InferenceLLM Evaluation

    Department of Electrical and Computer Engineering, Northeastern University, Boston, MA, USA. · Merck & Co., Inc., Rahway, NJ, USA.

  19. DuoRate: Two-Rate Per-Expert Precision Allocation for Mixture-of-Experts Serving

    Oct 9, 2026Mengqi Guo, Zhe Wei, Yuan Yuan +3Mixture-of-Experts QuantizationMixture-of-Experts Inference

    Huawei Technologies Ltd.

  20. Democratizing MoE inference on commodity GPUs with CoMoE

    Oct 7, 2026Ruwen Fan, Yuezhi Zu, Minhui Xie +6Communication-Efficient Mixture-of-ExpertsMixture-of-Experts Inference

    Jimmy · Alibaba Cloud Computing · Tsinghua University

  21. Cascadia: Resident 975B MoE Inference on Eleven AI PCs

    Oct 5, 2026Tate Berenbaum, Matias Parij, Muthaiah VenkatachalamLLM InferenceMixture-of-Experts Inference

    Not Community Labs Inc. · Intel Corporation

  22. How Sparse Probability Maps Shape Mixture-of-Experts Routing

    Oct 5, 2026Tomás Brogueira, Marcos Treviso, Miguel CouceiroMixture-of-Experts Language ModelsMixture-of-Experts Inference

    Técnico, Universidade de Lisboa · INESC-ID · Instituto de Telecomunicações +2

  23. ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models

    Oct 1, 2026Lianjun Liu, Shipeng Li, You Huang +5Tucker DecompositionMixture-of-Experts Language Models

    Hainan University, Haikou, China · Xiamen University, Xiamen, China · Zhejiang Normal University, Jinhua, China

  24. Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUs

    Sep 30, 2026Jaehwan Lee, Sangmin Lee, Chaewon Kim +2Expert ParallelismMixture-of-Experts Inference

    Dept. of Computer Science and Engineering Seoul National University · Dept. of Computer Science and Engineering Graduate School of Data Science Seoul National University

  25. DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference

    Sep 30, 2026Xudong Tan, Peng Ye, Ming Xie +4Mixture-of-Experts PruningEfficient Multimodal Inference

    Fudan University · The Chinese University of Hong Kong

  26. Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging

    Sep 29, 2026Sanjali Yadav, Bahar AsgariMixture-of-Experts InferenceMemory-Efficient Inference

    University of Maryland, College Park

  27. Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts

    Sep 29, 2026Kwanhee Lee, Namhoon Lee, Dan AlistarhMixture-of-Experts QuantizationMixture-of-Experts Inference

    POSTECH & ISTA · POSTECH · ISTA

  28. MoRE: Scaling mixture of experts with hardware-aware low-rank routing

    Sep 28, 2026Honam Wong, Surbhi Goel, Enric Boix-AdseràParameter-Free Mixture-of-Experts RoutingLow-Rank Matrix Decomposition

    University of Pennsylvania · The Wharton School, University of Pennsylvania

  29. BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding

    Sep 28, 2026Ali Abbasi, Justin Shi, Soheil KolouriMixture-of-Experts InferenceLLM Inference Acceleration

    College of Connected Computing, Vanderbilt University

  30. SlimWise: Decoupling Expert Pruning Across Prefill and Decode for Efficient MoE Serving

    Sep 28, 2026Gunho Park, Kyoungho Jeun, Juntaek Oh +3Mixture-of-Experts PruningDisaggregated LLM Serving

    a2sys · KAIST