Mixture-of-Experts Inference

Overview

Mixture-of-experts inference studies how to select and execute only the experts needed for each token or input, aiming to reduce computation without losing model quality. Methods include adaptive routing and expert skipping, quantization and pruning, predictive prefetching and caching, speculative decoding, and expert-parallel scheduling across GPUs or heterogeneous devices. Systems are evaluated in settings such as prefill versus token-by-token decoding, long-context serving, distributed GPU clusters, and memory-limited edge devices, using measures like latency, throughput, memory traffic, and task accuracy. A central tension is that activating fewer parameters does not necessarily make inference faster: moving expert weights and coordinating irregular workloads can dominate the computation, especially when memory or network bandwidth is limited.

Latest papers

CardsList

200 papers28 papers in the last four weeks, down 3% on the four weeks before. 0.2% of all new papers.

  1. Communicate the Change: Efficient Expert Parallelism via Activation Compression

    Nov 17, 2026Fengyuan Liu, Jay Gala, Siva Reddy +1Communication-Efficient Mixture-of-ExpertsMixture-of-Experts Inference

  2. GEAR2^2: Pruned Experts Live On through Gated Reuse of Activated Experts and a Missing-Mass Residual

    Nov 16, 2026Linliang Chen, Yan Zhong, Wang Kang +3Mixture-of-Experts PruningMixture-of-Experts Language Models

  3. CECAR: Cache & Expert Co-Aware Routing Accelerates On-Device Inference of MoE LLMs

    Nov 15, 2026Byeongju Kim, Hoonki Lee, Byungjun Kim +4On-Device Language Model InferenceMixture-of-Experts Inference

  4. Fuse-All MoE Inference Kernel with Load Balancing on TPUs

    Nov 13, 2026Rupeng Liu, Bangsheng Tang, Zongwei Zhou +2Mixture-of-Experts InferenceLLM Inference Acceleration

  5. Fewer Expert Transfers Without Changing Routing: Expert Eviction for MoE Offloading

    Nov 11, 2026Pierre-Luc Bacon, Diego Calanzone, Jean-Maxime LaroucheMixture-of-Experts InferenceLLM Inference Acceleration

  6. Direct GPU Energy Measurements of Batch-Level Sharing in Mixture-of-Experts and Recurrent Models

    Nov 8, 2026Rafael Pina, Varuna De SilvaEnergy-Efficient MLMixture-of-Experts Inference

  7. Multi-GPU MoE Inference with Adaptive Expert Offloading

    Nov 6, 2026William X. Zheng, Zhongjie Chen, Shuo Yang +9Mixture-of-Experts InferenceExpert Offloading

  8. FastAFD: When Does Attention-FFN Disaggregation Pay on Rack-Scale NVLink?

    Nov 4, 2026Yichao Fu, Yuxuan Zhang, Minshen Zhang +3LLM Inference AccelerationMixture-of-Experts Inference

  9. Matching the Training-Time Weight Rule in Inference-Time Expert Deactivation

    Nov 4, 2026Hao Cheng, Zhiyu Yang, Bo Ma +1Mixture-of-Experts InferenceExpert Routing

  10. Accelerating Offloading-Based MoE Inference by Interleaving Concurrent Requests

    Nov 3, 2026Hongseung Yu, In Gim, Kyunghan Lee +1LLM Inference AccelerationMixture-of-Experts Inference

  11. HybridSpec: Throughput-Aware Speculative Decoding for Hybrid GDN-MoE Models

    Nov 2, 2026Haiyu Wang, Sai Qian Zhang, Zechun Liu +1Speculative DecodingLLM Inference Acceleration

  12. MatMoE: Matryoshka Mixture-of-Experts with Dynamic Mixed-Precision Quantization for Efficient Inference

    Nov 1, 2026Renjie Wei, Haochen Huang, Chuyu Qiu +3Mixture-of-Experts InferenceMixture-of-Experts Quantization

  13. RankLane: Efficient MoE Prefill on Accelerators with Ranked Expert Capacities

    Oct 30, 2026Wentao Ni, Chihao Yu, Hong Cai +4Mixture-of-Experts InferenceLLM Inference Acceleration

  14. Characterizing Recoverable Expert-Utilization Signatures in Fine-Tuned Mixture-of-Experts LLMs

    Oct 29, 2026Fei Wang, Baochun Li, Bo LiMixture-of-Experts Language ModelsMemorization in Language Models

  15. CRISP: Co-designing Pruning and Scheduling for Efficient MoE Inference on Edge Servers

    Oct 27, 2026Congwei Zhang, Youming Tao, Yifei Zou +6Mixture-of-Experts InferenceMemory-Efficient Inference

  16. COPE: Composable Performance Emulation for Mixture-of-Experts Offloading Policies

    Oct 24, 2026Hanming Zhang, Erhu FengExpert OffloadingMixture-of-Experts Inference

  17. Low-Latency MoE Inference in Space Data Centers: A Wireless-Power-Transfer Perspective

    Oct 22, 2026Haotian Zheng, Zhanwei Wang, Kaibin HuangEnergy-Aware LLM InferenceLLM Inference

  18. Don't Skip the Baseline: A Controlled Comparison of Dynamic Expert Skipping in MoEs

    Oct 20, 2026Jakob Steimle, Thomas Elsken, Daniel Ehrhardt +3Mixture-of-Experts InferenceMixture-of-Experts Pruning

  19. TempoMoE: Identifying and Steering Fast-Slow Thinking Experts for Efficient Multimodal Reasoning

    Oct 19, 2026Xiyue Zhang, Keru Zhou, Ziyi Wang +2Mixture-of-Experts InferenceEfficient Multimodal Inference