cs.AINov 6, 2026

Multi-GPU MoE Inference with Adaptive Expert Offloading

Authors: William X. Zheng, Zhongjie Chen, Shuo Yang, Ziming Mao, Xiaoze Fan, Yi Pan, Kunjun Li, Hanfei Yu, 

Organizations: University of California, Berkeley · Microsoft · Shanghai Jiao-tong University · University of Texas at Austin, Department of Computer Science · 

Abstract

Modern Mixture-of-Experts (MoE) inference is constrained by GPU memory. This constraint is mitigated by expert offloading, which stores part of the model's weight in host memory and executes on demand. Though MoE architecture has computation sparsity, more GPUs are needed for serving increasingly larger frontier models. However, existing expert offloading systems overlook this need and fail to scale performance with additional GPUs. We present AdaOffload, a general and efficient multi-GPU hybrid expert execution framework that effectively handles heterogeneous hardware and workloads. We co-design GPU-CPU parallelism and GPU-CPU co-execution for decode and prefill with online workload observation that adapts to different hardware, workload, and batch size heterogeneity. We evaluated AdaOffload on five different platforms with varying GPU capability (L4, Pro6000, B200), number of GPUs, CPU capability, and NUMA topology on three different frontier open-weight models (284B to 753B). AdaOffload achieves 1.9×1.9\times to 15.5×15.5\times throughput speedups across different models over state-of-the-art systems.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. TriMoE: Augmenting GPU with AMX-Enabled CPU and DIMM-NDP for High-Throughput MoE Inference via Offloading

    Mar 1, 2026Yudong Pan, Yintao He, Tianhua Han +7Mixture-of-Experts InferenceLLM Inference Acceleration

  2. OOM: Offload-Optimized Full-Parameter MoE Training via Memory-Execution Co-Design

    Nov 8, 2026Ruijia Yang, Shiyuan Lin, Chengxi Liao +1Memory-Efficient TrainingMixture-of-Experts Models