Multi-GPU MoE Inference with Adaptive Expert Offloading
Organizations: University of California, Berkeley · Microsoft · Shanghai Jiao-tong University · University of Texas at Austin, Department of Computer Science ·
Abstract
Modern Mixture-of-Experts (MoE) inference is constrained by GPU memory. This constraint is mitigated by expert offloading, which stores part of the model's weight in host memory and executes on demand. Though MoE architecture has computation sparsity, more GPUs are needed for serving increasingly larger frontier models. However, existing expert offloading systems overlook this need and fail to scale performance with additional GPUs. We present AdaOffload, a general and efficient multi-GPU hybrid expert execution framework that effectively handles heterogeneous hardware and workloads. We co-design GPU-CPU parallelism and GPU-CPU co-execution for decode and prefill with online workload observation that adapts to different hardware, workload, and batch size heterogeneity. We evaluated AdaOffload on five different platforms with varying GPU capability (L4, Pro6000, B200), number of GPUs, CPU capability, and NUMA topology on three different frontier open-weight models (284B to 753B). AdaOffload achieves to throughput speedups across different models over state-of-the-art systems.
Figures & tables
| System | Hardware aware | Batch-size aware | Multi GPU | Supported parallelism |
|---|---|---|---|---|
| KTransformers | – | – | TP | |
| Ollama | – | – | PP | |
| llama.cpp | – | – | PP | |
| FreeToken | – | – | – | |
| AdaOffload (ours) | HTP |
| Provider | GPUs | Eff. compute (TFLOP/s) | CPU (cores) | Eff. compute (TFLOP/s) | DRAM (GiB) | PCIe/link (GB/s) | Aggregate H2D (GB/s) |
|---|---|---|---|---|---|---|---|
| Mibura | L4 | 3.390 | Xeon 4410Y (24) | 0.428 | 256 | Gen4 (32) | 98.0 |
| GCP | L4 | 6.317 | Xeon (48) | 0.510 | 402 | Gen3 (16) | 33.3 |
| NVIDIA | PRO 6000 | 6.829 | TR PRO 7955WX (16) | 0.314 | 480 | Gen5 (64) | 103.2 |
| AWS | PRO 6000 | 8.751 | Xeon 8559C (24) | 0.463 | 500 | Gen5 (64) | 50.8 |
| Model | Expert format | Total params (B) | Active params (B) | Expert weights (GiB) | Non-expert weights (GiB) | Total weights (GiB) |
|---|---|---|---|---|---|---|
| DSV4-Flash | MXFP4 | 284 | 13 | 137.06 | 8.24 | 145.30 |
| GLM-5.3-Flash | NVFP4 | 320 | 18 | 159.47 | 16.62 | 176.09 |
| GLM-5.2 | NVFP4 | 753 | 40 | 379.69 | 34.65 | 414.34 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Mixture-of-Experts Inference
Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.
See all 200 papers