Multi-GPU MoE Inference with Adaptive Expert Offloading
Authors: William X. Zheng, Zhongjie Chen, Shuo Yang, Ziming Mao, Xiaoze Fan, Yi Pan, Kunjun Li, Hanfei Yu, Chenfeng Xu, Yang Zhou, Rishabh Iyer, Ion Stoica
Organizations: University of California, Berkeley · Microsoft · Shanghai Jiao-tong University · University of Texas at Austin, Department of Computer Science · University of California, Davis
Modern Mixture-of-Experts (MoE) inference is constrained by GPU memory. This constraint is mitigated by expert offloading, which stores part of the model's weight in host memory and executes on demand. Though MoE architecture has computation sparsity, more GPUs are needed for serving increasingly larger frontier models. However, existing expert offloading systems overlook this need and fail to scale performance with additional GPUs. We present AdaOffload, a general and efficient multi-GPU hybrid expert execution framework that effectively handles heterogeneous hardware and workloads. We co-design GPU-CPU parallelism and GPU-CPU co-execution for decode and prefill with online workload observation that adapts to different hardware, workload, and batch size heterogeneity. We evaluated AdaOffload on five different platforms with varying GPU capability (L4, Pro6000, B200), number of GPUs, CPU capability, and NUMA topology on three different frontier open-weight models (284B to 753B). AdaOffload achieves 1.9× to 15.5× throughput speedups across different models over state-of-the-art systems.
Figures & tables
Figure 1: Mean TTFT and decode throughput when serving 16 concurrent requests with L4 GPUs with expert offloading ( AdaOffload ). 8-fit indicates fitting all experts in VRAM.
Figure 2: Decode throughput of DeepSeek-V4-Flash on eight L4 GPUs with 16 concurrent requests. Cold-KV offloading keeps all experts on GPU, with indexer keys on GPU or CPU as indicated. Expert offloading instead uses a half-VRAM expert cache while keeping KV and indexer keys on GPU.
Figure 3: Decode expert footprint across batch sizes for DSV4-Flash on AIME, Chemistry ( Hendrycks et al., 2021 ) , and ShareGPT ( anon8231489123, 2023 ) workloads. Left: number of unique experts; right: median token load per expert.
System
Hardware aware
Batch-size aware
Multi GPU
Supported parallelism
KTransformers
–
–
✓
TP
Ollama
–
–
✓
PP
llama.cpp
–
–
✓
PP
FreeToken
✓
–
–
–
AdaOffload (ours)
✓
✓
✓
HTP
Table 1: Comparison of AdaOffload with prior works.
Figure 4: Overview of AdaOffload . HTP employs distinct TP domains for CPUs and GPUs. The CPU is split by NUMA node; hardware profiling selects intra-NUMA sharding, resulting in TP4. Offline profiling provides hardware- and model-specific cost tables. Online expert-residency and token-load samples guide asynchronous policy updates. In this example, q=1/3 , Expert 3 is fetched to the GPU for minimum completion time.
Figure 5: CPU execution efficiency and task latency. (a) CPU kernel efficiency on two different single-NUMA-node configurations at different expert intermediate widths. Intermediate widths 2048/1024/512/256 correspond to NUMA-local sharding factors of 1/2/4/8. (b) CPU task wall-time distributions with spin waiting off/on.
Figure 6: Four-point latency prediction on L4 × 8 for DSV4-Flash at U=8 . Held-out WAPE is 2.6% and 4.0%, respectively.
Provider
GPUs
Eff. compute (TFLOP/s)
CPU (cores)
Eff. compute (TFLOP/s)
DRAM (GiB)
PCIe/link (GB/s)
Aggregate H2D (GB/s)
Mibura
L4 ×4
3.390
Xeon 4410Y (24)
0.428
256
Gen4 (32)
98.0
GCP
L4 ×8
6.317
Xeon (48)
0.510
402
Gen3 (16)
33.3
NVIDIA
PRO 6000 ×2
6.829
TR PRO 7955WX (16)
0.314
480
Gen5 (64)
103.2
AWS
PRO 6000 ×2
8.751
Xeon 8559C (24)
0.463
500
Gen5 (64)
50.8
Table 2: Hardware specifications across different multi-GPU nodes. Effective compute is measured under one representative workload.
Figure 7: End-to-end performance of AdaOffload and baselines on AIME reasoning tasks. Columns show concurrency 1, 4, and 8; rows show streaming decode throughput and mean TTFT.
Figure 8: End-to-end performance of AdaOffload and baselines on coding tasks. Columns show concurrency 1, 4, and 8; rows show streaming decode throughput and mean visible TTFT. Ollama timeouts all session except for concurrency 1.
Figure 9: Decode throughput of AdaOffload versus the best and worst measured fixed- q policies at concurrency C=1 (left) and C=8 (right).
Figure 10: Prefill forward latency of AdaOffload and KTransformers for coding tasks at concurrency C=8 .
Figure 11: Decode throughput and mean TTFT for different numbers of GCP L4 GPUs. Framework can use all CPUs.
Model
Expert format
Total params (B)
Active params (B)
Expert weights (GiB)
Non-expert weights (GiB)
Total weights (GiB)
DSV4-Flash
MXFP4
284
13
137.06
8.24
145.30
GLM-5.3-Flash
NVFP4
320
18
159.47
16.62
176.09
GLM-5.2
NVFP4
753
40
379.69
34.65
414.34
Table 3: Model specification.
Figure 12: End-to-end performance of AdaOffload and baseline on AIME reasoning tasks with GLM5.3-Flash NVFP4.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 13: Decode throughput of AdaOffload versus the best and worst measured fixed-q policies at concurrency C=1 and C=8 for GLM5.3-Flash.
Figure 14: AIME performance of AdaOffload , llama.cpp, and Ollama with GLM-5.2 NVFP4 on two B200 GPUs as concurrency varies. (a) Aggregate streaming decode throughput; (b) mean TTFT (log scale). The output cap is 512 tokens. TTFT includes the first request.
Mixture-of-Experts (MoE) models reduce inference computation through sparse expert activation, but their large expert memory footprint often exceeds GPU capacity, making expert offloading necessary for memory-constrained inference. However, existing systems typically handle GPU-missed experts by either transferring them to the GPU, incurring substantial PCIe transfer latency, or executing them on the CPU, making the slower CPU computation the bottleneck. To address these limitations, we present \method, a CPU-GPU collaborative MoE inference system that adaptively assigns these missed experts to the CPU or GPU for efficient execution. By exploiting differences in expert token-loads, \method selectively transfers high-load experts to the GPU while executing low-load experts on the CPU. To minimize inference latency, \method adaptively adjusts expert assignments to balance the CPU computation path against the GPU transfer-and-computation path, allowing both execution paths to complete as early as possible. Extensive experiments on three representative MoE LLMs show that \method improves prefill speed by up to 51.9% and decode throughput by up to 39.5% over the strongest baseline across the evaluated batch sizes, with consistent gains under different GPU-memory budgets.
To deploy large Mixture-of-Experts (MoE) models cost-effectively, offloading-based single-GPU heterogeneous inference is crucial. While GPU-CPU architectures that offload cold experts are constrained by host memory bandwidth, emerging GPU-NDP architectures utilize DIMM-NDP to offload non-hot experts. However, non-hot experts are not a homogeneous memory-bound group: a significant subset of warm experts exists is severely penalized by high GPU I/O latency yet can saturate NDP compute throughput, exposing a critical compute gap. We present TriMoE, a novel GPU-CPU-NDP architecture that fills this gap by synergistically leveraging AMX-enabled CPU to precisely map hot, warm, and cold experts onto their optimal compute units. We further introduce a bottleneck-aware expert scheduling policy and a prediction-driven dynamic relayout/rebalancing scheme. Experiments demonstrate that TriMoE achieves up to 2.83x speedup over state-of-the-art solutions.
Yudong Pan, Yintao He, Tianhua Han +7
SKLP, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China
Offloading model state to host memory and streaming layer weights let a single multi-GPU node train large mixture-of-experts (MoE) models. GPU memory then becomes a performance budget, shared by resident expert state, retained activations, and execution worksets. The best allocation depends on the expert execution path, batch size, and sequence length. We present OOM (Offload-Optimized MoE), a full-parameter MoE training system that co-designs expert execution and memory allocation. For expert-parallel execution, OOM updates resident experts on their owning GPUs and jointly plans state residency and MoE activation retention according to pipeline dependencies. For local execution, OOM bounds GPU worksets through dependency-aware partitioning; in hybrid models, it also retains attention results on the host to avoid costly recomputation and selectively regenerates layer boundaries through cheaper layers to make room. Calibrated cost models guide joint configuration within GPU and host budgets to minimize step time; OOM selects among the configured paths through short executions. On four H100s, OOM trains with 1M-token contexts, whereas FSDP2, ZeRO-Offload, and MegaTrain exhaust GPU memory at 512K. It also trains a 122B-parameter MoE model, on which all three exhaust host memory. On four 24 GB RTX 4090s, it trains 512K-token contexts and 1,024 1K-token sequences per step without gradient accumulation. Across matched-global-batch comparisons, OOM delivers 1.58x and 4.08x geometric-mean (up to 2.14x and 7.77x) throughput over CPU-offloaded FSDP2 on the two platforms while using the least host memory among the evaluated systems -- as little as 1/3.0 of FSDP2's.
Ruijia Yang, Shiyuan Lin, Chengxi Liao +1
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China.