cs.LGOct 24, 2026

COPE: Composable Performance Emulation for Mixture-of-Experts Offloading Policies

Authors: Hanming Zhang, Erhu Feng

Organizations: Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

As LLMs scale, mixture-of-experts (MoE) architectures increase model capacity while keeping per-token computation manageable through sparse expert activation. This sparsity also enables memory-efficient deployment by retaining most expert weights in CPU memory and transferring or executing only the routed experts on demand. Existing MoE offloading systems typically target specific model-platform configurations, yet differences in model sparsity, CPU and GPU throughput, VRAM capacity, and host-device bandwidth can change the policy ranking. Because implementing and exhaustively evaluating all candidate policies incurs substantial engineering cost, efficiently selecting the best-performing policy for each model-platform configuration remains a key challenge for practical MoE offloading. We present COPE (Composable Offloading Policy Emulator), which selects the best-performing MoE offloading policy for a target model-platform configuration before runtime integration. COPE comprises two components. Its Policy Matrix factors the design space into two independent axes, expert residency and MoE execution strategy, enabling mechanism reuse across candidate policies and exposing four new candidates. Its Virtual Heterogeneous Machine profiles combined-operator schedules on the target platform and combines these measurements with offline routing traces to estimate candidate performance. Compared with the evaluated baselines, COPE improves fastest-policy selection accuracy by 24%-100% and reduces performance-prediction error by 1.7×1.7\times to 6.8×6.8\times. By identifying the preferred policy before integration, COPE further reduces policy-specific implementation effort by up to 74%.

Figures & tables

Figure 1. Expert offloading choices. RAM retains all experts; VRAM caches some copies. Solid paths preserve the checkpoint: a hit runs on the GPU, while a miss runs on the CPU or stages to GPU. Dashed rerouting, a crossed-out skip arrow, and a smaller quantized CPU weight illustrate alternatives outside our scope.
Figure 2. DeepSeek-Coder-V2-Lite policy landscape for three resource slices on one A100. Throughput is normalized to the fastest candidate in each slice; color identifies execution and hatching identifies residency.
Figure 3. Overview of COPE. The routing trace contains routed and predicted expert IDs; the panel illustrates routed IDs across decode steps. The atomic-operator profile and routing trace are the only inputs to VHM cache replay.
Figure 4. Policy Matrix: three expert-residency families crossed with four canonical MoE execution schedules. Under Fixed residency, the initial set may follow expert-ID or offline popularity order ( Kamahori et al., 2025 ) . Prior systems’ workload and residency granularity may differ from COPE’s batch-1, per-expert implementations.
Figure 5. The layer-partition baseline and four cache execution schedules. Parallel sends misses to CPU; wait-then-GPU stages every miss; ready-first overlaps resident GPU work with staging; cost-selected assigns miss groups to CPU or staged-GPU execution. Prefetch is omitted.
Figure 6. Measured atomic-operator latency and HRM estimates for DeepSeek-V2-Lite on RTX 4090 with 32 CPU threads.
trace: routed IDs Trace[t,l]\mathrm{Trace}[t,l]
predicted next-layer IDs Pred[t,l]\mathrm{Pred}[t,l]
profile: Shared\mathrm{Shared} , MoE[e,h]\mathrm{MoE}[e,h] , Prefetch[r,q]\mathrm{Prefetch}[r,q]
policy p=(r,e)p=(r,e) : residency rr , execution ee
Algorithm 1 VHM replay: predict one policy’s TPOT from the profile and an offline routing trace.
ActionChange
ReplayAnother candidate or a workload with an existing valid trace
RetraceChanged workload or predictor without a valid trace
RecalibrateChanged model or target operating point, including cache budget and compute/transfer resources
UnsupportedMissing capabilities required by the candidate
Table 1. Action required after an input changes.
Figure 7. Measured throughput of five selected candidates under 10 A100 resource configurations.
ConfigurationDS-CoderQwen-3.5Qwen-3
CPU threads = 1Mutable + Ready-first (2.62 ×\times )Mutable + Cost-selected (2.78 ×\times )Mutable + Cost-selected (6.05 ×\times )
CPU threads = 2Mutable + Parallel (1.90 ×\times )Mutable + Cost-selected (1.74 ×\times )Mutable + Parallel (4.13 ×\times )
CPU threads = 16Fixed + Parallel (1.23 ×\times )Fixed + Parallel (1.02 ×\times )Fixed + Parallel (1.63 ×\times )
MPS = 50%Fixed + Parallel (1.31 ×\times )Mutable + Parallel (1.24 ×\times )Mutable + Parallel (1.99 ×\times )
VRAM = 16 GiBFixed + Parallel (1.29 ×\times )Mutable + Parallel (1.14 ×\times )Mutable + Parallel (1.53 ×\times )
Table 2. Measured TPOT winner at representative A100 model–configuration points. Parentheses give speedup over layer partition, TPOTlayer partition/TPOTwinner\mathrm{TPOT}_{\mathrm{layer\ partition}}/\mathrm{TPOT}_{\mathrm{winner}} . Each cell is tinted with its winner’s policy color from Figures 2 and 7 .
Figure 8. Mean Spearman rank correlation over the nine candidates on the A100 resource sweeps for DS-Coder, Qwen-3.5, and Qwen-3. Higher is better.
   threadsMPSVRAM
(threads, MPS, VRAM)(8,100,24)(8,100,24)(t,100,24)(t,100,24)(8,MPS,24)(8,\mathrm{MPS},24)(8,100,VRAM)(8,100,\mathrm{VRAM})
   12416205080162832
DS-CoderCOPE5/51/55/54/55/55/55/55/55/55/5–
Atomic-op sim.0/50/55/50/50/50/50/50/50/50/5–
HRM0/50/55/50/50/50/50/50/50/50/5–
Qwen-3.5COPE5/55/54/55/54/55/55/55/55/5–5/5
Table 3. Top-1 policy-selection accuracy across A100 resource sweeps for DS-Coder, Qwen-3.5, and Qwen-3. The measured lowest-TPOT policy among nine candidates is the ground truth for each prompt. Each x/5x/5 reports the number of prompts for which the predicted and measured winners agree; dashes indicate infeasible configurations.
Figure 9. RTX 4090 predicted versus measured TPOT for four models, five prompts, and nine candidates per model. Each measured point averages five fresh processes. The dashed identity line and shaded ±\pm 10% band provide an absolute-error reference.
Figure 10. Measured and predicted TPOT of the three fastest measured candidates on the RTX 4090.
ModelEstimatorTop-1Top-3ρ\rhoMAPE
DS-CoderCOPE5/515/151.00017.43%
Atomic-op sim.0/55/150.67840.11%
HRM0/55/150.67867.38%
DS-ChatCOPE5/515/151.00012.38%
Atomic-op sim.0/55/150.67840.62%
HRM0/55/150.67867.77%
Table 4. RTX 4090 selection counts, rank correlation, and candidate error for all estimators. Top-1 compares predicted and measured TPOT argmins; Top-3 is three-policy set overlap; MAPE averages over all prompt–candidate pairs.
Figure 11. Measured and predicted TPOT of the three fastest measured candidates in each of nine model–reservation cells on the RTX 5090.
ModelEstimatorTop-1Top-3ρ\rhoMAPE
DS-CoderCOPE15/1544/450.9919.10%
Atomic-op sim.0/1517/450.74523.03%
HRM0/1517/450.74553.81%
DS-ChatCOPE12/1545/450.9797.95%
Atomic-op sim.0/1515/450.71924.13%
HRM0/1515/450.73653.92%
Table 5. RTX 5090 prediction over three VRAM reservations per model. Denominators are 15 prompts and 45 top-three slots per model.
Figure 12. Policy-specific lines added to llama.cpp for the fixed-Parallel policy selected for Qwen3 on the RTX 4090, versus implementing every option along each Policy Matrix axis. Shared plumbing and diagnostics are excluded.
ProfileDS-CoderDS-ChatQwen-3.5Qwen-3
Atomic-operator profile2.82.83.43.5
Combined ops25.925.943.533.9
Table 6. RTX 4090 profiling time in minutes. Atomic-operator times are measured; combined-operator times are projected as 25% of measured exhaustive-schedule time, assuming only replay-reachable states are profiled.
More in this topic

Expert Offloading

Expert offloading manages where model experts reside and execute so sparse models can run efficiently on memory-constrained hardware.

See all 20 papers

Explore similar work

CardsList
Mar 9, 2026

Speculating Experts Accelerates Inference for Mixture-of-Experts

Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token compute. However, in memory-constrained inference settings, expert weights must be offloaded to CPU, creating a performance bottleneck from CPU-GPU transfers during decoding. We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts, enabling memory transfers to overlap with computation. Across multiple MoE architectures, we demonstrate that future experts can be reliably predicted by these internal representations. We also demonstrate that executing speculated experts generally maintains downstream task accuracy, thus preserving more effective compute-memory overlap by eliminating the need to re-fetch true router-selected experts. Integrated into an optimized inference engine, our approach achieves up to 14% reduction in time per output token (TPOT) over on-demand loading of experts from CPU memory. For MoEs where speculative execution alone yields suboptimal accuracy, we further examine lightweight estimators that improve expert prediction hit rates, thereby reducing performance degradation. Our code is released in open-source at https://github.com/axonn-ai/yalis/tree/offload_prefetch.

Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2

University of Maryland · TogetherAI

Mixture-of-Experts InferenceLLM Inference AccelerationMixture-of-Experts Language Models
AskPDF
Sep 1, 2026

PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Composition

Mixture-of-Experts (MoE) architectures scale Large Language Model (LLM) capacity efficiently by activating a sparse subset of experts per token. However, modern MoE inference remains heavily constrained by the rigid, whole-expert abstraction. Existing frameworks manage, schedule, or prune experts as atomic execution units, which fixes the optimization boundary too early and leaves fine-grained intra-expert computational redundancy underexplored. In this work, we present PCoMoE, a path-compositional execution framework that shifts MoE inference from coarse-grained expert selection to fine-grained path composition. PCoMoE incorporates a path-level formulation of expert computation, a compatibility-aware layer-wise pruning strategy to suppress low-value path combinations, and a hardware-friendly execution engine to exploit reusable sub-expert structures under strictly bounded overheads. Experimental results demonstrate that PCoMoE achieves up to a 1.31x end-to-end inference speedup while enhancing model accuracy by 10%. The code is available at https://github.com/gzyyy0/PCoMoE

Ziyan Gan, Fangxin Liu, Chenyang Guan +10

1Shanghai Jiao Tong University, China · 2Alibaba Group, China

Mixture-of-Experts PruningMixture-of-Experts InferenceLLM Inference AccelerationMixture-of-Experts Models
AskPDF
Mar 6, 2026

MoEless: Efficient MoE LLM Serving with Serverless Experts

Large Language Models (LLMs) increasingly adopt Mixture-of-Experts (MoE) architectures to scale efficiently under stringent resource constraints. However, MoE's sparse activation causes severe expert load imbalance, where a few experts become stragglers while others remain underutilized, leading to inflated inference latency and cost. Existing solutions assume static, serverful model deployments, limiting expert elasticity and often incurring costly expert swapping or degraded output quality. We present MoEless, an efficient serverless MoE serving framework that mitigates expert load imbalance via elastic expert execution. MoEless leverages lightweight, layer-aware predictors to estimate incoming expert load distributions and proactively identify stragglers. We design optimized scaling and placement strategies to improve function locality, GPU utilization, and cross-expert load balance. MoEless is prototyped on top of Megatron-LM and deployed on an eight-GPU testbed. Experiments with open-source MoE models and real-world workloads show that MoEless reduces inference latency by 43% and inference cost by 84% compared to state-of-the-art solutions.

Hanfei Yu, Bei Ouyang, Shwai He +2

University of Maryland College Park

LLM Inference EfficiencyExpert Load BalancingLLM ServingMixture-of-Experts Inference
AskPDF