COPE: Composable Performance Emulation for Mixture-of-Experts Offloading Policies

Hanming Zhang; Erhu Feng

cs.LGOct 24, 2026

COPE: Composable Performance Emulation for Mixture-of-Experts Offloading Policies

Authors: Hanming Zhang, Erhu Feng

Organizations: Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University

Abstract

As LLMs scale, mixture-of-experts (MoE) architectures increase model capacity while keeping per-token computation manageable through sparse expert activation. This sparsity also enables memory-efficient deployment by retaining most expert weights in CPU memory and transferring or executing only the routed experts on demand. Existing MoE offloading systems typically target specific model-platform configurations, yet differences in model sparsity, CPU and GPU throughput, VRAM capacity, and host-device bandwidth can change the policy ranking. Because implementing and exhaustively evaluating all candidate policies incurs substantial engineering cost, efficiently selecting the best-performing policy for each model-platform configuration remains a key challenge for practical MoE offloading. We present COPE (Composable Offloading Policy Emulator), which selects the best-performing MoE offloading policy for a target model-platform configuration before runtime integration. COPE comprises two components. Its Policy Matrix factors the design space into two independent axes, expert residency and MoE execution strategy, enabling mechanism reuse across candidate policies and exposing four new candidates. Its Virtual Heterogeneous Machine profiles combined-operator schedules on the target platform and combines these measurements with offline routing traces to estimate candidate performance. Compared with the evaluated baselines, COPE improves fastest-policy selection accuracy by 24%-100% and reduces performance-prediction error by $1.7\times$ to $6.8\times$ . By identifying the preferred policy before integration, COPE further reduces policy-specific implementation effort by up to 74%.

Figures & tables

Figure 1. Expert offloading choices. RAM retains all experts; VRAM caches some copies. Solid paths preserve the checkpoint: a hit runs on the GPU, while a miss runs on the CPU or stages to GPU. Dashed rerouting, a crossed-out skip arrow, and a smaller quantized CPU weight illustrate alternatives outside our scope.

Figure 2. DeepSeek-Coder-V2-Lite policy landscape for three resource slices on one A100. Throughput is normalized to the fastest candidate in each slice; color identifies execution and hatching identifies residency.

Panel of Figure 3 — Figure 3. Overview of COPE. The routing trace contains routed and predicted expert IDs; the panel illustrates routed IDs across decode steps. The atomic-operator profile and routing trace are the only inputs to VHM cache replay.

Panel of Figure 4 — Figure 4. Policy Matrix: three expert-residency families crossed with four canonical MoE execution schedules. Under Fixed residency, the initial set may follow expert-ID or offline popularity order ( Kamahori et al., 2025 ) . Prior systems’ workload and residency granularity may differ from COPE’s batch-1, per-expert implementations.

Figure 5. The layer-partition baseline and four cache execution schedules. Parallel sends misses to CPU; wait-then-GPU stages every miss; ready-first overlaps resident GPU work with staging; cost-selected assigns miss groups to CPU or staged-GPU execution. Prefetch is omitted.

Figure 6. Measured atomic-operator latency and HRM estimates for DeepSeek-V2-Lite on RTX 4090 with 32 CPU threads.

trace: routed IDs

\mathrm{Trace}[t,l]

predicted next-layer IDs

\mathrm{Pred}[t,l]

profile:

\mathrm{Shared}

\mathrm{MoE}[e,h]

\mathrm{Prefetch}[r,q]

policy

p=(r,e)

: residency

r

, execution

e

Algorithm 1 VHM replay: predict one policy’s TPOT from the profile and an offline routing trace.

Action	Change
Replay	Another candidate or a workload with an existing valid trace
Retrace	Changed workload or predictor without a valid trace
Recalibrate	Changed model or target operating point, including cache budget and compute/transfer resources
Unsupported	Missing capabilities required by the candidate

Table 1. Action required after an input changes.

Figure 7. Measured throughput of five selected candidates under 10 A100 resource configurations.

Configuration	DS-Coder	Qwen-3.5	Qwen-3
CPU threads = 1	Mutable + Ready-first (2.62 $\times$ )	Mutable + Cost-selected (2.78 $\times$ )	Mutable + Cost-selected (6.05 $\times$ )
CPU threads = 2	Mutable + Parallel (1.90 $\times$ )	Mutable + Cost-selected (1.74 $\times$ )	Mutable + Parallel (4.13 $\times$ )
CPU threads = 16	Fixed + Parallel (1.23 $\times$ )	Fixed + Parallel (1.02 $\times$ )	Fixed + Parallel (1.63 $\times$ )
MPS = 50%	Fixed + Parallel (1.31 $\times$ )	Mutable + Parallel (1.24 $\times$ )	Mutable + Parallel (1.99 $\times$ )
VRAM = 16 GiB	Fixed + Parallel (1.29 $\times$ )	Mutable + Parallel (1.14 $\times$ )	Mutable + Parallel (1.53 $\times$ )

Table 2. Measured TPOT winner at representative A100 model–configuration points. Parentheses give speedup over layer partition,

\mathrm{TPOT}_{\mathrm{layer\ partition}}/\mathrm{TPOT}_{\mathrm{winner}}

. Each cell is tinted with its winner’s policy color from Figures 2 and 7 .

Figure 8. Mean Spearman rank correlation over the nine candidates on the A100 resource sweeps for DS-Coder, Qwen-3.5, and Qwen-3. Higher is better.

			threads				MPS			VRAM
(threads, MPS, VRAM)		$(8,100,24)$	$(t,100,24)$				$(8,\mathrm{MPS},24)$			$(8,100,\mathrm{VRAM})$
			1	2	4	16	20	50	80	16	28	32
DS-Coder	COPE	5/5	1/5	5/5	4/5	5/5	5/5	5/5	5/5	5/5	5/5	–
	Atomic-op sim.	0/5	0/5	5/5	0/5	0/5	0/5	0/5	0/5	0/5	0/5	–
	HRM	0/5	0/5	5/5	0/5	0/5	0/5	0/5	0/5	0/5	0/5	–
Qwen-3.5	COPE	5/5	5/5	4/5	5/5	4/5	5/5	5/5	5/5	5/5	–	5/5

Table 3. Top-1 policy-selection accuracy across A100 resource sweeps for DS-Coder, Qwen-3.5, and Qwen-3. The measured lowest-TPOT policy among nine candidates is the ground truth for each prompt. Each

x/5

reports the number of prompts for which the predicted and measured winners agree; dashes indicate infeasible configurations.

Figure 9. RTX 4090 predicted versus measured TPOT for four models, five prompts, and nine candidates per model. Each measured point averages five fresh processes. The dashed identity line and shaded

\pm

10% band provide an absolute-error reference.

Figure 10. Measured and predicted TPOT of the three fastest measured candidates on the RTX 4090.

Model	Estimator	Top-1	Top-3	$\rho$	MAPE
DS-Coder	COPE	5/5	15/15	1.000	17.43%
	Atomic-op sim.	0/5	5/15	0.678	40.11%
	HRM	0/5	5/15	0.678	67.38%
DS-Chat	COPE	5/5	15/15	1.000	12.38%
	Atomic-op sim.	0/5	5/15	0.678	40.62%
	HRM	0/5	5/15	0.678	67.77%

Table 4. RTX 4090 selection counts, rank correlation, and candidate error for all estimators. Top-1 compares predicted and measured TPOT argmins; Top-3 is three-policy set overlap; MAPE averages over all prompt–candidate pairs.

Figure 11. Measured and predicted TPOT of the three fastest measured candidates in each of nine model–reservation cells on the RTX 5090.

Model	Estimator	Top-1	Top-3	$\rho$	MAPE
DS-Coder	COPE	15/15	44/45	0.991	9.10%
	Atomic-op sim.	0/15	17/45	0.745	23.03%
	HRM	0/15	17/45	0.745	53.81%
DS-Chat	COPE	12/15	45/45	0.979	7.95%
	Atomic-op sim.	0/15	15/45	0.719	24.13%
	HRM	0/15	15/45	0.736	53.92%

Table 5. RTX 5090 prediction over three VRAM reservations per model. Denominators are 15 prompts and 45 top-three slots per model.

Figure 12. Policy-specific lines added to llama.cpp for the fixed-Parallel policy selected for Qwen3 on the RTX 4090, versus implementing every option along each Policy Matrix axis. Shared plumbing and diagnostics are excluded.

Profile	DS-Coder	DS-Chat	Qwen-3.5	Qwen-3
Atomic-operator profile	2.8	2.8	3.4	3.5
Combined ops	25.9	25.9	43.5	33.9

Table 6. RTX 4090 profiling time in minutes. Atomic-operator times are measured; combined-operator times are projected as 25% of measured exhaustive-schedule time, assuming only replay-reachable states are profiled.

Expert Offloading

Expert offloading manages where model experts reside and execute so sparse models can run efficiently on memory-constrained hardware.

See all 20 papers

Speculating Experts Accelerates Inference for Mixture-of-Experts
Mar 9, 2026Vivan Madan, Prajwal Singhania, Abhinav Bhatele +2Mixture-of-Experts Inference LLM Inference Acceleration
University of Maryland · TogetherAI
PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Composition
Sep 1, 2026Ziyan Gan, Fangxin Liu, Chenyang Guan +10Mixture-of-Experts Pruning Mixture-of-Experts Inference
1Shanghai Jiao Tong University, China · 2Alibaba Group, China
MoEless: Efficient MoE LLM Serving with Serverless Experts
Mar 6, 2026Hanfei Yu, Bei Ouyang, Shwai He +2LLM Inference Efficiency Expert Load Balancing
University of Maryland College Park

Mar 9, 2026J/K move · Enter open · S save

Speculating Experts Accelerates Inference for Mixture-of-Experts

Vivan Madan, Prajwal Singhania, Abhinav Bhatele, Tom Goldstein+1

University of Maryland · TogetherAI

Mixture-of-Experts Inference LLM Inference Acceleration Mixture-of-Experts Language Models

Open paper AskPDF arXiv

Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token compute. However, in memory-constrained inference settings, expert weights must be offloaded to CPU, creating a performance bottleneck from CPU-GPU transfers during decoding. We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts, enabling memory transfers to overlap with computation. Across multiple MoE architectures, we demonstrate that future experts can be reliably predicted by these internal representations. We also demonstrate that executing speculated experts generally maintains downstream task accuracy, thus preserving more effective compute-memory overlap by eliminating the need to re-fetch true router-selected experts. Integrated into an optimized inference engine, our approach achieves up to 14% reduction in time per output token (TPOT) over on-demand loading of experts from CPU memory. For MoEs where speculative execution alone yields suboptimal accuracy, we further examine lightweight estimators that improve expert prediction hit rates, thereby reducing performance degradation. Our code is released in open-source at https://github.com/axonn-ai/yalis/tree/offload_prefetch.

Abstract

Expert Offloading

Explore similar work

Speculating Experts Accelerates Inference for Mixture-of-Experts

PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Composition

MoEless: Efficient MoE LLM Serving with Serverless Experts