Organizations: Shanghai Jiao Tong University, Shanghai, China · Shanghai Jiao Tong University, China · Hong Kong University of Science and Technology, Hongkong, China
Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine speculative decoding with expert offloading. Its lightweight quantized draft model runs efficiently on device while preserving high-fidelity expert activation patterns. This provides accurate multi-step lookahead over future expert usage, allowing the runtime to organize expert movement and execution ahead of verification and improve both communication and execution efficiency. Under single-request inference, Specter achieves an average 1.50x (up to 1.88x) speedup in time per output token (TPOT) over the state-of-the-art system. Specter makes memory-constrained, single-GPU MoE inference more practical on commodity hardware. Our code is available at https://github.com/wwwwwf/specter_asplos.
Figures & tables
Figure 1. Conventional offloading versus Specter . Aligned quantized speculative decoding lays the foundation for accurate prefetching and efficient system design. Two diagrams and timelines contrast intra-token prefetching with multi-step speculative offloading. The conventional path alternates PCIe transfers and GPU inference, leaving idle intervals. The Specter path uses a quantized draft model to produce candidate tokens and an expert lookahead list, allowing expert prefetching to overlap with drafting and target verification.
Figure 2. Expert-ID selection fidelity between an INT4 Qwen1.5-MoE draft model and its FP16 target model. Three histograms show expert-ID match-rate distributions for all observations, per-layer means, and per-request means. The horizontal axes show match rate and the vertical axis shows frequency, both in percent. All three distributions concentrate around 90 to 92 percent, with a reported global mean of 90.8 percent.
Model@(GPTQ&Q4)
Method@(DS&Q4)
Precision@(GGUF&DS)
Phi
DS
Qwen
GGUF
GPTQ
AWQ
Q2
Q4
Q8
Max
99.79
99.45
93.48
99.99
99.45
99.45
99.98
99.99
99.99
Min
95.15
94.49
86.30
95.09
94.49
93.17
90.10
95.09
98.66
Mean
96.50
96.33
90.82
96.53
96.33
95.72
92.96
96.53
99.38
Table 1. Expert activation similarity (%) across MoE models (using native top- k configuration and GPTQ method), diverse quant methods ( Frantar et al., 2022 ; Lin et al., 2024 ) , and GGUF-format precisions (based on DeepSeekV2-Lite).
Figure 3. Increasing the speculative depth yields an approximately linear benefit in dense models, but a markedly nonlinear trend in MoE models, as deeper speculation rapidly increases I/O demand under memory constraints. Schematic timelines compare speculative decoding in dense and MoE models. Light blocks represent drafting, dark blocks represent verification, and an orange outline marks additional overhead. Increasing draft depth adds little verification time in the dense example, whereas the MoE example has a substantially longer verification interval and lower speedup because more experts are activated.
Figure 4. Measured expert working-set growth with speculation depth. Three line charts plot distinct experts per layer against speculation depths of 1, 2, 4, 8, and 16 for DeepSeekV2-Lite, Qwen1.5-MoE, and Phi-3.5-MoE. Orange lines show the mean and shaded bands show the minimum-to-maximum range. Expert demand rises in all three models, with annotated growth factors of 5.29, 7.18, and 4.65, respectively.
Figure 5. The high-level architecture of Specter . The system integrates a Speculative Inference Controller, a Predictive I/O Orchestrator, and a Streamlined Execution Engine to transform I/O latency into productive computation through proactive data orchestration and adaptive control. A layered diagram connects the speculative decoding workflow to three system modules and the memory hierarchy. The controller initializes models, profiles speculative depth, and manages decoding state. The execution engine dispatches fused drafting and reordered expert verification. The I/O orchestrator analyzes routing patterns, plans prefetches, and manages residency. GPU memory holds quantized experts, an expert cache, and shared parameters, while target expert weights reside in CPU memory and transfer over PCIe.
Figure 6. Overview of parameter distribution and data layout orchestration in Specter . Diagram of parameter placement across GPU and CPU memory. The draft and target models share GPU-resident parameters and a KV cache. Quantized draft experts remain on the GPU, while full-precision target experts are prefetched or loaded from CPU memory into a GPU expert cache.
Figure 7. Speculative depth profiling results for DeepSeekV2-Lite, Qwen1.5-MoE and Phi-3.5-MoE. The reported TPOT is averaged over prompts and repeated runs. Three line plots show TPOT against speculation depth for DeepSeekV2-Lite, Qwen1.5-MoE, and Phi-3.5-MoE. Highlighted minima occur at depths 16, 8, and 16, respectively. Increasing speculation depth beyond these minima does not consistently reduce TPOT.
Figure 8. State lifecycle across one speculative decoding iteration, including verifier-guided token commitment and shared-prefix KV management. State-transition diagram for one speculative decoding iteration. Persistent tokens and shared prefix KV states feed the draft and target models. Drafting creates temporary tokens and KV states; causally masked target verification produces verification states. Sampling commits accepted state to the persistent prefix, while temporary draft state is released.
Figure 9. Execution timeline of adaptive ELB-guided prefetching during drafting. Timeline divided into Cold Start, Prefetch Overlap, and Final Cleanup. ELB counters collect routing metadata during draft computation. Snapshots at chunk boundaries trigger prefetch transfers on a separate CUDA I/O stream, overlapping adaptive draft chunks. The final chunk discards new routing metadata and clears the ELB before verification begins.
Figure 10. Performance comparison of different backends: (a) prefill stage latency, (b) decode stage latency. Lower values indicate better performance. Two plots compare Marlin, ExLV1, ExLV2, and Torch. The prefill plot shows latency across sequence lengths from 4 to 128, and the decode plot shows one latency bar per backend. Marlin has the lowest latency and Torch the highest in both plots.
Figure 11. Impact of quantization-aware fused expert kernels on throughput and latency in MoE models. Grouped bar charts compare FP16, INT4, and fused INT4 execution for Qwen1.5-MoE and Phi-3.5-MoE. For both models, INT4 lowers TPOT and raises throughput relative to FP16, and fused INT4 further lowers TPOT and raises throughput.
Figure 12. Comparison of conventional token-wise verification and Specter’s cache-aware expert-wise execution. Two diagrams compare token-wise and expert-wise verification. Naive computation traverses a token-by-expert grid and uses GEMV operations with experts in GPU or CPU memory. Reordering groups tokens by expert for GEMM computation and separates cache-hit and cache-miss experts around the expert cache.
Phi-3.5-MoE ( Abdin et al., 2024 )
Qwen1.5-MoE ( Qwen Team, 2024 )
DeepSeekV2-Lite ( DeepSeek-AI, 2024 )
Total Param.
41.9B
14.3B
15.7B
Activated Param.
6.6B
2.7B
2.4B
Total Weight
78GB
29GB
34GB
MoE Layer Number
32
24
26
First Dense Layer
0
0
1
Experts (Per Layer)
16
60
64
Table 2. Configuration of evaluated models.
Figure 13. Dataset-induced expert-routing diversity on DeepSeekV2-Lite with K=16 . Left: mean distinct experts per layer of each dataset, with 10th–90th percentile ranges. Right: pairwise Jensen–Shannon distances between normalized routing distributions. Two panels compare five datasets. The bar chart shows distinct experts per layer with percentile intervals: HumanEval has the largest mean and WikiText-2 the smallest. The heat map shows zero self-distances and nonzero distances between different datasets, with WikiText-2 generally the most distinct.
Figure 14. End-to-end inference performance of Specter and other approaches on varying datasets, memory settings and models. The figures in the upper row represent experiments conducted under higher GPU memory conditions, while the figures in the bottom row represent experiments performed under lower GPU memory conditions. Under shared-weight accounting, the reference high/low memory budgets are 24/18 GB, 20/16 GB, and 40/32 GB for DeepSeekV2-Lite, Qwen1.5-MoE, and Phi-3.5-MoE, respectively. Six grouped bar charts show TPOT in milliseconds per token for seven systems across five datasets and their average. Columns represent DeepSeekV2-Lite, Qwen1.5-MoE, and Phi-3.5-MoE; rows represent high and low memory budgets. Specter has the lowest average TPOT in each panel, while Transformers has the highest.
Figure 15. TPOT and throughput scaling with batch size on DeepSeekV2-Lite and Qwen1.5-MoE. Four line charts compare Specter, SpecMoEOff, MO-SM, and Transformers as batch size increases from 1 to 32. Columns represent DeepSeekV2-Lite and Qwen1.5-MoE; the upper row shows throughput and the lower row shows TPOT on logarithmic vertical scales. Specter has the highest throughput and lowest TPOT at every displayed batch size.
DeepSeekV2-Lite
Qwen1.5-MoE
Method
GK
WT
HE
GP
C4
AVG
GK
WT
HE
GP
C4
AVG
TPOT (ms/token) ↓
Transformers
1504.7
1451.2
1489.6
1485.9
1536.7
1493.6
1171.0
1192.2
1129.0
1195.6
1249.2
1187.4
MO-SM
1267.3
1206.5
1254.9
1268.6
1317.0
1262.9
1059.5
1020.6
982.8
1038.5
1067.1
1033.7
SpecMoEOff
707.6
722.0
705.7
705.7
749.0
718.0
686.2
600.2
593.2
641.1
648.1
633.8
Specter
125.9
125.0
100.2
122.9
122.1
119.2
286.6
250.0
241.6
256.3
267.1
260.3
Table 3. Absolute performance across workloads at B=32 .
Figure 16. Expert working-set growth with batch size. Three line charts show mean distinct experts per layer and shaded minimum-to-maximum ranges for batch sizes 1, 2, 4, and 8. Expert counts increase for DeepSeekV2-Lite, Qwen1.5-MoE, and Phi-3.5-MoE, with Qwen1.5-MoE exhibiting the largest relative increase.
Scenario
Draft
Target
Shared
Total
Independent
6.13
7.27
—
13.40
+ Shared Experts
4.58
5.72
1.55
11.85
+ Other Params
2.66
3.80
3.47
9.93
+ KV Cache
0.41
1.55
5.72
7.68 ↓ 43%
Table 4. Comparative GPU memory footprints (GB) of full-precision and 4-bit quantized Qwen1.5-MoE with a 12K-token context.
Figure 17. Cache hit rates of different policies on DeepSeekV2-Lite under three GPU-memory budgets. Grouped bars compare cache hit percentages for LRU, LRU-S, SP-E, SP-L, and Specter at memory budgets of 16, 24, and 32 GB. Specter remains close to 100 percent and has the highest hit rate in each group. LRU and LRU-S improve as the memory budget increases, with LRU-S approaching the prefetch policies at 32 GB.
Metric
DeepSeekV2-Lite
Qwen1.5-MoE
Phi-3.5-MoE
Draft
674.0
223.3
952.5
Verification
136.1
88.2
82.2
Total H2D Load
243.4
89.7
2089.2
Exposed H2D Stall
108.3
35.4
1621.3
Hidden H2D
135.1
54.3
467.9
Hidden Ratio
55.5%
60.5%
22.4%
Table 5. Per-iteration latency breakdown across models and cumulative component ablation on DeepSeekV2-Lite.
(a) Controlled expert-prediction errors
Error Rate
0%
25%
50%
75%
Prefetch Acc.
1.000
0.785
0.575
0.410
TPOT
63.0
66.5
82.6
84.2
Ratio
1.00 ×
1.06 ×
1.31 ×
1.34 ×
Table 6. Robustness to expert-prediction errors and natural variations. TPOT is reported in ms/token.
The Mixture-of-Experts (MoE) architecture has emerged as a promising approach to mitigate the rising computational costs of large language models (LLMs) by selectively activating parameters. However, its high memory requirements and sub-optimal parameter efficiency pose significant challenges for efficient deployment. Although CPU-offloaded MoE inference systems have been proposed in the literature, they offer limited efficiency, particularly for large batch sizes. In this work, we propose SpecMoE, a memory-efficient MoE inference system based on our self-assisted speculative decoding algorithm. SpecMoE demonstrates the effectiveness of applying speculative decoding to MoE inference without requiring additional model training or fine-tuning. Our system improves inference throughput by up to 4.30×, while significantly reducing bandwidth requirements of both memory and interconnect on memory-constrained systems.
Mixture-of-Experts (MoE) models have gained popularity as a means of scaling the capacity of large language models (LLMs) while maintaining sparse activations and reduced per-token compute. However, in memory-constrained inference settings, expert weights must be offloaded to CPU, creating a performance bottleneck from CPU-GPU transfers during decoding. We propose an expert prefetching scheme that leverages currently computed internal model representations to speculate future experts, enabling memory transfers to overlap with computation. Across multiple MoE architectures, we demonstrate that future experts can be reliably predicted by these internal representations. We also demonstrate that executing speculated experts generally maintains downstream task accuracy, thus preserving more effective compute-memory overlap by eliminating the need to re-fetch true router-selected experts. Integrated into an optimized inference engine, our approach achieves up to 14% reduction in time per output token (TPOT) over on-demand loading of experts from CPU memory. For MoEs where speculative execution alone yields suboptimal accuracy, we further examine lightweight estimators that improve expert prediction hit rates, thereby reducing performance degradation. Our code is released in open-source at https://github.com/axonn-ai/yalis/tree/offload_prefetch.
Speculative decoding accelerates Large Language Model (LLM) inference by verifying multiple drafted tokens in parallel. However, for Mixture-of-Experts (MoE) models, this parallelism introduces a severe bottleneck: large draft trees activate many unique experts, significantly increasing memory pressure and diminishing speedups from speculative decoding relative to autoregressive decoding. Prior methods reduce speculation depth when MoE verification becomes expensive. We propose MoE-Spec, a training-free verification-time expert budgeting method that decouples speculation depth from memory cost by enforcing a fixed expert capacity limit at each layer, loading only the experts that contribute most to verification and dropping the long tail of rarely used experts that drive bandwidth overhead. Experiments across multiple model scales and datasets show that this method yields 10--30% higher throughput than state-of-the-art speculative decoding baselines (EAGLE-3) at comparable quality, with flexibility to trade accuracy for further latency reductions through tighter budgets.
Bradley McDanel, Steven Li, Sruthikesh Surineni +1