cs.LGNov 18, 2025
SaveSpecter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
Organizations: Shanghai Jiao Tong University, Shanghai, China · Shanghai Jiao Tong University, China · Hong Kong University of Science and Technology, Hongkong, China
Abstract
Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine speculative decoding with expert offloading. Its lightweight quantized draft model runs efficiently on device while preserving high-fidelity expert activation patterns. This provides accurate multi-step lookahead over future expert usage, allowing the runtime to organize expert movement and execution ahead of verification and improve both communication and execution efficiency. Under single-request inference, Specter achieves an average 1.50x (up to 1.88x) speedup in time per output token (TPOT) over the state-of-the-art system. Specter makes memory-constrained, single-GPU MoE inference more practical on commodity hardware. Our code is available at https://github.com/wwwwwf/specter_asplos.
Figures & tables
Figure 1. Conventional offloading versus Specter . Aligned quantized speculative decoding lays the foundation for accurate prefetching and efficient system design. Two diagrams and timelines contrast intra-token prefetching with multi-step speculative offloading. The conventional path alternates PCIe transfers and GPU inference, leaving idle intervals. The Specter path uses a quantized draft model to produce candidate tokens and an expert lookahead list, allowing expert prefetching to overlap with drafting and target verification.
Figure 2. Expert-ID selection fidelity between an INT4 Qwen1.5-MoE draft model and its FP16 target model. Three histograms show expert-ID match-rate distributions for all observations, per-layer means, and per-request means. The horizontal axes show match rate and the vertical axis shows frequency, both in percent. All three distributions concentrate around 90 to 92 percent, with a reported global mean of 90.8 percent.
| Model@(GPTQ&Q4) | Method@(DS&Q4) | Precision@(GGUF&DS) | |||||||
| Phi | DS | Qwen | GGUF | GPTQ | AWQ | Q2 | Q4 | Q8 | |
| Max | 99.79 | 99.45 | 93.48 | 99.99 | 99.45 | 99.45 | 99.98 | 99.99 | 99.99 |
| Min | 95.15 | 94.49 | 86.30 | 95.09 | 94.49 | 93.17 | 90.10 | 95.09 | 98.66 |
| Mean | 96.50 | 96.33 | 90.82 | 96.53 | 96.33 | 95.72 | 92.96 | 96.53 | 99.38 |
Table 1. Expert activation similarity (%) across MoE models (using native top- configuration and GPTQ method), diverse quant methods ( Frantar et al., 2022 ; Lin et al., 2024 ) , and GGUF-format precisions (based on DeepSeekV2-Lite).
Figure 3. Increasing the speculative depth yields an approximately linear benefit in dense models, but a markedly nonlinear trend in MoE models, as deeper speculation rapidly increases I/O demand under memory constraints. Schematic timelines compare speculative decoding in dense and MoE models. Light blocks represent drafting, dark blocks represent verification, and an orange outline marks additional overhead. Increasing draft depth adds little verification time in the dense example, whereas the MoE example has a substantially longer verification interval and lower speedup because more experts are activated.
Figure 4. Measured expert working-set growth with speculation depth. Three line charts plot distinct experts per layer against speculation depths of 1, 2, 4, 8, and 16 for DeepSeekV2-Lite, Qwen1.5-MoE, and Phi-3.5-MoE. Orange lines show the mean and shaded bands show the minimum-to-maximum range. Expert demand rises in all three models, with annotated growth factors of 5.29, 7.18, and 4.65, respectively.
Figure 5. The high-level architecture of Specter . The system integrates a Speculative Inference Controller, a Predictive I/O Orchestrator, and a Streamlined Execution Engine to transform I/O latency into productive computation through proactive data orchestration and adaptive control. A layered diagram connects the speculative decoding workflow to three system modules and the memory hierarchy. The controller initializes models, profiles speculative depth, and manages decoding state. The execution engine dispatches fused drafting and reordered expert verification. The I/O orchestrator analyzes routing patterns, plans prefetches, and manages residency. GPU memory holds quantized experts, an expert cache, and shared parameters, while target expert weights reside in CPU memory and transfer over PCIe.
Figure 6. Overview of parameter distribution and data layout orchestration in Specter . Diagram of parameter placement across GPU and CPU memory. The draft and target models share GPU-resident parameters and a KV cache. Quantized draft experts remain on the GPU, while full-precision target experts are prefetched or loaded from CPU memory into a GPU expert cache.
Figure 7. Speculative depth profiling results for DeepSeekV2-Lite, Qwen1.5-MoE and Phi-3.5-MoE. The reported TPOT is averaged over prompts and repeated runs. Three line plots show TPOT against speculation depth for DeepSeekV2-Lite, Qwen1.5-MoE, and Phi-3.5-MoE. Highlighted minima occur at depths 16, 8, and 16, respectively. Increasing speculation depth beyond these minima does not consistently reduce TPOT.
Figure 8. State lifecycle across one speculative decoding iteration, including verifier-guided token commitment and shared-prefix KV management. State-transition diagram for one speculative decoding iteration. Persistent tokens and shared prefix KV states feed the draft and target models. Drafting creates temporary tokens and KV states; causally masked target verification produces verification states. Sampling commits accepted state to the persistent prefix, while temporary draft state is released.
Figure 9. Execution timeline of adaptive ELB-guided prefetching during drafting. Timeline divided into Cold Start, Prefetch Overlap, and Final Cleanup. ELB counters collect routing metadata during draft computation. Snapshots at chunk boundaries trigger prefetch transfers on a separate CUDA I/O stream, overlapping adaptive draft chunks. The final chunk discards new routing metadata and clears the ELB before verification begins.
Figure 10. Performance comparison of different backends: (a) prefill stage latency, (b) decode stage latency. Lower values indicate better performance. Two plots compare Marlin, ExLV1, ExLV2, and Torch. The prefill plot shows latency across sequence lengths from 4 to 128, and the decode plot shows one latency bar per backend. Marlin has the lowest latency and Torch the highest in both plots.
Figure 11. Impact of quantization-aware fused expert kernels on throughput and latency in MoE models. Grouped bar charts compare FP16, INT4, and fused INT4 execution for Qwen1.5-MoE and Phi-3.5-MoE. For both models, INT4 lowers TPOT and raises throughput relative to FP16, and fused INT4 further lowers TPOT and raises throughput.
Figure 12. Comparison of conventional token-wise verification and Specter’s cache-aware expert-wise execution. Two diagrams compare token-wise and expert-wise verification. Naive computation traverses a token-by-expert grid and uses GEMV operations with experts in GPU or CPU memory. Reordering groups tokens by expert for GEMM computation and separates cache-hit and cache-miss experts around the expert cache.
| Phi-3.5-MoE ( Abdin et al., 2024 ) | Qwen1.5-MoE ( Qwen Team, 2024 ) | DeepSeekV2-Lite ( DeepSeek-AI, 2024 ) | |
| Total Param. | 41.9B | 14.3B | 15.7B |
| Activated Param. | 6.6B | 2.7B | 2.4B |
| Total Weight | 78GB | 29GB | 34GB |
| MoE Layer Number | 32 | 24 | 26 |
| First Dense Layer | 0 | 0 | 1 |
| Experts (Per Layer) | 16 | 60 | 64 |
Table 2. Configuration of evaluated models.
Figure 13. Dataset-induced expert-routing diversity on DeepSeekV2-Lite with . Left: mean distinct experts per layer of each dataset, with 10th–90th percentile ranges. Right: pairwise Jensen–Shannon distances between normalized routing distributions. Two panels compare five datasets. The bar chart shows distinct experts per layer with percentile intervals: HumanEval has the largest mean and WikiText-2 the smallest. The heat map shows zero self-distances and nonzero distances between different datasets, with WikiText-2 generally the most distinct.
Figure 14. End-to-end inference performance of Specter and other approaches on varying datasets, memory settings and models. The figures in the upper row represent experiments conducted under higher GPU memory conditions, while the figures in the bottom row represent experiments performed under lower GPU memory conditions. Under shared-weight accounting, the reference high/low memory budgets are 24/18 GB, 20/16 GB, and 40/32 GB for DeepSeekV2-Lite, Qwen1.5-MoE, and Phi-3.5-MoE, respectively. Six grouped bar charts show TPOT in milliseconds per token for seven systems across five datasets and their average. Columns represent DeepSeekV2-Lite, Qwen1.5-MoE, and Phi-3.5-MoE; rows represent high and low memory budgets. Specter has the lowest average TPOT in each panel, while Transformers has the highest.
Figure 15. TPOT and throughput scaling with batch size on DeepSeekV2-Lite and Qwen1.5-MoE. Four line charts compare Specter, SpecMoEOff, MO-SM, and Transformers as batch size increases from 1 to 32. Columns represent DeepSeekV2-Lite and Qwen1.5-MoE; the upper row shows throughput and the lower row shows TPOT on logarithmic vertical scales. Specter has the highest throughput and lowest TPOT at every displayed batch size.
| DeepSeekV2-Lite | Qwen1.5-MoE | |||||||||||
| Method | GK | WT | HE | GP | C4 | AVG | GK | WT | HE | GP | C4 | AVG |
| TPOT (ms/token) | ||||||||||||
| Transformers | 1504.7 | 1451.2 | 1489.6 | 1485.9 | 1536.7 | 1493.6 | 1171.0 | 1192.2 | 1129.0 | 1195.6 | 1249.2 | 1187.4 |
| MO-SM | 1267.3 | 1206.5 | 1254.9 | 1268.6 | 1317.0 | 1262.9 | 1059.5 | 1020.6 | 982.8 | 1038.5 | 1067.1 | 1033.7 |
| SpecMoEOff | 707.6 | 722.0 | 705.7 | 705.7 | 749.0 | 718.0 | 686.2 | 600.2 | 593.2 | 641.1 | 648.1 | 633.8 |
| Specter | 125.9 | 125.0 | 100.2 | 122.9 | 122.1 | 119.2 | 286.6 | 250.0 | 241.6 | 256.3 | 267.1 | 260.3 |
Table 3. Absolute performance across workloads at .
Figure 16. Expert working-set growth with batch size. Three line charts show mean distinct experts per layer and shaded minimum-to-maximum ranges for batch sizes 1, 2, 4, and 8. Expert counts increase for DeepSeekV2-Lite, Qwen1.5-MoE, and Phi-3.5-MoE, with Qwen1.5-MoE exhibiting the largest relative increase.
| Scenario | Draft | Target | Shared | Total |
| Independent | 6.13 | 7.27 | — | 13.40 |
| + Shared Experts | 4.58 | 5.72 | 1.55 | 11.85 |
| + Other Params | 2.66 | 3.80 | 3.47 | 9.93 |
| + KV Cache | 0.41 | 1.55 | 5.72 | 7.68 43% |
Table 4. Comparative GPU memory footprints (GB) of full-precision and 4-bit quantized Qwen1.5-MoE with a 12K-token context.
Figure 17. Cache hit rates of different policies on DeepSeekV2-Lite under three GPU-memory budgets. Grouped bars compare cache hit percentages for LRU, LRU-S, SP-E, SP-L, and Specter at memory budgets of 16, 24, and 32 GB. Specter remains close to 100 percent and has the highest hit rate in each group. LRU and LRU-S improve as the memory budget increases, with LRU-S approaching the prefetch policies at 32 GB.
| Metric | DeepSeekV2-Lite | Qwen1.5-MoE | Phi-3.5-MoE |
| Draft | 674.0 | 223.3 | 952.5 |
| Verification | 136.1 | 88.2 | 82.2 |
| Total H2D Load | 243.4 | 89.7 | 2089.2 |
| Exposed H2D Stall | 108.3 | 35.4 | 1621.3 |
| Hidden H2D | 135.1 | 54.3 | 467.9 |
| Hidden Ratio | 55.5% | 60.5% | 22.4% |
Table 5. Per-iteration latency breakdown across models and cumulative component ablation on DeepSeekV2-Lite.
| (a) Controlled expert-prediction errors | ||||
| Error Rate | 0% | 25% | 50% | 75% |
| Prefetch Acc. | 1.000 | 0.785 | 0.575 | 0.410 |
| TPOT | 63.0 | 66.5 | 82.6 | 84.2 |
| Ratio | 1.00 | 1.06 | 1.31 | 1.34 |
Table 6. Robustness to expert-prediction errors and natural variations. TPOT is reported in ms/token.
More in this topic
Mixture-of-Experts Inference
Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.
See all 175 papers