cs.AIApr 11, 2026
SaveSpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding
Organizations: KAIST · UNIST
Abstract
The Mixture-of-Experts (MoE) architecture has emerged as a promising approach to mitigate the rising computational costs of large language models (LLMs) by selectively activating parameters. However, its high memory requirements and sub-optimal parameter efficiency pose significant challenges for efficient deployment. Although CPU-offloaded MoE inference systems have been proposed in the literature, they offer limited efficiency, particularly for large batch sizes. In this work, we propose SpecMoE, a memory-efficient MoE inference system based on our self-assisted speculative decoding algorithm. SpecMoE demonstrates the effectiveness of applying speculative decoding to MoE inference without requiring additional model training or fine-tuning. Our system improves inference throughput by up to , while significantly reducing bandwidth requirements of both memory and interconnect on memory-constrained systems.
Figures & tables
Fig. 1: (a) Conventional dense Transformer block and (b) sparsely activated MoE block. The number of activated experts chosen per token ( ) is set during training, and the model is referred to as a top- gating model (e.g., top-2 gating in NLLB-MoE [ 49 ] ).
Fig. 2: Overview of CPU offloading-based MoE inference system. Dense layers are always activated and stored on the GPU, while sparse layers are offloaded to the CPU. In this example, three input tokens activate Exp #0 and Exp #2 , fetched from CPU memory.
Fig. 3: (a) Latency breakdown of a single NLLB-MoE’s decoding step with different batch sizes using MoE-OnDemand on a CPU offloading-based system with a single NVIDIA H100 GPU. (b) The size of activated expert parameters vs. non-expert parameters when varying the input batch size in NLLB-MoE. The non-MoE parameters (black) are kept in the GPU while the activated experts (yellow) are migrated over the PCIe channel on-demand.
Fig. 4: Workflow of a speculative decoding step. The draft model generates speculated tokens auto-regressively (e.g., 5 tokens in the figure) and feeds them into the target model. The earliest rejected token (“own”) and all following tokens are discarded (“own”, “skills”), while the corrected token (“best”) is appended to the list of accepted tokens. In this example, 3 tokens (“experts”, “share”, “their”) are accepted and 4 tokens (“experts”, “share”, “their”, “best”) are generated in a single speculative decoding step.
Fig. 5: High-level overview of SpecMoE system for a top-1 gating MoE model. All non-expert model parameters and a few selected experts are on the GPU (green), while the remaining experts are offloaded to the CPU (blue). As a self-assisted speculative decoding model, SpecMoE utilizes parts of the target model (parameters in green) as the draft model. With this draft model, whose parameters are entirely on the GPU, speculated draft tokens are generated. The target model then verifies the speculated tokens by accessing the required experts, which are fetched from CPU if they are not already on GPU memory (experts in blue). To maintain high draft model quality, the experts used to form the draft model are periodically replaced with our proposed expert replacement policy.
Fig. 6: Expert hotness of NLLB-MoE in the English-French translation task on the WMT-14 dataset [ 4 ] . NLLB-MoE contains 6 MoE blocks, each containing 128 experts. All the MoE blocks within this MoE model (y-axis, showing different MoE block ID) are depicted, showing how different experts within an MoE block (x-axis, showing expert ID within that MoE block) exhibit different hotness.
Fig. 7: The workflow of SpecMoE’s self-assisted speculative decoding algorithm. Example assumes a top-1 gating MoE model. 1 For each MoE block, only =1 expert is pinned in the GPU as a draft expert and used to generate draft tokens (i.e., draft experts in green). 2 The draft tokens are then verified by the target MoE model, migrating any required experts not already on the GPU (experts in blue) from the CPU, on demand. 3 During verification, the temporally hot experts for each MoE block are identified (e.g., Exp #1 of MoE block #0 is assumed as the most activated expert in this example). 4 Consequently, the identified temporally hottest experts are pinned in the GPU as the next draft expert, evicting any draft experts that were not identified as the hottest experts. In this example, the draft experts in MoE block #0 changes from Exp #3 to Exp #1 , while those in MoE block #1 remains unchanged.
Fig. 8: Affinity-based expert selection mechanism in the SpecMoE speculation phase. (a) Expert selection when the number of draft experts ( ) equals to the number of routed experts per token ( ). (b) When , each token is routed to the most suitable experts among draft experts by referring to the affinity table.
Fig. 9: Comparison of data transfer size for expert migration between MoE-OnDemand and SpecMoE. SpecMoE reduces transfers by keeping frequently activated draft experts in GPU and minimizing redundant migrations. With (top-1 gating), SpecMoE retains Exp #1 in GPU, while MoE-OnDemand offloads all experts to CPU. In (a), MoE-OnDemand incurs CPU-to-GPU transfers every step. In (b), SpecMoE avoids migration in the speculation phase by generating all speculated tokens with Exp #1 in GPU. Migration occurs only in the verification phase when experts are accessed based on the gating function. Here, only Exp #0 and Exp #3 migrate once over 5 tokens, significantly reducing data transfer. (c) shows execution timelines, highlighting SpecMoE’s efficiency.
| Model | Experts | Layers (MoE / total) | Routing | Skewness | Capacity (GB) |
|---|---|---|---|---|---|
| NLLB-MoE | 128 | 6 / 24 | Top-2 | 0.84 | 54 |
| Mixtral-8x7B | 8 | 32 / 32 | Top-2 | 0.32 | 94 |
| Llama-4-Scout | 16 | 48 / 48 | Top-1 | 0.59 | 218 |
TABLE I: MoE model configurations. For brevity, we only include information about the decoder blocks for NLLB-MoE. Skewness is the portion of routed tokens covered by the top-25% hottest experts. A higher skewness means the model has a more skewed expert activation distribution.
| Metric | Random | HotGlobal | HotTemporal |
|---|---|---|---|
| Tokens generated per step | 6.812 | 7.220 | 7.265 |
| Performance speedup | 1.000 | 1.084 | 1.143 |
TABLE II: Average number of generated tokens ( ) for different expert selection policies (batch size: 256) and their effect on end-to-end inference performance (speedup normalized to Random ). =10 in our evaluations.
Fig. 10: Normalized end-to-end inference latency (NLLB-MoE).
Fig. 11: End-to-end inference throughput (tokens per second).
Fig. 12: CPU-to-GPU data transfer size in end-to-end inference.
Fig. 13: Average number of tokens generated per speculative decoding step as the number of draft tokens varies (batch size: 128).
| Metric | Number of experts in draft model ( ) | |||
|---|---|---|---|---|
| 2 | 4 | 8 | 16 | |
| Tokens generated per step | 6.929 | 7.265 | 7.620 | 7.819 |
| Throughput (tokens/sec) | 116.401 | 122.974 | 121.309 | 117.157 |
TABLE III: Average number of tokens generated per step ( ) and achieved throughput for when the number of draft experts ( ) is changed in NLLB-MoE (batch size: 256).
Fig. 14: End-to-end inference throughput of Mixtral-8x7B.
Fig. 15: End-to-end inference throughput of LLama-4-Scout.
Fig. 16: Normalized end-to-end inference throughput of NLLB-MoE when expert parameters are offloaded to the SSD.
More in this topic
Mixture-of-Experts Inference
Mixture-of-experts inference studies how to run sparsely activated expert models efficiently while preserving their quality.
See all 175 papers