The Mixture-of-Experts (MoE) architecture has emerged as a promising approach to mitigate the rising computational costs of large language models (LLMs) by selectively activating parameters. However, its high memory requirements and sub-optimal parameter efficiency pose significant challenges for efficient deployment. Although CPU-offloaded MoE inference systems have been proposed in the literature, they offer limited efficiency, particularly for large batch sizes. In this work, we propose SpecMoE, a memory-efficient MoE inference system based on our self-assisted speculative decoding algorithm. SpecMoE demonstrates the effectiveness of applying speculative decoding to MoE inference without requiring additional model training or fine-tuning. Our system improves inference throughput by up to 4.30×, while significantly reducing bandwidth requirements of both memory and interconnect on memory-constrained systems.
Figures & tables
Fig. 1: (a) Conventional dense Transformer block and (b) sparsely activated MoE block. The number of activated experts chosen per token ( K ) is set during training, and the model is referred to as a top- K gating model (e.g., top-2 gating in NLLB-MoE [ 49 ] ).
Fig. 2: Overview of CPU offloading-based MoE inference system. Dense layers are always activated and stored on the GPU, while sparse layers are offloaded to the CPU. In this example, three input tokens activate Exp #0 and Exp #2 , fetched from CPU memory.
Fig. 3: (a) Latency breakdown of a single NLLB-MoE’s decoding step with different batch sizes using MoE-OnDemand on a CPU offloading-based system with a single NVIDIA H100 GPU. (b) The size of activated expert parameters vs. non-expert parameters when varying the input batch size in NLLB-MoE. The non-MoE parameters (black) are kept in the GPU while the activated experts (yellow) are migrated over the PCIe channel on-demand.
Fig. 4: Workflow of a speculative decoding step. The draft model generates speculated tokens auto-regressively (e.g., 5 tokens in the figure) and feeds them into the target model. The earliest rejected token (“own”) and all following tokens are discarded (“own”, “skills”), while the corrected token (“best”) is appended to the list of accepted tokens. In this example, 3 tokens (“experts”, “share”, “their”) are accepted and 4 tokens (“experts”, “share”, “their”, “best”) are generated in a single speculative decoding step.
Fig. 5: High-level overview of SpecMoE system for a top-1 gating MoE model. All non-expert model parameters and a few selected experts are on the GPU (green), while the remaining experts are offloaded to the CPU (blue). As a self-assisted speculative decoding model, SpecMoE utilizes parts of the target model (parameters in green) as the draft model. With this draft model, whose parameters are entirely on the GPU, speculated draft tokens are generated. The target model then verifies the speculated tokens by accessing the required experts, which are fetched from CPU if they are not already on GPU memory (experts in blue). To maintain high draft model quality, the experts used to form the draft model are periodically replaced with our proposed expert replacement policy.
Fig. 6: Expert hotness of NLLB-MoE in the English-French translation task on the WMT-14 dataset [ 4 ] . NLLB-MoE contains 6 MoE blocks, each containing 128 experts. All the MoE blocks within this MoE model (y-axis, showing different MoE block ID) are depicted, showing how different experts within an MoE block (x-axis, showing expert ID within that MoE block) exhibit different hotness.
Fig. 7: The workflow of SpecMoE’s self-assisted speculative decoding algorithm. Example assumes a top-1 gating MoE model. 1 For each MoE block, only N =1 expert is pinned in the GPU as a draft expert and used to generate draft tokens (i.e., draft experts in green). 2 The draft tokens are then verified by the target MoE model, migrating any required experts not already on the GPU (experts in blue) from the CPU, on demand. 3 During verification, the temporally hot experts for each MoE block are identified (e.g., Exp #1 of MoE block #0 is assumed as the most activated expert in this example). 4 Consequently, the identified N temporally hottest experts are pinned in the GPU as the next draft expert, evicting any draft experts that were not identified as the hottest N experts. In this example, the draft experts in MoE block #0 changes from Exp #3 to Exp #1 , while those in MoE block #1 remains unchanged.
Fig. 8: Affinity-based expert selection mechanism in the SpecMoE speculation phase. (a) Expert selection when the number of draft experts ( N ) equals to the number of routed experts per token ( K=1 ). (b) When N>K , each token is routed to the most suitable K experts among N draft experts by referring to the affinity table.
Fig. 9: Comparison of data transfer size for expert migration between MoE-OnDemand and SpecMoE. SpecMoE reduces transfers by keeping frequently activated draft experts in GPU and minimizing redundant migrations. With N=K=1 (top-1 gating), SpecMoE retains Exp #1 in GPU, while MoE-OnDemand offloads all experts to CPU. In (a), MoE-OnDemand incurs CPU-to-GPU transfers every step. In (b), SpecMoE avoids migration in the speculation phase by generating all speculated tokens with Exp #1 in GPU. Migration occurs only in the verification phase when experts are accessed based on the gating function. Here, only Exp #0 and Exp #3 migrate once over 5 tokens, significantly reducing data transfer. (c) shows execution timelines, highlighting SpecMoE’s efficiency.
Model
Experts
Layers (MoE / total)
Routing
Skewness
Capacity (GB)
NLLB-MoE
128
6 / 24
Top-2
0.84
54
Mixtral-8x7B
8
32 / 32
Top-2
0.32
94
Llama-4-Scout
16
48 / 48
Top-1
0.59
218
TABLE I: MoE model configurations. For brevity, we only include information about the decoder blocks for NLLB-MoE. Skewness is the portion of routed tokens covered by the top-25% hottest experts. A higher skewness means the model has a more skewed expert activation distribution.
Metric
Random
HotGlobal
HotTemporal
Tokens generated per step
6.812
7.220
7.265
Performance speedup
1.000
1.084
1.143
TABLE II: Average number of generated tokens ( τ(γ) ) for different expert selection policies (batch size: 256) and their effect on end-to-end inference performance (speedup normalized to Random ). γ =10 in our evaluations.
Fig. 11: End-to-end inference throughput (tokens per second).
Fig. 12: CPU-to-GPU data transfer size in end-to-end inference.
Fig. 13: Average number of tokens generated per speculative decoding step as the number of draft tokens varies (batch size: 128).
Metric
Number of experts in draft model ( N )
2
4
8
16
Tokens generated per step
6.929
7.265
7.620
7.819
Throughput (tokens/sec)
116.401
122.974
121.309
117.157
TABLE III: Average number of tokens generated per step ( τ(γ) ) and achieved throughput for γ=10 when the number of draft experts ( N ) is changed in NLLB-MoE (batch size: 256).
Fig. 14: End-to-end inference throughput of Mixtral-8x7B.
Fig. 15: End-to-end inference throughput of LLama-4-Scout.
Fig. 16: Normalized end-to-end inference throughput of NLLB-MoE when expert parameters are offloaded to the SSD.
Deploying large language models (LLMs) for inference on edge devices is challenging due to severe memory and bandwidth constraints. While speculative decoding and Mixture-of-Experts (MoE) have been proposed to improve inference efficiency, naively combining them often incurs excessive verification overhead and poor expert reuse, limiting their effectiveness in memory-bound edge settings. In this work, we propose S2-MoE, an efficient self-speculative decoding framework for MoE inference on edge devices. S2-MoE reduces redundant verification through routing-aware adaptive speculative expansion, improves verification efficiency with reuse-aware expert gating, and aligns draft and target execution via shared context. Implemented in llama.cpp, S2-MoE achieves up to 5.3× speedup (about 2.0× on average) over standard autoregressive decoding across diverse MoE models and datasets on edge devices. Code is available at https://github.com/angerybob/S2-MoE.
Haochen Huang, Shengxuan Qiu, Meng Li
Institute for Artificial Intelligence, Peking University, Beijing, China · School of Integrated Circuits, Peking University, Beijing, China · School of Electronics Engineering and Computer Science, Peking University, Beijing, China
Speculative decoding accelerates Large Language Model (LLM) inference by verifying multiple drafted tokens in parallel. However, for Mixture-of-Experts (MoE) models, this parallelism introduces a severe bottleneck: large draft trees activate many unique experts, significantly increasing memory pressure and diminishing speedups from speculative decoding relative to autoregressive decoding. Prior methods reduce speculation depth when MoE verification becomes expensive. We propose MoE-Spec, a training-free verification-time expert budgeting method that decouples speculation depth from memory cost by enforcing a fixed expert capacity limit at each layer, loading only the experts that contribute most to verification and dropping the long tail of rarely used experts that drive bandwidth overhead. Experiments across multiple model scales and datasets show that this method yields 10--30% higher throughput than state-of-the-art speculative decoding baselines (EAGLE-3) at comparable quality, with flexibility to trade accuracy for further latency reductions through tighter budgets.
Bradley McDanel, Steven Li, Sruthikesh Surineni +1
Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine speculative decoding with expert offloading. Its lightweight quantized draft model runs efficiently on device while preserving high-fidelity expert activation patterns. This provides accurate multi-step lookahead over future expert usage, allowing the runtime to organize expert movement and execution ahead of verification and improve both communication and execution efficiency. Under single-request inference, Specter achieves an average 1.50x (up to 1.88x) speedup in time per output token (TPOT) over the state-of-the-art system. Specter makes memory-constrained, single-GPU MoE inference more practical on commodity hardware. Our code is available at https://github.com/wwwwwf/specter_asplos.
Wenfeng Wang, Xiaofeng Hou, Xinfeng Xia +5
Shanghai Jiao Tong University, Shanghai, China · Shanghai Jiao Tong University, China · Hong Kong University of Science and Technology, Hongkong, China