MoE-Spec: Expert Budgeting for Efficient Speculative Decoding

Bradley McDanel; Steven Li; Sruthikesh Surineni; Harshit Khaitan

cs.LGFeb 17, 2026

MoE-Spec: Expert Budgeting for Efficient Speculative Decoding

Authors: Bradley McDanel, Steven Li, Sruthikesh Surineni, Harshit Khaitan

Organizations: Franklin and Marshall College · Meta Reality Labs

Abstract

Speculative decoding accelerates Large Language Model (LLM) inference by verifying multiple drafted tokens in parallel. However, for Mixture-of-Experts (MoE) models, this parallelism introduces a severe bottleneck: large draft trees activate many unique experts, significantly increasing memory pressure and diminishing speedups from speculative decoding relative to autoregressive decoding. Prior methods reduce speculation depth when MoE verification becomes expensive. We propose MoE-Spec, a training-free verification-time expert budgeting method that decouples speculation depth from memory cost by enforcing a fixed expert capacity limit at each layer, loading only the experts that contribute most to verification and dropping the long tail of rarely used experts that drive bandwidth overhead. Experiments across multiple model scales and datasets show that this method yields 10--30% higher throughput than state-of-the-art speculative decoding baselines (EAGLE-3) at comparable quality, with flexibility to trade accuracy for further latency reductions through tighter budgets.

Figures & tables

Figure 1: Overview of MoE-Spec. Standard MoE speculative decoding loads all N experts (N = 8) activated across the draft tree (top path). MoE-Spec ranks experts by aggregate routing probability and enforces a budget

B

(here

B=4

), loading only top-scoring experts (bottom path). Tokens are color-coded by the number of missing experts from their natural routing. In this example, both paths accept the same three tokens despite MoE-Spec loading half as many experts.

Panel of Figure 2 — Figure 2: Motivation for expert budgeting. (a) During verification, each token routes to $k$ experts; the target model must load all unique experts $\mathcal{E}$ across the draft tree. (b) As tree size $M$ grows, unique experts per layer approaches $N$ , negating sparse activation benefits. (c) Routing probabilities are heavy-tailed: the top 32 of 64 experts capture 93% of routing weight for a tree size of 63.

GSM8K

MATH500

HumanEval

MBPP

CNN/DM

Model

Method

Spd

Acc

\tau

Spd

Acc

\tau

Spd

P@1

\tau

Spd

P@1

\tau

Spd

R-L

\tau

Temperature = 0

Mixtral-8x7B

1.0

\times

–

1.0

\times

–

1.0

\times

–

1.0

\times

–

1.0

\times

–

EAGLE

1.5

\times

2.8

1.4

\times

2.5

1.6

\times

3.2

1.7

\times

3.2

1.3

\times

2.3

MoE-Spec

1.8

\times

2.9

2.0

\times

2.5

1.9

\times

3.1

2.3

\times

3.1

1.5

\times

2.2

OLMoE-1B-7B

1.0

\times

–

1.0

\times

–

1.0

\times

–

1.0

\times

–

1.0

\times

–

Table 1: Speedup over autoregressive decoding, mean acceptance length (

\tau

), and task metrics at T=0 and T=1. MoE-Spec uses substitution policy with the expert budget maximizing throughput while maintaining quality comparable to EAGLE-3 (within one standard deviation, measured from T=1 runs with 5 seeds). T=1 results report mean over 5 random seeds.

Figure 3: Quality-speedup tradeoff at

T=1

, averaged across five benchmarks. For each benchmark, quality and speedup are normalized relative to AR (100%), then averaged. Error bars combine cross-benchmark variance with seed-to-seed variance from 5 runs. Individual per-benchmark curves appear in Appendix E .

Figure 4: Expert activation and speedup on OLMoE-1B-7B. ( 4 ) Unique experts activated during verification as tree size increases. EAGLE-3 loads over 50 of 64 experts at large trees; MoE-Spec with

B=32

saturates at the budget. ( 4 ) Speedup relative to AR. EAGLE-3 peaks at tree size 31 then declines as expert loading dominates; MoE-Spec continues improving at larger trees. ( 4 ) Speedup vs. active experts. At tree size 255, EAGLE-3 loads 54 experts for 1.85

\times

speedup; MoE-Spec loads 32 experts for 2.1

\times

speedup.

Figure 5: Design ablations on OLMoE-1B-7B at

T=1

, normalized to autoregressive (AR) baseline. (a) Selection methods: Static ranking fails at low budgets while Router-based ranking tracks the Oracle upper bound. (b) Coverage policies: Truncation and Substitution perform comparably across budgets.

Figure 6: Reconstruction error by selection method on OLMoE-1B-7B, averaged across all MoE layers and 5 benchmarks. Oracle matches Router error using 25% fewer experts.

Figure 7: Routing probability coverage by expert budget across datasets on OLMoE-1B-7B with tree size 63. Datasets are ordered by coverage (best to worst): MBPP reaches 95% coverage with 8 experts, while CNN/DM requires 24 experts to reach 90%.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Draft tree size (main)	63
Draft tree size (sweep)	3–511
Decoding temperature	0.0, 1.0
Samples per benchmark	80
Max generation length	1024
Shortlist method	Router

Appendix

Table 2: General experimental settings.

Model	Experts	Budget ( $B$ )	Draft / GPUs
OLMoE-1B-7B	64 ( $k$ =8)	8–56	EAGLE-3 / 1
Qwen3-30B-A3B	128 ( $k$ =8)	16–64	EAGLE-3 / 1
Mixtral-8x7B	8 ( $k$ =2)	2–7	EAGLE / 2

Appendix

Table 3: Model-specific configurations.

Input: Hidden states

\{\mathbf{h}_{1},\ldots,\mathbf{h}_{M}\}

, budget

B

, policy

Output: MoE output for each token

// Compute routing probabilities

for

t=1,\ldots,M

g_{i}(\mathbf{h}_{t})\leftarrow\mathrm{softmax}(\mathbf{W}_{\mathrm{gate}}\mathbf{h}_{t})

for all

i

// Aggregate importance scores

for

i=1,\ldots,N

Appendix

Algorithm 1 MoE-Spec Expert Budgeting

Benchmark	Mode	Quality	Throughput	Accept
GSM8K	Sequential	0.663	17.1 tok/s	2.98
GSM8K	Batched	0.688	25.2 tok/s	3.00
HumanEval	Sequential	0.304	23.6 tok/s	4.16
HumanEval	Batched	0.304	35.0 tok/s	4.20
CNN/DM	Sequential	0.180	12.3 tok/s	1.93
CNN/DM	Batched	0.173	18.4 tok/s	1.96

Appendix

Table 4: Comparison of sequential vs. batched MoE computation on OLMoE-1B-7B (EAGLE-3, no expert budget). Quality differences are within noise; throughput improves 1.4–1.5

\times

Figure 8: Expert co-activation heatmaps for all 16 MoE layers in OLMoE-1B-7B. Each cell shows normalized pairwise co-activation frequency. Experts are reordered by hierarchical clustering. Bright diagonal blocks indicate expert groups that frequently activate together, with concentration ratios of 10–33 $\times$ above uniform-random expectation.

Figure 9: Quality vs. speedup at

T=0

across three models (rows) and five benchmarks (columns). Diamonds mark EAGLE; connected points show MoE-Spec at varying expert budgets.

Figure 10: Quality vs. speedup at

T=1

across three models (rows) and five benchmarks (columns). Error bars show standard deviation over 5 seeds.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 311 papers

Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
Jul 14, 2026Jincheng Xie, Runheng Liu, Heyan Huang +4Mixture-of-Experts Inference Speculative Decoding
Department of Mathematical Sciences, Tsinghua University, Beijing, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JDT AI Infra +2
SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding
Apr 11, 2026Jehyeon Bang, Eunyeong Cho, Ranggi Hwang +2Mixture-of-Experts Inference LLM Inference Acceleration
KAIST · UNIST
SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference
Oct 11, 2026Hongliang Cao, Xiuyu Li, Siqi Wang +3LLM Inference Acceleration Speculative Decoding
Beihang University · Monash University · The Hong Kong University of Science and Technology

Jul 14, 2026J/K move · Enter open · S save

Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

Jincheng Xie, Runheng Liu, Heyan Huang, Yawen Ling+3

Department of Mathematical Sciences, Tsinghua University, Beijing, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JDT AI Infra+2

Mixture-of-Experts Inference Speculative Decoding Mixture-of-Experts Models

Open paper AskPDF arXiv

Sparse Mixture-of-Experts (MoE) models have become an important approach for scaling Large Language Models (LLMs), but their inference efficiency depends strongly on expert activation patterns. Speculative decoding (SD) accelerates autoregressive generation by verifying multiple draft tokens in parallel, yet existing draft selection strategies primarily optimize acceptance likelihood. In large-scale MoE models, however, selecting draft tokens also determines the union of experts activated during verification. We observe that confidence-driven SD can introduce expert scattering: high-probability draft tokens may route to disjoint experts, increasing expert-weight memory traffic and reducing the speedup from speculation. Motivated by this observation, we revisit draft-tree selection under the non-uniform memory-cost structure of MoE inference. We propose EcoSpec, a cost-aware speculative decoding framework that incorporates predicted marginal expert activation cost into draft selection. With a lightweight expert predictor and a dynamic expert buffer, EcoSpec favors draft paths that preserve high acceptance likelihood while reusing experts already covered by the current verification set, without modifying the target-model verification rule. We evaluate EcoSpec on three large-scale MoE models, including DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B, across reasoning, coding, question-answering, and dialogue benchmarks. EcoSpec consistently reduces active expert footprints and improves end-to-end decoding speed, achieving up to

1.62\times

speedup. These results show that accounting for expert activation cost is important for efficient speculative decoding in large-scale MoE models.

Abstract

Speculative Decoding

Explore similar work

Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding

SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference