Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models

Ekagra Ranjan; Komal K. Teru; Bharat Venkitesh; Acyr Locatelli

cs.LGOct 28, 2026

Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models

Authors: Ekagra Ranjan, Komal K. Teru, Bharat Venkitesh, Acyr Locatelli

Organizations: Cohere

Abstract

Speculative decoding (SD) accelerates autoregressive inference by leveraging the memory bound nature of decoding, verifying multiple draft tokens in a single forward pass. For mixture-of-experts (MoE) models, prior work predicts a concave speedup-batch size relationship, with gains peaking at intermediate batch sizes due to expert saturation. We evaluate this across 10 sparse MoE models (30B--1T parameters, 8--512 experts) and find recent fine-grained models break this trend, showing higher speedup at low batch size. We find verification cost is governed not only by activated-expert bytes, but also by temporal overlap across draft tokens and GPU execution dynamics. At low batch size, MoE layers in fine-grained models account for a much smaller share of runtime than of weights; on H100, kernel profiling attributes this mainly to higher bandwidth utilization of MoE kernels relative to attention, with L2 absorbing repeated reads of co-routed experts. Across draft tokens, routing overlap further reduces the unique experts activated during verification relative to independence assumptions. We model this effect using a maximum-entropy Iterative Proportional Fitting (IPF) framework that predicts unique expert counts from routing-overlap statistics with less than 4% error, and substantially narrows the gap to real-model curves and captures the qualitative low-batch trend from synthetic routing.

Figures & tables

Figure 1: TPOT speedup versus batch size for four representative MoE models. Mixtral-8x7B exhibits the predicted concave curve, whereas the other three are non-concave and peak at or near batch size 1.

Model	$C_{\text{expert}}$ (MB/GPU)	$C_{\text{dense}}$ (MB/GPU)	MoE wt%	$\text{unique}_{\text{sd}}$	TE. pred.	TE observed	Gap
Mixtral-8x7B (bf16, TP2)	168.0	40.0	89.4	4.99	0.428	0.505	+0.08
Mixtral-8x7B (fp8, TP1)	168.0	40.0	89.4	4.99	0.428	0.488	+0.06
Qwen3-30B (bf16, TP1)	9.0	36.0	66.7	19.49	0.511	0.630	+0.12
Qwen3-30B (bf16, TP2)	4.5	18.0	66.7	19.49	0.511	0.697	+0.19
Qwen3-30B (fp8, TP1)	4.5	18.0	66.7	19.49	0.511	0.818	+0.31
Qwen3-30B (fp8, TP2)	2.25	9.0	66.7	19.49	0.511	0.860	+0.35

Table 1: TE predicted by the activated-expert-bytes model vs. measured TE at batch size 1 using per-GPU weights.

Model	$k/N$	Uniform	Indep	Empirical	Temporal	Temp/Total
Mixtral-8x7B	0.250	0.250	0.253	0.355	0.102	28.7%
Qwen3-30B	0.063	0.063	0.119	0.431	0.312	72.4%
Qwen3-235B	0.063	0.063	0.098	0.447	0.349	78.2%
Kimi-K2.5	0.021	0.021	0.040	0.296	0.256	86.4%

Table 2: Step-1 expert overlap decomposition. Temporal = empirical

-

independence; Temporal/Total = fraction of overlap attributable to temporal overlap.

Model	$k$	$N$	Unif	Indep	Emp	Emp/ $k$	Uniform IPF	Err	Skewed IPF	Err
			Unique experts (4 tokens)				IPF prediction
Mixtral-8x7B	2	8	5.5	5.5	5.0	2.5 $\times$	4.99	+0.1%	4.99	+0.1%
Qwen2-57B	8	64	26.5	26.0	24.6	3.1 $\times$	24.72	+0.5%	24.63	+0.1%
Qwen3-30B	8	128	29.1	26.9	19.5	2.4 $\times$	20.08	+3.1%	19.72	+1.2%
Kimi-K2.5	8	384	31.0	30.2	23.3	2.9 $\times$	23.88	+2.7%	23.66	+1.7%

Table 3: Unique experts (

K{=}3

) and IPF prediction accuracy. Emp = empirical; Unif = uniform.

Metric	Qwen3-30B	Mixtral-8x7B
Expert size (FP8)	4.5 MB	168 MB
MoE share of weight	66.7%	89.4%
MoE share of GPU time	33%	75%
Attn CTAs / 132 SMs	16 (12%)	96 (73%)
Attn HBM BW (% peak)	22–25%	62%
MoE CTAs / 132 SMs	96–128	132

Table 5: GPU profiling comparison at batch size 1 on H100.

Panel of Figure 2 — Figure 2: Target efficiency under different routing strategies.

Panel of Figure 3 — Figure 3: SD speedup vs. batch size for $K{=}3/5/7$ on MT-Bench (H100). Each model keeps its curve shape across $K$ ; larger $K$ mostly shifts the curve down, except for Qwen3-235B, whose stronger drafter keeps $K{=}5$ competitive.

Figure 4: Target efficiency vs. batch size for Qwen3-30B (FP8, TP2, EAGLE-3, $K{=}3$ ) on H100: MT-Bench vs. SPEED-Bench 32k.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$k/N$	$E_{\text{dim}}$	$\text{SE}_{\text{dim}}$	Hidden	Layers	TP	SD Method
Mixtral-8x7B	2/8	14336	n/a	4096	32	2	EAGLE
Qwen2-57B	8/64	2560	20480	3584	28	2	Draft model (0.5B)
GLM-4.7-Flash	4/64	1536	10240	2048	40	1	MTP
Qwen3-30B	8/128	768	n/a	2048	48	1	EAGLE-3
Qwen3-235B	8/128	1536	n/a	4096	94	4	EAGLE-3
GPT-OSS-120B	4/128	2880	n/a	2880	56	4	EAGLE-3

Appendix

Table 6: MoE models evaluated. Here

N

denotes the total number of experts,

k

the number of experts per token,

E_{\text{dim}}

the expert intermediate size,

\text{SE}_{\text{dim}}

the shared-expert intermediate size ( n/a if none), and TP the tensor parallelism.

Batch size	Num prompts	Batch size	Num prompts
1	80	16	320
2	80	32	640
4	80	64	1280
8	160	128	2560

Appendix

Table 7: Benchmark schedule for the MT-Bench end-to-end SD sweep. The 80 base prompts are oversampled with replacement when the configured number of requests exceeds 80; all runs use fixed 256-token outputs and disable prefix caching.

Model	Target weights	TP	GPU
Mixtral-8x7B	mistralai/Mixtral-8x7B-Instruct-v0.1	2	H100
Qwen2-57B	Qwen/Qwen2-57B-A14B-Instruct	2	H100
GLM-4.7-Flash	zai-org/GLM-4.7-Flash	1	H100
Qwen3-30B	Qwen/Qwen3-30B-A3B-FP8	1	H100
Qwen3-235B	Qwen/Qwen3-235B-A22B-Instruct-2507-FP8	4	H100
GPT-OSS-120B	openai/gpt-oss-120b	4	H100

Appendix

Table 8: Target-weight identifiers and runtime configuration used in the reported experiments.

Model	Draft weights	Method
Mixtral-8x7B	yuhuili/EAGLE-mixtral-instruct-8x7B	EAGLE
Qwen2-57B	Qwen/Qwen2-0.5B-Instruct	Draft model
GLM-4.7-Flash	(built-in MTP heads)	MTP
Qwen3-30B	RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3	EAGLE-3
Qwen3-235B	RedHatAI/Qwen3-235B-A22B-Instruct-2507-speculator.eagle3	EAGLE-3
GPT-OSS-120B	nvidia/gpt-oss-120b-Eagle3-long-context	EAGLE-3

Appendix

Table 9: Draft-weight identifiers and SD methods used in the reported experiments.

Model	SD method	Target eff.	Model	SD method	Target eff.
Qwen2-57B (bf16, TP2)	Separate draft	0.270	Qwen3-30B (fp8, TP2)	EAGLE-3	0.860
Mixtral-8x7B (bf16, TP2)	Separate draft	0.192	Qwen3-235B (bf16, TP8)	EAGLE-3	0.786
Mixtral-8x7B (bf16, TP2)	EAGLE	0.505	Qwen3-235B (fp8, TP4)	EAGLE-3	0.889
Mixtral-8x7B (fp8, TP1)	EAGLE	0.488	Qwen3-Next-80B (bf16, TP4)	MTP	0.615
GLM-4.7-Flash (bf16, TP1)	MTP	0.554	Qwen3-Next-80B (fp8, TP2)	MTP	0.707
GLM-4.7-Flash (bf16, TP2)	MTP	0.568	Qwen3.5-35B (bf16, TP2)	MTP	0.546

Appendix

Table 10: Target efficiency at batch size 1 for

K{=}3

. Higher values indicate non-concave speedup curves as observed in Figure 6 .

Figure 5: Qwen3-30B TPOT speedup versus batch size across TP/precision settings and hardware. Panels show BF16 TP1 H100, BF16 TP2 H100, FP8 TP2 H100, and FP8 TP2 B200 (126 MB L2).

Figure 6: TPOT speedup versus batch size for all 10 MoE models ( $K{=}3$ ).

Panel of Figure 7 — Figure 7: TPOT speedup versus batch size across four datasets for two representative non-concave models. Each curve corresponds to InstructCoder [ 33 ] , MT-Bench [ 56 ] , ShareGPT [ 6 ] , or Spec-Bench [ 54 ] . Although the absolute speedups vary by dataset, both models retain the same qualitative non-concave, batch-size-1-peaked behavior.

Step	Correction	Target efficiency	$\Delta R$	% gap closed
0	Weight model baseline	0.511	n/a	n/a
1	MoE fraction: 67% $\to$ 33% (GPU time $\neq$ weight)	0.679	+0.168	44%
2	MoE scaling: $S$ =2.44 $\to$ 1.57 (L2 cache + dilution)	0.843	+0.164	43%
3	Non-MoE: $S$ =1.0 $\to$ 0.94 (amortization)	0.872	+0.029	8%
	Measured (kernel-level sum)	0.872
	Measured (sweep wall-clock)	0.896

Appendix

Table 11: Three-step correction for Qwen3-30B at batch size 1 on H100.

Metric	Qwen3-235B (H100, TP4)	Qwen3-30B (B200, TP1)	Mixtral-8x7B (B200, TP1)
MoE weight reads (SD)	per (token, expert) pair	once per expert	once per expert
Expert size (per GPU)	4.5 MB	4.5 MB	168 MB
MoE share of weight	67.9%	66.7%	89.4%
MoE share of GPU time	34%	35%	69%
Attn CTAs / SMs	18 (14%)	16 (11%)	48 (32%)
Attn HBM BW (% peak)	19–26%	12–13%	27–37%

Appendix

Table 12: GPU profiling at batch size 1 for Qwen3-235B on H100 and for Qwen3-30B and Mixtral on B200 (all FP8). H100 has 132 SMs and B200 has 148 SMs.

Model	$k/N$	Uniform	Indep	Empirical	Temporal	Temp/Total
Mixtral-8x7B	0.250	0.250	0.253	0.355	0.102	28.7%
Qwen2-57B	0.125	0.125	0.140	0.204	0.064	31.5%
GLM-4.7-Flash	0.063	0.063	0.097	0.319	0.222	69.4%
Qwen3-30B	0.063	0.063	0.119	0.431	0.312	72.4%
Qwen3-235B	0.063	0.063	0.098	0.447	0.349	78.2%
Qwen3.5-35B	0.031	0.031	0.055	0.332	0.277	83.4%

Appendix

Table 13: Step-1 expert overlap decomposition for all measured models. Temporal = empirical

-

independence; Temporal/Total = fraction of overlap attributable to temporal overlap.

Model	Step	Acc	1st Rej	$\Delta$	$n_{\text{1st}}$
Qwen3-30B	1	0.433	0.441	$-$ 0.008	4991
	2	0.409	0.448	$-$ 0.039	3021
	3	0.392	0.420	$-$ 0.028	1374
Qwen3-235B	1	0.447	0.488	$-$ 0.040	8420
	2	0.422	0.475	$-$ 0.053	5936
	3	0.407	0.458	$-$ 0.051	3752

Appendix

Table 14: Deconfounded consecutive overlap (accepted vs. first-rejection). Negative

\Delta

means rejected tokens have higher overlap.

Model	Step	$\Delta$	Cohen’s $d$	$p$	Size
Qwen3-30B	1	$-$ 0.008	$-$ 0.06	2.3e-3	negligible
	2	$-$ 0.039	$-$ 0.26	5.9e-22	small
	3	$-$ 0.028	$-$ 0.19	5.4e-7	negligible
Qwen3-235B	1	$-$ 0.040	$-$ 0.29	6.2e-111	small
	2	$-$ 0.053	$-$ 0.38	8.2e-129	small
	3	$-$ 0.051	$-$ 0.37	5.3e-79	small

Appendix

Table 15: Cohen’s

d

for consecutive overlap (accepted vs. first-rejection, deconfounded). Negative

d

= rejected tokens have higher overlap.

Figure 9: Anchor overlap (step 0 $\to$ step $j$ ) with SEM error bars and Cohen’s $d$ annotations, by acceptance status and step. Whereas consecutive overlap is consistently higher for rejected tokens, anchor overlap tends to be similar or slightly lower, indicating that the inertia is local rather than global.

Model	Data	$\Delta_{1}$	$\Delta_{2}$	$\Delta_{3}$	Anchor drop
Qwen3-30B	Spec-Bench	+0.000	$-$ 0.017	$-$ 0.017	+0.085
	Aya	+0.002	$-$ 0.018	$-$ 0.015	+0.095
Qwen3-235B	Spec-Bench	$-$ 0.018	$-$ 0.021	$-$ 0.028	+0.070
	Aya	$-$ 0.002	$-$ 0.020	$-$ 0.015	+0.106

Appendix

Table 16: Held-out deconfounded routing summary. Entries report median

\Delta_{\text{Con}}

across held-out slices, where negative values mean rejected tokens have higher consecutive overlap, together with the median anchor-drop mean.

Model	Step 1	Step 2	Step 3	Step 4	Indep
Mixtral-8x7B	0.355	0.303	0.298	0.288	0.253
Qwen2-57B	0.204	0.150	0.149	0.142	0.140
GLM-4.7-Flash	0.319	0.277	0.245	0.233	0.097
Qwen3-30B	0.431	0.356	0.328	0.312	0.119
Qwen3-235B	0.447	0.378	0.354	0.337	0.098
Qwen3.5-35B	0.332	0.272	0.243	0.225	0.055

Appendix

Table 17: Overlap ratio by step distance for the measured models. The independence baseline (last column) indicates the overlap expected in the absence of temporal effects.

Data slice	Step 1	Step 2	Step 3	Step 4	Indep	Verify ratio
MT-Bench	0.431	0.356	0.328	0.312	0.119	0.415
Spec-Bench mean	0.438 (0.407–0.468)	0.358 (0.324–0.391)	0.330 (0.289–0.366)	0.313 (0.274–0.348)	0.172 (0.122–0.207)	0.418 (0.400–0.434)
Aya mean	0.477 (0.447–0.515)	0.396 (0.364–0.458)	0.364 (0.333–0.424)	0.345 (0.310–0.404)	0.176 (0.148–0.228)	0.439 (0.423–0.462)

Appendix

Table 18: Held-out overlap decay for Qwen3-30B. Parentheses report the range across held-out slices.

Figure 10: Per-layer step-1 overlap ratio for Qwen3-30B ( $K{=}3$ ). The empirical overlap (blue) far exceeds the independence baseline (orange) at every layer except the first and last, with mid-network layers peaking at 4–5 $\times$ above baseline.

Qwen3-30B		Qwen3-235B
Layer	Overlap (indep)	Layer	Overlap (indep)
0	0.094 (0.089)	0	0.110 (0.078)
1	0.265 (0.099)	1	0.168 (0.082)
2	0.306 (0.118)	2	0.247 (0.085)
3	0.389 (0.121)	3	0.328 (0.088)
22	0.433 (0.111)	45	0.399 (0.090)

Appendix

Table 19: Per-layer step-1 overlap ratio for representative layers in Qwen3-30B (48 layers) and Qwen3-235B (94 layers).

Model	$k$	$N$	Unif	Indep	Emp	Emp/ $k$	Uniform IPF	Err	Skewed IPF	Err
			Unique experts				IPF prediction
Mixtral-8x7B	2	8	5.5	5.5	5.0	2.5 $\times$	4.99	+0.1%	4.99	+0.1%
Qwen2-57B	8	64	26.5	26.0	24.6	3.1 $\times$	24.72	+0.5%	24.63	+0.1%
GLM-4.7-Flash	4	64	14.6	13.9	11.0	2.7 $\times$	11.16	+1.6%	11.07	+0.8%
Qwen3-30B	8	128	29.1	26.9	19.5	2.4 $\times$	20.08	+3.1%	19.72	+1.2%
Qwen3-235B	8	128	29.1	27.7	19.0	2.4 $\times$	19.67	+3.5%	19.45	+2.3%

Appendix

Table 20: Unique experts (

K{=}3

, four-token window) and IPF prediction accuracy for the measured models.

Panel of Figure 11 — Figure 11: Dense draft cost for Qwen3-30B ( $L{=}48$ , $N{=}128$ ). The dense drafter becomes more expensive at large batch sizes.

Panel of Figure 12 — Figure 12: Synthetic MoE draft analysis for $N{=}896$ (extrapolated from Qwen3-30B with dummy weights). The MoE layer remains bandwidth-bound to larger batch sizes, keeping the draft ratio flat where the dense drafter’s ratio rises.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 314 papers

Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
Jul 14, 2026Jincheng Xie, Runheng Liu, Heyan Huang +4Mixture-of-Experts Inference Speculative Decoding
Department of Mathematical Sciences, Tsinghua University, Beijing, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JDT AI Infra +2
MoE-Spec: Expert Budgeting for Efficient Speculative Decoding
Feb 17, 2026Bradley McDanel, Steven Li, Sruthikesh Surineni +1Speculative Decoding LLM Inference Acceleration
Franklin and Marshall College · Meta Reality Labs
Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation
Sep 18, 2026Kumari Nishu, Han-Byul Kim, Santosh Chilkunda +13Mixture-of-Experts Inference Mixture-of-Experts Models
Apple

Jul 14, 2026J/K move · Enter open · S save

Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

Jincheng Xie, Runheng Liu, Heyan Huang, Yawen Ling+3

Department of Mathematical Sciences, Tsinghua University, Beijing, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JDT AI Infra+2

Mixture-of-Experts Inference Speculative Decoding Mixture-of-Experts Models

Open paper AskPDF arXiv

Sparse Mixture-of-Experts (MoE) models have become an important approach for scaling Large Language Models (LLMs), but their inference efficiency depends strongly on expert activation patterns. Speculative decoding (SD) accelerates autoregressive generation by verifying multiple draft tokens in parallel, yet existing draft selection strategies primarily optimize acceptance likelihood. In large-scale MoE models, however, selecting draft tokens also determines the union of experts activated during verification. We observe that confidence-driven SD can introduce expert scattering: high-probability draft tokens may route to disjoint experts, increasing expert-weight memory traffic and reducing the speedup from speculation. Motivated by this observation, we revisit draft-tree selection under the non-uniform memory-cost structure of MoE inference. We propose EcoSpec, a cost-aware speculative decoding framework that incorporates predicted marginal expert activation cost into draft selection. With a lightweight expert predictor and a dynamic expert buffer, EcoSpec favors draft paths that preserve high acceptance likelihood while reusing experts already covered by the current verification set, without modifying the target-model verification rule. We evaluate EcoSpec on three large-scale MoE models, including DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B, across reasoning, coding, question-answering, and dialogue benchmarks. EcoSpec consistently reduces active expert footprints and improves end-to-end decoding speed, achieving up to

1.62\times

speedup. These results show that accounting for expert activation cost is important for efficient speculative decoding in large-scale MoE models.

Abstract

Speculative Decoding

Explore similar work

Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

MoE-Spec: Expert Budgeting for Efficient Speculative Decoding

Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation