Speculative Decoding for Mixture-of-Experts Models: Expert Bytes, Kernel Bandwidth and the Best Draft Length

Yuzheng Lan

cs.LGOct 25, 2026

Speculative Decoding for Mixture-of-Experts Models: Expert Bytes, Kernel Bandwidth and the Best Draft Length

Authors: Yuzheng Lan

Organizations: ReClan AI Studio, LLC

Abstract

Speculative decoding checks several drafted tokens in one verification pass. On a mixture-of-experts (MoE) model, consecutive tokens often select the same experts. A deduplicating engine reads each selected expert once per pass, so it requests fewer bytes than a per-slot engine, which reads it once per token. We write a step's time as the bytes each kernel requests over its effective bandwidth, plus fixed time. Under geometric acceptance, we derive from this step time when speculation is faster than plain decoding. The best draft length is a monotone staircase in the acceptance rate. We bound how a bandwidth error moves each result, and we check the main steps in Lean. Deduplication is faster exactly where its share $\bar u(n)$ of the per-slot bytes is below its ratio $r(n)$ of the per-slot bandwidth. On llama.cpp Metal, MLX-LM and llama.cpp CUDA, which we modified to deduplicate, the per-slot kernel has the higher effective bandwidth (r = 0.32 to 0.75). Thus deduplication is faster only from 5 to 8 tokens per pass, or never. We fit the bandwidths on one layer's expert operation, run alone, in under an hour per model. Bytes from routing traces over these bandwidths predict the time that deduplication saves. The measured saving lies inside its error bound at every point on CUDA and MLX-LM and at 36 of 48 points on Metal. The crossover is the first pass size at which deduplication is faster. The prediction places it within one pass size in 10 of 12 configurations of engine, model and depth. We computed these predictions after the measurements. On DeepSeek-V2-Lite, operation times measured before its speculative runs give the best draft length on CUDA within one at all 22 acceptance values.

Figures & tables

Figure 1: Consecutive-token reuse

\rho_{l}(n)

, layer mean, on the WikiText-103 fit documents, against the number

n

of consecutive tokens. Dotted: independent uniform routing,

1-q^{\,n-1}

( Huang et al.,, 2025 ) . In every figure, color marks the model: blue OLMoE, green Qwen3, orange gpt-oss and brown DeepSeek-V2-Lite.

Figure 2: Effective bandwidth of each kernel on the expert operation alone. Per-slot kernel: on requested bytes

nmb

(solid line, filled markers). Deduplicating kernel: on union bytes

Ub

(dashed line, open markers). Median and interquartile range over all MoE layers and windows of recorded routing; color: model; dotted: nominal DRAM bandwidth. CUDA was measured on OLMoE, gpt-oss and DeepSeek-V2-Lite, and MLX-LM on OLMoE.

engine	model	cost	$\beta_{\mathrm{slot}}$	$\beta_{\mathrm{dedup}}$	$r$	$\bar{u}$	$\beta_{\mathrm{slot}}$	$\beta_{\mathrm{dedup}}$	$r$	$\bar{u}$
		re-read	$n=2$				$n=8$
llama.cpp CUDA	OLMoE	0.18-0.22	618	464	0.75	0.82	933	546	0.58	0.50
llama.cpp CUDA	gpt-oss	0.34	654	346	0.53	0.72	1018	434	0.43	0.36
llama.cpp CUDA	DeepSeek-V2-Lite	$-$	523	355	0.68	0.83	814	480	0.59	0.62
llama.cpp Metal	OLMoE	0.88-0.90	250	147	0.59	0.82	428	219	0.51	0.50
llama.cpp Metal	Qwen3	0.92-0.98	131	65	0.50	0.77	280	98	0.35	0.43

Table 1: Effective bandwidth of each kernel in GB/s, over all MoE layers and windows of recorded routing, including the fixed work of the operation. The values of

r

and

\bar{u}

come from bytes and times summed over layers and windows. Re-read cost:

c_{\mathrm{re}}/c_{\mathrm{u}}

of ( 4 ), as a range over two layers (

-

: no fit on chosen ids). The deduplicating kernel is faster where

\bar{u}<r

( 7 ).

prediction	bound
verify time	$\|\hat{\tau}_{p}(n)-\tau_{p}(n)\|\leq\frac{\varepsilon}{1-\varepsilon}M_{p}(n)\leq\frac{\varepsilon}{1-\varepsilon}\tau_{p}(n)$ (17)
saving	$\|\hat{S}(n)-S(n)\|\leq\frac{\varepsilon}{1-\varepsilon}\bigl(M^{\mathrm{e}}_{\mathrm{slot}}(n)+M^{\mathrm{e}}_{\mathrm{dedup}}(n)\bigr)$ (18)
faster kernel	sides of ( 7 ) agree if $\|\ln r(n)-\ln\bar{u}(n)\|>\ln\frac{1+\varepsilon}{1-\varepsilon}$ (19)
best draft length	$T_{a}(\hat{k}^{})\geq\frac{1-\delta}{1+\delta}T_{a}(k^{})$ : at most $\frac{2\delta}{1+\delta}$ of the throughput lost; $\hat{k}^{}=k^{}$ if $\frac{1+\delta}{1-\delta}T_{a}(i)<T_{a}(k^{})$ for every $i\neq k^{}$ (20)
best speedup	$\frac{1-\delta}{1+\delta}\leq\frac{\hat{T}_{a}(\hat{k}^{})/\hat{T}_{a}(0)}{T_{a}(k^{})/T_{a}(0)}\leq\frac{1+\delta}{1-\delta}$ (21)

Table 2: Error bounds for a relative bandwidth error

|e|\leq\varepsilon

(first three rows) or a relative step-time error

\delta

(last two).

Figure 3: Bytes. Left: DRAM bytes read by the CUDA expert kernels over the union bytes

Ub

, against

nm/U

(Nsight Compute, chosen ids, one layer of OLMoE and gpt-oss). Filled: per-slot kernel; open: deduplicating kernel; color: model. The deduplicating kernel reads

Ub

(horizontal line). The per-slot kernel reads between

Ub

and its requested

nmb

(dotted). Right: the share of a pass’s

nm

slots that name distinct experts,

U/(nm)

, at

n=2,4,6,8

. It is shown on the windows the whole-model runs verified, against WikiText-103 windows of the same model. Both axes lie between

1/n

(every token repeats the first token’s experts) and 1 (no expert repeats). A point above the dotted line means the verified text repeated experts less often than WikiText-103. Color: model; shape: the engine of the runs (circle: llama.cpp CUDA, triangle: llama.cpp Metal). The labeled gpt-oss points at

d=4096

below the line are its repeating continuations.

engine	model	$d$	wiki	text	op. time	meas.	wiki	text	op. time	wiki	text	op. time
			saving error (ms)			crossover				$k^{*}$ within one
Metal	OLMoE	512	0.43	0.42	0.77	7	7	7	8	9	9	9
Metal	OLMoE	4096	0.98	0.91	2.28	7	7	7	8	8	8	8
Metal	Qwen3	512	1.51	0.92	2.24	$-$	$-$	$-$	$-$	7	8	7
Metal	Qwen3	4096	3.70	1.83	1.25	$-$	$-$	$-$	$-$	8	8	8
Metal	gpt-oss	512	2.23	3.77	3.89	8	$-$	$-$	$-$	10	9	10

Table 3: The prediction with WikiText-103 routing (wiki) and with the routing of the verified text (text), and the measured-time version (op. time). All three are compared with the same measurements, one row per configuration. Saving error: largest

|\hat{S}(n)-S(n)|

over

n=2,\dots,9

(

n\leq 8

on CUDA), in ms. Crossover as defined in Section 2 (meas.: measured;

-

: none).

k^{*}

: acceptance values, of 11, at which the predicted best draft length is within one of the measured one. Metal and CUDA are llama.cpp.

Figure 4: Saving

S(n)

. Black and grey: measured on the six validation prompts, with its 95% bootstrap interval. Red: the prediction, dotted to an open circle where

n=9

is extrapolated. Shaded red: the band of the saving bound ( 2 ) for the engine’s

\varepsilon

. Grey dashed: the measured-time version. The number in each panel counts the measured points inside the band. Positive: the deduplicating kernel is faster. CUDA is shown to

n=8

, since it runs the deduplicating kernel at

n=9

with deduplication off and on.

Figure 5: Best draft length and the bounds ( 2 ) and ( 2 ). Each configuration has one point under the measured-time version (filled) and one under the calibrated version (open), with

\delta

as in Section 4.3 . Color: model; shape: engine (circle: llama.cpp CUDA, triangle: llama.cpp Metal, square: MLX-LM). Left: predicted against measured

k^{*}

, measured-time version; area grows with the number of acceptance values; dotted: exact prediction; shaded: within one. Middle: largest share of throughput lost by drafting

\hat{k}^{*}

, against its bound ( 2 ). Right: largest relative error of the predicted best speedup, against its bound ( 2 ). In the middle and right panels the solid line is where the error equals its bound. The theory requires every point to lie in the shaded region on or below it, and every point does.

engine	version	$d$	crossover	sign	saving error	$k^{*}$ within 1	loss	$\tau$ error / $\delta$
llama.cpp CUDA	measured-time	512	$-$ / $-$	7 of 7	0.23	11	2.9%	9.9% / 0.072
	measured-time	4096	$-$ / $-$	7 of 7	0.28	11	4.1%	10.1% / 0.077
llama.cpp Metal	measured-time	512	$-$ / $-$	8 of 8	1.6	9	20.1%	34.0% / 0.27
	measured-time	4096	$-$ / $-$	8 of 8	4.4	6	22.5%	28.4% / 0.25
	calibrated	512	$-$ / $-$	8 of 8	1.5	11	6.4%	4.3% / 0.036
	calibrated	4096	$-$ / $-$	8 of 8	4.4	9	8.3%	10.5% / 0.098

Table 4: DeepSeek-V2-Lite-Chat: the measured-time version and the calibrated version, written before its speculative runs. Crossover: predicted / measured (

-

: none), over

n\leq 8

on CUDA. Sign: values of

n

with the measured sign of

S(n)

, of those whose interval excludes 0. Saving error in ms.

k^{*}

: acceptance values, of 11, with

\hat{k}^{*}

within one of the measured best. Loss: largest share of the best throughput lost at

\hat{k}^{*}

\tau

error: largest relative error of the predicted verify time, over both kernels and

n=2,\dots,9

\delta

as in Section 4.3 . The best speedup lies inside its bound ( 2 ) in every row.

Figure 6: Saving

S(n)

on DeepSeek-V2-Lite, as in Figure 4 . Black and grey: measured, with its 95% interval. Red: the prediction from fits run after the whole-model runs, with the band of the saving bound ( 2 ) for the engine’s

\varepsilon

. Grey dashed: the measured-time version, written before its speculative runs. The number in each panel counts the measured points inside the band.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$a$	acceptance rate (geometric acceptance)
$a_{0}$	$V(1)/V(0)-1$ , the acceptance above which one draft is faster than none
$a_{j}$	$j$ -th step point
$b$	bytes of one expert of the timed layer, $b=w_{l}$
$B_{k}$	$\beta_{\mathrm{dedup}}(k+1)$
$c_{i}$	$V(i)-V(i-1)$ , the time the $i$ -th draft adds

Appendix

Table 11

engine	tested sizes	other sizes	where the source decides
llama.cpp Metal b11617	per-slot	per-slot for $n\leq 31$ ; tiled (32 tokens per tile) for $32\leq n\leq 512$ , deduplicating only at $n=32$ for two models	dispatch test “batch $\geq 32$ rows” in ggml-metal-common.cpp ; matrix-vector kernel in mul_mv.metal , one grid slice per token-slot pair
MLX-LM 0.32.0	per-slot	per-slot for $n<4N/m$ (32 or 64); tiled (16 per tile), neither, above	dispatch branch on rows per expert in mlx/backend/metal/quantized.cpp ; MoE call in mlx_lm/models/switch_layers.py
llama.cpp CUDA b11617	upstream; run in Sec. 4	both at $n=1$ ; per-slot for $2\leq n\leq 8$ ; deduplicating for $9\leq n\leq 128$ (OLMoE, Qwen3), $16/15$ times union (gpt-oss)	batch limit 8 for the matrix-vector kernel in ggml-cuda/mmvq.cuh ; dispatch in ggml_cuda_mul_mat_id ( ggml-cuda.cu ); tiles in ggml-cuda/mmq.cu

Appendix

Table 5: Kernel that runs the expert products, read from source.

n

is the number of tokens verified in one pass, and the tests verify

n\leq 9

tokens per step. “Neither”: the requested bytes equal neither the union nor the per-slot count on every trace. Bytes are requested bytes. The source does not determine how many requests on-chip caches serve. The CUDA row at

2\leq n\leq 8

is the source reading of the matrix-vector kernel, which requests each slot’s expert once. Profiled L2 requests of this kernel fall below

nmb

when repeated experts sit in the same slot of every token. They fall as far as 0.14 of

nmb

, at

n=8

and

U=8

. Two causes would both produce this: L1 hits under per-slot requests, and a pattern that requests each expert once per slot position. The hardware counters do not decide between them, so the row rests on the source. For gpt-oss, the factor

16/15

counts the loads of the tiled kernel in the source reading. The factor

f_{l}=500/459

of Table 6 instead counts the padded rows of each weight tensor, the byte unit of Appendix E .

engine	switch	deduplication off	deduplication on	$f_{l}$ , deduplicating
llama.cpp Metal, modified	deduplicating kernel from 2 rows (32 upstream)	mul_mv_id , per-slot, $n\leq 9$	mul_mm_id , deduplicating, $2\leq n\leq 9$	1 (OLMoE Q8_0, gpt-oss MXFP4); 2.14 or 2.33 (Qwen3 Q4_K_M, K-quant reload)
MLX-LM, modified	sort rows by expert, gather_qmm_rhs	gather_qmv , per-slot, $n\leq 9$	sorted rows, deduplicating, $2\leq n\leq 9$	10/9 (16-row tile)
llama.cpp CUDA, modified	per-slot batch cap 8 to 1	MMVQ, per-slot, $n\leq 8$ ; MMQ at $n=9$	MMQ, deduplicating, $2\leq n\leq 9$	1 (OLMoE); 500/459 (gpt-oss, padded unit)

Appendix

Table 6: The deduplicating engines. The kernel each setting runs is read from source at

n=1,\dots,9

verified tokens. At

n=1

, an engine runs the same kernel with deduplication off and on.

f_{l}

: bytes the deduplicating kernel reads per distinct expert, over the expert’s bytes. On Qwen3 Q4_K_M, the tiled kernel reads each K-quant expert more than once. MMVQ and MMQ: llama.cpp CUDA’s quantized matrix-vector and matrix-matrix kernels.

				$\varepsilon$		verify-time error
engine	model	$d$	amplification	implied	engine	prediction	measured-time	calibrated
Metal	OLMoE	512	6.0 to 79	0.014	0.031	19%	20%	6%
Metal	OLMoE	4096	6.2 to 86	0.028	0.031	24%	24%	6%
Metal	Qwen3	512	3.9 to 7.8	0.020	0.031	21%	22%	6%
Metal	Qwen3	4096	4.2 to 19	0.047	0.031	25%	31%	4%
Metal	gpt-oss	512	3.3 to 343	0.033	0.031	20%	15%	9%

Appendix

Table 7: Per configuration, three quantities. Amplification: the factor

(\hat{M}^{\mathrm{e}}_{\mathrm{slot}}+\hat{M}^{\mathrm{e}}_{\mathrm{dedup}})/|S(n)|

by which the saving bound ( 2 ) turns a relative bandwidth error into a relative saving error. It is given as its range over

n=2,\dots,9

(

n\leq 8

on CUDA;

|S(n)|

floored at 0.05 ms), and it is largest where

S(n)

crosses 0. Implied

\varepsilon

: the bandwidth error that the prediction’s largest saving error requires under ( 2 ), against the engine’s

\varepsilon

. Verify-time error: the largest relative error of the verify time with deduplication on over

n=2,\dots,8

, for the prediction, the measured-time version and the calibrated version. Metal and CUDA are llama.cpp.

rule	CUDA 512	CUDA 4096	Metal 512	Metal 4096
crossover within one	$-$ / $-$ , holds	$-$ /9, holds	$-$ / $-$ , holds	$-$ / $-$ , holds
$\hat{k}^{*}$ within one at $\geq 9$ of 11 values	11, holds	11, holds	9, holds	6, fails
loss at most 6% at every value	2.9%, holds	4.1%, holds	20.1%, fails	22.5%, fails
sign of $S(n)$ where the interval excludes 0	7 of 7, holds	7 of 7, holds	8 of 8, holds	8 of 8, holds
verify time within 15%, both kernels	9.9%, holds	10.1%, holds	34.0%, fails	28.4%, fails
best speedup inside ( 2 )	$\delta=0.072$ , holds	0.077, holds	0.27, holds	0.25, holds

Appendix

Table 8: Scoring rules for DeepSeek-V2-Lite and the measured values, measured-time version. The speedup rule uses

\delta

, the largest relative error of the predicted step time. The crossover rule is scored over

n=2,\dots,9

, and a measured crossover at

n=9

counts as within one of none. On CUDA, the deduplicating kernel runs at

n=9

with deduplication off and on. So Table 4 , like Table 3 , reads the CUDA crossover over

n\leq 8

Figure 7: Bandwidth of a held-out model, predicted from the other models of the same engine. It is plotted against the bandwidth measured on the expert operation of the held-out model, at

n=2,4,6,8

. Left: each kernel’s bandwidth (filled: per-slot; open: deduplicating), log scale. Right: the ratio

r(n)

. Color: the held-out model; shape: engine (circle: llama.cpp CUDA, triangle: llama.cpp Metal). Dotted: equal.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 311 papers

Feb 17, 2026

MoE-Spec: Expert Budgeting for Efficient Speculative Decoding

Speculative decoding accelerates Large Language Model (LLM) inference by verifying multiple drafted tokens in parallel. However, for Mixture-of-Experts (MoE) models, this parallelism introduces a severe bottleneck: large draft trees activate many unique experts, significantly increasing memory pressure and diminishing speedups from speculative decoding relative to autoregressive decoding. Prior methods reduce speculation depth when MoE verification becomes expensive. We propose MoE-Spec, a training-free verification-time expert budgeting method that decouples speculation depth from memory cost by enforcing a fixed expert capacity limit at each layer, loading only the experts that contribute most to verification and dropping the long tail of rarely used experts that drive bandwidth overhead. Experiments across multiple model scales and datasets show that this method yields 10--30% higher throughput than state-of-the-art speculative decoding baselines (EAGLE-3) at comparable quality, with flexibility to trade accuracy for further latency reductions through tighter budgets.

Bradley McDanel, Steven Li, Sruthikesh Surineni +1

Franklin and Marshall College · Meta Reality Labs

Speculative Decoding LLM Inference Acceleration Mixture-of-Experts Inference

Ask PDF

Jul 14, 2026

Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

Sparse Mixture-of-Experts (MoE) models have become an important approach for scaling Large Language Models (LLMs), but their inference efficiency depends strongly on expert activation patterns. Speculative decoding (SD) accelerates autoregressive generation by verifying multiple draft tokens in parallel, yet existing draft selection strategies primarily optimize acceptance likelihood. In large-scale MoE models, however, selecting draft tokens also determines the union of experts activated during verification. We observe that confidence-driven SD can introduce expert scattering: high-probability draft tokens may route to disjoint experts, increasing expert-weight memory traffic and reducing the speedup from speculation. Motivated by this observation, we revisit draft-tree selection under the non-uniform memory-cost structure of MoE inference. We propose EcoSpec, a cost-aware speculative decoding framework that incorporates predicted marginal expert activation cost into draft selection. With a lightweight expert predictor and a dynamic expert buffer, EcoSpec favors draft paths that preserve high acceptance likelihood while reusing experts already covered by the current verification set, without modifying the target-model verification rule. We evaluate EcoSpec on three large-scale MoE models, including DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B, across reasoning, coding, question-answering, and dialogue benchmarks. EcoSpec consistently reduces active expert footprints and improves end-to-end decoding speed, achieving up to

1.62\times

speedup. These results show that accounting for expert activation cost is important for efficient speculative decoding in large-scale MoE models.

Jincheng Xie, Runheng Liu, Heyan Huang +4

Department of Mathematical Sciences, Tsinghua University, Beijing, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JDT AI Infra +2

Mixture-of-Experts Inference Speculative Decoding Mixture-of-Experts Models

Ask PDF

Oct 11, 2026

SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference

Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D) expert transfers enter the critical path of decoding. Speculative decoding verifies several draft tokens in one forward pass, yet in an MoE model each verified token activates its own experts, so a larger draft tree transfers more experts. Because of this H2D cost, speculative decoding commits several tokens per verification cycle yet is sometimes even slower than autoregressive decoding. Based on this observation, we propose SpecFit, which chooses the verified draft subtree to maximize predicted throughput rather than acceptance length. SpecFit estimates the expected number of committed tokens from draft path probabilities and predicts cycle time from the node count and the per-layer union of the experts the subtree activates; this union is a monotone submodular coverage function, so each node pays only for the experts it adds. An exact search over ancestor-closed subtrees returns the subtree with the highest predicted throughput; run after each draft level, the same search also decides when to stop drafting, and the unmodified target verifies the result. On CNN/DailyMail summarization and a GSM8K subset with Qwen3-30B-A3B, SpecFit decodes 21.6% and 7.0% faster than EAGLE-3 with the same expert cache and 1.97x and 1.72x as fast as vanilla EAGLE-3. Code is available at https://github.com/Chlience/SpecFit.

Hongliang Cao, Xiuyu Li, Siqi Wang +3

Beihang University · Monash University · The Hong Kong University of Science and Technology

LLM Inference Acceleration Speculative Decoding Mixture-of-Experts Inference Expert Offloading

Ask PDF

Abstract

Speculative Decoding

Explore similar work

MoE-Spec: Expert Budgeting for Efficient Speculative Decoding

Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference