The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts

Aidar Amankulov; Denis Mamatin

cs.LGAug 26, 2026

The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts

Authors: Aidar Amankulov, Denis Mamatin

Abstract

Speculative decoding in Mixture-of-Experts (MoE) models faces the problem of unstable verification cost caused by input-dependent expert loading. To study the physics of this process, we formulate speculation-budget selection as an offline Stochastic Shortest Path (SSP) problem over reference sequences and build a diagnostic Oracle that uses counterfactual simulation to account for MoE verification cost. A detailed analysis of the Oracle's decisions on the Qwen3-Coder and EAGLE-3 pairing, in the space of marginal deltas (Delta Space), shows that rejected candidates form a strict linear boundary. This result demonstrates that a complex global optimization is locally governed by a necessary condition balancing marginal cost against expected progress ( $\frac{Δ\mathbb{E}[Cost]}{Δ\mathbb{E}[a]}$ ), providing a rigorous mathematical reference point for designing future adaptive online heuristics.

Figures & tables

Figure 1: $T_{T,r}$ against $\bar{U}_{r}$ for each $K$ , with the fitted line and coefficients of model ( 4 ). The extended robustness protocol is given in Appendix A .

Figure 2: Quantitative bounds of the MoE Stochastic Oracle under varying speculation budgets

L_{max}

. As

L_{max}

increases, Theoretical Maximum Speedup (

S_{max}

), Mean

k

, and Mean Accepted Length exhibit diminishing returns and saturation. The Oracle selectively caps

k

to avoid costly MoE expert activations on low-probability draft tokens.

Figure 3: Delta Space Analysis ( $\Delta\mathbb{E}[\text{Cost}]$ vs. $\Delta\mathbb{E}[a]$ ) of rejected draft candidates relative to the Oracle’s choice $(0,0)$ . Quadrants II and IV are strictly empty, proving strict monotonicity. Candidates in Quadrant I (risk zone) and Quadrant III (caution zone) form a linear boundary passing through the origin.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Target	Qwen/Qwen3-Coder-30B-A3B-Instruct; MoE with 128 experts and top- $k_{e}=8$ routing
Draft head	lmsys/SGLang-EAGLE3-Qwen3-Coder-30B-A3B-Instruct-SpecForge
GPU and stack	NVIDIA A100 80GB; vLLM fork with per-round trace; FULL CUDA graphs
Mode	$B=1$ , synchronous scheduling
Sampling	$T=0.7$ , top_p $=1.0$ , seed $=0$
Other	bfloat16, max_model_len $=4096$ , gpu_mem_util $=0.85$ , max_new_tokens $=1000$

Appendix

Table 4

$K$	$m$	Rounds	Median $T_{T}$ , ms	Median $\bar{U}_{r}$	Median $L$
1	2	127410	8.98	12.1	605
2	3	106110	10.81	15.5	606
3	4	96819	12.63	18.2	605
4	5	90813	10.94	20.6	605
5	6	87401	11.51	22.6	600
6	7	85147	12.07	24.5	591

Appendix

Table A1: Dataset summary by run. Median

L

ranges between 586 and 606 tokens. The non-monotonicity of

T_{T}

K=3\rightarrow 4

coincides with a change in the estimated

\beta_{0}

; without a randomized repeat sweep it cannot be causally attributed to a CUDA-graph mode.

Factor	What it measures	Mechanistic hypothesis
$\bar{U}_{r}$	Number of unique experts per pass, averaged across layers	Expert-weight traffic from HBM
$L$	Total context, tokens	KV-cache read and attention
$m$	Number of positions in the verification pass	Microbatch and CUDA-graph size
$E_{\max}$	Maximum number of tokens per single expert	Load imbalance
KV usage	KV-cache occupancy	Allocator pressure
Graph bucket	Size of the captured graph	Padding; at $B=1$ the bucket is equivalent to $m$

Appendix

Table 6

Factor	OOS $R^{2}$ , alone	OOS RMSE, ms	$\Delta$ RMSE beyond $\bar{U}_{r}$ , ms	corr with $\bar{U}_{r}$
$\bar{U}_{r}$	$0.787\pm 0.026$	$0.831\pm 0.037$	—	1.00
$m=K+1$	$0.515\pm 0.033$	$1.256\pm 0.020$	$+0.014\pm 0.003$	0.86
Graph bucket	$0.515\pm 0.033$	$1.256\pm 0.020$	$+0.014\pm 0.003$	0.86
$E_{\max}$	$0.362\pm 0.029$	$1.442\pm 0.014$	$+0.015\pm 0.002$	0.75
KV usage	$0.077\pm 0.017$	$1.734\pm 0.051$	$+0.139\pm 0.007$	0.03
$L$	$0.079\pm 0.016$	$1.732\pm 0.051$	$+0.146\pm 0.008$	0.03

Appendix

Table A2: Descriptive predictive importance of factors on the pool of

925\,752

rounds; intervals are 95% CIs across folds.

\bar{U}_{r}

is the strongest single predictor, and

L

gives the largest additional error reduction on top of it.

Category	Rounds	OOS $R^{2}$ : $\bar{U}_{r}$	OOS $R^{2}$ : $\bar{U}_{r}+L$	$\Delta$ RMSE from $L$ , ms	Final $R^{2}$
code_math	326429	$0.870\pm 0.023$	$0.903\pm 0.018$	$+0.098\pm 0.007$	0.9993
extraction	5839	$0.861\pm 0.097$	$0.864\pm 0.090$	$+0.005\pm 0.019$	0.9991
math_reasoning	57788	$0.843\pm 0.039$	$0.851\pm 0.033$	$+0.015\pm 0.014$	0.9990
coding	47116	$0.809\pm 0.079$	$0.904\pm 0.064$	$+0.281\pm 0.081$	0.9991
translation	16594	$0.797\pm 0.023$	$0.795\pm 0.022$	$-0.002\pm 0.008$	0.9991
rag	63141	$0.766\pm 0.011$	$0.774\pm 0.007$	$+0.012\pm 0.024$	0.9982

Appendix

Table A3: Factor importance by category, sorted by the OOS

R^{2}

of the

\bar{U}_{r}

-only model. The summary row corresponds to the full pool.

Figure A1: Out-of-sample

R^{2}

for predicting

T_{T}

by category:

\bar{U}_{r}

alone versus

\bar{U}_{r}+L

Figure A2: Round scatter and fitted line for each of the ten values of $K$ on shared axes. As $K$ grows, the cloud shifts to the right, and at the $m=4/5$ boundary the slope increases roughly threefold.

$K$	$\alpha$ , ms/expert	$c$ , $10^{-3}$ ms/token	$\beta_{0}$ , ms	$N$	$R^{2}$	OOS MAPE
1	0.0839 (0.0003)	1.098 (0.001)	7.294 (0.003)	127410	0.9896	0.44%
2	0.0824 (0.0003)	1.101 (0.002)	8.854 (0.005)	106110	0.9806	0.51%
3	0.0779 (0.0004)	1.106 (0.002)	10.532 (0.008)	96819	0.9669	0.59%
4	0.2675 (0.0002)	1.099 (0.001)	4.739 (0.003)	90813	0.9977	0.33%
5	0.2664 (0.0003)	1.103 (0.001)	4.780 (0.006)	87401	0.9982	0.32%
6	0.2654 (0.0001)	1.106 (0.001)	4.881 (0.003)	85147	0.9985	0.30%

Appendix

Table A4: OLS estimates of the model

T_{T}=\beta_{0}+\alpha\bar{U}_{r}+cL

by budget

K

. Parentheses give clustered standard errors; each run has 641 cluster-generations. Under these standard errors, all reported coefficients have

p<0.001

; because of the large number of nested rounds, the emphasis is on effect sizes, intervals, and OOS error. OOS MAPE is measured on held-out generation groups across five folds.

Category	$\alpha$ at $K=8$	$c$ , $10^{-3}$	$N$	$R^{2}$
coding	0.2625 (0.0005)	1.110 (0.001)	47116	0.9991
code_math	0.2640 (0.0001)	1.096 (0.001)	326429	0.9993
qa	0.2643 (0.0004)	1.165 (0.010)	75787	0.9987
stem	0.2644 (0.0003)	1.094 (0.002)	42275	0.9986
math_reasoning	0.2644 (0.0005)	1.109 (0.003)	57788	0.9989
translation	0.2647 (0.0005)	1.190 (0.041)	16594	0.9991

Appendix

Table A5: Model by category, sorted by

\alpha

. Parentheses give clustered standard errors. Coefficients are close across all domains:

\alpha\in[0.262,0.266]

ms/expert and

c\in[1.05,1.19]\times 10^{-3}

ms/token.

Figure A3: Model coefficients by category: the slope

\alpha

K=8

with 95% CI, and the error on held-out generations. The dashed line shows the estimate on the full pool.

Figure A4: Six dimensionless quality metrics by category:

1-R^{2}

, MedAPE, MAPE, nMAE, nRMSE, and P95 APE. Typical error is 0.3–0.4%, and the 95th percentile does not exceed 1%.

Model	$R^{2}$	CV RMSE, ms	CV MAPE
$\bar{U}_{r}+L$	0.856	0.685	4.42%
$\bar{U}_{r}+L+\gamma K$	0.861	0.675	4.33%
$\bar{U}_{r}\times C(K)+L$	0.999	0.056	0.37%

Appendix

Table A6: Budget as a linear coefficient versus the categorical form on the full pool; CV uses GroupKFold(5) by generation.

Figure A5: The pooled model with a linear coefficient on $K$ , at the median $L$ of each run. The red line shows the pooled model; the gray dashed lines show the separate per- $K$ fits. The pooled line reproduces the large- $m$ regime but is systematically off at $m\leq 4$ .

Figure A6: Draft time versus budget and the model $T_{D}=0.459K+0.073$ ms. Each point corresponds to a round; points on the zero line represent records with a missing timing.

Check	$\alpha$ at $K=8$ , ms/expert	$c$ , $10^{-3}$ ms/token	$\beta_{0}$ , ms
OLS, clustered SE	0.2643 [0.2642, 0.2645]	1.1022	4.938
Bootstrap by generation, 999	0.2643 [0.2642, 0.2645]	1.1022	4.938
Huber regression	0.2644	1.1013	4.937
1% outlier trimming	0.2644	1.1008	4.936
First half of run	0.2642	1.1003	4.942
Second half of run	0.2650	1.1046	4.919

Appendix

Table A7: Model coefficients under robustness checks on the

K=8

subsample. For OLS and bootstrap, 95% intervals are given; for leave-one-category-out, the range across fourteen refits.

Figure A7: Estimates of

\alpha

K=8

and

c

under robustness checks. The dashed line marks the baseline OLS estimate; the entire spread stays within a fraction of a percent.

Figure B1: A regular fragment and two major boundaries. In the examples shown, a regular pattern coincides with a long

k^{\star}

, while a structural or language transition coincides with the stopping point.

Figure B2: Three limitations of the simple boundary hypothesis. A token mismatch by itself does not imply a sharp change in routing, and the general label “linguistic boundary” does not determine the cost of a transition.

Figure B3: Context-wise distributions of the chosen length. On the left, the shares of

k^{\star}=1

and

k^{\star}\geq 4

; on the right, the mean

k^{\star}

by boundary type. Intervals are a 95% clustered bootstrap at the response level and reflect between-response variability.

Figure B4: Four cases where expert cost shifts the boundary. In all cases, the full objective chooses

k^{\star}

, while without expert cost the neighboring length

k^{\star}+1

becomes cheaper. The local increase in new experts serves as a diagnostic, not a standalone causal test.

Figure B5: New experts at script change and at the first draft mismatch. The left pair compares positions without a script change to Latin

\leftrightarrow

Cyrillic transitions; the right pair compares the position before a mismatch with the first mismatch position. A higher value means the continuation brings in more previously unused experts. Intervals are a 95% clustered bootstrap at the response level.

Figure B6: Four selected eight-token matches from different genres and two structural boundaries.

Figure B7: Two major boundaries and four counterexamples. Neither a formal boundary, nor a full match, nor a semantic mismatch alone determines the Oracle’s decision.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 311 papers

MoE-Spec: Expert Budgeting for Efficient Speculative Decoding
Feb 17, 2026Bradley McDanel, Steven Li, Sruthikesh Surineni +1Speculative Decoding LLM Inference Acceleration
Franklin and Marshall College · Meta Reality Labs
Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
Jul 14, 2026Jincheng Xie, Runheng Liu, Heyan Huang +4Mixture-of-Experts Inference Speculative Decoding
Department of Mathematical Sciences, Tsinghua University, Beijing, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JDT AI Infra +2
AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding
Aug 4, 2026Shuang Liang, Hao Mark Chen, Zhiwen Mo +4Mixture-of-Experts Inference Speculative Decoding
Imperial College London · Tile-AI

Feb 17, 2026J/K move · Enter open · S save

MoE-Spec: Expert Budgeting for Efficient Speculative Decoding

Bradley McDanel, Steven Li, Sruthikesh Surineni, Harshit Khaitan

Franklin and Marshall College · Meta Reality Labs

Speculative Decoding LLM Inference Acceleration Mixture-of-Experts Inference

Open paper AskPDF arXiv

Speculative decoding accelerates Large Language Model (LLM) inference by verifying multiple drafted tokens in parallel. However, for Mixture-of-Experts (MoE) models, this parallelism introduces a severe bottleneck: large draft trees activate many unique experts, significantly increasing memory pressure and diminishing speedups from speculative decoding relative to autoregressive decoding. Prior methods reduce speculation depth when MoE verification becomes expensive. We propose MoE-Spec, a training-free verification-time expert budgeting method that decouples speculation depth from memory cost by enforcing a fixed expert capacity limit at each layer, loading only the experts that contribute most to verification and dropping the long tail of rarely used experts that drive bandwidth overhead. Experiments across multiple model scales and datasets show that this method yields 10--30% higher throughput than state-of-the-art speculative decoding baselines (EAGLE-3) at comparable quality, with flexibility to trade accuracy for further latency reductions through tighter budgets.

Abstract

Speculative Decoding

Explore similar work

MoE-Spec: Expert Budgeting for Efficient Speculative Decoding

Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding