The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts

Aidar Amankulov; Denis Mamatin

cs.LGAug 26, 2026

The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts

Authors: Aidar Amankulov, Denis Mamatin

Abstract

Speculative decoding in Mixture-of-Experts (MoE) models faces the problem of unstable verification cost caused by input-dependent expert loading. To study the physics of this process, we formulate speculation-budget selection as an offline Stochastic Shortest Path (SSP) problem over reference sequences and build a diagnostic Oracle that uses counterfactual simulation to account for MoE verification cost. A detailed analysis of the Oracle's decisions on the Qwen3-Coder and EAGLE-3 pairing, in the space of marginal deltas (Delta Space), shows that rejected candidates form a strict linear boundary. This result demonstrates that a complex global optimization is locally governed by a necessary condition balancing marginal cost against expected progress ( $\frac{Δ\mathbb{E}[Cost]}{Δ\mathbb{E}[a]}$ ), providing a rigorous mathematical reference point for designing future adaptive online heuristics.

Figures & tables

Figure 1: $T_{T,r}$ against $\bar{U}_{r}$ for each $K$ , with the fitted line and coefficients of model ( 4 ). The extended robustness protocol is given in Appendix A .

Figure 2: Quantitative bounds of the MoE Stochastic Oracle under varying speculation budgets

L_{max}

. As

L_{max}

increases, Theoretical Maximum Speedup (

S_{max}

), Mean

k

, and Mean Accepted Length exhibit diminishing returns and saturation. The Oracle selectively caps

k

to avoid costly MoE expert activations on low-probability draft tokens.

Figure 3: Delta Space Analysis ( $\Delta\mathbb{E}[\text{Cost}]$ vs. $\Delta\mathbb{E}[a]$ ) of rejected draft candidates relative to the Oracle’s choice $(0,0)$ . Quadrants II and IV are strictly empty, proving strict monotonicity. Candidates in Quadrant I (risk zone) and Quadrant III (caution zone) form a linear boundary passing through the origin.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Target	Qwen/Qwen3-Coder-30B-A3B-Instruct; MoE with 128 experts and top- $k_{e}=8$ routing
Draft head	lmsys/SGLang-EAGLE3-Qwen3-Coder-30B-A3B-Instruct-SpecForge
GPU and stack	NVIDIA A100 80GB; vLLM fork with per-round trace; FULL CUDA graphs
Mode	$B=1$ , synchronous scheduling
Sampling	$T=0.7$ , top_p $=1.0$ , seed $=0$
Other	bfloat16, max_model_len $=4096$ , gpu_mem_util $=0.85$ , max_new_tokens $=1000$

Appendix

Table 4

$K$	$m$	Rounds	Median $T_{T}$ , ms	Median $\bar{U}_{r}$	Median $L$
1	2	127410	8.98	12.1	605
2	3	106110	10.81	15.5	606
3	4	96819	12.63	18.2	605
4	5	90813	10.94	20.6	605
5	6	87401	11.51	22.6	600
6	7	85147	12.07	24.5	591

Appendix

Table A1: Dataset summary by run. Median

L

ranges between 586 and 606 tokens. The non-monotonicity of

T_{T}

K=3\rightarrow 4

coincides with a change in the estimated

\beta_{0}

; without a randomized repeat sweep it cannot be causally attributed to a CUDA-graph mode.

Factor	What it measures	Mechanistic hypothesis
$\bar{U}_{r}$	Number of unique experts per pass, averaged across layers	Expert-weight traffic from HBM
$L$	Total context, tokens	KV-cache read and attention
$m$	Number of positions in the verification pass	Microbatch and CUDA-graph size
$E_{\max}$	Maximum number of tokens per single expert	Load imbalance
KV usage	KV-cache occupancy	Allocator pressure
Graph bucket	Size of the captured graph	Padding; at $B=1$ the bucket is equivalent to $m$

Appendix

Table 6

Factor	OOS $R^{2}$ , alone	OOS RMSE, ms	$\Delta$ RMSE beyond $\bar{U}_{r}$ , ms	corr with $\bar{U}_{r}$
$\bar{U}_{r}$	$0.787\pm 0.026$	$0.831\pm 0.037$	—	1.00
$m=K+1$	$0.515\pm 0.033$	$1.256\pm 0.020$	$+0.014\pm 0.003$	0.86
Graph bucket	$0.515\pm 0.033$	$1.256\pm 0.020$	$+0.014\pm 0.003$	0.86
$E_{\max}$	$0.362\pm 0.029$	$1.442\pm 0.014$	$+0.015\pm 0.002$	0.75
KV usage	$0.077\pm 0.017$	$1.734\pm 0.051$	$+0.139\pm 0.007$	0.03
$L$	$0.079\pm 0.016$	$1.732\pm 0.051$	$+0.146\pm 0.008$	0.03

Appendix

Table A2: Descriptive predictive importance of factors on the pool of

925\,752

rounds; intervals are 95% CIs across folds.

\bar{U}_{r}

is the strongest single predictor, and

L

gives the largest additional error reduction on top of it.

Category	Rounds	OOS $R^{2}$ : $\bar{U}_{r}$	OOS $R^{2}$ : $\bar{U}_{r}+L$	$\Delta$ RMSE from $L$ , ms	Final $R^{2}$
code_math	326429	$0.870\pm 0.023$	$0.903\pm 0.018$	$+0.098\pm 0.007$	0.9993
extraction	5839	$0.861\pm 0.097$	$0.864\pm 0.090$	$+0.005\pm 0.019$	0.9991
math_reasoning	57788	$0.843\pm 0.039$	$0.851\pm 0.033$	$+0.015\pm 0.014$	0.9990
coding	47116	$0.809\pm 0.079$	$0.904\pm 0.064$	$+0.281\pm 0.081$	0.9991
translation	16594	$0.797\pm 0.023$	$0.795\pm 0.022$	$-0.002\pm 0.008$	0.9991
rag	63141	$0.766\pm 0.011$	$0.774\pm 0.007$	$+0.012\pm 0.024$	0.9982

Appendix

Table A3: Factor importance by category, sorted by the OOS

R^{2}

of the

\bar{U}_{r}

-only model. The summary row corresponds to the full pool.

Figure A1: Out-of-sample

R^{2}

for predicting

T_{T}

by category:

\bar{U}_{r}

alone versus

\bar{U}_{r}+L

Figure A2: Round scatter and fitted line for each of the ten values of $K$ on shared axes. As $K$ grows, the cloud shifts to the right, and at the $m=4/5$ boundary the slope increases roughly threefold.

$K$	$\alpha$ , ms/expert	$c$ , $10^{-3}$ ms/token	$\beta_{0}$ , ms	$N$	$R^{2}$	OOS MAPE
1	0.0839 (0.0003)	1.098 (0.001)	7.294 (0.003)	127410	0.9896	0.44%
2	0.0824 (0.0003)	1.101 (0.002)	8.854 (0.005)	106110	0.9806	0.51%
3	0.0779 (0.0004)	1.106 (0.002)	10.532 (0.008)	96819	0.9669	0.59%
4	0.2675 (0.0002)	1.099 (0.001)	4.739 (0.003)	90813	0.9977	0.33%
5	0.2664 (0.0003)	1.103 (0.001)	4.780 (0.006)	87401	0.9982	0.32%
6	0.2654 (0.0001)	1.106 (0.001)	4.881 (0.003)	85147	0.9985	0.30%

Appendix

Table A4: OLS estimates of the model

T_{T}=\beta_{0}+\alpha\bar{U}_{r}+cL

by budget

K

. Parentheses give clustered standard errors; each run has 641 cluster-generations. Under these standard errors, all reported coefficients have

p<0.001

; because of the large number of nested rounds, the emphasis is on effect sizes, intervals, and OOS error. OOS MAPE is measured on held-out generation groups across five folds.

Category	$\alpha$ at $K=8$	$c$ , $10^{-3}$	$N$	$R^{2}$
coding	0.2625 (0.0005)	1.110 (0.001)	47116	0.9991
code_math	0.2640 (0.0001)	1.096 (0.001)	326429	0.9993
qa	0.2643 (0.0004)	1.165 (0.010)	75787	0.9987
stem	0.2644 (0.0003)	1.094 (0.002)	42275	0.9986
math_reasoning	0.2644 (0.0005)	1.109 (0.003)	57788	0.9989
translation	0.2647 (0.0005)	1.190 (0.041)	16594	0.9991

Appendix

Table A5: Model by category, sorted by

\alpha

. Parentheses give clustered standard errors. Coefficients are close across all domains:

\alpha\in[0.262,0.266]

ms/expert and

c\in[1.05,1.19]\times 10^{-3}

ms/token.

Figure A3: Model coefficients by category: the slope

\alpha

K=8

with 95% CI, and the error on held-out generations. The dashed line shows the estimate on the full pool.

Figure A4: Six dimensionless quality metrics by category:

1-R^{2}

, MedAPE, MAPE, nMAE, nRMSE, and P95 APE. Typical error is 0.3–0.4%, and the 95th percentile does not exceed 1%.

Model	$R^{2}$	CV RMSE, ms	CV MAPE
$\bar{U}_{r}+L$	0.856	0.685	4.42%
$\bar{U}_{r}+L+\gamma K$	0.861	0.675	4.33%
$\bar{U}_{r}\times C(K)+L$	0.999	0.056	0.37%

Appendix

Table A6: Budget as a linear coefficient versus the categorical form on the full pool; CV uses GroupKFold(5) by generation.

Figure A5: The pooled model with a linear coefficient on $K$ , at the median $L$ of each run. The red line shows the pooled model; the gray dashed lines show the separate per- $K$ fits. The pooled line reproduces the large- $m$ regime but is systematically off at $m\leq 4$ .

Figure A6: Draft time versus budget and the model $T_{D}=0.459K+0.073$ ms. Each point corresponds to a round; points on the zero line represent records with a missing timing.

Check	$\alpha$ at $K=8$ , ms/expert	$c$ , $10^{-3}$ ms/token	$\beta_{0}$ , ms
OLS, clustered SE	0.2643 [0.2642, 0.2645]	1.1022	4.938
Bootstrap by generation, 999	0.2643 [0.2642, 0.2645]	1.1022	4.938
Huber regression	0.2644	1.1013	4.937
1% outlier trimming	0.2644	1.1008	4.936
First half of run	0.2642	1.1003	4.942
Second half of run	0.2650	1.1046	4.919

Appendix

Table A7: Model coefficients under robustness checks on the

K=8

subsample. For OLS and bootstrap, 95% intervals are given; for leave-one-category-out, the range across fourteen refits.

Figure A7: Estimates of

\alpha

K=8

and

c

under robustness checks. The dashed line marks the baseline OLS estimate; the entire spread stays within a fraction of a percent.

Figure B1: A regular fragment and two major boundaries. In the examples shown, a regular pattern coincides with a long

k^{\star}

, while a structural or language transition coincides with the stopping point.

Figure B2: Three limitations of the simple boundary hypothesis. A token mismatch by itself does not imply a sharp change in routing, and the general label “linguistic boundary” does not determine the cost of a transition.

Figure B3: Context-wise distributions of the chosen length. On the left, the shares of

k^{\star}=1

and

k^{\star}\geq 4

; on the right, the mean

k^{\star}

by boundary type. Intervals are a 95% clustered bootstrap at the response level and reflect between-response variability.

Figure B4: Four cases where expert cost shifts the boundary. In all cases, the full objective chooses

k^{\star}

, while without expert cost the neighboring length

k^{\star}+1

becomes cheaper. The local increase in new experts serves as a diagnostic, not a standalone causal test.

Figure B5: New experts at script change and at the first draft mismatch. The left pair compares positions without a script change to Latin

\leftrightarrow

Cyrillic transitions; the right pair compares the position before a mismatch with the first mismatch position. A higher value means the continuation brings in more previously unused experts. Intervals are a 95% clustered bootstrap at the response level.

Figure B6: Four selected eight-token matches from different genres and two structural boundaries.

Figure B7: Two major boundaries and four counterexamples. Neither a formal boundary, nor a full match, nor a semantic mismatch alone determines the Oracle’s decision.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 311 papers

Feb 17, 2026

MoE-Spec: Expert Budgeting for Efficient Speculative Decoding

Speculative decoding accelerates Large Language Model (LLM) inference by verifying multiple drafted tokens in parallel. However, for Mixture-of-Experts (MoE) models, this parallelism introduces a severe bottleneck: large draft trees activate many unique experts, significantly increasing memory pressure and diminishing speedups from speculative decoding relative to autoregressive decoding. Prior methods reduce speculation depth when MoE verification becomes expensive. We propose MoE-Spec, a training-free verification-time expert budgeting method that decouples speculation depth from memory cost by enforcing a fixed expert capacity limit at each layer, loading only the experts that contribute most to verification and dropping the long tail of rarely used experts that drive bandwidth overhead. Experiments across multiple model scales and datasets show that this method yields 10--30% higher throughput than state-of-the-art speculative decoding baselines (EAGLE-3) at comparable quality, with flexibility to trade accuracy for further latency reductions through tighter budgets.

Bradley McDanel, Steven Li, Sruthikesh Surineni +1

Franklin and Marshall College · Meta Reality Labs

Speculative Decoding LLM Inference Acceleration Mixture-of-Experts Inference

Ask PDF

Jul 14, 2026

Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

Sparse Mixture-of-Experts (MoE) models have become an important approach for scaling Large Language Models (LLMs), but their inference efficiency depends strongly on expert activation patterns. Speculative decoding (SD) accelerates autoregressive generation by verifying multiple draft tokens in parallel, yet existing draft selection strategies primarily optimize acceptance likelihood. In large-scale MoE models, however, selecting draft tokens also determines the union of experts activated during verification. We observe that confidence-driven SD can introduce expert scattering: high-probability draft tokens may route to disjoint experts, increasing expert-weight memory traffic and reducing the speedup from speculation. Motivated by this observation, we revisit draft-tree selection under the non-uniform memory-cost structure of MoE inference. We propose EcoSpec, a cost-aware speculative decoding framework that incorporates predicted marginal expert activation cost into draft selection. With a lightweight expert predictor and a dynamic expert buffer, EcoSpec favors draft paths that preserve high acceptance likelihood while reusing experts already covered by the current verification set, without modifying the target-model verification rule. We evaluate EcoSpec on three large-scale MoE models, including DeepSeek-V3.1 (671B), Qwen3-235B-A22B, and GPT-OSS-120B, across reasoning, coding, question-answering, and dialogue benchmarks. EcoSpec consistently reduces active expert footprints and improves end-to-end decoding speed, achieving up to

1.62\times

speedup. These results show that accounting for expert activation cost is important for efficient speculative decoding in large-scale MoE models.

Jincheng Xie, Runheng Liu, Heyan Huang +4

Department of Mathematical Sciences, Tsinghua University, Beijing, China · School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · JDT AI Infra +2

Mixture-of-Experts Inference Speculative Decoding Mixture-of-Experts Models

Ask PDF

Aug 4, 2026

AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding

Speculative decoding verifies a tree of draft tokens in one target-model forward pass. For a mixture-of-experts (MoE) target, however, parallel verification can activate the union of the experts selected by all tree nodes, even though only a small subset of those nodes reaches the accepted output. Token count, activated-expert union size, and expert-weight traffic are therefore distinct cost measures: reducing the token workload need not shrink the expert union proportionally, and under offloading, transfer traffic also depends on cache residency. We introduce AcceptMoE, a verifier-side expert selector that combines target-router scores with offline-estimated commitment probabilities and automatically adjusts the number of eligible experts for each verification block, eliminating the need for a user-specified expert budget. Under offloading, AcceptMoE conditions expert eligibility on cache residency instead of predicting natural routes and prefetching the corresponding expert weights. Although constraining target-expert eligibility changes the model distribution, across 12 model-task pairs spanning three MoE targets and four benchmarks, AcceptMoE's mean accuracy is 0.27 percentage points lower than that of EAGLE-3 speculative decoding with natural routing. Served with SGLang at batch size one, it reaches 1.290 times the throughput of this baseline with all expert weights in GPU memory, and 2.06 times under physical expert offloading, while reducing host-to-device traffic by 73.6 percent to 77.1 percent.

Shuang Liang, Hao Mark Chen, Zhiwen Mo +4

Imperial College London · Tile-AI

Mixture-of-Experts Inference Speculative Decoding

Ask PDF

Abstract

Speculative Decoding

Explore similar work

MoE-Spec: Expert Budgeting for Efficient Speculative Decoding

Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts

AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding