Speculative Decoding Scaling Laws (SDSL): Throughput Optimization Made Simple

Amirhossein Bozorgkhoo; Igor Molybog

cs.CLFeb 25, 2026

Speculative Decoding Scaling Laws (SDSL): Throughput Optimization Made Simple

Authors: Amirhossein Bozorgkhoo, Igor Molybog

Organizations: Independent Researcher · University of Hawai’i at Manoa, Honolulu, HI, USA.

Abstract

Speculative decoding is a technique that uses multiple language models to accelerate infer- ence. Previous works have used an experi- mental approach to optimize the throughput of the inference pipeline, which involves LLM training and can be costly. This study of spec- ulative decoding proposes a theory that ana- lytically connects the key hyperparameters of pre-trained LLMs to the throughput efficiency of a downstream SD-based inference system. The theory allows the prediction of throughput- optimal hyperparameters for the components of an inference system before their pre-training.

Figures & tables

	Draft Model	OPT-125M	OPT-350M	OPT-1.3B	OPT-2.7B
Target Model	Perplexity	29.79318619	25.32799339	19.36876869	17.36415863
OPT-13B	15.58453751	${0.5959}_{\pm 0.0018}$	${0.6281}_{\pm 0.0017}$	${0.6694}_{\pm 0.0016}$	${0.6793}_{\pm 0.0016}$
OPT-30B	15.30226898	${0.6779}_{\pm 0.0017}$	${0.6984}_{\pm 0.0016}$	${0.7325}_{\pm 0.0015}$	${0.7447}_{\pm 0.0015}$
OPT-66B	14.58085632	${0.7118}_{\pm 0.0016}$	${0.7342}_{\pm 0.0015}$	${0.7659}_{\pm 0.0014}$	${0.7847}_{\pm 0.0014}$
Qwen1.5-14B	11.7312355	${0.599}_{\pm 0.0013}$	${0.6178}_{\pm 0.0012}$	${0.6569}_{\pm 0.0012}$	${0.6708}_{\pm 0.0012}$
Qwen1.5-32B	10.41608143	${0.5842}_{\pm 0.0013}$	${0.6086}_{\pm 0.0013}$	${0.6458}_{\pm 0.0012}$	${0.6617}_{\pm 0.0012}$

Table 1: Estimated

\alpha

values along with their associated 95% confidence intervals (associated with stochasticity in evaluation and not training) and measured perplexity for selected pairs of all target models and OPT draft models.

	Draft Model	Qwen2.5-0.5B	Qwen2.5-1.5B	Qwen2.5-3B	Qwen1.5-0.5B	Qwen1.5-1.8B	Qwen1.5-4B
Target Model	Perplexity	17.91464424	13.69491291	12.5883894	18.37218094	14.10007191	13.90737915
OPT-13B	15.58453751	${0.6516}_{\pm 0.0017}$	${0.6678}_{\pm 0.0017}$	${0.6725}_{\pm 0.0016}$	${0.649}_{\pm 0.0017}$	${0.6672}_{\pm 0.0017}$	${0.6716}_{\pm 0.0017}$
OPT-30B	15.30226898	${0.7282}_{\pm 0.0016}$	${0.7414}_{\pm 0.0015}$	${0.749}_{\pm 0.0015}$	${0.7241}_{\pm 0.0016}$	${0.7372}_{\pm 0.0015}$	${0.7483}_{\pm 0.0015}$
OPT-66B	14.58085632	${0.7678}_{\pm 0.0014}$	${0.7881}_{\pm 0.0013}$	${0.7882}_{\pm 0.0013}$	${0.7634}_{\pm 0.0013}$	${0.7808}_{\pm 0.0014}$	${0.7875}_{\pm 0.0014}$
Qwen1.5-14B	11.7312355	${0.6898}_{\pm 0.0011}$	${0.7244}_{\pm 0.0011}$	${0.7323}_{\pm 0.0011}$	${0.6838}_{\pm 0.0011}$	${0.7131}_{\pm 0.0011}$	${0.7343}_{\pm 0.0011}$
Qwen1.5-32B	10.41608143	${0.6698}_{\pm 0.0012}$	${0.7069}_{\pm 0.0011}$	${0.7152}_{\pm 0.0011}$	${0.6585}_{\pm 0.0012}$	${0.6873}_{\pm 0.0012}$	${0.7084}_{\pm 0.0011}$

Table 2: Estimated

\alpha

Figure 1: Visualization of the fitted affine plane relating the estimated scaling parameter $\alpha$ to draft model perplexity ( $x$ ) and target model perplexity ( $y$ ). Scatter points represent empirical $(x,y,\alpha)$ observations from Tables 1 and 2 , while the surface corresponds to the least-squares fit of Equation 5 .

Panel of Figure 2 — Figure 2: Throughput (tokens per FLOP) predicted by Equation ( 9 ) as a function of draft model size $N$ (in billions of parameters) for different target models and draft model families based on Table 5 . Each curve corresponds to a single target model and is annotated directly on the curve. Black markers indicate the draft model sizes used in experiments, while star markers denote the optimal draft size $N^{\ast}$ that maximizes predicted throughput for each target model.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Curves show estimated $\alpha$ values as a function of draft model perplexity, based on data from Tables 1 and 2 . The $\alpha$ values were computed based on the method outlined in Appendix B , which estimates them using accepted token statistics. Function details appear in Table 4 .

Panel of Figure 4 — Figure 4: Latency metrics (TTFT, TTOT, and TPOT) for the OPT-13B target model plotted as a function of the normalized deviation $|N-N^{\ast}|/M$ from the analytically predicted optimal draft size. Each point corresponds to an individual draft model from the OPT, Qwen1.5, or Qwen2.5 families, with error bars indicating 95% confidence intervals across prompts.The vertical dashed line marks the predicted optimum $N^{\ast}$ .Across all metrics and draft families, latency increases with distance from $N^{\ast}$ , supporting the accuracy of the throughput-based optimal draft size prediction.

Parameter	Estimate	Std. Error	Margin of Error	CI Lower (95%)	CI Upper (95%)
A	$-0.0067$	$0.000607$	$\pm 0.001201$	$-0.007901$	$-0.005499$
B	$0.012971$	$0.001545$	$\pm 0.003056$	$0.009914$	$0.016027$
C	$0.642084$	$0.021228$	$\pm 0.042006$	$0.600078$	$0.684090$
Model Performance
Mean Squared Error (MSE)			$0.001284$
$R^{2}$			$0.602296$

Appendix

Table 3: Model coefficients for Equation 5 , including standard errors, margins of error, and 95% confidence intervals, along with overall model performance metrics (mean squared error and

R^{2}

Target Model	Function	Param 1	MoE 1	CI 1	Param 2	MoE 2	CI 2	MSE	$R^{2}$
	Linear	-0.008262	0.001653	[-0.009915, -0.006609]	0.832953	0.031375	[0.801578, 0.864329]	0.000113	0.943203
Qwen1.5-14B	Logarithmic	-0.165206	0.026300	[-0.191506, -0.138905]	1.155850	0.075721	[1.080129, 1.231571]	0.000073	0.963273
	Power-law	1.382782	0.158956	[1.223826, 1.541738]	-0.247168	0.040405	[-0.287573, -0.206763]	0.000073	0.963076
	Linear	-0.007686	0.001206	[-0.008892, -0.006480]	0.804857	0.022886	[0.781971, 0.827743]	0.000060	0.964297
Qwen1.5-32B	Logarithmic	-0.153441	0.017053	[-0.170494, -0.136388]	1.104542	0.049097	[1.055444, 1.153639]	0.000031	0.981757
	Power-law	1.302398	0.100870	[1.201528, 1.403268]	-0.235315	0.027208	[-0.262523, -0.208108]	0.000032	0.981142

Appendix

Table 4: Complete regression results for linear, logarithmic, and power-law fits of

\alpha

as a function of draft model perplexity, including parameter estimates, margins of error (MoE), 95% confidence intervals (CI), mean squared error (MSE), and coefficient of determination (

R^{2}

Target Model	M (target size)	$D^{\prime}$ (training target size)	Draft Family	$D$ (training draft size)	$N^{\ast}$ (Optimal N)	$\mathcal{T}(N^{\ast})$
			OPT	180B	124039820.3	8.947e-11
Qwen1.5-14B	14167290880	3T	Qwen1.5	2.4T	115390938.2	9.165e-11
			Qwen2.5	18T	112685392.4	9.235e-11
			OPT	180B	198605884.7	4.061e-11
Qwen1.5-32B	32512218112	3T	Qwen1.5	2.4T	184494249.2	4.154e-11
			Qwen2.5	18T	180081907.1	4.183e-11

Appendix

Table 5: Let optimal draft model characteristics for each target model. For each target, we report the model size

M

, training dataset size

D

, and the optimal draft model size

N

that maximizes decoding throughput. The last column reports the achieved throughput in tokens per FLOP for each target model in the measured optimal draft model. This table provides the empirical foundation for our scaling law analysis, illustrating how optimal draft configurations vary with model scale and data.

Draft Model	Draft Family	$N^{\ast}$ (Optimal N)	$\|N-N^{\ast}\|/M$	TTFT (s)	TTOT (s)	TPOT (s)
OPT-125M	OPT	117M	0.000617	$0.04864\pm 0.0017$	$2.514\pm 0.2042$	$0.01006\pm 0.00082$
OPT-350M	OPT	117M	0.01664	$0.050003\pm 0.00126$	$3.686\pm 0.20972$	$0.01474\pm 0.00084$
OPT-1.3B	OPT	117M	0.09324	$0.04779\pm 0.00118$	$3.507\pm 0.1593$	$0.01403\pm 0.00064$
OPT-2.7B	OPT	117M	0.1972	$0.04762\pm 0.00101$	$4.387\pm 0.1858$	$0.01755\pm 0.00074$
Qwen1.5-0.5B	Qwen1.5	109M	0.0276	$0.1001\pm 0.00816$	$33.18\pm 8.925$	$0.1327\pm 0.0357$
Qwen1.5-1.8B	Qwen1.5	109M	0.1344	$0.1008\pm 0.01120$	$34.49\pm 8.817$	$0.138\pm 0.03527$

Appendix

Table 6: Latency-based validation of the analytically predicted optimal draft size

N^{\ast}

for speculative decoding with an OPT-13B target model. For each evaluated draft model, we report the predicted optimum

N^{\ast}

, the normalized distance

|N-N^{\ast}|/M

, and empirical latency measurements obtained from speculative decoding on 50 HellaSwag prompts. Latency metrics include time-to-first-token (TTFT), total generation time (TTOT), and time per output token (TPOT), reported as mean

\pm

margin of error corresponding to 95% confidence intervals.

Parameter	Range	Number of Grid Points	Description
$N$	$10^{8}$ to $10^{10}$	10000	Initial draft model size
$M$	$13\times 10^{9}$ to $110\times 10^{9}$	8	Target model size
$D$	$10^{12}$ to $10^{13}$	6	Number of tokens used in draft model training
$D\prime$	$10^{12}$ to $10^{13}$	6	Number of training tokens used in target model training

Appendix

Table 7: Ranges for

N

D

, and

M

used in the analysis.

Parameter	Estimate	MoE (95%)	95% CI (lower)	95% CI (upper)
$\mu$	$4.89\times 10^{-3}$	$7.64\times 10^{-4}$	$4.12\times 10^{-3}$	$5.65\times 10^{-3}$
$M_{0}$	$7.23\times 10^{7}$	$1.46\times 10^{6}$	$7.09\times 10^{7}$	$7.38\times 10^{7}$
$\gamma$	$-6.07\times 10^{-5}$	$1.79\times 10^{-5}$	$-7.86\times 10^{-5}$	$-4.28\times 10^{-5}$
$\gamma^{\prime}$	$1.16\times 10^{-6}$	$1.78\times 10^{-5}$	$-1.67\times 10^{-5}$	$1.90\times 10^{-5}$
$R^{2}$	$0.978$
Adj. $R^{2}$	$0.978$

Appendix

Table 8: Scaling-law regression results for the throughput-optimal draft size. The model

N^{\*}/M=\mu+M_{0}/M+\gamma\log D+\gamma^{\prime}\log D^{\prime}

is fit using ordinary least squares with heteroskedasticity-robust (HC3) standard errors. Margins of error (MoE) correspond to half-widths of the reported 95% confidence intervals.

Parameter	Estimate	MoE (95%)	95% CI (lower)	95% CI (upper)
$\mu$	$2.71\times 10^{-3}$	$4.38\times 10^{-5}$	$2.67\times 10^{-3}$	$2.75\times 10^{-3}$
$M_{0}$	$8.71\times 10^{7}$	$2.18\times 10^{6}$	$8.50\times 10^{7}$	$8.93\times 10^{7}$
$R^{2}$	$0.9865$
Observations	$288$
Number of Grid Points	$\|M\|=8,\;\|D\|=6,\;\|D^{\prime}\|=6$
Regression type	OLS with HC3 robust standard errors

Appendix

Table 9: Pooled regression results for the leading-order scaling law

N^{\*}=\mu M+M_{0}

. The regression is performed over all

(M,D,D^{\prime})

configurations, treating dataset-induced variation as part of the residual. Margins of error (MoE) correspond to half-widths of the reported 95% confidence intervals.

Panel of Figure 5 — Figure 5: Numerical characterization of the throughput-optimal draft model size $N^{\ast}$ as a function of the target model size $M$ , the draft training dataset size $D$ , and the target training dataset size $D^{\prime}$ . Each row presents the raw dependence of $N^{\ast}$ (left) alongside the corresponding normalized view $N^{\ast}/M$ (right). The results show that the dominant scaling of the optimal draft size is approximately linear in the target model size, while the effects of the training dataset sizes act as weaker, second-order corrections.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 309 papers

Mar 3, 2026

Speculative Speculative Decoding

Autoregressive decoding is bottlenecked by its sequential nature. Speculative decoding has become a standard way to accelerate inference by using a fast draft model to predict upcoming tokens from a slower target model, and then verifying them in parallel with a single target model forward pass. However, speculative decoding itself relies on a sequential dependence between speculation and verification. We introduce speculative speculative decoding (SSD) to parallelize these operations. While a verification is ongoing, the draft model predicts likely verification outcomes and prepares speculations pre-emptively for them. If the actual verification outcome is then in the predicted set, a speculation can be returned immediately, eliminating drafting overhead entirely. We identify three key challenges presented by speculative speculative decoding, and suggest principled methods to solve each. The result is Saguaro, an optimized SSD algorithm. Our implementation is on average 30% faster than optimized speculative decoding baselines and up to 5x faster than autoregressive decoding with open source inference engines.

Tanishq Kumar, Tri Dao, Avner May

Department of Computer Science, Stanford University · Department of Computer Science, Princeton University · Together AI

Speculative Decoding LLM Inference Acceleration Parallel Speculative Decoding Autoregressive Decoding

Ask PDF

Feb 24, 2026

MineDraft: A Framework for Batch Parallel Speculative Decoding

Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to propose draft tokens that are subsequently verified by a larger target model. However, the performance of standard SD is often limited by the strictly sequential execution of these drafting and verification stages. To address this, this paper proposes MineDraft, a batch parallel speculative decoding (PSD) framework designed to effectively hide drafting latency by overlapping it with verification. Our theoretical analysis shows that PSD is substantially more efficient than standard SD. MineDraft realizes the PSD through a novel batch-parallel design that maintains two batches of requests, overlapping drafting for one batch with verification for the other. Our experimental results show significant improvements of \alg{} in both throughput (up to 75%) and end-to-end latency (up to 39%) over standard SD. Furthermore, we have implemented MineDraft as a plugin for vLLM, demonstrating its practicality for production-ready inference systems.

Zhenwei Tang, Arun Verma, Zijian Zhou +4

College of Computing and Data Science, Nanyang Technological University, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre · Department of Computer Science, National University of Singapore, Republic of Singapore +1

Speculative Decoding LLM Inference Acceleration Parallel Speculative Decoding Parallel LLM Inference

Ask PDF

May 29, 2026

Speculative Pipeline Decoding: Higher-Accruacy and Zero-Bubble Speculation via Pipeline Parallelism

Speculative Decoding (SD) accelerates low-concurrency LLM inference by employing a draft-then-verify paradigm. However, mainstream methods typically rely on multi-token prediction, which introduces escalating prediction difficulty and serial drafting latency. To address these, we propose Speculative Pipeline Decoding (SPD), a groundbreaking framework that unlocks the true potential of pipeline parallelism. By partitioning the target LLM into

n

pipeline stages, SPD allows LLM to process

n

tokens within single sequence in parallel to accelerate decoding. To continuous fill the pipeline in single sequence decoding, a speculation module aggregates intermediate features across different pipeline depths to predict the next token, executing strictly in parallel with the target model's pipeline step, to realize bounded difficulty, higher acceptance rates, and zero latency bubbles. Our experiments demonstrate that SPD achieves significantly higher theoretical and wall-clock speedup compared to mainstream baselines at moderate pipeline depth, though more aggressive settings require further improvement. Our code is available at https://github.com/yuyijiong/speculative_pipeline_decoding

Yijiong Yu, Huazheng Wang, Shuai Yuan +2

aOregon State University · bDeepSolution

Speculative Decoding LLM Inference Acceleration Parallel Decoding Pipeline Parallelism Parallel Speculative Decoding

Ask PDF

Abstract

Speculative Decoding

Explore similar work

Speculative Speculative Decoding

MineDraft: A Framework for Batch Parallel Speculative Decoding

Speculative Pipeline Decoding: Higher-Accruacy and Zero-Bubble Speculation via Pipeline Parallelism