cs.CLFeb 25, 2026

Speculative Decoding Scaling Laws (SDSL): Throughput Optimization Made Simple

Authors: Amirhossein Bozorgkhoo, Igor Molybog

Organizations: Independent Researcher · University of Hawai’i at Manoa, Honolulu, HI, USA.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Speculative decoding is a technique that uses multiple language models to accelerate infer- ence. Previous works have used an experi- mental approach to optimize the throughput of the inference pipeline, which involves LLM training and can be costly. This study of spec- ulative decoding proposes a theory that ana- lytically connects the key hyperparameters of pre-trained LLMs to the throughput efficiency of a downstream SD-based inference system. The theory allows the prediction of throughput- optimal hyperparameters for the components of an inference system before their pre-training.

Figures & tables

 Draft ModelOPT-125MOPT-350MOPT-1.3BOPT-2.7B
Target ModelPerplexity29.7931861925.3279933919.3687686917.36415863
OPT-13B15.584537510.5959±0.0018{0.5959}_{\pm 0.0018}0.6281±0.0017{0.6281}_{\pm 0.0017}0.6694±0.0016{0.6694}_{\pm 0.0016}0.6793±0.0016{0.6793}_{\pm 0.0016}
OPT-30B15.302268980.6779±0.0017{0.6779}_{\pm 0.0017}0.6984±0.0016{0.6984}_{\pm 0.0016}0.7325±0.0015{0.7325}_{\pm 0.0015}0.7447±0.0015{0.7447}_{\pm 0.0015}
OPT-66B14.580856320.7118±0.0016{0.7118}_{\pm 0.0016}0.7342±0.0015{0.7342}_{\pm 0.0015}0.7659±0.0014{0.7659}_{\pm 0.0014}0.7847±0.0014{0.7847}_{\pm 0.0014}
Qwen1.5-14B11.73123550.599±0.0013{0.599}_{\pm 0.0013}0.6178±0.0012{0.6178}_{\pm 0.0012}0.6569±0.0012{0.6569}_{\pm 0.0012}0.6708±0.0012{0.6708}_{\pm 0.0012}
Qwen1.5-32B10.416081430.5842±0.0013{0.5842}_{\pm 0.0013}0.6086±0.0013{0.6086}_{\pm 0.0013}0.6458±0.0012{0.6458}_{\pm 0.0012}0.6617±0.0012{0.6617}_{\pm 0.0012}
Table 1: Estimated α\alpha values along with their associated 95% confidence intervals (associated with stochasticity in evaluation and not training) and measured perplexity for selected pairs of all target models and OPT draft models.
 Draft ModelQwen2.5-0.5BQwen2.5-1.5BQwen2.5-3BQwen1.5-0.5BQwen1.5-1.8BQwen1.5-4B
Target ModelPerplexity17.9146442413.6949129112.588389418.3721809414.1000719113.90737915
OPT-13B15.584537510.6516±0.0017{0.6516}_{\pm 0.0017}0.6678±0.0017{0.6678}_{\pm 0.0017}0.6725±0.0016{0.6725}_{\pm 0.0016}0.649±0.0017{0.649}_{\pm 0.0017}0.6672±0.0017{0.6672}_{\pm 0.0017}0.6716±0.0017{0.6716}_{\pm 0.0017}
OPT-30B15.302268980.7282±0.0016{0.7282}_{\pm 0.0016}0.7414±0.0015{0.7414}_{\pm 0.0015}0.749±0.0015{0.749}_{\pm 0.0015}0.7241±0.0016{0.7241}_{\pm 0.0016}0.7372±0.0015{0.7372}_{\pm 0.0015}0.7483±0.0015{0.7483}_{\pm 0.0015}
OPT-66B14.580856320.7678±0.0014{0.7678}_{\pm 0.0014}0.7881±0.0013{0.7881}_{\pm 0.0013}0.7882±0.0013{0.7882}_{\pm 0.0013}0.7634±0.0013{0.7634}_{\pm 0.0013}0.7808±0.0014{0.7808}_{\pm 0.0014}0.7875±0.0014{0.7875}_{\pm 0.0014}
Qwen1.5-14B11.73123550.6898±0.0011{0.6898}_{\pm 0.0011}0.7244±0.0011{0.7244}_{\pm 0.0011}0.7323±0.0011{0.7323}_{\pm 0.0011}0.6838±0.0011{0.6838}_{\pm 0.0011}0.7131±0.0011{0.7131}_{\pm 0.0011}0.7343±0.0011{0.7343}_{\pm 0.0011}
Qwen1.5-32B10.416081430.6698±0.0012{0.6698}_{\pm 0.0012}0.7069±0.0011{0.7069}_{\pm 0.0011}0.7152±0.0011{0.7152}_{\pm 0.0011}0.6585±0.0012{0.6585}_{\pm 0.0012}0.6873±0.0012{0.6873}_{\pm 0.0012}0.7084±0.0011{0.7084}_{\pm 0.0011}
Table 2: Estimated α\alpha values along with their associated 95% confidence intervals (associated with stochasticity in evaluation and not training) and measured perplexity for selected pairs of all target models and Qwen draft models.
Figure 1: Visualization of the fitted affine plane relating the estimated scaling parameter α\alpha to draft model perplexity ( xx ) and target model perplexity ( yy ). Scatter points represent empirical (x,y,α)(x,y,\alpha) observations from Tables 1 and 2 , while the surface corresponds to the least-squares fit of Equation 5 .
Figure 2: Throughput (tokens per FLOP) predicted by Equation ( 9 ) as a function of draft model size NN (in billions of parameters) for different target models and draft model families based on Table 5 . Each curve corresponds to a single target model and is annotated directly on the curve. Black markers indicate the draft model sizes used in experiments, while star markers denote the optimal draft size N∗N^{\ast} that maximizes predicted throughput for each target model.
Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Curves show estimated α\alpha values as a function of draft model perplexity, based on data from Tables 1 and 2 . The α\alpha values were computed based on the method outlined in Appendix B , which estimates them using accepted token statistics. Function details appear in Table 4 .
Figure 4: Latency metrics (TTFT, TTOT, and TPOT) for the OPT-13B target model plotted as a function of the normalized deviation ∣N−N∗∣/M|N-N^{\ast}|/M from the analytically predicted optimal draft size. Each point corresponds to an individual draft model from the OPT, Qwen1.5, or Qwen2.5 families, with error bars indicating 95% confidence intervals across prompts.The vertical dashed line marks the predicted optimum N∗N^{\ast} .Across all metrics and draft families, latency increases with distance from N∗N^{\ast} , supporting the accuracy of the throughput-based optimal draft size prediction.
ParameterEstimateStd. ErrorMargin of ErrorCI Lower (95%)CI Upper (95%)
A−0.0067-0.00670.0006070.000607±0.001201\pm 0.001201−0.007901-0.007901−0.005499-0.005499
B0.0129710.0129710.0015450.001545±0.003056\pm 0.0030560.0099140.0099140.0160270.016027
C0.6420840.6420840.0212280.021228±0.042006\pm 0.0420060.6000780.6000780.6840900.684090
Model Performance
Mean Squared Error (MSE)0.0012840.001284
R2R^{2}0.6022960.602296
Appendix
Table 3: Model coefficients for Equation 5 , including standard errors, margins of error, and 95% confidence intervals, along with overall model performance metrics (mean squared error and R2R^{2} ).
Target ModelFunctionParam 1MoE 1CI 1Param 2MoE 2CI 2MSER2R^{2}
 Linear-0.0082620.001653[-0.009915, -0.006609]0.8329530.031375[0.801578, 0.864329]0.0001130.943203
Qwen1.5-14BLogarithmic-0.1652060.026300[-0.191506, -0.138905]1.1558500.075721[1.080129, 1.231571]0.0000730.963273
 Power-law1.3827820.158956[1.223826, 1.541738]-0.2471680.040405[-0.287573, -0.206763]0.0000730.963076
 Linear-0.0076860.001206[-0.008892, -0.006480]0.8048570.022886[0.781971, 0.827743]0.0000600.964297
Qwen1.5-32BLogarithmic-0.1534410.017053[-0.170494, -0.136388]1.1045420.049097[1.055444, 1.153639]0.0000310.981757
 Power-law1.3023980.100870[1.201528, 1.403268]-0.2353150.027208[-0.262523, -0.208108]0.0000320.981142
Appendix
Table 4: Complete regression results for linear, logarithmic, and power-law fits of α\alpha as a function of draft model perplexity, including parameter estimates, margins of error (MoE), 95% confidence intervals (CI), mean squared error (MSE), and coefficient of determination ( R2R^{2} ).
Target ModelM (target size)D′D^{\prime} (training target size)Draft FamilyDD (training draft size)N∗N^{\ast} (Optimal N)T(N∗)\mathcal{T}(N^{\ast})
   OPT180B124039820.38.947e-11
Qwen1.5-14B141672908803TQwen1.52.4T115390938.29.165e-11
   Qwen2.518T112685392.49.235e-11
   OPT180B198605884.74.061e-11
Qwen1.5-32B325122181123TQwen1.52.4T184494249.24.154e-11
   Qwen2.518T180081907.14.183e-11
Appendix
Table 5: Let optimal draft model characteristics for each target model. For each target, we report the model size MM , training dataset size DD , and the optimal draft model size NN that maximizes decoding throughput. The last column reports the achieved throughput in tokens per FLOP for each target model in the measured optimal draft model. This table provides the empirical foundation for our scaling law analysis, illustrating how optimal draft configurations vary with model scale and data.
Draft ModelDraft FamilyN∗N^{\ast} (Optimal N)∣N−N∗∣/M|N-N^{\ast}|/MTTFT (s)TTOT (s)TPOT (s)
OPT-125MOPT117M0.0006170.04864±0.00170.04864\pm 0.00172.514±0.20422.514\pm 0.20420.01006±0.000820.01006\pm 0.00082
OPT-350MOPT117M0.016640.050003±0.001260.050003\pm 0.001263.686±0.209723.686\pm 0.209720.01474±0.000840.01474\pm 0.00084
OPT-1.3BOPT117M0.093240.04779±0.001180.04779\pm 0.001183.507±0.15933.507\pm 0.15930.01403±0.000640.01403\pm 0.00064
OPT-2.7BOPT117M0.19720.04762±0.001010.04762\pm 0.001014.387±0.18584.387\pm 0.18580.01755±0.000740.01755\pm 0.00074
Qwen1.5-0.5BQwen1.5109M0.02760.1001±0.008160.1001\pm 0.0081633.18±8.92533.18\pm 8.9250.1327±0.03570.1327\pm 0.0357
Qwen1.5-1.8BQwen1.5109M0.13440.1008±0.011200.1008\pm 0.0112034.49±8.81734.49\pm 8.8170.138±0.035270.138\pm 0.03527
Appendix
Table 6: Latency-based validation of the analytically predicted optimal draft size N∗N^{\ast} for speculative decoding with an OPT-13B target model. For each evaluated draft model, we report the predicted optimum N∗N^{\ast} , the normalized distance ∣N−N∗∣/M|N-N^{\ast}|/M , and empirical latency measurements obtained from speculative decoding on 50 HellaSwag prompts. Latency metrics include time-to-first-token (TTFT), total generation time (TTOT), and time per output token (TPOT), reported as mean ±\pm margin of error corresponding to 95% confidence intervals.
ParameterRangeNumber of Grid PointsDescription
NN10810^{8} to 101010^{10}10000Initial draft model size
MM13×10913\times 10^{9} to 110×109110\times 10^{9}8Target model size
DD101210^{12} to 101310^{13}6Number of tokens used in draft model training
D′D\prime101210^{12} to 101310^{13}6Number of training tokens used in target model training
Appendix
Table 7: Ranges for NN , DD , and MM used in the analysis.
ParameterEstimateMoE (95%)95% CI (lower)95% CI (upper)
μ\mu4.89×10−34.89\times 10^{-3}7.64×10−47.64\times 10^{-4}4.12×10−34.12\times 10^{-3}5.65×10−35.65\times 10^{-3}
M0M_{0}7.23×1077.23\times 10^{7}1.46×1061.46\times 10^{6}7.09×1077.09\times 10^{7}7.38×1077.38\times 10^{7}
γ\gamma−6.07×10−5-6.07\times 10^{-5}1.79×10−51.79\times 10^{-5}−7.86×10−5-7.86\times 10^{-5}−4.28×10−5-4.28\times 10^{-5}
γ′\gamma^{\prime}1.16×10−61.16\times 10^{-6}1.78×10−51.78\times 10^{-5}−1.67×10−5-1.67\times 10^{-5}1.90×10−51.90\times 10^{-5}
R2R^{2}0.9780.978
Adj. R2R^{2}0.9780.978
Appendix
Table 8: Scaling-law regression results for the throughput-optimal draft size. The model N\*/M=μ+M0/M+γlog⁡D+γ′log⁡D′N^{\*}/M=\mu+M_{0}/M+\gamma\log D+\gamma^{\prime}\log D^{\prime} is fit using ordinary least squares with heteroskedasticity-robust (HC3) standard errors. Margins of error (MoE) correspond to half-widths of the reported 95% confidence intervals.
ParameterEstimateMoE (95%)95% CI (lower)95% CI (upper)
μ\mu2.71×10−32.71\times 10^{-3}4.38×10−54.38\times 10^{-5}2.67×10−32.67\times 10^{-3}2.75×10−32.75\times 10^{-3}
M0M_{0}8.71×1078.71\times 10^{7}2.18×1062.18\times 10^{6}8.50×1078.50\times 10^{7}8.93×1078.93\times 10^{7}
R2R^{2}0.98650.9865
Observations288288
Number of Grid Points∣M∣=8,  ∣D∣=6,  ∣D′∣=6|M|=8,\;|D|=6,\;|D^{\prime}|=6
Regression typeOLS with HC3 robust standard errors
Appendix
Table 9: Pooled regression results for the leading-order scaling law N\*=μM+M0N^{\*}=\mu M+M_{0} . The regression is performed over all (M,D,D′)(M,D,D^{\prime}) configurations, treating dataset-induced variation as part of the residual. Margins of error (MoE) correspond to half-widths of the reported 95% confidence intervals.
Figure 5: Numerical characterization of the throughput-optimal draft model size N∗N^{\ast} as a function of the target model size MM , the draft training dataset size DD , and the target training dataset size D′D^{\prime} . Each row presents the raw dependence of N∗N^{\ast} (left) alongside the corresponding normalized view N∗/MN^{\ast}/M (right). The results show that the dominant scaling of the optimal draft size is approximately linear in the target model size, while the effects of the training dataset sizes act as weaker, second-order corrections.
More in this topic

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 309 papers

Explore similar work

CardsList

  1. Speculative Speculative Decoding

    Mar 3, 2026Tanishq Kumar, Tri Dao, Avner MaySpeculative DecodingLLM Inference Acceleration

    Department of Computer Science, Stanford University · Department of Computer Science, Princeton University · Together AI

  2. MineDraft: A Framework for Batch Parallel Speculative Decoding

    Feb 24, 2026Zhenwei Tang, Arun Verma, Zijian Zhou +4Speculative DecodingLLM Inference Acceleration

    College of Computing and Data Science, Nanyang Technological University, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre · Department of Computer Science, National University of Singapore, Republic of Singapore +1

  3. Speculative Pipeline Decoding: Higher-Accruacy and Zero-Bubble Speculation via Pipeline Parallelism

    May 29, 2026Yijiong Yu, Huazheng Wang, Shuai Yuan +2Speculative DecodingLLM Inference Acceleration

    aOregon State University · bDeepSolution