Speculative decoding (SD) accelerates large language model (LLM) inference, but its optimal configuration depends jointly on workload characteristics, GPU hardware, and available memory. For example, on the RTX5090, the state-of-the-art speculative drafter DFlash achieves 3.50× the throughput of autoregressive (AR) decoding at low concurrency, but only 0.76× at high concurrency with short outputs. We present SpecSearch, a hardware-aware search engine that jointly optimizes decoding strategy, draft length, and precision across workload--GPU pairs. Its analytical cost model captures memory and compute constraints to identify feasible configurations and narrow the search space for throughput optimization. Guided by this model, calibration measurements are collected from multiple GPUs. Using the measurements, a Jev-based predictor learns to select high-throughput decoding configurations on held-out GPUs without requiring deployment-GPU calibration. The predictor is parameter-efficiently trained on calibration data, while adapting a frozen Qwen3-4B using LoRA adapters and a linear correction layer. Across NVIDIA RTX5090, H100, L40 and AMD MI300X GPUs, SpecSearch with full calibration achieves an average of 2.01× the throughput of AR and 1.36× the throughput of SD, both served by vLLM. The proposed Jev-based predictor closely matches this search performance with 1.30× throughput improvement over SD with vLLM, while eliminating the need for held-out GPU calibration. Additionally, it retains 97.36% of the throughput achieved by SpecSearch that utilizes full calibration.
Figures & tables
Fig. 1: Overview of SpecSearch with full calibration : Given the GPU and the workload (1), the proposed cost model discards deployment configurations that exceed the memory budget and determines the critical batch size B⋆ of the device (2). Calibration on the deployment GPU runs AR and SD where quantization and optimal draft lengths k are explored within vLLM (3). SpecSearch explores dynamic draft-length mapping for vLLM that sets k from the effective batch size at every decoding step (4). The most profitable configuration for each workload is frozen into a policy lookup table to be reused during real-time inference (5).
Fig. 2: The best decoding configuration and its memory cost depend on the workload. (a) vLLM SD-to-AR throughput ratio with 256-token outputs and fp16 KV; the dashed line marks the critical batch size B⋆ predicted by SpecSearch cost model (Section IV ). (b) With longer outputs (2,048 tokens), vLLM SD with fp8 KV is fastest from concurrency 32 to 128 . (c) With shorter outputs (up to 256 tokens), vLLM SD has 3.5 – 3.9× throughput at concurrency 1 – 4 , but its advantage over AR disappears by concurrency 64 . (d) The drafter use reduces KV cache capacity to 0.73× that of AR at equal KV precision.
TABLE II: Validation of the cost model across policies. Predicted (pred.) and measured (meas.) denote cost-model predictions for memory footprint and realized allocation, respectively. Additionally, GPU memory budgets are emulated, ranging from 9 to 15.7 GiB, to determine deployment feasibility. Thus, each cell contains two symbols: the first is the cost model prediction for feasibility and the second is the vLLM deployment outcome. A ✓ means feasible (runs successfully), × means out-of-memory (fails).
MI300X
RTX 5090
vLLM AR
vLLM SD
SpecSearch
Improvement vs.
AR
vLLM SD
SpecSearch
Improvement vs.
ℓ
B
fp16
fp8
fp16
fp8
fp16
fp8
best AR
best vLLM SD
fp16
fp8
fp16
fp8
fp16
fp8
best AR
best SD
256
8
1.88
1.90
4.26
4.55
4.32
4.69
2.47×
1.03×
1.02
1.01
3.91
3.89
3.91
3.84
3.83×
1.00×
32
5.15
5.75
7.18
8.14
8.38
8.88
1.54×
1.09×
3.65
3.67
5.13
5.19
7.36
7.54
2.01×
1.42×
64
8.21
9.34
8.45
8.93
10.88
11.91
1.28×
1.33×
6.37
6.65
5.44
5.53
9.46
9.31
1.42×
1.71×
128
11.98
13.91
9.38
10.13
14.99
16.24
1.17×
1.60×
10.01
10.76
5.67
5.78
10.48
10.89
1.00×
1.86×
TABLE III: Throughput (k tokens/s) of decoding policies at 15.7 GiB device budget. Bold : highest mean value per workload. Underlined : the policy selected by SpecSearch. Within each method (AR, vLLM SD, SpecSearch), the darker shading marks fp8 KV in contrast to fp16 KV cache.
MI300X
RTX 5090
vLLM AR
vLLM SD
SpecSearch
Improvement vs.
AR
Baseline SD
SpecSearch
Improvement vs.
ℓ
B
fp16
fp8
fp16
fp8
fp16
fp8
best AR
best vLLM SD
fp16
fp8
fp16
fp8
fp16
fp8
best AR
best SD
256
8
1.00 ×
1.01 ×
2.27 ×
2.42 ×
2.30 ×
2.50 ×
2.46×
1.03×
1.00 ×
0.99 ×
3.84 ×
3.82 ×
3.84 ×
3.77 ×
3.84×
1.00×
32
1.00 ×
1.12 ×
1.39 ×
1.58 ×
1.62 ×
1.72 ×
1.54×
1.09×
1.00 ×
1.01 ×
1.41 ×
1.43 ×
2.02 ×
2.06 ×
2.00×
1.41×
64
1.00 ×
1.13 ×
1.03 ×
1.08 ×
1.32 ×
1.45 ×
1.28×
1.35×
1.00 ×
1.05 ×
0.85 ×
0.87 ×
1.49 ×
1.46 ×
1.42×
1.71×
128
1.00 ×
1.16 ×
0.78 ×
0.85 ×
1.25 ×
1.36 ×
1.17×
1.60×
1.00 ×
1.08 ×
0.57 ×
0.58 ×
1.05 ×
1.09 ×
1.00×
1.87×
TABLE IV: Speedup of decoding policies at 15.7 GiB device budget. Bold : highest mean value per workload. Underlined : the policy selected by SpecSearch. Within each method (AR, vLLM SD, SpecSearch), the darker shading marks fp8 KV in contrast to fp16 KV cache.
Fig. 3: Throughput (in k token/s) comparison of MagicDec vs. SpecSearch for Qwen-3-4B. Each cell reports MagicDec throughput on RTX5090, while its ratio to SpecSearch is given in parentheses. Gray cells denote out-of-memory (OOM) failures.
ℓ
B=8
B=32
B=64
B=128
MI300X
256
k=7 , fp8
k=3 , fp8
k=3 , fp8
k=3 , fp8
1,024
k=7 , fp8
k=3 , fp8
k=3 , fp8
k=3 , fp8
2,048
k† , fp8
k=7 , fp8
k=7 , fp8
k=3 , fp8
RTX 5090
256
k=15 , fp16
k=3 , fp16
k=3 , fp16
AR, fp8
1,024
k=15 , fp16
k† , fp16
k=3 , fp8
k=3 , fp8
2,048
k=15 , fp16
k† , fp8
k† , fp8
k=3 , fp8
TABLE V: Policy selected by SpecSearch in Table III and IV : AR vs. SD decoding (with draft length k ) and KV precision. k† : dynamic draft-length mapping for SD (Table VI ).
GPU
ℓ
1–20
21–48
49–96
97–128
MI300X
256
7/7
7/3
3/3
3/3
1,024
15/15
7/3
7/3
3/3
2,048
15/15
7/7
7/3
7/3
RTX 5090
256
15/15
3/3
3/3
3/3
1,024
15/15
3/3
3/3
3/3
2,048
15/15
7/7
3/3
3/3
TABLE VI: Dynamic draft-length mappings of SpecSearch. Each entry reports the selected draft length k for fp16/fp8 KV-cache precision, respectively, at the indicated effective-batch range.
policy
target
draft
KV cache
weights (GiB)
KV tokens
max conc.
ktok/s
wall (s)
speedup
AR
fp16
–
fp16
7.56
AR
fp16
–
fp8
7.59
AR
fp8
–
fp16
4.24
19,712
4.81×
8.617
5.942
0.63×
AR
fp8
–
fp8
4.24
39,568
9.66×
13.661
3.748
1.00×
SD
fp16
fp16
fp16
8.61
SD
fp16
fp16
fp8
8.65
TABLE VII: Quantization explored under an 8 GiB memory budget on the RTX 5090. Shaded rows indicate policies that exceed the device memory budget. Target and draft denote the target and draft model precisions, respectively. ℓ=4,096 .
batch size
drafter
weights
acceptance
throughput
(GiB)
τˉ
retention
k tok/s
retention
32
fp16
1.01
9.87
1.0
8.98
1.0×
fp8
0.51
9.90
1.0
9.16
1.0×
int4
0.26
9.81
0.9
9.00
1.0×
64
fp16
1.01
9.81
1.0
8.63
1.0×
fp8
0.51
10.03
1.0
9.43
1.0×
TABLE VIII: Drafter quantization on the RTX 5090. τˉ is the mean acceptance length of draft tokens across datasets. Retention is the rate at which mean acceptance and throughput are preserved relative to fp16.
Fig. 4: AR decoding: fp8-to-fp16 KV throughput ratio versus generated tokens per request, with concurrency 48 .
RTX 5090
MI300X
policy
capacity (tokens)
max conc. (queries)
capacity (tokens)
max conc. (queries)
AR, fp16 KV
54,800
13.4×
49,632
12.1×
AR, fp8 KV
109,168
26.7×
99,264
24.2×
vLLM SD, k=15 , fp16 KV
40,000
9.8×
33,808
8.3×
vLLM SD, k=15 , fp8 KV
79,600
19.4×
67,632
16.5×
Concurrency retention rate
0.73
0.68
TABLE X: Relationship between KV cache capacity, in tokens, and maximum concurrency. The workload has ℓ=4,096 across runs.
Improvement vs.
ℓ
B
AR
vLLM SD
SpecSearch
best AR
vLLM SD
256
8
0.86
3.04
2.69
3.14×
0.89×
32
2.97
6.42
6.27
2.11×
0.98×
64
4.99
7.77
8.74
1.75×
1.12×
128
7.48
8.49
9.98
1.33×
1.18×
1,024
8
0.83
2.88
2.87
3.47×
1.00×
TABLE XI: Throughput (k tokens/s) with target model quantization (fp8) on the MI300X. SpecSearch uses a dynamic draft-length mapping with the fp8 target. The improvement columns report SpecSearch throughput relative to the vLLM baseline. Bold : highest mean throughput per workload.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
H100
L40
vLLM AR
vLLM SD
SpecSearch
Improvement vs.
AR
vLLM SD
SpecSearch
Improvement vs.
ℓ
B
fp16
fp8
fp16
fp8
fp16
fp8
best AR
best SD
fp16
fp8
fp16
fp8
fp16
fp8
best AR
best SD
256
8
1.20
1.20
4.19
4.16
4.19
4.18
3.48×
1.00×
0.61
0.62
1.85
1.76
1.85
1.82
2.94×
0.99×
32
3.88
3.87
7.46
7.51
9.53
9.56
2.47×
1.27×
1.97
2.05
2.16
2.37
3.65
3.65
1.78×
1.54×
64
6.90
6.83
8.33
8.21
12.57
12.45
1.80×
1.49×
3.34
3.54
2.42
2.46
4.23
4.37
1.23×
1.77×
128
11.57
11.74
9.15
9.03
14.20
13.96
1.21×
1.55×
5.06
5.44
2.57
2.61
4.88
4.93
1.00×
2.09×
Appendix
TABLE XII: Throughput (k tokens/s) of decoding policies at 15.7 GiB device budget. Bold : highest mean value per workload. Underlined : the policy selected by SpecSearch. Within each method (AR, vLLM SD, SpecSearch), the darker shading marks fp8 KV in contrast to fp16 KV cache.
H100
L40
vLLM AR
vLLM SD
SpecSearch
Improvement vs.
AR
Baseline SD
SpecSearch
Improvement vs.
ℓ
B
fp16
fp8
fp16
fp8
fp16
fp8
best AR
best SD
fp16
fp8
fp16
fp8
fp16
fp8
best AR
best SD
256
8
1.00 ×
1.00 ×
3.49 ×
3.48 ×
3.49 ×
3.49 ×
3.49×
1.00×
1.00 ×
1.01 ×
3.02 ×
2.87 ×
3.02 ×
2.97 ×
2.94×
0.98×
32
1.00 ×
1.00 ×
1.92 ×
1.94 ×
2.46 ×
2.47 ×
2.47×
1.27×
1.00 ×
1.04 ×
1.09 ×
1.20 ×
1.85 ×
1.85 ×
1.78×
1.54×
64
1.00 ×
0.99 ×
1.21 ×
1.19 ×
1.83 ×
1.81 ×
1.81×
1.49×
1.00 ×
1.06 ×
0.73 ×
0.74 ×
1.27 ×
1.31 ×
1.23×
1.78×
128
1.00 ×
1.01 ×
0.79 ×
0.78 ×
1.23 ×
1.21 ×
1.21×
1.55×
1.00 ×
1.08 ×
0.51 ×
0.51 ×
0.97 ×
0.98 ×
1.00×
2.09×
Appendix
TABLE XIII: Speedup of decoding policies at 15.7 GiB device budget. Bold : highest mean value per workload. Underlined : the policy selected by SpecSearch. Within each method (AR, vLLM SD, SpecSearch), the darker shading marks fp8 KV in contrast to fp16 KV cache.
k=3
k=7
k=15
ℓ
B
fp16
fp8
fp16
fp8
fp16
fp8
256
8
3.49
3.75
4.32
4.63
4.26
4.55
32
8.15
8.90
8.35
9.12
7.18
8.14
64
10.69
11.30
10.09
11.28
8.45
8.93
128
13.15
14.21
12.35
12.53
9.38
10.13
1,024
8
2.94
3.47
3.68
4.31
3.58
4.18
Appendix
TABLE XIV: MI300X throughput (k tokens/s) for every static draft length and KV-cache precision at the 15.7 GiB budget. Bold : highest mean throughput at that workload.
k=3
k=7
k=15
ℓ
B
fp16
fp8
fp16
fp8
fp16
fp8
256
8
1.86×
2.01×
2.31×
2.47 ×
2.27×
2.42×
32
1.52×
1.67×
1.56×
1.70 ×
1.39×
1.58×
64
1.34×
1.41 ×
1.26×
1.41 ×
1.03×
1.08×
128
1.21×
1.31 ×
1.13×
1.15×
0.78×
0.85×
1,024
8
1.64×
1.93×
2.05×
2.41 ×
1.99×
2.34×
Appendix
TABLE XV: MI300X wall-clock speedup over AR decoding with fp16 KV for every static draft length and KV-cache precision. Bold : highest mean speedup at that workload.
Method
GPU
ℓ
B=8
B=32
B=64
B=128
IDW
MI300X
256
k=7 , fp8
k=7 , fp8
k=7 , fp8
k=3 , fp8
1,024
k=7 , fp8
k=7 , fp8
k=7 , fp8
k=3 , fp8
2,048
k=7 , fp8
k=7 , fp8
k=7 , fp8
k=3 , fp8
IDW
RTX 5090
256
k=15 , fp16
k=3 , fp8
k=3 , fp8
AR, fp8
1,024
k=15 , fp8
k=3 , fp8
k=3 , fp8
k=3 , fp8
2,048
k=15 , fp8
k=3 , fp8
k=3 , fp8
k=3 , fp8
Appendix
TABLE XVI: Policies selected by IDW and source-tuned Jev at each workload: AR vs. vLLM SD and KV precision.
Method
GPU
ℓ
1–20
21–48
49–96
97–128
IDW
MI300X
256
7/7
7/7
7/7
3/3
1,024
7/7
7/7
7/7
3/3
2,048
7/7
7/7
7/7
3/3
IDW
RTX 5090
256
15/15
3/3
3/3
3/3
1,024
15/15
3/3
3/3
3/3
2,048
15/15
3/3
3/3
3/3
Appendix
TABLE XVII: Predicted dynamic draft-length mappings of IDW and source-tuned Jev. Each entry reports k for fp16/fp8 KV, respectively, at the indicated effective-batch range.
Speculative decoding is a technique that uses multiple language models to accelerate infer- ence. Previous works have used an experi- mental approach to optimize the throughput of the inference pipeline, which involves LLM training and can be costly. This study of spec- ulative decoding proposes a theory that ana- lytically connects the key hyperparameters of pre-trained LLMs to the throughput efficiency of a downstream SD-based inference system. The theory allows the prediction of throughput- optimal hyperparameters for the components of an inference system before their pre-training.
Amirhossein Bozorgkhoo, Igor Molybog
Independent Researcher · University of Hawai’i at Manoa, Honolulu, HI, USA.
Autoregressive decoding is bottlenecked by its sequential nature. Speculative decoding has become a standard way to accelerate inference by using a fast draft model to predict upcoming tokens from a slower target model, and then verifying them in parallel with a single target model forward pass. However, speculative decoding itself relies on a sequential dependence between speculation and verification. We introduce speculative speculative decoding (SSD) to parallelize these operations. While a verification is ongoing, the draft model predicts likely verification outcomes and prepares speculations pre-emptively for them. If the actual verification outcome is then in the predicted set, a speculation can be returned immediately, eliminating drafting overhead entirely. We identify three key challenges presented by speculative speculative decoding, and suggest principled methods to solve each. The result is Saguaro, an optimized SSD algorithm. Our implementation is on average 30% faster than optimized speculative decoding baselines and up to 5x faster than autoregressive decoding with open source inference engines.
Tanishq Kumar, Tri Dao, Avner May
Department of Computer Science, Stanford University · Department of Computer Science, Princeton University · Together AI
Speculative decoding accelerates memory-bound LLM decoding with a lightweight drafter, and parallel drafters extend this speedup by drafting a whole block in one forward pass. However, it remains unclear whether their released systems (stock) attain the full speedup that parallel drafting permits. We identify three challenges. First, with one request, each step accepts few tokens but reads every target weight. Second, with batching, per-request verification work roughly halves the speedup over autoregressive (AR) decoding. Third, speculative decoding is deployment-specific, and no universal optimization fits every serving regime, drafter, target model and GPU. To address these challenges, we present SpecPilot, an autonomous agentic loop that discovers algorithms and systems optimizations for each deployment. In each round, a cost model locates the largest gaps between the current system and its ceiling. Parallel agents explore directions and integrate the successful ones. Token forcing checks every output token against stock to detect reward hacking. A structured memory carries validated mechanisms to new rounds and deployments. SpecPilot discovered a general lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification and three batching mechanisms. We evaluate it across four target models, three drafters, two GPU platforms and five benchmarks. Without training or quality loss, it raises the speedup over AR from stock's 4.46× to 10.87× with one request, and from 1.94× to 3.93× with batching.
Kunjun Li, Wenhao Chai, Yihao Liang +7
University of California, Berkeley · Princeton University · Stanford University +1