cs.AINov 17, 2026

SpecSearch: Towards Hardware-Aware Efficient Speculative Decoding

Authors: Selin Yildirim, Ali Zeyrek, Deming Chen

Organizations: University of Illinois Urbana-Champaign, IL, USA

Abstract

Speculative decoding (SD) accelerates large language model (LLM) inference, but its optimal configuration depends jointly on workload characteristics, GPU hardware, and available memory. For example, on the RTX5090, the state-of-the-art speculative drafter DFlash achieves 3.50×3.50\times the throughput of autoregressive (AR) decoding at low concurrency, but only 0.76×0.76\times at high concurrency with short outputs. We present SpecSearch, a hardware-aware search engine that jointly optimizes decoding strategy, draft length, and precision across workload--GPU pairs. Its analytical cost model captures memory and compute constraints to identify feasible configurations and narrow the search space for throughput optimization. Guided by this model, calibration measurements are collected from multiple GPUs. Using the measurements, a Jev-based predictor learns to select high-throughput decoding configurations on held-out GPUs without requiring deployment-GPU calibration. The predictor is parameter-efficiently trained on calibration data, while adapting a frozen Qwen3-4B using LoRA adapters and a linear correction layer. Across NVIDIA RTX5090, H100, L40 and AMD MI300X GPUs, SpecSearch with full calibration achieves an average of 2.01×2.01\times the throughput of AR and 1.36×1.36\times the throughput of SD, both served by vLLM. The proposed Jev-based predictor closely matches this search performance with 1.30×1.30\times throughput improvement over SD with vLLM, while eliminating the need for held-out GPU calibration. Additionally, it retains 97.36%97.36\% of the throughput achieved by SpecSearch that utilizes full calibration.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Speculative Decoding Scaling Laws (SDSL): Throughput Optimization Made Simple

    Feb 25, 2026Amirhossein Bozorgkhoo, Igor MolybogSpeculative DecodingLLM Inference Acceleration

  2. Speculative Speculative Decoding

    Mar 3, 2026Tanishq Kumar, Tri Dao, Avner MaySpeculative DecodingLLM Inference Acceleration

  3. SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding

    Nov 2, 2026Kunjun Li, Wenhao Chai, Yihao Liang +7Speculative DecodingLLM Inference Acceleration