cs.ETOct 28, 2026

TableTune: Measured, Self-Correcting Draft-Length Tables for Dynamic Speculative Decoding

Authors: Abhishek Malvankar, Paul Castro, Nick Mitchell, Lionel Villard

Organizations: IBM Research.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Speculative decoding speeds up large language model (LLM) serving only while the GPU has spare compute, so the best draft length, the number of tokens a small drafter guesses per step, shrinks as the batch grows. vLLM lets operators set the draft length for each batch size with a hand-written table, but nothing measures or corrects that table. We present TableTune. Its tuner times the decoding step for every batch size and draft length on vLLM's own serving path and computes, in minutes, the table that maximizes throughput. Its controller, a vLLM plugin, tracks how many guesses are accepted in live traffic and rebuilds the table while serving. We measure TableTune only against other ways to speculate. On an H100, the tuned table delivers at least 94% of the throughput of the best fixed draft length at every load in four of five drafter settings (88% in the fifth) without the operator having to know which length that is. Against reimplementations of TurboSpec- and Nightjar-style control on the same engine, its worst-case loss is never higher than theirs, and with the EAGLE-3 drafter it is 3% against 9% and 12%. On a mixture-of-experts model, vLLM's documented table loses up to 18% and the tuned table at most 3%. Starting from mistuned tables, the controller raises throughput by up to 58% (up to 45% on vLLM's default runner), and after a workload switch with the DFlash drafter it recovers 96% of a stale table's loss. In llm-d's optimized-baseline deployment as shipped, two Qwen3-32B replicas on four GPUs, the tuned table is up to 32% faster than a fixed draft length of 3 and up to 25% faster than vLLM's documented table at high load, and the controller raises the documented table's throughput by up to 29%.

Figures & tables

Figure 1: No fixed draft length is best at every load. H100 (V2 runner), Qwen3-4B + EAGLE-3, code prompts at temperature 1.0, mean of three runs: throughput as % of the best fixed draft length at each load.
Figure 2: TableTune. The tuner measures step time on vLLM’s serving path and acceptance on sample prompts, and writes a table in vLLM’s own format. While serving, the controller plugin reads vLLM’s acceptance counters, re-estimates acceptance and rebuilds the table with the tuner’s step times.
 H100 V2H100 V1Older GPUs
SettingTableBest fixedTableBest fixedGPUTableBest fixed
EAGLE-3, code4.03.4 (2)39.53.1 (2)T4 / A10G4.0 / 2.39.6 (2) / 15.0 (1)
EAGLE-3, 4-bit AWQ target3.0 ‡4.6 (2)43.04.2 (2)T420.6 †9.2 (1) †
n-gram, code-apply5.7 §4.5 (5)5.20.0 (5)T42.87.6 (5)
Qwen3-0.6B draft model, coden/a0.0 (1)35.60.0 (1)T43.5 †12.2 (1) †
DFlash, code11.619.1 (3)29.520.4 (3)A10G5.218.0 (3)
Table 1: Worst loss (%) against the best fixed draft length at each load, over 1–128 open requests (Spec-Bench: 1–64), mean of three runs. Table is TableTune’s tuned table; Best fixed is the one fixed draft length with the smallest worst-case loss (length in brackets), which an operator could only find by benchmarking every length. Lower is better; bold marks the better of the two. H100 V2: full CUDA graphs per draft length; H100 V1: vLLM’s default runner, which switches to piecewise CUDA graphs whenever a table is set. Older GPUs: the same settings from our T4 and A10G study, each run on the GPU named (EAGLE-3 code on both). † single run. ‡ the 0/1/3 table , built from the tuner’s length-0, 1 and 3 profiles only (the all-length table , built from every profiled length, steps from length 2 to 1, which stalls this vLLM nightly; Section 6.6 ). § the tuned table is that single fixed length, so the two differ only by run-to-run noise. n/a: the tuner’s V2 profile of the draft model is invalid (Section 6.7 ).
Figure 3: TableTune against TurboSpec-style and Nightjar-style control on the H100 (V2 runner), code prompts at temperature 1.0, mean of three runs. Throughput as % of the best fixed draft length at each load; every method chooses among the lengths of its starting table: with EAGLE-3, 0/1/3 for TurboSpec- and Nightjar-style, 2/3 for the all-length tuned table and 1/3 for the 0/1/3 table (also the controller’s start); with DFlash, 2/4/6/8 for all.
Figure 4: The controller. (a) Repairing the same mistuned table on three GPUs: vLLM’s documented table with its ranges divided by 4 (length 0 above a batch of 32), started with a mismatched chat prior, EAGLE-3 on code. Lines: gain of the controller over the same table without it; hollow circles: gain of that GPU’s tuned table, the reference the controller should reach (no sweep point at 40). (b) A workload switch: DFlash on the H100 (V2), 96 open requests, code prompts, then math from the fourth 60-second window. Mean of three runs.
Starting tableH100 V2H100 V1Older GPUs
Documented table / 4 (Figure 4 a)+58% (40)+45% (64)T4 +36% (64), A10G +14% (40)
Documented table+2% (16)+8% (96)T4 +15% (64), A10G +16% (64)
n-gram, hand-written table+30% (4) †+22% (4) †T4 +24% (16)
Draft model, EAGLE-3’s table+40% (96) †§+2% †T4 +1% †
DFlash, EAGLE-3’s table+23% (64)+37% (64)A10G +11% (4)
4-bit AWQ, documented table+5% (64) †−- 9 to −- 24% †T4 +23% (64) †
Table 2: The controller: best gain over the same table without it, at the load where it occurs (loads 4–96; n-gram 4–128), mean of three runs unless marked ( † single run). Top: steady traffic, starting from a mistuned table. Bottom: a mid-run workload switch (Spec-Bench math and translation with EAGLE-3; DFlash: code then math at 96 requests; greedy, A10G protocol), the stale table’s loss and what the controller with exploration and prior decay does about it. Older GPUs: the same experiment from our T4 and A10G study, on the GPU named. § still 60% below no speculation, which beats every draft length of this drafter under V2.
Figure 5: llm-d optimized-baseline with EAGLE-3 behind the llm-d router, code prompts at temperature 1.0, mean of three runs: output tokens/s against fixed draft length 3 (0 = fixed length 3). (a) The guide’s own model and layout, 2 ×\times Qwen3-32B at TP=2 on four H100s. (b) 4 ×\times Qwen3-8B at TP=1 on the same GPUs; the 32-request dip is shared by every speculative configuration. The tuned table with the controller is within 2% of the table alone up to 512 requests in (b), and up to 4% above it in (a); it is omitted.
More in this topic

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,101 papers

Explore similar work

CardsList
May 10, 2026

Test-Time Speculation

Speculative decoding accelerates LLM inference by using a fast draft model to generate tokens and a more accurate target model to verify them. Its performance depends on the acceptance length\textit{acceptance length}, or number of draft tokens accepted by the target. Our studies show that the acceptance length of even state-of-the-art speculators, like DFlash, EAGLE-3 and PARD degrade with generation length, reaching values close to 1 (i.e. no speedup) within just a few thousand output tokens, making speculators ineffective for long-response tasks. Acceptance lengths decline because most speculators are trained offline on short sequences, but are forced to match the target model on much longer outputs at inference, well beyond their training distribution. To address this issue, we propose Test-Time Speculation (TTS)\textit{Test-Time Speculation (TTS)}, an online distillation approach that continuously adapts the speculator at test-time. TTS leverages the key insight that the token verification step already invokes the target model for each draft token, providing the training signal needed to adapt the draft at no additional cost. Treating the draft as the student and the target as a teacher, TTS adjusts the draft over several speculation rounds, with each update improving the draft's accuracy as generation proceeds. Our results across multiple models from the Qwen-3, Qwen-3.5, and Llama3.1 families show that TTS improves acceptance lengths over state-of-the-art speculators by up to 72%72\% and 41%41\% on average, with the benefits scaling with increased generation lengths.

Avinash Kumar, Sujay Sanghavi, Poulami Das

The University of Texas at Austin

Test-Time AdaptationTest-Time OptimizationSpeculative DecodingLLM Inference AccelerationLanguage Model Distillation
AskPDF
Feb 5, 2026

TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference

Speculative decoding can substantially accelerate LLM inference, but realizing its benefits in practice is challenging due to evolving workloads. We present TIDE (Temporal Incremental Draft Engine), a serving-engine-native framework that integrates online draft adaptation directly into high-performance LLM inference systems. TIDE reuses target model's intermediate hidden states generated during inference as training signals for draft adaptation, thereby avoiding additional target model computation and serving-time overhead. It employs adaptive runtime control to activate speculation and draft model training only when beneficial. TIDE exploits heterogeneous clusters by mapping inference and training to appropriate GPU classes. Across diverse real-world workloads, TIDE achieves up to 1.66×\times throughput over no-speculation baselines while recovering performance on misaligned workloads where static draft models degrade throughput. TIDE also reduces training time by up to 3.02×\times and storage requirements by 24×\times compared to existing draft training approaches, and improves system throughput by up to 1.22×\times on heterogeneous GPU clusters.

Jiyoung Park, Hankyu Jang, Changseok Song +1

Moreh, Inc.

Adaptive InferenceLLM Inference ServingSpeculative DecodingLLM Inference Acceleration
AskPDF
Mar 2, 2026

Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning

Speculative decoding accelerates large language model (LLM) inference by using a small draft model to generate candidate tokens for a larger target model to verify. The efficacy of this technique hinges on the trade-off between the time spent on drafting candidates and verifying them. However, current state-of-the-art methods rely on a static time allocation, while recent dynamic approaches optimize for proxy metrics like acceptance length, often neglecting the true time cost and treating the drafting and verification phases in isolation. To address these limitations, we introduce Learning to Draft (LTD), a novel method that directly optimizes for throughput of each draft-and-verify cycle. We formulate the problem as a reinforcement learning environment and train two co-adaptive policies to dynamically coordinate the draft and verification phases. This encourages the policies to adapt to each other and explicitly maximize decoding efficiency. We conducted extensive evaluations on five diverse LLMs and four distinct tasks. Our results show that LTD achieves speedup ratios ranging from 2.24x to 4.32x, outperforming the state-of-the-art method Eagle3 up to 36.4%.

Jiebin Zhang, Zhenghan Yu, Liang Wang +8

National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University · Microsoft Research Asia

Speculative DecodingLLM Inference Acceleration
AskPDF