TableTune: Measured, Self-Correcting Draft-Length Tables for Dynamic Speculative Decoding

Abhishek Malvankar; Paul Castro; Nick Mitchell; Lionel Villard

cs.ETOct 28, 2026

TableTune: Measured, Self-Correcting Draft-Length Tables for Dynamic Speculative Decoding

Authors: Abhishek Malvankar, Paul Castro, Nick Mitchell, Lionel Villard

Organizations: IBM Research.

Abstract

Speculative decoding speeds up large language model (LLM) serving only while the GPU has spare compute, so the best draft length, the number of tokens a small drafter guesses per step, shrinks as the batch grows. vLLM lets operators set the draft length for each batch size with a hand-written table, but nothing measures or corrects that table. We present TableTune. Its tuner times the decoding step for every batch size and draft length on vLLM's own serving path and computes, in minutes, the table that maximizes throughput. Its controller, a vLLM plugin, tracks how many guesses are accepted in live traffic and rebuilds the table while serving. We measure TableTune only against other ways to speculate. On an H100, the tuned table delivers at least 94% of the throughput of the best fixed draft length at every load in four of five drafter settings (88% in the fifth) without the operator having to know which length that is. Against reimplementations of TurboSpec- and Nightjar-style control on the same engine, its worst-case loss is never higher than theirs, and with the EAGLE-3 drafter it is 3% against 9% and 12%. On a mixture-of-experts model, vLLM's documented table loses up to 18% and the tuned table at most 3%. Starting from mistuned tables, the controller raises throughput by up to 58% (up to 45% on vLLM's default runner), and after a workload switch with the DFlash drafter it recovers 96% of a stale table's loss. In llm-d's optimized-baseline deployment as shipped, two Qwen3-32B replicas on four GPUs, the tuned table is up to 32% faster than a fixed draft length of 3 and up to 25% faster than vLLM's documented table at high load, and the controller raises the documented table's throughput by up to 29%.

Figures & tables

Figure 1: No fixed draft length is best at every load. H100 (V2 runner), Qwen3-4B + EAGLE-3, code prompts at temperature 1.0, mean of three runs: throughput as % of the best fixed draft length at each load.

Panel of Figure 2 — Figure 2: TableTune. The tuner measures step time on vLLM’s serving path and acceptance on sample prompts, and writes a table in vLLM’s own format. While serving, the controller plugin reads vLLM’s acceptance counters, re-estimates acceptance and rebuilds the table with the tuner’s step times.

Setting	Table	Best fixed	Table	Best fixed	GPU	Table	Best fixed
	H100 V2		H100 V1		Older GPUs
EAGLE-3, code	4.0	3.4 (2)	39.5	3.1 (2)	T4 / A10G	4.0 / 2.3	9.6 (2) / 15.0 (1)
EAGLE-3, 4-bit AWQ target	3.0 ‡	4.6 (2)	43.0	4.2 (2)	T4	20.6 †	9.2 (1) †
n-gram, code-apply	5.7 §	4.5 (5)	5.2	0.0 (5)	T4	2.8	7.6 (5)
Qwen3-0.6B draft model, code	n/a	0.0 (1)	35.6	0.0 (1)	T4	3.5 †	12.2 (1) †
DFlash, code	11.6	19.1 (3)	29.5	20.4 (3)	A10G	5.2	18.0 (3)

Table 1: Worst loss (%) against the best fixed draft length at each load, over 1–128 open requests (Spec-Bench: 1–64), mean of three runs. Table is TableTune’s tuned table; Best fixed is the one fixed draft length with the smallest worst-case loss (length in brackets), which an operator could only find by benchmarking every length. Lower is better; bold marks the better of the two. H100 V2: full CUDA graphs per draft length; H100 V1: vLLM’s default runner, which switches to piecewise CUDA graphs whenever a table is set. Older GPUs: the same settings from our T4 and A10G study, each run on the GPU named (EAGLE-3 code on both). † single run. ‡ the 0/1/3 table , built from the tuner’s length-0, 1 and 3 profiles only (the all-length table , built from every profiled length, steps from length 2 to 1, which stalls this vLLM nightly; Section 6.6 ). § the tuned table is that single fixed length, so the two differ only by run-to-run noise. n/a: the tuner’s V2 profile of the draft model is invalid (Section 6.7 ).

Figure 3: TableTune against TurboSpec-style and Nightjar-style control on the H100 (V2 runner), code prompts at temperature 1.0, mean of three runs. Throughput as % of the best fixed draft length at each load; every method chooses among the lengths of its starting table: with EAGLE-3, 0/1/3 for TurboSpec- and Nightjar-style, 2/3 for the all-length tuned table and 1/3 for the 0/1/3 table (also the controller’s start); with DFlash, 2/4/6/8 for all.

Figure 4: The controller. (a) Repairing the same mistuned table on three GPUs: vLLM’s documented table with its ranges divided by 4 (length 0 above a batch of 32), started with a mismatched chat prior, EAGLE-3 on code. Lines: gain of the controller over the same table without it; hollow circles: gain of that GPU’s tuned table, the reference the controller should reach (no sweep point at 40). (b) A workload switch: DFlash on the H100 (V2), 96 open requests, code prompts, then math from the fourth 60-second window. Mean of three runs.

Starting table	H100 V2	H100 V1	Older GPUs
Documented table / 4 (Figure 4 a)	+58% (40)	+45% (64)	T4 +36% (64), A10G +14% (40)
Documented table	+2% (16)	+8% (96)	T4 +15% (64), A10G +16% (64)
n-gram, hand-written table	+30% (4) †	+22% (4) †	T4 +24% (16)
Draft model, EAGLE-3’s table	+40% (96) †§	+2% †	T4 +1% †
DFlash, EAGLE-3’s table	+23% (64)	+37% (64)	A10G +11% (4)
4-bit AWQ, documented table	+5% (64) †	$-$ 9 to $-$ 24% †	T4 +23% (64) †

Table 2: The controller: best gain over the same table without it, at the load where it occurs (loads 4–96; n-gram 4–128), mean of three runs unless marked ( † single run). Top: steady traffic, starting from a mistuned table. Bottom: a mid-run workload switch (Spec-Bench math and translation with EAGLE-3; DFlash: code then math at 96 requests; greedy, A10G protocol), the stale table’s loss and what the controller with exploration and prior decay does about it. Older GPUs: the same experiment from our T4 and A10G study, on the GPU named. § still 60% below no speculation, which beats every draft length of this drafter under V2.

Figure 5: llm-d optimized-baseline with EAGLE-3 behind the llm-d router, code prompts at temperature 1.0, mean of three runs: output tokens/s against fixed draft length 3 (0 = fixed length 3). (a) The guide’s own model and layout, 2

\times

Qwen3-32B at TP=2 on four H100s. (b) 4

\times

Qwen3-8B at TP=1 on the same GPUs; the 32-request dip is shared by every speculative configuration. The tuned table with the controller is within 2% of the table alone up to 512 requests in (b), and up to 4% above it in (a); it is omitted.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,101 papers

Test-Time Speculation
May 10, 2026Avinash Kumar, Sujay Sanghavi, Poulami DasTest-Time Adaptation Test-Time Optimization
The University of Texas at Austin
TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference
Feb 5, 2026Jiyoung Park, Hankyu Jang, Changseok Song +1Adaptive Inference LLM Inference Serving
Moreh, Inc.
Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning
Mar 2, 2026Jiebin Zhang, Zhenghan Yu, Liang Wang +8Speculative Decoding LLM Inference Acceleration
National Key Laboratory for Multimedia Information Processing, Peking University · School of Computer Science, Peking University · Microsoft Research Asia

May 10, 2026J/K move · Enter open · S save

Test-Time Speculation

Avinash Kumar, Sujay Sanghavi, Poulami Das

The University of Texas at Austin

Test-Time Adaptation Test-Time Optimization Speculative Decoding LLM Inference Acceleration Language Model Distillation

Open paper AskPDF arXiv

Speculative decoding accelerates LLM inference by using a fast draft model to generate tokens and a more accurate target model to verify them. Its performance depends on the

\textit{acceptance length}

, or number of draft tokens accepted by the target. Our studies show that the acceptance length of even state-of-the-art speculators, like DFlash, EAGLE-3 and PARD degrade with generation length, reaching values close to 1 (i.e. no speedup) within just a few thousand output tokens, making speculators ineffective for long-response tasks. Acceptance lengths decline because most speculators are trained offline on short sequences, but are forced to match the target model on much longer outputs at inference, well beyond their training distribution. To address this issue, we propose

\textit{Test-Time Speculation (TTS)}

, an online distillation approach that continuously adapts the speculator at test-time. TTS leverages the key insight that the token verification step already invokes the target model for each draft token, providing the training signal needed to adapt the draft at no additional cost. Treating the draft as the student and the target as a teacher, TTS adjusts the draft over several speculation rounds, with each update improving the draft's accuracy as generation proceeds. Our results across multiple models from the Qwen-3, Qwen-3.5, and Llama3.1 families show that TTS improves acceptance lengths over state-of-the-art speculators by up to

72\%

and

41\%

on average, with the benefits scaling with increased generation lengths.

Abstract

LLM Inference Acceleration

Explore similar work

Test-Time Speculation

TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference

Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learning