APEX: Speculate smarter, not deeper

Manvi Jha; Zach Zhang; Zhichao Xu; Linbo Liu; Sai Muralidhar Jayanthi; Vinayak Arannil

cs.CLOct 6, 2026

APEX: Speculate smarter, not deeper

Authors: Manvi Jha, Zach Zhang, Zhichao Xu, Linbo Liu, Sai Muralidhar Jayanthi, Vinayak Arannil

Organizations: University of Illinois Urbana-Champaign AWS AI · AWS AI

Abstract

Speculative decoding reduces large language model inference latency by drafting multiple tokens before target-model verification, but its effectiveness depends on both the proposal mechanism and draft depth. Fixed configurations cannot respond to changes in predictability, repetition, and acceptance during generation, so deeper drafting can increase wasted computation without proportional speedup. We introduce APEX, a learned controller that balances decoding speed and draft-token waste through request-level expert selection and block-level depth adaptation. APEX-Router selects among EAGLE-3, n-gram, and draft-model speculation for each request, while APEX-Depth adjusts draft length at each verification block using causal decoding signals and recent verifier feedback. APEX models accepted draft length as censored survival feedback, learning position-wise rejection hazards, block execution costs, and an action utility that balances throughput, accepted progress, and wasted tokens. This allows the controller to adapt speculation while retaining the target model's verification procedure. We integrate APEX into vLLM and evaluate it with Qwen3-8B across six workloads, achieving up to 5.24X speedup over autoregressive decoding. Across the aggregate evaluation, APEX-S achieves 4.27X speedup, while APEX-B achieves 3.27X speedup with a 41.0% relative reduction in wasted-token percentage compared with fixed n-gram speculation at k=16, providing distinct operating points for balancing acceleration and draft-token utilization.

Figures & tables

Figure 1: Speedup over autoregressive decoding across six workloads and three speculative mechanisms. Each cell reports highest speedup and its depth $k$ ; boxes mark the best method per workload.

Panel of Figure 2 — Figure 2: Online signals associated with SD performance (a) Median speedup over autoregressive decoding versus acceptance rate, with shaded variability. (b) Median speedup versus token entropy and repetition density; the gray plane marks autoregressive performance.

Figure 3: APEX overview. The router selects a speculative expert and initial depth from prompt features. After each verifier block, the depth controller uses the updated decoding state and prior verifier feedback to select the next depth. Observed acceptance and runtime update the controller state; network parameters remain fixed during inference.

Method	Depth	Speedup vs. AR $\uparrow$	Wasted Token % $\downarrow$	WAS $\uparrow$
EAGLE-3	$k=2$	1.71 $\times$	64.71%	2.64
	$k=4$	2.35 $\times$	66.25%	3.55
	$k=8$	1.94 $\times$	87.11%	2.23
	$k=16$	1.37 $\times$	97.67%	1.41
n-gram SD	$k=2$	1.76 $\times$	54.56%	3.23
	$k=4$	2.79 $\times$	54.14%	5.16

Table 1: Overall decoding performance. APEX is compared against autoregressive decoding and fixed speculative baselines. Higher speedup is better; lower wasted-token percentage is better.

Figure 4: Speedup vs Token Wastage. The plot shows the trend of Speedup (vs AR) vs Token Wastage across different depths and mechanisms. The horizontal axis reports non-wasted draft tokens (

100-\mathrm{Waste}\%

), and the vertical axis reports speedup over autoregressive decoding.

Method	Depth	Speedup vs. AR $\uparrow$	Wasted Token % $\downarrow$	WAS $\uparrow$
EAGLE-3	$k=2$	2.11 $\times$	44.50%	4.74
	$k=4$	2.78 $\times$	55.50%	5.01
	$k=8$	2.01 $\times$	87.38%	2.30
	$k=16$	1.27 $\times$	98.31%	1.29
n-gram SD	$k=2$	2.08 $\times$	46.00%	4.52
	$k=4$	2.31 $\times$	67.25%	3.43

Table 2: Workload-level speed–waste comparison for code generation .

Method	Depth	Speedup vs. AR $\uparrow$	Wasted Token % $\downarrow$	WAS $\uparrow$
EAGLE-3	$k=2$	1.55 $\times$	72.50%	2.14
	$k=4$	2.50 $\times$	62.50%	4.00
	$k=8$	3.22 $\times$	72.25%	4.46
	$k=16$	2.05 $\times$	93.44%	2.19
n-gram SD	$k=2$	1.89 $\times$	55.50%	3.41
	$k=4$	2.85 $\times$	53.75%	5.30

Table 3: Workload-level speed–waste comparison for mathematical reasoning .

Method	Depth	Speedup vs. AR $\uparrow$	Wasted Token % $\downarrow$	WAS $\uparrow$
EAGLE-3	$k=2$	1.59 $\times$	70.50%	2.26
	$k=4$	1.89 $\times$	77.75%	2.43
	$k=8$	0.51 $\times$	99.02%	0.52
	$k=16$	1.26 $\times$	98.38%	1.28
n-gram SD	$k=2$	1.22 $\times$	37.41%	3.26
	$k=4$	3.32 $\times$	34.50%	9.62

Table 4: Workload-level speed–waste comparison for long-context completion .

Method	Depth	Speedup vs. AR $\uparrow$	Wasted Token % $\downarrow$	WAS $\uparrow$
EAGLE-3	$k=2$	1.46 $\times$	77.00%	1.90
	$k=4$	2.13 $\times$	71.75%	2.97
	$k=8$	1.64 $\times$	92.00%	1.78
	$k=16$	1.37 $\times$	97.69%	1.40
n-gram SD	$k=2$	1.70 $\times$	65.00%	2.62
	$k=4$	2.46 $\times$	63.50%	3.87

Table 5: Workload-level speed–waste comparison for long-chain reasoning .

Method	Depth	Speedup vs. AR $\uparrow$	Wasted Token % $\downarrow$	WAS $\uparrow$
EAGLE-3	$k=2$	1.78 $\times$	61.00%	2.92
	$k=4$	2.35 $\times$	66.25%	3.55
	$k=8$	2.61 $\times$	79.88%	3.27
	$k=16$	1.21 $\times$	98.69%	1.23
n-gram SD	$k=2$	1.84 $\times$	58.00%	3.17
	$k=4$	3.05 $\times$	48.75%	6.26

Table 6: Workload-level speed–waste comparison for conversational generation .

Method	Depth	Speedup vs. AR $\uparrow$	Wasted Token % $\downarrow$	WAS $\uparrow$
EAGLE-3	$k=2$	1.67 $\times$	66.50%	2.51
	$k=4$	2.45 $\times$	63.75%	3.84
	$k=8$	0.98 $\times$	99.39%	0.99
	$k=16$	1.24 $\times$	98.50%	1.26
n-gram SD	$k=2$	1.76 $\times$	62.00%	2.84
	$k=4$	2.50 $\times$	62.50%	4.00

Table 7: Workload-level speed–waste comparison for swe-style tasks.

Profile	Variant	Speedup	Waste $\%$	WAS
APEX-S	Full APEX-S	2.44 $\times$	59.22 $\%$	4.12
	w/o survival loss	1.62 $\times$	68.16 $\%$	2.37
	w/o cost/TPS objective	2.41 $\times$	62.69%	3.85
	w/o entropy features	2.49 $\times$	62.41 $\%$	3.99
	w/o repetition features	2.42 $\times$	62.22 $\%$	3.89
	w/o verifier-history features	1.14 $\times$	54.74 $\%$	2.09

Table 8: Ablation study across operating profiles . Each variant removes one component from the corresponding full APEX controller. Speedup is measured relative to autoregressive decoding AR=1.00

\times

Method	Candidate $k$ set	Mean $k$	Speedup vs. AR	Wasted Token %	WAS
APEX-S	$k=1\text{--}16$	3.61	3.34 $\times$	47.18%	7.07
	$k=3\text{--}16$	4.54	3.43 $\times$	50.57%	6.79
	$k\in\{2,4,8,12,16\}$	3.92	3.39 $\times$	47.95%	7.06
APEX-B	$k=1\text{--}16$	3.09	3.25 $\times$	42.76%	7.60
	$k=3\text{--}16$	4.28	3.34 $\times$	48.55%	6.89
	$k\in\{2,4,8,12,16\}$	3.57	3.31 $\times$	45.10%	7.33

Table 9: APEX candidate-

k

sweep under the quality-balanced operating profile. Higher speedup and WAS are better; lower wasted-token percentage is better.

Explore similar work

May 10, 2026cs.CL

Test-Time Speculation

Speculative decoding accelerates LLM inference by using a fast draft model to generate tokens and a more accurate target model to verify them. Its performance depends on the

\textit{acceptance length}

, or number of draft tokens accepted by the target. Our studies show that the acceptance length of even state-of-the-art speculators, like DFlash, EAGLE-3 and PARD degrade with generation length, reaching values close to 1 (i.e. no speedup) within just a few thousand output tokens, making speculators ineffective for long-response tasks. Acceptance lengths decline because most speculators are trained offline on short sequences, but are forced to match the target model on much longer outputs at inference, well beyond their training distribution. To address this issue, we propose

\textit{Test-Time Speculation (TTS)}

, an online distillation approach that continuously adapts the speculator at test-time. TTS leverages the key insight that the token verification step already invokes the target model for each draft token, providing the training signal needed to adapt the draft at no additional cost. Treating the draft as the student and the target as a teacher, TTS adjusts the draft over several speculation rounds, with each update improving the draft's accuracy as generation proceeds. Our results across multiple models from the Qwen-3, Qwen-3.5, and Llama3.1 families show that TTS improves acceptance lengths over state-of-the-art speculators by up to

72\%

and

41\%

on average, with the benefits scaling with increased generation lengths.

Avinash Kumar, Sujay Sanghavi, Poulami Das

The University of Texas at Austin

Speculative Decoding Self-Speculative LLM Inference Optimization

Ask PDF

May 31, 2026cs.CL

Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding

Large Language Model (LLM) generation remains expensive because autoregressive decoding calls the model once for each new token. Speculative decoding reduces this cost by drafting multiple tokens and verifying them with the target model in one step, but its speedup depends on how many drafted tokens are accepted. Parameter-free draft sources can propose long continuations at low cost in structured and agentic workloads, yet a cache match that looks promising at one generation step may have low payoff at the next. We propose Hybrid Verified Decoding, which predicts the accepted length of a cache draft before verification and uses this payoff estimate to choose between cache verification and a model-based drafter. Across three LLMs and sixteen datasets, Hybrid Verified Decoding is especially effective on agentic workflows, where it outperforms EAGLE3 in every setting with a 2.73x average speedup. Our analysis shows how prompt structure creates cache opportunities, how high-payoff cache drafts concentrate in a small part of the draft space, and how payoff-guided selection reduces sequential decoding work, pointing to runtime draft selection as a promising direction for speculative decoding.

Xin Su, Dawid Majchrowski, Fangyuan Yu +5

Thoughtworks · Nvidia

Speculative Decoding Decoding Cache

Ask PDF

Jul 16, 2026cs.CL

D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decoding

Speculative decoding accelerates large language model (LLM) inference without compromising output quality. Recent parallel drafting methods further improve single-request performance by decoupling draft length from drafting latency, enabling longer drafts and higher mean accepted tokens (MAT). However, under high request concurrency, long drafts waste substantial computation on rejected tokens, increasing verification cost and potentially making speculative decoding slower than autoregressive decoding. We present D-Cut, an adaptive pruning method that selects draft tokens jointly across the batch and concentrates the verification budget on tokens most likely to be accepted. D-Cut is motivated by two observations. First, acceptance lengths vary considerably across concurrent requests; D-Cut therefore performs cross-request pruning, allocating the verification budget adaptively according to draft confidence. Second, verification cost depends strongly on the deployment environment, including GPU architecture and parallelism strategy; D-Cut incorporates a runtime cost model to adapt its pruning depth to the target environment. Experiments on dense and mixture-of-experts (MoE) models show that, under high concurrency, D-Cut improves the average speedup from

1.26\times

1.65\times

, restores acceleration in dense-model configurations where long-draft baselines are slower than autoregressive decoding, and achieves up to

3.0\times

speedup over autoregressive decoding on MoE models.

Tianyu Liu, Yuhao Shen, Rui Cen +7

1Tencent · 2Independent Researcher

Speculative Decoding Drafter Cuts

Ask PDF