SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scaling

Jeonghoon Park; Seongwoon Jo; Jongwon Lee; Taesik Gong

cs.AIOct 4, 2026

SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scaling

Authors: Jeonghoon Park, Seongwoon Jo, Jongwon Lee, Taesik Gong

Organizations: UNIST

Abstract

Long-form reasoning makes inference expensive, and speculative decoding mitigates this cost by verifying multiple draft tokens in parallel. Its speedup, however, can fade as context grows and draft acceptance declines. We focus on attention-mass dilution: as softmax normalizes over more visible Keys, the mass concentrated on the highest-scoring Keys can decrease. We introduce SharpDraft, a training-free method that counteracts this effect through cardinality-aware Query scaling, without the computational overhead of online adaptation. Under explicit assumptions, we derive an exact top- $k$ mass correction and deploy a closed-form fixed-slope approximation. Across AIME-26, GPQA-Diamond, and LongGenBench Diary, SharpDraft achieves $2.59$ - $3.19\times$ geometric-mean end-to-end speedups over target-only autoregressive decoding when applied to DFlash, PARD, and EAGLE 3.1. With DFlash, it improves decoding speed and outperforms full-parameter and LoRA-based online adaptation in end-to-end speedup, while matching the unmodified drafter's reported peak allocated GPU memory.

Figures & tables

Panel of Figure 1 — Figure 1: Long-generation behavior of Qwen3-8B on LongGenBench Diary. (a) Decode throughput of different methods in tokens per second. (b) Top-32 attention mass of different methods.

Model	Method	Trainable Params $\downarrow$	Peak Mem. $\downarrow$	Online Update Cost (ms/update) $\downarrow$	Speedup $\uparrow$
Model	Method	Trainable Params $\downarrow$	Peak Mem. $\downarrow$	Online Update Cost (ms/update) $\downarrow$	AIME-26	GPQA-D	LGB(Diary)	Overall
Qwen3-4B	Target-only AR	$0$	$12.54\,\mathrm{GiB}$	$0$	$1.000\times$	$1.000\times$	$1.000\times$	$1.000\times$
	DFlash (Frozen)	$0$	$16.68\,\mathrm{GiB}$	$0$	$1.267\times$	$1.598\times$	$1.637\times$	$1.491\times$
	$+$ Full Adaptation	$537.43\mathrm{M}$	$24.46\,\mathrm{GiB}$	$103.14$	$1.368\times$	$1.981\times$	$2.478\times$	$1.886\times$
	$+$ LoRA ( $r=32$ )	$9.67\mathrm{M}$	$19.95\,\mathrm{GiB}$	$51.26$	$1.434\times$	$1.952\times$	$2.568\times$	$1.930\times$
	$+$ LoRA ( $r=16$ )	$4.83\mathrm{M}$	$19.84\,\mathrm{GiB}$	$49.91$	$1.589\times$	$2.183\times$	$2.865\times$	$2.150\times$

Table 1: End-to-end speedup and adaptation overhead on long-generation workloads. Speedups are relative to target-only AR; Overall is the geometric mean across the three benchmarks. Peak memory and update cost are measured on AIME-26 and GPQA-Diamond only.

Panel of Figure 2 — Figure 2: Mean end-to-end latency and recorded GPU training time per AIME-26 request.

Method	MAL $\uparrow$			Speedup $\uparrow$
Method	AIME-26	GPQA-D	LGB(Diary)	AIME-26	GPQA-D	LGB(Diary)	Overall
Target-only AR	–	–	–	$1.000\times$	$1.000\times$	$1.000\times$	$1.000\times$
PARD	$2.368$	$2.728$	$2.252$	$2.691\times$	$3.389\times$	$2.729\times$	$2.919\times$
$+$ LoRA	$1.637$	$2.224$	$1.390$	$1.015\times$	$1.567\times$	$0.923\times$	$1.137\times$
$+$ SharpDraft	$\mathbf{2.441}$	$\mathbf{2.906}$	$\mathbf{2.424}$	$\mathbf{2.773\times}$	$\mathbf{3.939\times}$	$\mathbf{2.975\times}$	$\mathbf{3.191\times}$
EAGLE 3.1	$3.152$	$3.659$	$5.668$	$1.922\times$	$2.529\times$	$3.515\times$	$2.575\times$

Table 2: MAL and end-to-end speedup across draft model families with Qwen3-8B. Speedups are relative to target-only AR; Overall is their geometric mean across the three benchmarks.

Panel of Figure 3 — Figure 3: Draft acceptance during multi-turn MATH-500 generation, up to 32K tokens in (a)–(c) and 100K in (d). Lines show mean session MAL; bands indicate pointwise 95% bootstrap confidence intervals.

Method	Final output length			All problems
Method	0–10K	10–20K	20–32K	Overall MAL $\uparrow$
Frozen ( $\beta=1$ )	4.092	3.113	2.596	3.182
Fixed $\beta=e^{0.1}$	4.273 (+0.181)	3.524 (+0.411)	3.318 (+0.723)	3.660 (+0.477)
Fixed $\beta=e^{0.2}$	4.305 (+0.214)	3.726 (+0.613)	3.785 (+1.189)	3.925 (+0.743)
Fixed $\beta=e^{0.3}$	4.284 (+0.193)	3.784 (+0.671)	3.989 (+1.393)	4.023 (+0.841)
Fixed $\beta=e^{0.4}$	4.235 (+0.143)	3.756 (+0.643)	4.028 (+1.433)	4.018 (+0.835)

Table 3: Fixed versus dynamic Query scaling on Qwen3-8B/DFlash using 30 output-matched AIME-26 problems. Values are mean whole-request MAL, grouped by final output length. Parentheses show gains over Frozen.

Panel of Figure 4 — Figure 4: Fixed and dynamic scaling during AIME-26 generation with Qwen3-8B. (a) Top-32 attention mass on matched prefixes. (b) Mean cumulative MAL relative to SharpDraft .

Panel of Figure 5 — Figure 5: Fixed-slope sensitivity and layerwise calibration on Qwen3-8B/AIME-26. (a) MAL across slope settings. (b,c) Top-32 mass error and MAL. Shading shows pointwise 95% bootstrap confidence intervals.

Panel of Figure 6 — Figure 6: Sensitivity to the reference cardinality across models and datasets. Verifier-call-rate savings are relative to 2K for Qwen/Llama and 4K for GLM.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Effective slope inferred from the measured reference contrast for Qwen3-8B/DFlash. The dashed line marks the deployed setting

s=5

Explore similar work

LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter
Sep 29, 2026Hao-Yuan He, Peng-Fei Liu, Si Shen +1Speculative Decoding Block-Diffusion Drafter
BudgetDraft: Acceptance-Aware Multi-View Training for Sparse-KV Speculative Decoding
May 29, 2026Liang He, Jingbo Wen, Qishi Zhan +4Speculative Decoding Graphics Processing Unit Memory
Shanghai Institute of Optics and Fine Mechanics · The University of Sydney · Marquette University +4
DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting
Jul 8, 2026Hong-Kai Zheng, Piji LiSpeculative Decoding Drafter
College of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, China · MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, China · The Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China

Sep 29, 2026, cs.LGJ/K move · Enter open · S save

LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter

Hao-Yuan He, Peng-Fei Liu, Si Shen, Ming Li

Speculative Decoding Block-Diffusion Drafter Drafter Autoregressive Model Long-Context Reasoning

Open paper AskPDF arXiv

Speculative decoding accelerates autoregressive inference by verifying multiple draft tokens in a single target forward pass. However, as the context grows, existing state-of-the-art drafters become increasingly expensive, eroding the very efficiency advantage they are designed to provide. We argue that this scaling is unnecessary. A standalone language model must grow with its prefix because it is solely responsible for every token it produces. A drafter, by contrast, only proposes candidates; the target catches and corrects every error before any token is committed. The drafter's decoding cost can therefore be made entirely independent of the prefix length. We introduce LongSpark, a block-diffusion drafter that achieves this by extracting fixed-size, multiscale views from the target's verification pass, thereby eliminating the need for a growing persistent state. Extensive evaluations demonstrate that LongSpark achieves state-of-the-art end-to-end efficiency across multiple model scales and realistic serving conditions. Notably, it delivers the lowest time-per-output-token on long-context tasks while reducing the drafter's context state by several orders of magnitude.

Abstract

Explore similar work

LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter

BudgetDraft: Acceptance-Aware Multi-View Training for Sparse-KV Speculative Decoding

DeLS-Spec: Decoupled Long-Short Contexts for Parallel Speculative Drafting