ForeSpark: Joint Draft and Verification Budget Scheduling for Parallel Speculative Decoding
Organizations: Meta · Seoul National University.
Abstract
Parallel drafters accelerate LLM decoding by proposing a whole block of draft tokens in a single forward pass, making them attractive for LLM serving. Recent systems optimize goodput (i.e., accepted tokens per second) because rejected drafts steal compute from other requests. However, these systems control only how many tokens are verified, so every request still drafts a full block. At high serving load, we find that drafting consumes up to 40% of step time, of which a majority is discarded because it is never verified. Yet, draft-length control from autoregressive drafters does not carry over, because a parallel drafter commits to its length before its single pass. Moreover, draft and verification lengths are coupled across the batch, so they must be selected jointly for every request while keeping the decision computationally cheap at every step. In this paper, we propose work-conserving speculation (WCS), a design that jointly allocates draft and verification budgets per request in LLM serving, and present ForeSpark as its prototype. ForeSpark realizes WCS with (i) a context-aware pre-draft estimator that predicts acceptance for every candidate length from the target model's context features, (ii) a lightweight joint draft-verify scheduler that selects both lengths through a global sort, and (iii) a post-draft budget distribution that spends verification on the most promising tokens. Implemented on vLLM, ForeSpark improves throughput by up to 1.77x over the state-of-the-art DSpark and by 2.49x over autoregressive decoding at a matched per-request generation rate, without changing the output distribution. It also tolerates higher load, sustaining up to 1.51x the request rate of DSpark under latency service level objectives (SLOs).
Figures & tables
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Qwen3-14B | GLM5.3-Flash | GLM5.3 | |
|---|---|---|---|
| Captured target layers | 1, 10, 19, 28, 37 | 6, 15, 25, 34, 43 | 6, 20, 34, 48, 62, 76 |
| Drafter layers hidden size | |||
| Training GPUs (server / trainer), steps | 2 / 6, 112K | 4 / 4, 154K | 4 / 4, 162K |
| Serving GPUs and parallelism | 8 H100, TP8 | 8 B200, TP8 + EP8 | 8 B200, TP8 + EP8 |
| Weights / KV cache | BF16 / BF16 | NVFP4 experts / FP8 | NVFP4 experts / FP8 |
| Max. requests / tokens per step | 512 / 24,576 | 256 / 24,576 | 256 / 24,576 |
| Qwen3-14B | GLM5.3-Flash | GLM5.3 | ||||
|---|---|---|---|---|---|---|
| Category | DSpark | ForeSpark | DSpark | ForeSpark | DSpark | ForeSpark |
| Coding | 5.918 | 5.863 | 3.753 | 3.855 | 4.038 | 3.979 |
| Humanities | 4.495 | 4.715 | 2.947 | 2.950 | 3.101 | 3.021 |
| Math | 6.211 | 6.556 | 3.185 | 3.154 | 3.291 | 3.425 |
| Multilingual | 2.929 | 3.115 | 3.424 | 3.438 | 3.201 | 3.232 |
| QA | 3.335 | 3.277 | 3.027 | 2.889 | 3.336 | 2.929 |
| Qwen3-14B | GLM5.3-Flash | GLM5.3 | ||||||||||
| DSpark | ForeSpark | DSpark | ForeSpark | DSpark | ForeSpark | |||||||
| Set | =7 | =15 | =7 | =15 | =7 | =15 | =7 | =15 | =7 | =15 | =7 | =15 |
| GSM8K | 5.85 | 7.47 | 5.94 | 7.93 | 4.85 | 5.35 | 4.84 | 5.53 | 4.67 | 5.12 | 4.82 | 5.27 |
| MATH-500 | 6.21 | 8.04 | 6.32 | 8.75 | 4.50 | 5.29 | 4.49 | 5.13 | 4.58 | 4.90 | 4.56 | 5.30 |
| AIME25 | 5.77 | 7.66 | 6.00 | 8.07 | 3.46 | 3.68 | 3.53 | 3.67 | 3.73 | 3.93 | 3.75 | 3.73 |
| HumanEval | 5.70 | 7.37 | 5.98 | 8.04 | 6.76 | 9.96 | 6.82 | 10.26 | 6.00 | 7.76 | 5.32 | 8.00 |
DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference
SGLang and evaluate it across multiple draft--target model pairs, reasoning benchmarks, real-world long-context workloads, and a wide range of batch sizes. Results show that SPECTRE consistently improves large-model serving throughput while causing only minor interference to the native workloads of tail-model services. In large-model deployments, including Qwen3-235B-A22B with TP=8, SPECTRE achieves up to \textbf{2.28 speedup} over autoregressive decoding and up to an additional 66% relative improvement over the strongest speculative decoding baselines. Talk is cheap, we show you the code: https://github.com/sgl-project/sglang/pull/22272.