Test-time scaling has recently emerged as a powerful approach for improving LLM reasoning by allocating additional computation during inference, substantially enhancing accuracy on challenging tasks such as mathematics and coding. To accelerate the exploration of reasoning paths, recent studies proposed speculative execution. However, we show that supporting speculative execution poses two unique challenges for LLM serving systems: (1) an explosion in the search space of candidate paths and (2) frequent, fine-grained verification tasks for candidates. To address these challenges, this paper proposes SpecScale, a serving system for efficient speculative execution. We introduce three techniques to reconcile the trade-off between latency and computational overhead: (1) early pruning of low-quality candidate paths, (2) deduplicating computation across redundant candidate paths, and (3) deferring fine-grained verification tasks. We evaluate SpecScale on challenging reasoning benchmarks, including MATH and Olympiad. Our results show that SpecScale significantly outperforms both non-speculative and recent speculative approaches, delivering substantial improvements in throughput and latency while preserving answer quality.
Figures & tables
Figure 1. Beam search with Process Reward Model (PRM), where beam size ( N ) is 4 and beam width ( M ) is 2
Dataset
Qwen2.5-3B
Qwen2.5-32B
Qwen2.5-3B w/ PRM (8 beams)
GSM8K
88.90 %
93.90 %
94.70 %
MATH-500
60.60 %
80.30 %
76.24 %
OlympiadBench
19.52 %
32.65 %
30.40 %
Table 1. Effectiveness of test-time scaling on the math datasets in terms of accuracy
Figure 2. Distribution of waiting time ratio for test-time scaling with beam search
Figure 3. Test-time scaling with two approaches: (a) example search tree, (b) execution timeline of non-speculation and speculation. The numeric subscript on each rectangle indicates the order of the decoding phases, while the subscript v denotes verification.
Figure 4. Number of candidate paths according to concurrent requests on MATH dataset using Qwen2.5-3B-Instruct
Figure 5. Normalized token latency with regard to concurrent requests on MATH dataset using Qwen2.5-3B-Instruct
Figure 6. Cumulative number of tokens per verification batch on MATH dataset using Qwen2.5-3B-Instruct
Figure 7. Overview of our proposed system: SpecScale
Figure 8. Speculative execution with early pruning, where beam size ( N ) and beam width ( M ) are both 4, implying top- k=1
Figure 9. Step score distribution
Figure 10. Baseline vs. computation deduplication approach
Figure 11. Common token (%) across candidates of each step
Figure 12. Throughput of verification in a microbenchmark
Figure 13. Token throughput performance for GSM, MATH, and Olympiad datasets on Qwen2.5 and Llama3 models
Figure 14. Serving performance for MATH on Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct
Figure 15. Quantitative analysis of SpecScale and baselines for MATH on Qwen2.5-7B-Instruct with beam size 4 ( E : Early Pruning, D : Computation Deduplication, and L : Lazy Verification)
Table 2. Answer accuracy (%) of Qwen2.5-3B-Instruct with a PRM verifier. The numbers in parentheses denote the accuracy difference in percentage points compared to the baseline ( Non-spec ).
Figure 16. Normalized output token throughput with varying batching token thresholds for lazy verification
Figure 17. Normalized KV memory usage (left y-axis) and token latency (right y-axis) as the beam size increases
Figure 18. Token throughput according to the number of concurrent requests