Test-time scaling has recently emerged as a powerful approach for improving LLM reasoning by allocating additional computation during inference, substantially enhancing accuracy on challenging tasks such as mathematics and coding. To accelerate the exploration of reasoning paths, recent studies proposed speculative execution. However, we show that supporting speculative execution poses two unique challenges for LLM serving systems: (1) an explosion in the search space of candidate paths and (2) frequent, fine-grained verification tasks for candidates. To address these challenges, this paper proposes SpecScale, a serving system for efficient speculative execution. We introduce three techniques to reconcile the trade-off between latency and computational overhead: (1) early pruning of low-quality candidate paths, (2) deduplicating computation across redundant candidate paths, and (3) deferring fine-grained verification tasks. We evaluate SpecScale on challenging reasoning benchmarks, including MATH and Olympiad. Our results show that SpecScale significantly outperforms both non-speculative and recent speculative approaches, delivering substantial improvements in throughput and latency while preserving answer quality.
Figures & tables
Figure 1. Beam search with Process Reward Model (PRM), where beam size ( N ) is 4 and beam width ( M ) is 2
Dataset
Qwen2.5-3B
Qwen2.5-32B
Qwen2.5-3B w/ PRM (8 beams)
GSM8K
88.90 %
93.90 %
94.70 %
MATH-500
60.60 %
80.30 %
76.24 %
OlympiadBench
19.52 %
32.65 %
30.40 %
Table 1. Effectiveness of test-time scaling on the math datasets in terms of accuracy
Figure 2. Distribution of waiting time ratio for test-time scaling with beam search
Figure 3. Test-time scaling with two approaches: (a) example search tree, (b) execution timeline of non-speculation and speculation. The numeric subscript on each rectangle indicates the order of the decoding phases, while the subscript v denotes verification.
Figure 4. Number of candidate paths according to concurrent requests on MATH dataset using Qwen2.5-3B-Instruct
Figure 5. Normalized token latency with regard to concurrent requests on MATH dataset using Qwen2.5-3B-Instruct
Figure 6. Cumulative number of tokens per verification batch on MATH dataset using Qwen2.5-3B-Instruct
Figure 7. Overview of our proposed system: SpecScale
Figure 8. Speculative execution with early pruning, where beam size ( N ) and beam width ( M ) are both 4, implying top- k=1
Figure 9. Step score distribution
Figure 10. Baseline vs. computation deduplication approach
Figure 11. Common token (%) across candidates of each step
Figure 12. Throughput of verification in a microbenchmark
Figure 13. Token throughput performance for GSM, MATH, and Olympiad datasets on Qwen2.5 and Llama3 models
Figure 14. Serving performance for MATH on Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct
Figure 15. Quantitative analysis of SpecScale and baselines for MATH on Qwen2.5-7B-Instruct with beam size 4 ( E : Early Pruning, D : Computation Deduplication, and L : Lazy Verification)
Table 2. Answer accuracy (%) of Qwen2.5-3B-Instruct with a PRM verifier. The numbers in parentheses denote the accuracy difference in percentage points compared to the baseline ( Non-spec ).
Figure 16. Normalized output token throughput with varying batching token thresholds for lazy verification
Figure 17. Normalized KV memory usage (left y-axis) and token latency (right y-axis) as the beam size increases
Figure 18. Token throughput according to the number of concurrent requests
Test-time compute (TTC) scaling has emerged as a powerful paradigm for improving large language model (LLM) reasoning by allocating additional compute during inference, e.g., via multi-sample generation and verifier-based reranking. Existing TTC scaling strategies and reasoning scorers remain fragmented, evaluated under inconsistent protocols, and are rarely analyzed through the lens of quality-cost trade-offs. We introduce ThinkBooster, a unified framework for seamless test-time compute scaling of LLM reasoning, which consists of (i) a modular Python library implementing state-of-the-art TTC scaling strategy and scorer families, (ii) a benchmark that jointly evaluates performance and computational efficiency, and (iii) a deployable OpenAI-compatible proxy service that enables drop-in integration of adaptive reasoning into real-world applications. We further provide a demo visual debugger for inspecting the reasoning trajectories, intermediate selection decisions, and alternative reasoning paths. Empirical results on mathematical and coding tasks reveal the performance-compute trade-offs of TTC scaling strategies and scoring methods and demonstrate that ThinkBooster provides practical gains in real-world tasks. The code is available online under an MIT license.
Tree-of-Thought (ToT) reasoning structures Large Language Model (LLM) inference as a tree-based search, demonstrating strong potential for solving complex mathematical and programming tasks. However, its efficiency is constrained by the reward dependency barrier -- a synchronization bottleneck caused by sequential reward-guided exploration that limits search parallelism and introduces substantial latency. Prior system optimizations, mainly designed for linear Chain-of-Thought (CoT) reasoning, cannot address these challenges, leaving the efficiency of ToT underexplored. To enhance ToT reasoning efficiency, we observe that the reasoning paths can be explored speculatively to break the reward synchronization barrier. Therefore, in this paper, we propose SPEX and introduce three key techniques: (i) intra-query speculative path selection to predict and expand high-potential branches of ToT, (ii) inter-query budget allocation to balance speculative resource allocation across queries dynamically, and (iii) adaptive early termination to prune deep and redundant branches for a skewed search tree. We implement SPEX on top of the SGLang framework and evaluate it across diverse ToT algorithms and LLMs. Extensive experiments show that SPEX achieves 1.2∼3× speedup for different ToT reasoning algorithms. Moreover, SPEX synergizes with token-level speculative decoding, achieving cumulative speedups of up to 4.1×. Ablation studies further confirm the contributions of each technique. Overall, SPEX represents a significant step toward efficient and scalable ToT reasoning, unlocking the parallelism required for high-performance inference-time scaling for LLMs.
Shuzhang Zhong, Haochen Huang, Shengxuan Qiu +3
Institute for Artificial Intelligence, Peking University · School of Integrated Circuits, Peking University · 3ByteDance Seed
Scaling test-time reasoning has substantially improved the problem-solving ability of large language models (LLMs), but standard autoregressive decoding still executes long reasoning traces sequentially, creating severe latency for difficult tasks (up to days and weeks). Parallel reasoning offers a natural remedy. However, prior systems primarily focus on Subtask Parallelism, where the model learns to decompose a high-level task into smaller chunks that can be solved independently. This approach overlooks another pervasive form of parallelism: Trial Parallelism, where multiple speculative attempts explore, verify, and aggregate competing hypotheses in parallel. In this paper, we introduce Parason, which reveals and learns both forms of parallelism in LLM reasoning. Our analysis identifies Trial Parallelism as the majority of parallelizable reasoning computation (65.5% in DeepSeek-V4's reasoning steps in HLE), and it becomes increasingly dominant on hard problems. Guided by this taxonomy, Parason converts sequential reasoning traces into structured parallel trajectories with a context-free grammar, then trains models with Parallelism-Aware Group Relative Policy Optimization (PA-GRPO), whose reward jointly balances accuracy, latency, and the two parallelism ratios. At inference time, Parason executes the learned parallel structure through tool calls, translating theoretical savings to real-world wall-clock acceleration. Experiments on mathematical reasoning benchmarks including AIME24 and AIME25 show that Parason achieves an average acceleration about 1.7× while maintaining competitive accuracy.
Zhengyang Zhang, Zijian Zhang, Jiaxuan Gao +4
Tsinghua University · NVIDIA · Independent Researcher +1