Scaling inference-time compute for Large Language Models (LLMs) has unlocked unprecedented reasoning capabilities. However, existing inference-time scaling methods typically rely on inefficient and suboptimal discrete search algorithms or trial-and-error prompting to improve the online policy. In this paper, we propose ∇-Reasoner, an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop to refine the policy on the fly. Our core component, Differentiable Textual Optimization (DTO), leverages gradient signals from both the LLM's likelihood and a reward model to refine textual representations. ∇-Reasoner further incorporates rejection sampling and acceleration design to robustify and speed up decoding. Theoretically, we show that performing inference-time gradient descent in the sample space to maximize reward is dual to aligning an LLM policy via KL-regularized reinforcement learning. Empirically, ∇-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark, while reducing number of model calls by approximately 10-40% compared to strong baselines. Overall, our work introduces a paradigm shift from zeroth-order search to first-order optimization at test time, offering a cost-effective path to amplify LLM reasoning.
Figures & tables
Figure 1: LLM reasoning can be formulated as a maximization problem over the reward landscape. (left) Traditional inference-time scaling methods are zeroth-order methods, sampling numerous candidate responses and evaluating them to identify higher-quality solutions. (right) This paper introduces a first-order method for inference-time scaling, where reward gradients are directly leveraged to guide the search process toward highly rewarding responses.
Figure 2: Basic implementation of ∇ -Reasoner. ∇ -Reasoner is an iterative decoding algorithm driven by DTO. At each decoding step, DTO applies gradient descent on the logits initialized from the base model to optimize a reward-informed loss to refine the policy. The updated policy is then combined with rejection sampling, leading to high-reward responses. The pseudocode for the full implementation with acceleration techniques (Sec. 3.3 ) is deferred to Appendix B .
Models
Methods
MATH-500
AMC
AIME24
AIME25
Qwen-2.5-7B
Greedy decoding
43.8
33.0
6.7
6.7
SC ( Xie et al., 2024 ) ( N=8 )
69.8
49.4
22.5
20.0
BoN ( Stiennon et al., 2020 ) ( N=8 )
70.2
50.1
22.5
13.3
ToT ( Yao et al., 2024 )
57.8
42.4
6.7
10.0
RAP ( Hao et al., 2023 )
68.6
50.1
18.3
14.2
SFT ( Ouyang et al., 2022 )
65.8
36.4
6.3
11.7
Table 1: Accuracy (%) on math reasoning datasets compared with baseline methods, including both test-time and training-time approaches. We skip results on AIME datasets for Llama-3.1-8B as it is incapable of generating reasonable performance. We mark the best performer in bold and the runner-up with underline . Our method outperforms all test-time baselines and even achieves performance on par with the training-based methods (SFT and GRPO), respectively.
Figure 3: A comparison of computational cost, measured by the number of model calls. Our method reduces costs by up to 40.2% compared to baselines.
Figure 4: Test-time scaling curves comparing our method with BoN and SC. We change the number of samples N for BoN and SC and number of rollouts Nmax for our method. The results show ∇ -Reasoner achieves superior performance with reduced cost across multiple models.
Model
Skywork-Qwen3-4B
Skywork-Qwen3-8B
MATH-500
80.4
80.8 (+0.4)
AMC
56.8
57.1 (+0.3)
Table 2: Ablation study on reward model choice.
Model
Baseline
∇ -Reasoner
Qwen-2.5
65.9
32.8
Qwen-2.5-Instruct
66.5
28.9
Llama-3-Instruct
66.9
40.1
Table 3: Analysis of rejection rate (%) in rejection sampling. We set N=8 for the BoN baseline and also set Nmax=8 for our ∇ -Reasoner. The theoretical rejection rate of the baseline is 66.0% .
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Generation Hyperparameters
Temperature
0.5
Top-p
0.95
Max Generation Length (AMC, MATH-500)
1024
Max Generation Length (AIME)
3072
Evaluation
Appendix
Table 4: Summary of Experimental Settings.
Method
Gen. (s)
Optim. (s)
Total (s)
FLOPs
BoN ( N=8 )
136.1
—
136.1
9.54×1015
∇ -Reasoner
106.2
45.9
152.1
2.46×1017
Appendix
Table 5: Comparison of wall-clock execution time for 83 prompts on the AMC dataset.
Recent work on test-time scaling for large language model (LLM) reasoning typically assumes that allocating more inference-time computation uniformly improves correctness. However, prior studies show that reasoning uncertainty is highly localized: a small subset of low-confidence tokens disproportionately contributes to reasoning errors and unnecessary output expansion. Motivated by this observation, we propose Thinking by Subtraction, a confidence-driven contrastive decoding approach that improves reasoning reliability through targeted token-level intervention. Our method, Confidence-Driven Contrastive Decoding, detects low-confidence tokens during decoding and intervenes selectively at these positions. It constructs a contrastive reference by replacing high-confidence tokens with minimal placeholders, and refines predictions by subtracting this reference distribution at low-confidence locations. Experiments show that CCD significantly improves accuracy across mathematical reasoning benchmarks while substantially reducing output length, with minimal KV-cache overhead. As a training-free method, CCD enhances reasoning reliability through targeted low-confidence intervention without computational redundancy. Our code will be made available at: https://github.com/bolo-web/CCD.
Lexiang Tang, Weihao Gao, Bingchen Zhao +4
Peking University, Beijing, China · University of Edinburgh, Edinburgh, UK.
Large Language Models (LLMs) often exhibit strong linguistic abilities while remaining unreliable on multi-step reasoning tasks, particularly when deployed without additional training or fine-tuning. In this work, we study inference-time techniques to improve the reasoning accuracy of LLMs. We systematically evaluate three classes of inference-time strategies: (i) self-consistency via stochastic decoding, where the model is sampled multiple times using controlled temperature and nucleus sampling and the most frequent final answer is selected; (ii) dual-model reasoning agreement, where outputs from two independent models are compared and only consistent reasoning traces are trusted; and (iii) self-reflection, where the model critiques and revises its own reasoning. Across all evaluated methods, we employ Chain-of-Thought (CoT) [1] prompting to elicit explicit intermediate reasoning steps before generating final answers. In this work, we provide a controlled comparative evaluation across three inference-time strategies under identical prompting and verification settings. Our experiments on LLM [2] show that self-consistency with nucleus sampling and controlled temperature value yields the substantial gains, achieving a 9% to 15% absolute improvement in accuracy over greedy single-pass decoding, well-suited for low-risk domains, offering meaningful gains with minimal overhead. The dual-model approach provides additional confirmation for model reasoning steps thus more appropriate for moderate-risk domains, where higher reliability justifies additional compute. Self-reflection offers only marginal improvements, suggesting limited effectiveness for smaller non-reasoning models at inference time.
Large language models (LLMs) struggle with complex, long-horizon reasoning due to instability caused by their frozen policy assumption. Current test-time scaling methods treat execution feedback merely as an external signal for filtering or rewriting trajectories, without internalizing it to improve the underlying reasoning strategy. Inspired by Popper's epistemology of "conjectures and refutations," we argue that intelligence requires real-time evolution of the model's policy through learning from failed attempts. We introduce Policy of Thoughts (PoT), a framework that recasts reasoning as a within-instance online optimization process. PoT first generates diverse candidate solutions via an efficient exploration mechanism, then uses Group Relative Policy Optimization (GRPO) to update a transient LoRA adapter based on execution feedback. This closed-loop design enables dynamic, instance-specific refinement of the model's reasoning priors. Experiments show that PoT dramatically boosts performance: a 4B model achieves 49.71% accuracy on LiveCodeBench, outperforming GPT-4o and DeepSeek-V3 despite being over 50 smaller.
Zhengbo Jiao, Hongyu Xian, Qinglong Wang +5
Zhejiang University · South China Normal University · Shanghai University of Finance and Economics +2