$\nabla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space

Peihao Wang; Ruisi Cai; Zhen Wang; Hongyuan Mei; Qiang Liu; Pan Li; Zhangyang Wang

cs.LGMar 5, 2026

$\nabla$ -Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space

Authors: Peihao Wang, Ruisi Cai, Zhen Wang, Hongyuan Mei, Qiang Liu, Pan Li, Zhangyang Wang

Organizations: The University of Texas at Austin · UC San Diego · TTIC · Georgia Tech

Abstract

Scaling inference-time compute for Large Language Models (LLMs) has unlocked unprecedented reasoning capabilities. However, existing inference-time scaling methods typically rely on inefficient and suboptimal discrete search algorithms or trial-and-error prompting to improve the online policy. In this paper, we propose $\nabla$ -Reasoner, an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop to refine the policy on the fly. Our core component, Differentiable Textual Optimization (DTO), leverages gradient signals from both the LLM's likelihood and a reward model to refine textual representations. $\nabla$ -Reasoner further incorporates rejection sampling and acceleration design to robustify and speed up decoding. Theoretically, we show that performing inference-time gradient descent in the sample space to maximize reward is dual to aligning an LLM policy via KL-regularized reinforcement learning. Empirically, $\nabla$ -Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark, while reducing number of model calls by approximately 10-40% compared to strong baselines. Overall, our work introduces a paradigm shift from zeroth-order search to first-order optimization at test time, offering a cost-effective path to amplify LLM reasoning.

Figures & tables

Figure 1: LLM reasoning can be formulated as a maximization problem over the reward landscape. (left) Traditional inference-time scaling methods are zeroth-order methods, sampling numerous candidate responses and evaluating them to identify higher-quality solutions. (right) This paper introduces a first-order method for inference-time scaling, where reward gradients are directly leveraged to guide the search process toward highly rewarding responses.

Figure 2: Basic implementation of

\mathcal{\nabla}

-Reasoner.

\mathcal{\nabla}

-Reasoner is an iterative decoding algorithm driven by DTO. At each decoding step, DTO applies gradient descent on the logits initialized from the base model to optimize a reward-informed loss to refine the policy. The updated policy is then combined with rejection sampling, leading to high-reward responses. The pseudocode for the full implementation with acceleration techniques (Sec. 3.3 ) is deferred to Appendix B .

Models	Methods	MATH-500	AMC	AIME24	AIME25
Qwen-2.5-7B	Greedy decoding	43.8	33.0	6.7	6.7
	SC ( Xie et al., 2024 ) ( $N=8$ )	69.8	49.4	22.5	20.0
	BoN ( Stiennon et al., 2020 ) ( $N=8$ )	70.2	50.1	22.5	13.3
	ToT ( Yao et al., 2024 )	57.8	42.4	6.7	10.0
	RAP ( Hao et al., 2023 )	68.6	50.1	18.3	14.2
	SFT ( Ouyang et al., 2022 )	65.8	36.4	6.3	11.7

Table 1: Accuracy (%) on math reasoning datasets compared with baseline methods, including both test-time and training-time approaches. We skip results on AIME datasets for Llama-3.1-8B as it is incapable of generating reasonable performance. We mark the best performer in bold and the runner-up with underline . Our method outperforms all test-time baselines and even achieves performance on par with the training-based methods (SFT and GRPO), respectively.

Figure 3: A comparison of computational cost, measured by the number of model calls. Our method reduces costs by up to 40.2% compared to baselines.

Figure 4: Test-time scaling curves comparing our method with BoN and SC. We change the number of samples

N

for BoN and SC and number of rollouts

N_{max}

for our method. The results show

\mathcal{\nabla}

-Reasoner achieves superior performance with reduced cost across multiple models.

Model	Skywork-Qwen3-4B	Skywork-Qwen3-8B
MATH-500	80.4	80.8 (+0.4)
AMC	56.8	57.1 (+0.3)

Table 2: Ablation study on reward model choice.

Model	Baseline	$\mathcal{\nabla}$ -Reasoner
Qwen-2.5	65.9	32.8
Qwen-2.5-Instruct	66.5	28.9
Llama-3-Instruct	66.9	40.1

Table 3: Analysis of rejection rate (%) in rejection sampling. We set

N=8

for the BoN baseline and also set

N_{max}=8

for our

\mathcal{\nabla}

-Reasoner. The theoretical rejection rate of the baseline is

66.0\%

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Generation Hyperparameters
Setting	Value
Temperature	$0.5$
Top-p	$0.95$
Max Generation Length (AMC, MATH-500)	1024
Max Generation Length (AIME)	3072
Evaluation

Appendix

Table 4: Summary of Experimental Settings.

Method	Gen. (s)	Optim. (s)	Total (s)	FLOPs
BoN ( $N=8$ )	136.1	—	136.1	$9.54\times 10^{15}$
$\mathcal{\nabla}$ -Reasoner	106.2	45.9	152.1	$2.46\times 10^{17}$

Appendix

Table 5: Comparison of wall-clock execution time for 83 prompts on the AMC dataset.

Explore similar work

Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoning
Feb 20, 2026Lexiang Tang, Weihao Gao, Bingchen Zhao +4Efficient Language Model Reasoning LLM Reliability
Peking University, Beijing, China · University of Edinburgh, Edinburgh, UK.
Enhancing reasoning accuracy in large language models during inference time
Mar 22, 2026Vinay Sharma, Manish JainLLM Inference LLM Reasoning
Policy of Thoughts: Scaling Test-Time Training for LLM Reasoning via Online Policy Evolution
Jan 28, 2026Zhengbo Jiao, Hongyu Xian, Qinglong Wang +5RL for Language Model Reasoning Test-Time Training
Zhejiang University · South China Normal University · Shanghai University of Finance and Economics +2

Feb 20, 2026J/K move · Enter open · S save

Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoning

Lexiang Tang, Weihao Gao, Bingchen Zhao, Lu Ma+3

Peking University, Beijing, China · University of Edinburgh, Edinburgh, UK.

Efficient Language Model Reasoning LLM Reliability Contrastive Decoding Numerical Reasoning in Language Models

Open paper AskPDF arXiv

Recent work on test-time scaling for large language model (LLM) reasoning typically assumes that allocating more inference-time computation uniformly improves correctness. However, prior studies show that reasoning uncertainty is highly localized: a small subset of low-confidence tokens disproportionately contributes to reasoning errors and unnecessary output expansion. Motivated by this observation, we propose Thinking by Subtraction, a confidence-driven contrastive decoding approach that improves reasoning reliability through targeted token-level intervention. Our method, Confidence-Driven Contrastive Decoding, detects low-confidence tokens during decoding and intervenes selectively at these positions. It constructs a contrastive reference by replacing high-confidence tokens with minimal placeholders, and refines predictions by subtracting this reference distribution at low-confidence locations. Experiments show that CCD significantly improves accuracy across mathematical reasoning benchmarks while substantially reducing output length, with minimal KV-cache overhead. As a training-free method, CCD enhances reasoning reliability through targeted low-confidence intervention without computational redundancy. Our code will be made available at: https://github.com/bolo-web/CCD.

Abstract

Explore similar work

Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoning

Enhancing reasoning accuracy in large language models during inference time

Policy of Thoughts: Scaling Test-Time Training for LLM Reasoning via Online Policy Evolution