Recent work on test-time scaling for large language model (LLM) reasoning typically assumes that allocating more inference-time computation uniformly improves correctness. However, prior studies show that reasoning uncertainty is highly localized: a small subset of low-confidence tokens disproportionately contributes to reasoning errors and unnecessary output expansion. Motivated by this observation, we propose Thinking by Subtraction, a confidence-driven contrastive decoding approach that improves reasoning reliability through targeted token-level intervention. Our method, Confidence-Driven Contrastive Decoding, detects low-confidence tokens during decoding and intervenes selectively at these positions. It constructs a contrastive reference by replacing high-confidence tokens with minimal placeholders, and refines predictions by subtracting this reference distribution at low-confidence locations. Experiments show that CCD significantly improves accuracy across mathematical reasoning benchmarks while substantially reducing output length, with minimal KV-cache overhead. As a training-free method, CCD enhances reasoning reliability through targeted low-confidence intervention without computational redundancy. Our code will be made available at: https://github.com/bolo-web/CCD.
Figures & tables
Figure 1: Trajectory-level relationship between token confidence and final answer correctness. Across 2,880 reasoning trajectories generated by Qwen3-8B on AIME24.
Figure 2: Overview of Confidence-Driven Contrastive Decoding (CCD). The decoding process consists of four key components: (1) online estimation of token-level confidence; (2) confidence-driven token selection that partitions chain-of-thought tokens into low-confidence (LC) and high-confidence (HC) sets; (3) contrastive decoding triggered at LC-CoT tokens to refine uncertain predictions; and (4) dual key–value (KV) cache maintenance to support selective intervention without disrupting standard autoregressive decoding.
Table 1: Performance evaluation on four mathematical reasoning benchmarks using Qwen3-Think. We compare Base (standard decoding), MTI and Ours (CCD) across 8 sampling runs (reported as Mean@8 accuracy %). Bold indicates the best results, and Results demonstrate that CCD consistently enhances reasoning accuracy by strategically applying contrastive decoding based on confidence thresholds.
Table 2: Performance comparison between Base and CCD on AIME2025 across MoE and Distilled architectures.
Figure 3: Effects of token masking strategies in the contrastive branch. (a) Masking high-confidence tokens induces a more uncertain (higher-entropy) contrastive distribution than masking mid- or low-confidence tokens. (b) Ablation on placeholder choices for replacement, where semantically minimal symbols provide the most effective contrastive signal.
Qwen3 Series (Dense)
(τcd,τrep)
4B
8B
14B
Base
63.75
67.91
68.75
(2,5)
64.08
67.91
72.92
(3,5)
65.00
71.67
71.25
(3,8)
67.08
70.83
73.51
(3,10)
67.92
68.33
73.75
Table 3: Hyperparameter ablation of (τcd,τrep) on AIME25. Base denotes standard decoding without CCD. We evaluate the transferability of the optimal setting across dense architectures.
Figure 4: Token-level confidence improvement at low-confidence decoding positions.
Figure 5: Trajectory-level confidence comparison before CCD intervention.
Figure 6: Trajectory-level confidence comparison after CCD intervention.
Figure 7: Average response length of the base output, CCD output, and contrastive distribution for Qwen3-8B across benchmarks. CCD substantially shortens responses on AIME-style tasks while preserving response length on HMMT25 and BRUMO25, whereas the contrastive distribution yields consistently longer outputs.
Figure 8: Frequency of explicit thinking tokens in model responses for Qwen3-8B. CCD significantly reduces thinking-token usage on AIME benchmarks while leaving HMMT25 and BRUMO25 largely unchanged, indicating selective suppression of overextended reasoning.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Case study on an AIME 2025 problem: comparison between Trivial CoT and CCD CoT on a combinatorics task.
Category
Target Keywords
Hesitation
wait, Wait, WAIT, hmm, Hmm, HMM
Correction
but, But, BUT, however, However, HOWEVER, wait,but, Wait,but, but wait, But wait, hold on, Hold on
Self-Correction
let me double-check, Let me double-check, looking back, Looking back
Long chains of thought (CoT) from current language models frequently contain logical gaps and unjustified leaps, limiting the gains from additional test-time compute. Improving reasoning quality directly would require process reward models, but the step-level annotations needed to train them are expensive and scarce. We find such a signal in how the model's confidence evolves during reasoning: premature confidence, the tendency to commit to an answer early and use the remaining tokens to rationalize it, strongly predicts flawed reasoning across tasks and model scales. We exploit this in progressive confidence shaping, a reinforcement learning objective that trains models to update their confidence as they reason rather than commit early -- rewarding gradual confidence growth and penalizing early commitment, with no external labels or reward models. The method improves accuracy and reasoning quality from 1.5B to 8B parameters across arithmetic (Countdown), math (DAPO, AIME), and science (ScienceQA): on Countdown, accuracy improves 3.2x (+42.0pp) and flawed reasoning drops 48pp; on AIME, Pass@64 improves 6.6pp. Consistent with this mechanism, the method also improves faithfulness: on a safety benchmark, our models more transparently surface misleading content in their reasoning traces rather than concealing it. Controlled experiments reveal that the problem and its remedy scale together: premature confidence grows with model size and task difficulty, and so do the gains from addressing it.
Jingchu Gai, Guanning Zeng, Christina Baek +4
1Carnegie Mellon University · 2Tsinghua University
Large language models (LLMs) have demonstrated impressive reasoning capabilities by scaling test-time compute via long Chain-of-Thought (CoT). However, recent findings suggest that raw token counts are unreliable proxies for reasoning quality: increased generation length does not consistently correlate with accuracy and may instead signal "overthinking," leading to performance degradation. In this work, we quantify inference-time effort by identifying deep-thinking tokens -- tokens where internal predictions undergo significant revisions in deeper model layers prior to convergence. Across four challenging mathematical and scientific benchmarks (AIME 24/25, HMMT 25, and GPQA-diamond) and a diverse set of reasoning-focused models (GPT-OSS, DeepSeek-R1, and Qwen3), we show that deep-thinking ratio (the proportion of deep-thinking tokens in a generated sequence) exhibits a robust and consistently positive correlation with accuracy, substantially outperforming both length-based and confidence-based baselines. Leveraging this insight, we introduce Think@n, a test-time scaling strategy that prioritizes samples with high deep-thinking ratios. We demonstrate that Think@n matches or exceeds standard self-consistency performance while significantly reducing inference costs by enabling the early rejection of unpromising generations based on short prefixes.
Scaling inference-time compute for Large Language Models (LLMs) has unlocked unprecedented reasoning capabilities. However, existing inference-time scaling methods typically rely on inefficient and suboptimal discrete search algorithms or trial-and-error prompting to improve the online policy. In this paper, we propose ∇-Reasoner, an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop to refine the policy on the fly. Our core component, Differentiable Textual Optimization (DTO), leverages gradient signals from both the LLM's likelihood and a reward model to refine textual representations. ∇-Reasoner further incorporates rejection sampling and acceleration design to robustify and speed up decoding. Theoretically, we show that performing inference-time gradient descent in the sample space to maximize reward is dual to aligning an LLM policy via KL-regularized reinforcement learning. Empirically, ∇-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark, while reducing number of model calls by approximately 10-40% compared to strong baselines. Overall, our work introduces a paradigm shift from zeroth-order search to first-order optimization at test time, offering a cost-effective path to amplify LLM reasoning.
Peihao Wang, Ruisi Cai, Zhen Wang +4
The University of Texas at Austin · UC San Diego · TTIC +1