Learning to Stop without Learning to Stop: Self-Supervised Confidence Training Improves Reasoning Efficiency

Parsa Hosseini; Akasha Tigalappanavara; Sumit Nawathe; Chenrui Fan; Sourya Basu; Genta Indra Winata; Anirban Das; Soheil Feizi

cs.AISep 25, 2026

Learning to Stop without Learning to Stop: Self-Supervised Confidence Training Improves Reasoning Efficiency

Authors: Parsa Hosseini, Akasha Tigalappanavara, Sumit Nawathe, Chenrui Fan, Sourya Basu, Genta Indra Winata, Anirban Das, Soheil Feizi, +1 more

Organizations: University of Maryland · AI Foundations, Capital One

Abstract

Reasoning models often generate very long reasoning traces, making inference computationally expensive. Existing approaches typically improve efficiency either through inference-time early-stopping mechanisms or by explicitly encouraging shorter reasoning during training, for example through reinforcement learning with length penalties. We show that substantial efficiency gains can instead emerge from a different kind of supervision: confidence. Using a self-supervised procedure, we fine-tune reasoning models to predict their confidence in the answer at intermediate points along their own reasoning trajectories using only 600 training problems. Confidence is used only as a training target: the loss contains no objective for reasoning length, efficiency, or stopping. At inference, the fine-tuned models use the standard generation procedure, with no confidence elicitation or early-stopping mechanism. Despite this, self-supervised confidence fine-tuning makes reasoning more efficient, reducing generated tokens by up to 25% at matched accuracy across Gemma, Qwen, Nemotron, and GPT-OSS models on mathematical, scientific, and coding reasoning benchmarks, with efficiency gains comparable to methods that explicitly optimize for shorter reasoning. Analysis of reasoning episodes further shows that confidence supervision largely preserves the base models' high-level reasoning composition rather than selectively suppressing particular behaviors. Our results suggest that efficient reasoning may emerge as a downstream consequence of learning metacognitive signals, without being directly optimized.

Figures & tables

Panel of Figure 1 — Figure 1: Teaching confidence instead of efficiency. Existing methods explicitly optimize for shorter reasoning traces; we supervise only confidence.

Figure 2: (a) Trial-answer correctness and (b) agreement with the final answer increase with confidence. (c) The model often continues generating after first reaching

c_{t}\geq 0.95

. (d) The expected accuracy gain from further reasoning decreases with confidence.

Panel of Figure 3 — Figure 3: Overview of ConfSFT. (a) The policy generates reasoning rollouts, constructs self-supervised confidence labels at intermediate states, and is fine-tuned only on the confidence targets. (b) At inference, the fine-tuned policy uses standard generation but can produce substantially shorter reasoning while reaching the same correct answer.

Nemotron-Nano-8B
Method	Math				Science		Coding				Average
	AIME2025		GSM8K		GPQA-Diamond		LiveCodeBench		HumanEval
	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Token Red.
Base	42.5	11,325	91.8	1,192	52.5	7,338	44.2	11,761	89.7	3,600	64.1	0.0%
DEER ( Yang et al., 2025 )	39.8	9,076 (-19.9%)	89.2	765 (-35.8%)	49.8	5,712 (-22.2%)	17.4	1,731 (-85.3%)	47.1	511 (-85.8%)	48.7	-49.8%
On-Policy SFT ( Zhao et al., 2026 )	39.0	10,487 (-7.4%)	91.1	999 (-16.2%)	50.3	6,146 (-16.2%)	41.8	11,364 (-3.4%)	89.3	3,174 (-11.8%)	62.3	-11.0%

Table 1: Main results. ConfSFT reduces generated tokens while maintaining base-model accuracy across model families and task domains.

Figure 4: Training Dynamics. (a) Accuracy remain stable. (b) Generated tokens decrease. (c) Confidence predictions improved. (d) Confidence predictions become more graded.

Figure 5: Different efficiency methods reshape reasoning in different ways. Following the Schoenfeld-style episode taxonomy, we compare each efficient policy with its corresponding base policy. Left: change in reasoning-token share for each episode, in percentage points. Right: total variation distance between the before- and after-training episode distributions.

	Gemma-4-E2B		Nemotron-Nano-8B
Method	Acc. $\uparrow$	Avg. Tokens $\downarrow$	Acc. $\uparrow$	Avg. Tokens $\downarrow$
Base	48.7	7,065	60.0	9,996
Position	46.5	6,550 (-7.3%)	61.0	10,122 (+1.3%)
Binary correctness	48.5	7,480 (+5.9%)	58.3	9,703 (-2.9%)
Shuffled confidence	48.7	7,057 (-0.1%)	57.1	10,392 (+4.0%)
ConfSFT	48.3	6,242 (-11.7%)	58.3	9,025 (-9.7%)

Table 2: Supervision-signal ablations. Replacing graded confidence with position, binary correctness, or shuffled confidence yields weaker or less consistent efficiency gains.

	Math				Science		Coding				Average
Marker	AIME2025		GSM8K		GPQA-Diamond		LiveCodeBench		HumanEval		Acc. $\uparrow$	Token Red. $\uparrow$
	Acc. $\uparrow$	Avg. Tokens $\downarrow$	Acc. $\uparrow$	Avg. Tokens $\downarrow$	Acc. $\uparrow$	Avg. Tokens $\downarrow$	Acc. $\uparrow$	Avg. Tokens $\downarrow$	Acc. $\uparrow$	Avg. Tokens $\downarrow$
Base	35.0	7,284	91.2	956	42.6	3,294	42.1	7,543	93.2	2,210	60.8	0.0%
Wait	32.1	5,987 (-17.8%)	91.4	938 (-1.9%)	40.8	2,968 (-9.9%)	42.4	6,403 (-15.1%)	93.8	2,059 (-6.8%)	60.1	-10.3%
\n\n	36.3	6,241 (-14.3%)	91.2	935 (-2.2%)	40.9	3,015 (-8.5%)	41.0	6,741 (-10.6%)	93.0	2,071 (-6.3%)	60.5	-8.4%

Table 3: Choice of decision-point marker on Gemma-4-E2B. ConfSFT remains effective when Wait is replaced with paragraph breaks ( \n\n ).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	Nemotron-Nano-8B	Gemma-4-E2B	Qwen3-4B
System prompt	detailed thinking on	(none)	(none)
Think-end marker	</think>	<channel\|>	</think>
$T$ / top- $p$ / top- $k$	$0.6$ / $0.95$ / $20$	$1.0$ / $0.95$ / $64$	$0.6$ / $0.95$ / $20$
Learning rate	$1\times 10^{-6}$	$2\times 10^{-6}$	$1\times 10^{-6}$
Groups / problems per round	$8$ / $87$	$8$ / $87$	$8$ / $87$
Rollouts / training problem	$8$	$8$	$8$

Appendix

Table 4: ConfSFT hyperparameters for the three models in Table 1 . Shared settings are in the text above.

Nemotron-Nano-8B
Method	Math				Science		Coding				Average
	AIME2025		GSM8K		GPQA-Diamond		LiveCodeBench		HumanEval
	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Token Red.
DEER ( Yang et al., 2025 )	$-2.7{\pm}2.3$	$-2{,}249{\pm}593$	$-2.6{\pm}0.5$	$-427{\pm}29$	$-2.7{\pm}1.5$	$-1{,}626{\pm}220$	$-26.8{\pm}4.4$	$-10{,}030{\pm}944$	$-42.6{\pm}4.7$	$-3{,}089{\pm}701$	$-15.5{\pm}1.4$	$-49.8{\pm}1.6$ %
On-Policy SFT ( Zhao et al., 2026 )	$-3.5{\pm}4.9$	$-838{\pm}351$	$-0.7{\pm}0.5$	$-193{\pm}25$	$-2.2{\pm}2.1$	$-1{,}192{\pm}248$	$-2.3{\pm}1.4$	$-397{\pm}189$	$-0.4{\pm}1.2$	$-426{\pm}234$	$-1.8{\pm}1.1$	$-11.0{\pm}1.6$ %
A&Z ( Arora and Zanette, 2025 )	$-1.0{\pm}5.4$	$-1{,}681{\pm}458$	$-0.8{\pm}0.3$	$+26{\pm}16$	$-1.6{\pm}1.8$	$-600{\pm}151$	$+0.3{\pm}1.3$	$-415{\pm}224$	$-1.3{\pm}1.2$	$-112{\pm}253$	$-0.9{\pm}1.2$	$-5.5{\pm}1.6$ %

Appendix

Table 5: Main results relative to the base model, with paired

95\%

CIs. Acc and tokens are differences from the corresponding base row of Table 1 .

Nemotron-Nano-8B
Method	Math				Science		Coding				Average
	AIME2025		GSM8K		GPQA-Diamond		LiveCodeBench		HumanEval
	P@8 $\uparrow$	Avg. Tokens $\downarrow$	P@8 $\uparrow$	Avg. Tokens $\downarrow$	P@8 $\uparrow$	Avg. Tokens $\downarrow$	P@8 $\uparrow$	Avg. Tokens $\downarrow$	P@8 $\uparrow$	Avg. Tokens $\downarrow$	P@8 $\uparrow$	Token Red.
Base	67.1	11,325	97.0	1,192	82.3	7,338	59.4	11,761	97.9	3,600	80.7	0.0%
DEER ( Yang et al., 2025 )	66.9	9,076 (-19.9%)	96.8	765 (-35.8%)	79.7	5,712 (-22.2%)	34.3	1,731 (-85.3%)	80.5	511 (-85.8%)	71.7	-49.8%
On-Policy SFT ( Zhao et al., 2026 )	67.7	10,487 (-7.4%)	96.3	999 (-16.2%)	81.9	6,146 (-16.2%)	58.7	11,364 (-3.4%)	97.5	3,174 (-11.8%)	80.4	-11.0%

Appendix

Table 6: Pass@8. Same methods and benchmarks as Table 1 .

Figure 6: Gemma-4-E2B counterpart of Figure 2 .

Figure 7: Gemma-4-E2B counterpart of Figure 4 .

Figure 8: Accuracy (left) and token reduction (right) over training rounds for the four supervision signals in Table 2 , AIME 2024 validation. Dashed lines mark the pre-SFT baseline.

Nemotron-Nano-8B
Method	Math				Science		Coding				Average
	AIME2025		GSM8K		GPQA-Diamond		LiveCodeBench		HumanEval
	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Token Red.
Base	42.5	11,325	91.8	1,192	52.5	7,338	44.2	11,761	89.7	3,600	64.1	0.0%
DEER ( $0.95$ )	39.8	9,076 (-19.9%)	89.2	765 (-35.8%)	49.8	5,712 (-22.2%)	17.4	1,731 (-85.3%)	47.1	511 (-85.8%)	48.7	-49.8%
DEER ( $0.98$ )	41.9	9,585 (-15.4%)	90.4	916 (-23.2%)	51.4	6,494 (-11.5%)	28.1	6,727 (-42.8%)	60.7	1,005 (-72.1%)	54.5	-33.0%

Appendix

Table 7: DEER confidence thresholds. Early exit at

0.95

and

0.98

thresholds.

Panel of Figure 9 — Figure 9: Length-penalty GRPO training curves.

Explore similar work

Apr 6, 2026

Early Stopping for Large Reasoning Models via Confidence Dynamics

Large reasoning models rely on long chain-of-thought generation to solve complex problems, but extended reasoning often incurs substantial computational cost and can even degrade performance due to overthinking. A key challenge is determining when the model should stop reasoning and produce the final answer. In this work, we study the confidence of intermediate answers during reasoning and observe two characteristic behaviors: correct reasoning trajectories often reach high-confidence answers early, while incorrect rollouts tend to produce long, unproductive reasoning traces and exhibit less reliable confidence dynamics. Motivated by these observations, we propose CoDE-Stop (Confidence Dynamics Early Stop), an early stopping method that leverages the dynamics of intermediate answer confidence to decide when to terminate reasoning, requiring no additional training and easily integrating into existing models. We evaluate CoDE-Stop on diverse reasoning and science benchmarks across multiple models. Compared to prior early stopping methods, it achieves a more favorable accuracy-compute tradeoff and reduces total token usage by 25-50% compared to standard full-length reasoning. In addition, we provide analyses of confidence dynamics during reasoning, offering insights into how confidence changes in both correct and incorrect trajectories.

Parsa Hosseini, Sumit Nawathe, Mahdi Salmani +2

University of Maryland · University of Southern California

Early Stopping Efficient Language Model Reasoning Large Reasoning Models Overthinking in Language Models Confidence Estimation in Language Models

Ask PDF

May 23, 2026

Understanding and Mitigating Premature Confidence for Better LLM Reasoning

Long chains of thought (CoT) from current language models frequently contain logical gaps and unjustified leaps, limiting the gains from additional test-time compute. Improving reasoning quality directly would require process reward models, but the step-level annotations needed to train them are expensive and scarce. We find such a signal in how the model's confidence evolves during reasoning: premature confidence, the tendency to commit to an answer early and use the remaining tokens to rationalize it, strongly predicts flawed reasoning across tasks and model scales. We exploit this in progressive confidence shaping, a reinforcement learning objective that trains models to update their confidence as they reason rather than commit early -- rewarding gradual confidence growth and penalizing early commitment, with no external labels or reward models. The method improves accuracy and reasoning quality from 1.5B to 8B parameters across arithmetic (Countdown), math (DAPO, AIME), and science (ScienceQA): on Countdown, accuracy improves 3.2x (+42.0pp) and flawed reasoning drops 48pp; on AIME, Pass@64 improves 6.6pp. Consistent with this mechanism, the method also improves faithfulness: on a safety benchmark, our models more transparently surface misleading content in their reasoning traces rather than concealing it. Controlled experiments reveal that the problem and its remedy scale together: premature confidence grows with model size and task difficulty, and so do the gains from addressing it.

Jingchu Gai, Guanning Zeng, Christina Baek +4

1Carnegie Mellon University · 2Tsinghua University

Confidence Estimation in Language Models RL for Language Model Reasoning LLM Uncertainty Estimation LLM Reasoning

Ask PDF

Sep 28, 2026

Efficient Reasoning via Constrained Optimization in Latent Space

Large Reasoning Models (LRMs) have shown remarkable reasoning capabilities, yet they still suffer from overthinking, generating redundant reasoning steps which incur substantial token consumption. Existing methods, such as suppressing reflective keywords or forcing shorter reasoning lengths, attempt to mitigate this issue but inevitably truncate necessary steps and induce underthinking, thereby compromising performance. To address this dilemma, we investigate the latent representations and observe that efficient reasoning steps naturally cluster into a concentrated region in latent space, while those deviating from this region tend to produce verbose sequences. To leverage this, we keep reasoning focused within this region via a quadratic program which projects deviating hidden states back into the region. Then we propose a novel training-free framework to achieve efficient reasoning that reduces token generation costs without sacrificing performance. Extensive experiments conducted on four models ranging from 1.5B to 14B, and across six benchmarks in math reasoning, coding, and scientific QA, validate the effectiveness of our method, up to a 12.1% improvement in accuracy while reducing generated tokens by 11.8% to 52.8%. Codes are available at https://github.com/hzn18/Opt4Reasoning.

Zhinan Hou, Xingchen Li, Keyou You

Tsinghua University

Overthinking in Language Models Efficient Language Model Reasoning Quadratic Programming Efficient Language Model Inference Latent Space Optimization Large Reasoning Models

Ask PDF