Learning to Stop without Learning to Stop: Self-Supervised Confidence Training Improves Reasoning Efficiency

Parsa Hosseini; Akasha Tigalappanavara; Sumit Nawathe; Chenrui Fan; Sourya Basu; Genta Indra Winata; Anirban Das; Soheil Feizi

cs.AISep 25, 2026

Learning to Stop without Learning to Stop: Self-Supervised Confidence Training Improves Reasoning Efficiency

Authors: Parsa Hosseini, Akasha Tigalappanavara, Sumit Nawathe, Chenrui Fan, Sourya Basu, Genta Indra Winata, Anirban Das, Soheil Feizi, +1 more

Organizations: University of Maryland · AI Foundations, Capital One

Abstract

Reasoning models often generate very long reasoning traces, making inference computationally expensive. Existing approaches typically improve efficiency either through inference-time early-stopping mechanisms or by explicitly encouraging shorter reasoning during training, for example through reinforcement learning with length penalties. We show that substantial efficiency gains can instead emerge from a different kind of supervision: confidence. Using a self-supervised procedure, we fine-tune reasoning models to predict their confidence in the answer at intermediate points along their own reasoning trajectories using only 600 training problems. Confidence is used only as a training target: the loss contains no objective for reasoning length, efficiency, or stopping. At inference, the fine-tuned models use the standard generation procedure, with no confidence elicitation or early-stopping mechanism. Despite this, self-supervised confidence fine-tuning makes reasoning more efficient, reducing generated tokens by up to 25% at matched accuracy across Gemma, Qwen, Nemotron, and GPT-OSS models on mathematical, scientific, and coding reasoning benchmarks, with efficiency gains comparable to methods that explicitly optimize for shorter reasoning. Analysis of reasoning episodes further shows that confidence supervision largely preserves the base models' high-level reasoning composition rather than selectively suppressing particular behaviors. Our results suggest that efficient reasoning may emerge as a downstream consequence of learning metacognitive signals, without being directly optimized.

Figures & tables

Panel of Figure 1 — Figure 1: Teaching confidence instead of efficiency. Existing methods explicitly optimize for shorter reasoning traces; we supervise only confidence.

Figure 2: (a) Trial-answer correctness and (b) agreement with the final answer increase with confidence. (c) The model often continues generating after first reaching

c_{t}\geq 0.95

. (d) The expected accuracy gain from further reasoning decreases with confidence.

Panel of Figure 3 — Figure 3: Overview of ConfSFT. (a) The policy generates reasoning rollouts, constructs self-supervised confidence labels at intermediate states, and is fine-tuned only on the confidence targets. (b) At inference, the fine-tuned policy uses standard generation but can produce substantially shorter reasoning while reaching the same correct answer.

Nemotron-Nano-8B
Method	Math				Science		Coding				Average
	AIME2025		GSM8K		GPQA-Diamond		LiveCodeBench		HumanEval
	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Token Red.
Base	42.5	11,325	91.8	1,192	52.5	7,338	44.2	11,761	89.7	3,600	64.1	0.0%
DEER ( Yang et al., 2025 )	39.8	9,076 (-19.9%)	89.2	765 (-35.8%)	49.8	5,712 (-22.2%)	17.4	1,731 (-85.3%)	47.1	511 (-85.8%)	48.7	-49.8%
On-Policy SFT ( Zhao et al., 2026 )	39.0	10,487 (-7.4%)	91.1	999 (-16.2%)	50.3	6,146 (-16.2%)	41.8	11,364 (-3.4%)	89.3	3,174 (-11.8%)	62.3	-11.0%

Table 1: Main results. ConfSFT reduces generated tokens while maintaining base-model accuracy across model families and task domains.

Figure 4: Training Dynamics. (a) Accuracy remain stable. (b) Generated tokens decrease. (c) Confidence predictions improved. (d) Confidence predictions become more graded.

Figure 5: Different efficiency methods reshape reasoning in different ways. Following the Schoenfeld-style episode taxonomy, we compare each efficient policy with its corresponding base policy. Left: change in reasoning-token share for each episode, in percentage points. Right: total variation distance between the before- and after-training episode distributions.

	Gemma-4-E2B		Nemotron-Nano-8B
Method	Acc. $\uparrow$	Avg. Tokens $\downarrow$	Acc. $\uparrow$	Avg. Tokens $\downarrow$
Base	48.7	7,065	60.0	9,996
Position	46.5	6,550 (-7.3%)	61.0	10,122 (+1.3%)
Binary correctness	48.5	7,480 (+5.9%)	58.3	9,703 (-2.9%)
Shuffled confidence	48.7	7,057 (-0.1%)	57.1	10,392 (+4.0%)
ConfSFT	48.3	6,242 (-11.7%)	58.3	9,025 (-9.7%)

Table 2: Supervision-signal ablations. Replacing graded confidence with position, binary correctness, or shuffled confidence yields weaker or less consistent efficiency gains.

	Math				Science		Coding				Average
Marker	AIME2025		GSM8K		GPQA-Diamond		LiveCodeBench		HumanEval		Acc. $\uparrow$	Token Red. $\uparrow$
	Acc. $\uparrow$	Avg. Tokens $\downarrow$	Acc. $\uparrow$	Avg. Tokens $\downarrow$	Acc. $\uparrow$	Avg. Tokens $\downarrow$	Acc. $\uparrow$	Avg. Tokens $\downarrow$	Acc. $\uparrow$	Avg. Tokens $\downarrow$
Base	35.0	7,284	91.2	956	42.6	3,294	42.1	7,543	93.2	2,210	60.8	0.0%
Wait	32.1	5,987 (-17.8%)	91.4	938 (-1.9%)	40.8	2,968 (-9.9%)	42.4	6,403 (-15.1%)	93.8	2,059 (-6.8%)	60.1	-10.3%
\n\n	36.3	6,241 (-14.3%)	91.2	935 (-2.2%)	40.9	3,015 (-8.5%)	41.0	6,741 (-10.6%)	93.0	2,071 (-6.3%)	60.5	-8.4%

Table 3: Choice of decision-point marker on Gemma-4-E2B. ConfSFT remains effective when Wait is replaced with paragraph breaks ( \n\n ).

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	Nemotron-Nano-8B	Gemma-4-E2B	Qwen3-4B
System prompt	detailed thinking on	(none)	(none)
Think-end marker	</think>	<channel\|>	</think>
$T$ / top- $p$ / top- $k$	$0.6$ / $0.95$ / $20$	$1.0$ / $0.95$ / $64$	$0.6$ / $0.95$ / $20$
Learning rate	$1\times 10^{-6}$	$2\times 10^{-6}$	$1\times 10^{-6}$
Groups / problems per round	$8$ / $87$	$8$ / $87$	$8$ / $87$
Rollouts / training problem	$8$	$8$	$8$

Appendix

Table 4: ConfSFT hyperparameters for the three models in Table 1 . Shared settings are in the text above.

Nemotron-Nano-8B
Method	Math				Science		Coding				Average
	AIME2025		GSM8K		GPQA-Diamond		LiveCodeBench		HumanEval
	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Token Red.
DEER ( Yang et al., 2025 )	$-2.7{\pm}2.3$	$-2{,}249{\pm}593$	$-2.6{\pm}0.5$	$-427{\pm}29$	$-2.7{\pm}1.5$	$-1{,}626{\pm}220$	$-26.8{\pm}4.4$	$-10{,}030{\pm}944$	$-42.6{\pm}4.7$	$-3{,}089{\pm}701$	$-15.5{\pm}1.4$	$-49.8{\pm}1.6$ %
On-Policy SFT ( Zhao et al., 2026 )	$-3.5{\pm}4.9$	$-838{\pm}351$	$-0.7{\pm}0.5$	$-193{\pm}25$	$-2.2{\pm}2.1$	$-1{,}192{\pm}248$	$-2.3{\pm}1.4$	$-397{\pm}189$	$-0.4{\pm}1.2$	$-426{\pm}234$	$-1.8{\pm}1.1$	$-11.0{\pm}1.6$ %
A&Z ( Arora and Zanette, 2025 )	$-1.0{\pm}5.4$	$-1{,}681{\pm}458$	$-0.8{\pm}0.3$	$+26{\pm}16$	$-1.6{\pm}1.8$	$-600{\pm}151$	$+0.3{\pm}1.3$	$-415{\pm}224$	$-1.3{\pm}1.2$	$-112{\pm}253$	$-0.9{\pm}1.2$	$-5.5{\pm}1.6$ %

Appendix

Table 5: Main results relative to the base model, with paired

95\%

CIs. Acc and tokens are differences from the corresponding base row of Table 1 .

Nemotron-Nano-8B
Method	Math				Science		Coding				Average
	AIME2025		GSM8K		GPQA-Diamond		LiveCodeBench		HumanEval
	P@8 $\uparrow$	Avg. Tokens $\downarrow$	P@8 $\uparrow$	Avg. Tokens $\downarrow$	P@8 $\uparrow$	Avg. Tokens $\downarrow$	P@8 $\uparrow$	Avg. Tokens $\downarrow$	P@8 $\uparrow$	Avg. Tokens $\downarrow$	P@8 $\uparrow$	Token Red.
Base	67.1	11,325	97.0	1,192	82.3	7,338	59.4	11,761	97.9	3,600	80.7	0.0%
DEER ( Yang et al., 2025 )	66.9	9,076 (-19.9%)	96.8	765 (-35.8%)	79.7	5,712 (-22.2%)	34.3	1,731 (-85.3%)	80.5	511 (-85.8%)	71.7	-49.8%
On-Policy SFT ( Zhao et al., 2026 )	67.7	10,487 (-7.4%)	96.3	999 (-16.2%)	81.9	6,146 (-16.2%)	58.7	11,364 (-3.4%)	97.5	3,174 (-11.8%)	80.4	-11.0%

Appendix

Table 6: Pass@8. Same methods and benchmarks as Table 1 .

Figure 6: Gemma-4-E2B counterpart of Figure 2 .

Figure 7: Gemma-4-E2B counterpart of Figure 4 .

Figure 8: Accuracy (left) and token reduction (right) over training rounds for the four supervision signals in Table 2 , AIME 2024 validation. Dashed lines mark the pre-SFT baseline.

Nemotron-Nano-8B
Method	Math				Science		Coding				Average
	AIME2025		GSM8K		GPQA-Diamond		LiveCodeBench		HumanEval
	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Avg. Tokens $\downarrow$	Acc $\uparrow$	Token Red.
Base	42.5	11,325	91.8	1,192	52.5	7,338	44.2	11,761	89.7	3,600	64.1	0.0%
DEER ( $0.95$ )	39.8	9,076 (-19.9%)	89.2	765 (-35.8%)	49.8	5,712 (-22.2%)	17.4	1,731 (-85.3%)	47.1	511 (-85.8%)	48.7	-49.8%
DEER ( $0.98$ )	41.9	9,585 (-15.4%)	90.4	916 (-23.2%)	51.4	6,494 (-11.5%)	28.1	6,727 (-42.8%)	60.7	1,005 (-72.1%)	54.5	-33.0%

Appendix

Table 7: DEER confidence thresholds. Early exit at

0.95

and

0.98

thresholds.

Panel of Figure 9 — Figure 9: Length-penalty GRPO training curves.

Explore similar work

Early Stopping for Large Reasoning Models via Confidence Dynamics
Apr 6, 2026Parsa Hosseini, Sumit Nawathe, Mahdi Salmani +2Early Stopping Efficient Language Model Reasoning
University of Maryland · University of Southern California
Understanding and Mitigating Premature Confidence for Better LLM Reasoning
May 23, 2026Jingchu Gai, Guanning Zeng, Christina Baek +4Confidence Estimation in Language Models RL for Language Model Reasoning
1Carnegie Mellon University · 2Tsinghua University
Efficient Reasoning via Constrained Optimization in Latent Space
Sep 28, 2026Zhinan Hou, Xingchen Li, Keyou YouOverthinking in Language Models Efficient Language Model Reasoning
Tsinghua University

Apr 6, 2026J/K move · Enter open · S save

Early Stopping for Large Reasoning Models via Confidence Dynamics

Parsa Hosseini, Sumit Nawathe, Mahdi Salmani, Meisam Razaviyayn+1

University of Maryland · University of Southern California

Early Stopping Efficient Language Model Reasoning Large Reasoning Models Overthinking in Language Models Confidence Estimation in Language Models

Open paper AskPDF arXiv

Large reasoning models rely on long chain-of-thought generation to solve complex problems, but extended reasoning often incurs substantial computational cost and can even degrade performance due to overthinking. A key challenge is determining when the model should stop reasoning and produce the final answer. In this work, we study the confidence of intermediate answers during reasoning and observe two characteristic behaviors: correct reasoning trajectories often reach high-confidence answers early, while incorrect rollouts tend to produce long, unproductive reasoning traces and exhibit less reliable confidence dynamics. Motivated by these observations, we propose CoDE-Stop (Confidence Dynamics Early Stop), an early stopping method that leverages the dynamics of intermediate answer confidence to decide when to terminate reasoning, requiring no additional training and easily integrating into existing models. We evaluate CoDE-Stop on diverse reasoning and science benchmarks across multiple models. Compared to prior early stopping methods, it achieves a more favorable accuracy-compute tradeoff and reduces total token usage by 25-50% compared to standard full-length reasoning. In addition, we provide analyses of confidence dynamics during reasoning, offering insights into how confidence changes in both correct and incorrect trajectories.

Abstract

Explore similar work

Early Stopping for Large Reasoning Models via Confidence Dynamics

Understanding and Mitigating Premature Confidence for Better LLM Reasoning

Efficient Reasoning via Constrained Optimization in Latent Space