SELFDOUBT: Uncertainty Quantification for Reasoning LLMs via the Hedge-to-Verify Ratio

Satwik Pandey; Suresh Raghu; Shashwat Pandey

cs.AIApr 7, 2026

SELFDOUBT: Uncertainty Quantification for Reasoning LLMs via the Hedge-to-Verify Ratio

Authors: Satwik Pandey, Suresh Raghu, Shashwat Pandey

Organizations: Independent Researcher · Zillow Group

Abstract

Uncertainty estimation for reasoning language models remains difficult to deploy in practice: sampling-based methods are computationally expensive, while common single-pass proxies such as verbalized confidence or trace length are often inconsistent across models. This problem is compounded for proprietary reasoning APIs that expose neither logits nor intermediate token probabilities, leaving practitioners with no reliable uncertainty signal at inference time. We propose SELFDOUBT, a single-pass uncertainty framework that resolves this impasse by extracting behavioral signals directly from the reasoning trace itself. Our key signal, the Hedge-to-Verify Ratio (HVR), detects whether a reasoning trace contains uncertainty markers and, if so, whether they are offset by explicit selfchecking behavior. Unlike methods that require multiple sampled traces or model internals, SELFDOUBT operates on a single observed reasoning trajectory, making it suitable for latency- and cost-constrained deployment over any proprietary API. We evaluate SELFDOUBT across seven models and three multi-step reasoning benchmarks (BBH, GPQA-Diamond, and MMLU-Pro). Most notably, traces containing no hedging markers are correct 96% of the time, revealing an emergent high-precision confidence gate at zero additional cost. For the remaining cases, the full SELFDOUBT score significantly outperforms sampling-based semantic entropy at 10x lower inference cost. A deployment cascade combining both stages attains 90% accuracy at 71% coverage without any task-specific labels. These results establish SELFDOUBT as a scalable, production-ready foundation for uncertainty estimation over proprietary reasoning models.

Figures & tables

Figure 1: Headline results. (a) Across 21 runs, SelfDoubt is the only method that leads both metrics at

O(1)

cost: mean AUROC 0.7895 and mean AURAC 0.8992 (slightly above SE at 0.8988).

Method	Cost	Requirements
Verb. Conf. ( Tian et al., 2023 ; Yoon et al., 2025 )	$O(1)$	Prompt mod.
Trace Length ( Devic et al., 2025 )	$O(1)$	Trace
TL+VB ( Devic et al., 2025 )	$O(1)$	Trace + prompt mod.
Lex. Hints ( Vanhoyweghen et al., 2025 )	$O(1)$	Trace + sentiment scorer
CoT-UQ ( Zhang and Zhang, 2025 )	$O(1)$	Token logits
EPR/WEPR ( Moslonka et al., 2026 )	$O(1)$	Top- $K$ logprobs

Table 1: Uncertainty quantification methods for language models.

O(1)

methods above the first rule;

O(N)

and

O(N^{2})

below.

Figure 2: SelfDoubt pipeline: calibration-time marker discovery and inference-time scoring.

Group	Model	$N_{0}/N$	Coverage	Accuracy (HVR = 0)	Wilson 95% CI
Full traces	Qwen 3 4B	31/722	4.3%	90.3%	[75.1%, 96.7%]
	Qwen 3 14B	77/794	9.7%	96.1%	[89.2%, 98.7%]
	GPT OSS 20B	233/771	30.2%	96.1%	[92.8%, 98.0%]
	GPT OSS 120B	205/798	25.7%	96.6%	[93.1%, 98.3%]
Thought summaries	Claude Sonnet 4.6	425/797	53.3%	98.1%	[96.3%, 99.0%]
	Grok 4.1 Fast	406/798	50.9%	94.8%	[92.2%, 96.6%]

Table 2: Precision and coverage of the HVR = 0 subset with 95% Wilson confidence intervals, grouped by trace format. Counts are shown as

N_{0}/N

Method	Cost	All		Trace		Summ.
		AUROC	AURAC	AUROC	AURAC	AUROC	AURAC
SelfDoubt (ours)	$O(1)$	0.7895	0.8992	0.7984	0.8977	0.7777	0.9013
TL+VB	$O(1)$	0.7754	0.8792	0.7659	0.8869	0.7880	0.8689
HVR (ours)	$O(1)$	0.7509	0.8883	0.7674	0.8884	0.7288	0.8882
Verb	$O(1)$	0.7453	0.8765	0.7475	0.8687	0.7424	0.8869
TL	$O(1)$	0.6959	0.8575	0.6738	0.8644	0.7255	0.8483

Table 3: Main results across 21 runs (7 models

\times

3 datasets). “All” is the mean over all runs; “Trace” and “Summ.” are means over the 12 full-trace and 9 thought-summary runs, respectively.

O(1)

methods above rule,

O(N)

below. Bold : best; underline : second-best.

Comparison	Metric	Mean $\Delta$	W–D–L	$W$	$p$ (one-sided)	Sig.?
vs. SE	AUROC	+0.049	17–0–4	198	0.001	Yes
vs. SE	AURAC	+0.000	16–0–5	159	0.069	No
vs. Verb	AUROC	+0.044	16–1–4	190	$<$ 0.001	Yes
vs. Verb	AURAC	+0.023	18–0–3	223	$<$ 0.001	Yes
vs. TL+VB	AUROC	+0.014	12–0–9	146	0.152	No
vs. TL+VB	AURAC	+0.020	11–0–10	142	0.187	No

Table 4: Paired Wilcoxon signed-rank tests (one-sided: SelfDoubt

>

baseline; 21 matched run pairs). W–D–L = wins–draws–losses for SelfDoubt .

Figure 3: AUROC–AURAC consistency on thought summary models. SelfDoubt occupies the upper-right quadrant among

O(1)

methods. SE shows strong AURAC but weak AUROC on compressed traces.

Figure 4: Accuracy–coverage curve for the SelfDoubt deployment cascade.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Model	N	SD	HVR	Verb	TL	TL+VB	SE	SV	GEO
BBH	gpt_120b	300	0.832	0.742	0.832	0.623	0.817	0.779	0.749	0.729
BBH	gpt_20b	292	0.773	0.703	0.778	0.568	0.780	0.819	0.776	0.784
BBH	qwen3_4b	300	0.747	0.733	0.667	0.649	0.702	0.732	0.628	0.505
BBH	qwen3_14b	300	0.799	0.746	0.785	0.524	0.723	0.727	0.537	0.611
GPQA	gpt_120b	198	0.844	0.813	0.835	0.761	0.839	0.760	0.758	0.697
GPQA	gpt_20b	180	0.736	0.721	0.727	0.767	0.777	0.646	0.689	0.556

Appendix

Table 5: Per-run AUROC: raw reasoning trace models.

Dataset	Model	N	SD	HVR	Verb	TL	TL+VB	SE	SV	GEO
BBH	claude	300	0.809	0.761	0.795	0.611	0.714	0.653	0.570	0.535
BBH	grok	300	0.826	0.771	0.821	0.723	0.838	0.785	0.563	0.700
BBH	gemini	293	0.640	0.645	0.532	0.827	0.770	0.663	0.516	0.614
GPQA	claude	198	0.787	0.686	0.807	0.772	0.819	0.679	0.564	0.503
GPQA	grok	198	0.831	0.791	0.801	0.809	0.887	0.709	0.635	0.687
GPQA	gemini	197	0.662	0.638	0.592	0.635	0.666	0.771	0.649	0.609

Appendix

Table 6: Per-run AUROC: thought summary models.

Dataset	Model	N	SD	HVR	Verb	TL	TL+VB	SE	SV	GEO
BBH	gpt_120b	300	0.975	0.947	0.962	0.932	0.971	0.955	0.972	0.970
BBH	gpt_20b	292	0.963	0.955	0.948	0.933	0.966	0.947	0.945	0.946
BBH	qwen3_4b	300	0.933	0.933	0.921	0.912	0.920	0.933	0.939	0.908
BBH	qwen3_14b	300	0.965	0.949	0.954	0.902	0.949	0.918	0.908	0.929
GPQA	gpt_120b	198	0.882	0.871	0.877	0.834	0.879	0.878	0.881	0.864
GPQA	gpt_20b	180	0.715	0.716	0.681	0.726	0.731	0.790	0.821	0.779

Appendix

Table 7: Per-run AURAC: raw reasoning trace models.

Dataset	Model	N	SD	HVR	Verb	TL	TL+VB	SE	SV	GEO
BBH	claude	300	0.953	0.948	0.949	0.955	0.967	0.916	0.941	0.938
BBH	grok	300	0.970	0.960	0.964	0.957	0.972	0.934	0.902	0.945
BBH	gemini	293	0.709	0.711	0.634	0.371	0.371	0.904	0.906	0.936
GPQA	claude	198	0.925	0.881	0.928	0.913	0.936	0.916	0.905	0.876
GPQA	grok	198	0.947	0.937	0.938	0.925	0.960	0.929	0.772	0.908
GPQA	gemini	197	0.790	0.785	0.771	0.785	0.792	0.864	0.820	0.803

Appendix

Table 8: Per-run AURAC: thought summary models.

Model	$\tau_{\text{verify}}$	$\tau_{\text{hedge}}$
Claude Sonnet 4.6	0.09	0.08
Gemini 2.5 Flash	0.12	0.05
gpt-oss-20b	0.14	0.20
gpt-oss-120b	0.10	0.15
Grok 4.1 Fast	0.10	0.10
Qwen3	0.17	0.20

Appendix

Table 9: Per-model Stage 2 thresholds for marker classification.

Variant / category	Count	% of 54	% of $N_{0}{=}1384$
Raw disagreements	54	100.0%	3.90%
Grading or answer-key issue	30	55.6%	2.17%
Ambiguous question / label	16	29.6%	1.16%
Genuinely wrong confident	8	14.8%	0.58%
Raw precision	1330 / 1384	—	96.10%
Audit-adjusted precision	1376 / 1384	—	99.42%

Appendix

Table 10: Audit of all 54 raw HVR = 0 disagreements.

Scenario	Genuine errors	Precision	Wilson 95% CI
As annotated	8	99.42%	[98.86%, 99.71%]
All ambiguous $\rightarrow$ genuine	24	98.27%	[97.32%, 98.89%]
Zero correction (raw labels only)	54	96.10%	[94.94%, 97.00%]

Appendix

Table 11: Sensitivity of HVR = 0 precision to audit trust level.

Traces	Sampling	Model	SD AUROC	HVR AUROC
15	stratified	Claude	— (empty markers)	—
30	stratified	Claude	0.821	0.671
60	stratified	Claude	0.829	0.751
90	stratified	Claude	0.833	0.756
120	stratified	Claude	0.831	0.748
180	stratified	Claude	0.827	0.743

Appendix

Table 12: Sample-size ablation under stratified sampling.

Traces	Sampling	Model	SD AUROC	HVR AUROC
15	random_pooled	Claude	—	—
30	random_pooled	Claude	—	—
60	random_pooled	Claude	0.835	0.754
90	random_pooled	Claude	0.831	0.751
120	random_pooled	Claude	0.833	0.759
180	random_pooled	Claude	0.828	0.746

Appendix

Table 13: Sample-size ablation under random pooled sampling.

Embedding	Model	BBH	GPQA	MMLU	Mean
SELFDOUBT AUROC
bge-m3	Claude	0.825	0.824	0.904	0.851
OpenAI-3-large	Claude	0.820	0.832	0.905	0.852
Qwen3-emb-8b	Claude	0.822	0.829	0.903	0.852
bge-m3	Qwen3	0.748	0.793	0.758	0.766
OpenAI-3-large	Qwen3	0.763	0.813	0.765	0.780

Appendix

Table 14: Embedding-model ablation for SelfDoubt AUROC across datasets.

Seed Set	Mean SD AUROC	$\Delta$ vs top_10
top_10	0.7895	0.000
top_8	0.7861	$-$ 0.003
top_20	0.7826	$-$ 0.007
top_2	0.7822	$-$ 0.007
top_4	0.7815	$-$ 0.008
top_12	0.7814	$-$ 0.008

Appendix

Table 15: Seed-set size ablation on mean SelfDoubt AUROC.

Model	Original	MuSR@90	$\Delta$
gpt-oss-120b	0.8406	0.8366	$-$ 0.004
gpt-oss-20b	0.7908	0.7867	$-$ 0.004
Qwen3	0.7745	0.7654	$-$ 0.009
Qwen3-14B	0.7876	0.7853	$-$ 0.002
Claude Sonnet 4.6	0.8358	0.8358	0.000
Grok 4.1 Fast	0.8101	0.8089	$-$ 0.001

Appendix

Table 16: Cross-dataset marker transfer: per-model mean SelfDoubt AUROC at MuSR@90.

Model	Dataset	Data-Driven	Seeds Only	$\Delta$ (HVR)
Claude	BBH	0.733	0.659	$-$ 0.074
Claude	MMLU-Pro	0.834	0.700	$\mathbf{-0.134}$
Grok	MMLU-Pro	0.720	0.678	$-$ 0.042

Appendix

Table 17: Stage 2 data-driven markers vs. seeds-only markers for HVR AUROC on summaries.

Tier-2 Ranker	Cascade AURAC	T2 AUROC (HVR $>$ 0)	Cost
z-sum (SD)	0.8944	0.7572	$O(1)$
HVR only	0.8883	0.6975	$O(1)$
SE	0.8830	0.7250	$O(N)$
Verb only	0.8756	0.7180	$O(1)$
TL+VB	0.8730	0.7149	$O(1)$
Random	0.8245	0.4917	—

Appendix

Table 18: Deployment Tier-2 ranker comparison on hard-subset ranking and cascade performance.

Group	Comparison	$n$	Mean $\Delta$ AUROC	W–D–L	$p$
Full traces	vs TL+VB	12	+0.032	9–0–3	0.026
Full traces	vs SE	12	+0.042	10–0–2	0.005
Full traces	vs Verb	12	+0.051	9–1–2	0.002
Summaries	vs TL+VB	9	$-$ 0.010	3–0–6	0.787
Summaries	vs SE	9	+0.058	7–0–2	0.037
Summaries	vs Verb	9	+0.035	7–0–2	0.049

Appendix

Table 19: Subgroup paired Wilcoxon results for SelfDoubt vs. baselines on AUROC and AURAC.

Panel of Figure 5 — Figure 5: Scaling vs. model size (raw traces only). (a) AUROC. (b) AURAC.

Answer type	Tasks and instruction
Boxed True / False	boolean_expressions : output the final answer exactly once in the form \boxed{...} using only True or False .
Boxed Yes / No	causal_judgement , navigate , web_of_lies : output the final answer exactly once in the form \boxed{...} using only Yes or No .
Boxed lowercase yes / no	sports_understanding : output the final answer exactly once in the form \boxed{...} using only yes or no .
Boxed valid / invalid	formal_fallacies : output the final answer exactly once in the form \boxed{...} using only valid or invalid .
Boxed option label	date_understanding , disambiguation_qa , geometric_shapes , hyperbaton , logical_deduction_three_objects , logical_deduction_five_objects , logical_deduction_seven_objects , movie_recommendation , penguins_in_a_table , reasoning_about_colored_objects , ruin_names , salient_translation_error_detection , snarks , temporal_sequences , tracking_shuffled_objects_three_objects , tracking_shuffled_objects_five_objects , tracking_shuffled_objects_seven_objects : output the final answer exactly once in the form \boxed{...} using only the option label in parentheses, e.g., (C) .
Boxed integer	multistep_arithmetic_two : output the final answer exactly once in the form \boxed{...} using only the final integer (with sign if needed). object_counting : output the final answer exactly once in the form \boxed{...} using only the final integer.

Appendix

Table 20: BBH task-family answer-format instructions used in verbalized-confidence prompting.

Hedge	Verify
possibly	check
seemingly	reassess
maybe	reevaluate
apparently	re-evaluate
probably	reinspect
presumably	rechecking

Appendix

Table 21: Example retained Stage-1 seeds after coherence filtering.

Multiplier	$\tau_{\text{verify}}$ range	$\tau_{\text{hedge}}$ range	Mean SD AUROC	$\Delta$ vs. 1.0 $\times$
$0.5\times$	[0.045, 0.085]	[0.025, 0.100]	0.7675	$-$ 0.022
$0.75\times$	[0.068, 0.128]	[0.038, 0.150]	0.7807	$-$ 0.009
$\mathbf{1.0\times}$ (default)	[0.090, 0.170]	[0.050, 0.200]	0.7895	0.000
$1.25\times$	[0.113, 0.213]	[0.063, 0.250]	0.7854	$-$ 0.004
$1.5\times$	[0.135, 0.255]	[0.075, 0.300]	0.7649	$-$ 0.025

Appendix

Table 22: Sensitivity of mean SelfDoubt AUROC to joint Stage 2 threshold scaling.

Explore similar work

How Uncertainty Estimation Scales with Sampling in Reasoning Models
Mar 19, 2026Maksym Del, Markus Kängsepp, Marharyta Domnich +4LLM Uncertainty Estimation Uncertainty Quantification
Institute of Computer Science, University of Tartu
The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Models
Mar 17, 2026Robert Welch, Emir Konuk, Kevin SmithVLM Reasoning
KTH Royal Institute of Technology, Stockholm, Sweden · Science for Life Laboratory, Stockholm, Sweden
Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs
Sep 29, 2026Feiyang Li, Shengjing Liu, Qi Zhan +7Uncertainty Quantification Confidence Estimation in Language Models
College of Computer Science and Software Engineering, Shenzhen University · Tsinghua University · Behavioral and Spatial AI Lab, Peking University & Tongji University

Mar 19, 2026J/K move · Enter open · S save

How Uncertainty Estimation Scales with Sampling in Reasoning Models

Maksym Del, Markus Kängsepp, Marharyta Domnich, Ardi Tampuu+3

Institute of Computer Science, University of Tartu

LLM Uncertainty Estimation Uncertainty Quantification Confidence Estimation in Language Models Large Reasoning Models Self-Consistency in Language Models

Open paper AskPDF arXiv

Uncertainty estimation is critical for deploying reasoning language models, yet remains poorly understood under extended chain-of-thought reasoning. We study parallel sampling as a fully black-box approach using verbalized confidence and self-consistency. Across three reasoning models and 17 tasks spanning mathematics, STEM, and humanities, we characterize how these signals scale. Both self-consistency and verbalized confidence scale in reasoning models, but self-consistency exhibits lower initial discrimination and lags behind verbalized confidence under moderate sampling. Most uncertainty gains, however, arise from signal combination: with just two samples, a hybrid estimator improves AUROC by up to

+12

on average and already outperforms either signal alone even when scaled to much larger budgets, after which returns diminish. These effects are domain-dependent: in mathematics, the native domain of RLVR-style post-training, reasoning models achieve higher uncertainty quality and exhibit both stronger complementarity and faster scaling than in STEM or humanities.

Abstract

Explore similar work

How Uncertainty Estimation Scales with Sampling in Reasoning Models

The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Models

Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs