When Should a Language Model Trust Itself? Same-Model Self-Verification as a Conditional Confidence Signal

Aditya Ajay Phalod

cs.CLApr 8, 2026

When Should a Language Model Trust Itself? Same-Model Self-Verification as a Conditional Confidence Signal

Authors: Aditya Ajay Phalod

Organizations: Independent Researcher

Abstract

Same-model self-verification, prompting a model to audit its own predicted answer, is a plausible confidence signal for selective prediction, but its practical value remains unclear once strong likelihood-based baselines are taken seriously. We evaluate self-verification against two such baselines, LL-AVG and LL-SUM, on ARC-Challenge and TruthfulQA-MC across multiple model families, scales, and prompt variants. We measure not only correctness ranking, but also abstention quality through AURC and operating-point analyses. The results are sharply task- and model-dependent. On ARC-Challenge, self-verification substantially improves over LL-AVG for Phi-2 and the Qwen models, with the largest gains appearing in Qwen-7B. On TruthfulQA-MC, however, the signal is less reliable: smaller models can become prompt-sensitive, DeepSeek-R1-Distill-8B degrades relative to LL-AVG, and LL-SUM often remains the stronger practical baseline. We therefore do not treat self-verification as a general-purpose uncertainty estimator. In this setting, it is better understood as a conditional confidence signal whose value depends on task type, model family, prompt formulation, and, crucially, the baseline it must beat.

Figures & tables

Panel of Figure 1 — Figure 1: AUROC by confidence signal across datasets under the default verification prompt. Self-Verify is strongly positive on ARC-Challenge for several models, but much less uniform on TruthfulQA-MC.

Panel of Figure 2 — Figure 2: AURC by confidence signal across datasets under the default verification prompt. Lower is better. ARC-Challenge shows large selective-prediction gains for Self-Verify on several models, while TruthfulQA-MC is much less favorable.

Dataset	Model	Prompt	Acc (LL-AVG)	AUROC (LL-AVG)	AUROC (Self-Verify)	AUROC (LL-SUM)	AURC (LL-AVG)	AURC (Self-Verify)	AURC (LL-SUM)
ARC-Challenge	DeepSeek-R1-Distill-8B	audit_v1	0.278	0.511	0.489	0.533	0.722	0.738	0.724
ARC-Challenge	DeepSeek-R1-Distill-8B	default	0.278	0.511	0.463	0.533	0.722	0.751	0.724
ARC-Challenge	Phi-2	audit_v1	0.534	0.547	0.755	0.693	0.427	0.278	0.233
ARC-Challenge	Phi-2	default	0.534	0.547	0.759	0.693	0.427	0.275	0.233
ARC-Challenge	Qwen-1.5B	audit_v1	0.492	0.557	0.770	0.693	0.472	0.296	0.273
ARC-Challenge	Qwen-1.5B	default	0.492	0.557	0.765	0.693	0.472	0.298	0.273

Table 1: Main results across datasets, models, and prompt variants. The table reports prediction accuracy under LL-AVG, along with AUROC and AURC for LL-AVG, Self-Verify, and LL-SUM. LL-AVG and LL-SUM values are repeated across prompt rows because only the verification prompt is varied. Lower AURC is better.

Dataset	Model	Prompt	$\Delta$ AUROC (SV–LLAVG)	$\Delta$ AURC (SV–LLAVG)	$\Delta$ AUROC (SV–LLSUM)	$\Delta$ AURC (SV–LLSUM)
ARC-Challenge	DeepSeek-R1-Distill-8B	audit_v1	-0.022	0.016	-0.043	0.014
ARC-Challenge	DeepSeek-R1-Distill-8B	default	-0.048	0.030	-0.069	0.027
ARC-Challenge	Phi-2	audit_v1	0.207	-0.149	0.061	0.045
ARC-Challenge	Phi-2	default	0.211	-0.153	0.066	0.041
ARC-Challenge	Qwen-1.5B	audit_v1	0.213	-0.176	0.077	0.023
ARC-Challenge	Qwen-1.5B	default	0.207	-0.173	0.071	0.025

Table 2: Pairwise deltas for Self-Verify relative to LL-AVG and LL-SUM. Negative

\Delta

AURC indicates lower risk for Self-Verify.

Dataset	Model	Prompt	Mean $\Delta$ AUROC	2.5% CI	97.5% CI
ARC-Challenge	DeepSeek-R1-Distill-8B	audit_v1	-0.023	-0.073	0.029
ARC-Challenge	DeepSeek-R1-Distill-8B	default	-0.049	-0.100	0.002
ARC-Challenge	Phi-2	audit_v1	0.208	0.168	0.251
ARC-Challenge	Phi-2	default	0.212	0.173	0.253
ARC-Challenge	Qwen-1.5B	audit_v1	0.213	0.169	0.257
ARC-Challenge	Qwen-1.5B	default	0.208	0.164	0.251

Table 3: Bootstrap estimates for

\Delta

AUROC

=\mathrm{AUROC}(\mathrm{Self\mbox{-}Verify})-\mathrm{AUROC}(\mathrm{LL\mbox{-}AVG})

. Positive values indicate that Self-Verify ranks correct predictions above incorrect predictions better than LL-AVG. Intervals are not corrected for multiple comparisons and should be interpreted as exploratory.

Dataset	Model	Signal	err@80%cov	err@50%cov	cov@20%err	cov@10%err
ARC-Challenge	Phi-2	LL-AVG	0.447	0.451	0.004	0.002
ARC-Challenge	Phi-2	Self-Verify	0.387	0.268	0.317	0.073
ARC-Challenge	Phi-2	LL-SUM	0.327	0.253	0.412	0.061
ARC-Challenge	Qwen-1.5B	LL-AVG	0.485	0.462	0.007	0.002
ARC-Challenge	Qwen-1.5B	Self-Verify	0.431	0.321	0.262	0.096
ARC-Challenge	Qwen-1.5B	LL-SUM	0.374	0.299	0.241	0.094

Table 4: Representative selective-prediction operating points under the default prompt for the models emphasized in the main text. Lower error at fixed coverage and higher coverage at fixed error indicate better abstention behavior. Full results appear in Appendix Table 7 .

Figure 3: Representative risk–coverage curves. ARC-Challenge shows clear selective-prediction gains for Self-Verify relative to LL-AVG, while TruthfulQA-MC is much less stable and more prompt-sensitive.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Model	Prompt	Brier (LL-AVG)	Brier (Self-Verify)	Brier (LL-SUM)	ECE10 (LL-AVG)	ECE10 (Self-Verify)	ECE10 (LL-SUM)
ARC-Challenge	DeepSeek-R1-Distill-8B	audit_v1	0.427	0.360	0.582	0.383	0.320	0.576
ARC-Challenge	DeepSeek-R1-Distill-8B	default	0.427	0.370	0.582	0.383	0.339	0.576
ARC-Challenge	Phi-2	audit_v1	0.269	0.210	0.228	0.134	0.080	0.131
ARC-Challenge	Phi-2	default	0.269	0.204	0.228	0.134	0.064	0.131
ARC-Challenge	Qwen-1.5B	audit_v1	0.272	0.289	0.268	0.119	0.275	0.216
ARC-Challenge	Qwen-1.5B	default	0.272	0.287	0.268	0.119	0.272	0.216

Appendix

Table 5: Additional calibration metrics for the main confidence signals, including Brier score and expected calibration error with 10 bins (ECE-10).

Dataset	Model	Prompt	AUROC (LL-AVG)	AUROC (SV)	AUROC (Margin)	AUROC (EntropyConf)	AUROC (LL-AVG-T)	AURC (LL-AVG)	AURC (SV)	AURC (Margin)	AURC (EntropyConf)	AURC (LL-AVG-T)
ARC-Challenge	Qwen-1.5B	audit_v1	0.557	0.770	0.578	0.531	0.545	0.471	0.295	0.459	0.488	0.483
ARC-Challenge	Qwen-1.5B	default	0.557	0.765	0.578	0.531	0.545	0.471	0.298	0.459	0.488	0.483
ARC-Challenge	Qwen-7B	audit_v1	0.555	0.879	0.585	0.515	0.550	0.363	0.148	0.348	0.384	0.367
ARC-Challenge	Qwen-7B	default	0.555	0.886	0.585	0.515	0.550	0.363	0.143	0.348	0.384	0.367
ARC-Challenge	TinyLlama-1.1B	audit_v1	0.484	0.525	0.505	0.471	0.481	0.731	0.702	0.722	0.735	0.734
ARC-Challenge	TinyLlama-1.1B	default	0.484	0.525	0.505	0.471	0.481	0.731	0.700	0.722	0.735	0.734

Appendix

Table 6: Auxiliary confidence baselines computed from the multiple-choice answer distribution. Margin denotes the gap between the top two LL-AVG option probabilities; EntropyConf denotes one minus normalized predictive entropy; LL-AVG-T denotes temperature-scaled LL-AVG fit on a held-out 20% calibration split.

Dataset	Model	Prompt	Signal	err@80%cov	err@50%cov	cov@20%err	cov@10%err
ARC-Challenge	DeepSeek-R1-Distill-8B	audit_v1	LL-AVG	0.720	0.720	0.000	0.000
ARC-Challenge	DeepSeek-R1-Distill-8B	audit_v1	LL-SUM	0.737	0.727	0.000	0.000
ARC-Challenge	DeepSeek-R1-Distill-8B	audit_v1	Self-Verify	0.719	0.729	0.000	0.000
ARC-Challenge	DeepSeek-R1-Distill-8B	default	LL-AVG	0.720	0.720	0.000	0.000
ARC-Challenge	DeepSeek-R1-Distill-8B	default	LL-SUM	0.737	0.727	0.000	0.000
ARC-Challenge	DeepSeek-R1-Distill-8B	default	Self-Verify	0.726	0.744	0.001	0.001

Appendix

Table 7: Full operating-point comparisons used in the selective-prediction analysis. Lower error at fixed coverage and higher coverage at fixed error indicate better abstention behavior.

Dataset	Model	AURC (audit_v1)	AURC (default)	AUROC (audit_v1)	AUROC (default)	$\|\Delta\mathrm{AUROC}\|$	$\|\Delta\mathrm{AURC}\|$
ARC-Challenge	DeepSeek-R1-Distill-8B	0.738	0.751	0.489	0.463	0.026	0.013
ARC-Challenge	Phi-2	0.278	0.275	0.755	0.759	0.004	0.003
ARC-Challenge	Qwen-1.5B	0.296	0.298	0.770	0.765	0.005	0.003
ARC-Challenge	Qwen-7B	0.149	0.143	0.879	0.886	0.008	0.005
ARC-Challenge	TinyLlama-1.1B	0.702	0.700	0.525	0.525	0.000	0.002
TruthfulQA-MC	DeepSeek-R1-Distill-8B	0.597	0.605	0.524	0.494	0.030	0.007

Appendix

Table 8: Prompt-ablation results for the Self-Verify confidence signal across datasets, models, and verification prompt variants.

Figure 4: AUROC comparison on ARC-Challenge under the default verification prompt.

Figure 5: AUROC comparison on ARC-Challenge under the audit-style verification prompt.

Figure 6: AUROC comparison on TruthfulQA-MC under the default verification prompt.

Figure 7: AUROC comparison on TruthfulQA-MC under the audit-style verification prompt.

Figure 8: AURC comparison on ARC-Challenge under the default verification prompt.

Figure 9: AURC comparison on ARC-Challenge under the audit-style verification prompt.

Figure 10: AURC comparison on TruthfulQA-MC under the default verification prompt.

Figure 11: AURC comparison on TruthfulQA-MC under the audit-style verification prompt.

Figure 12: Prompt-ablation AUROC summary across datasets and models.

Figure 13: Scale and family effects in self-verification performance.

Language Model Self-Verification

See all 16 papers

Measurement Validity and Representation Dynamics of Confidence Signals in Large Language Models
Oct 19, 2026Debojeet Ghosh, Shreyansh Kumar, Ayushman GargConfidence Estimation in Language Models Language Model Calibration
An Analysis of Training-Free Self-Reported Confidence in Language Models
Sep 17, 2026Lukas Meyer, Sofia Rossi, Wei Chen +2Confidence Estimation in Language Models Language Model Self-Assessment
DreamAI
Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulness
Feb 20, 2026Yegor Denisov-Blanch, Joshua Kazdan, Jessica Chudnovsky +4Correlated Errors in Language Models Inference-Time Scaling
Stanford Computer Science · Stanford Statistics · Stanford University +1

Oct 19, 2026J/K move · Enter open · S save

Measurement Validity and Representation Dynamics of Confidence Signals in Large Language Models

Debojeet Ghosh, Shreyansh Kumar, Ayushman Garg

Confidence Estimation in Language Models Language Model Calibration Verbalized Confidence Estimation Language Model Probing Semantic Entropy

Open paper AskPDF arXiv

Whether a language model "knows what it knows" matters for trustworthy deployment and for deciding when it should abstain. Prior work usually measures this through one channel: verbalized confidence, a linear probe on hidden states, or agreement among sampled answers (semantic entropy), and often treats these as interchangeable. We compare all three in a pre-registered audit of five Qwen2.5 models (0.5B-7B, including a base/instruct pair at 7B) on factual retrieval (PopQA, SimpleQA) and multi-step reasoning (GSM8K, MATH). First, probes trained on prefill states to predict whether sampled answers will disagree become informative earlier for retrieval (about 7% of layer depth on average) than for reasoning (about 28%). The pre-registered five-layer test supported this only marginally; a finer 21-layer follow-up, designed after those results, supports it more clearly. In most retrieval cells, however, a prompt-only TF-IDF classifier matches the probe, so the early signal may reflect surface features of the question. Second, verbalized confidence is unstable: slightly different prompt templates yield scores that barely correlate. Third, Expected Calibration Error is misleading: verbal confidence looks well calibrated because it clusters near the base rate while discriminating poorly, whereas sampling agreement has worse ECE but about twice the resolution. Of five pre-registered hypotheses, two were falsified, one was partly supported, one was not supported, and one could not be tested because the base model's verbal confidence was degenerate.

Abstract

Language Model Self-Verification

Explore similar work

Measurement Validity and Representation Dynamics of Confidence Signals in Large Language Models

An Analysis of Training-Free Self-Reported Confidence in Language Models

Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulness