BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence

Sean Wu; Fredrik K. Gustafsson; Edward Phillips; Boyan Gao; Anshul Thakur; David A. Clifton

cs.CLApr 3, 2026

BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence

Authors: Sean Wu, Fredrik K. Gustafsson, Edward Phillips, Boyan Gao, Anshul Thakur, David A. Clifton

Organizations: Department of Engineering Science, University of Oxford · Oxford Suzhou Centre for Advanced Research

Abstract

Large language models (LLMs) often produce confident but incorrect answers in settings where abstention would be safer. Standard evaluation protocols, however, require a response and do not account for how confidence should guide decisions under different risk preferences. To address this gap, we introduce the Behavioral Alignment Score (BAS), a decision-theoretic metric for evaluating how well LLM confidence supports abstention-aware decision making. BAS is derived from an explicit answer-or-abstain utility model and aggregates realized utility across a continuum of risk thresholds, yielding a measure of decision-level reliability that depends on both the magnitude and ordering of confidence. We show theoretically that truthful confidence estimates uniquely maximize expected BAS utility, linking calibration to decision-optimal behavior. BAS is related to proper scoring rules such as log loss, but differs structurally: log loss penalizes underconfidence and overconfidence symmetrically, whereas BAS imposes an asymmetric penalty that strongly prioritizes avoiding overconfident errors. Using BAS alongside widely used metrics such as ECE and AURC, we then construct a benchmark of self-reported confidence reliability across multiple LLMs and tasks. Our results reveal substantial variation in decision-useful confidence, and while larger and more accurate models tend to achieve higher BAS, even frontier models remain prone to severe overconfidence. Importantly, models with similar ECE or AURC can exhibit very different BAS due to highly overconfident errors, highlighting limitations of standard metrics. We further show that simple interventions, such as top- $k$ confidence elicitation and post-hoc calibration, can meaningfully improve confidence reliability. Overall, our work provides both a principled metric and a comprehensive benchmark for evaluating LLM confidence reliability.

Figures & tables

Figure 1: We introduce the Behavioral Alignment Score (BAS), a decision-theoretic metric derived from an explicit answer-or-abstain utility model, and use it to evaluate LLM confidence reliability across a diverse set of models and tasks. Reliability varies substantially across tasks, and while larger and more accurate models tend to also achieve higher BAS, even frontier models remain prone to overconfidence on complex, open-ended tasks.

Property	Accuracy	ECE	AURC	Brier Score	Log Loss	BAS
Confidence magnitude	✗	✓	✗	✓	✓	✓
Calibration	✗	✓	✗	✓	✓	✓
Ranking	✗	✗	✓	✓	✓	✓
Proper scoring rule	✗	✗	✗	✓	✓	✓
Strongly penalizes overconfidence	✗	✗	✗	✗	✓	✓
Asymmetric overconfidence penalty	✗	✗	✗	✗	✗	✓

Table 1: Conceptual comparison of BAS with existing evaluation metrics. BAS is most closely related to proper scoring rules such as log loss, sharing a sensitivity to high-confidence errors, but differs in imposing an asymmetric penalty that strongly prioritizes avoiding model overconfidence. In comparison, widely used calibration- and ranking-based metrics such as ECE and AURC capture only partial aspects of confidence reliability.

Frontier / Large Models (> 100B params)
	SimpleQA				MedQA				AIME
Model	Acc $\uparrow$	BAS $\uparrow$	ECE $\downarrow$	AURC $\downarrow$	Acc $\uparrow$	BAS $\uparrow$	ECE $\downarrow$	AURC $\downarrow$	Acc $\uparrow$	BAS $\uparrow$	ECE $\downarrow$	AURC $\downarrow$
DS-R1	29.9 2.9	-0.48 0.06	52.7 2.7	0.60 0.04	92.1 1.5	0.76 0.03	2.6 1.2	0.03 0.01	66.7 11.7	0.45 0.20	17.6 8.1	0.07 0.06
DS-V3.2	17.6 2.3	-0.63 0.06	49.0 2.9	0.75 0.04	91.8 1.5	0.76 0.03	3.3 1.3	0.02 0.01	65.0 11.7	0.24 0.28	24.2 10.0	0.10 0.07
GPT-4o	21.4 2.6	-1.30 0.06	70.5 2.5	0.76 0.04	91.3 1.6	0.70 0.05	3.5 1.5	0.05 0.01	11.7 7.5	-5.06 0.98	73.8 10.6	0.77 0.12
GPT-oss	12.0 2.0	-0.66 0.05	58.5 2.1	0.77 0.04	91.0 1.6	0.74 0.04	2.6 1.2	0.02 0.01	75.0 10.8	0.57 0.22	7.2 6.2	0.05 0.04

Table 2: Evaluation of models across SimpleQA, MedQA, and AIME. Results highlight that confidence reliability is strongly task-dependent and does not always align with accuracy, with even frontier models exhibiting severe overconfidence in challenging settings.

Panel of Figure 2 — Figure 2: Relationship between model scale, predictive performance, and confidence reliability. Top: Model size vs. accuracy and reliability metrics. Larger models tend to achieve higher accuracy and improved confidence reliability (higher BAS, lower ECE and AURC), although substantial variation remains across models. Bottom: Accuracy vs. reliability metrics. Models with higher accuracy tend to also exhibit more reliable confidence estimates.

Model	Direct Elicitation	Self-Reflection	Top- $k$	Top- $k$ + Self-Reflection
GPT-4o-mini	$-3.86\pm 0.21$	$-0.94\pm 0.09$	$\mathbf{-0.42\pm 0.02}$	$-0.91\pm 0.14$
GPT-oss-120B	$-0.66\pm 0.04$	$-0.61\pm 0.05$	$\mathbf{-0.22\pm 0.02}$	$-0.31\pm 0.03$
Grok-3-Mini	$-0.86\pm 0.07$	$-0.92\pm 0.08$	$\mathbf{-0.50\pm 0.05}$	$-0.58\pm 0.07$
Llama-3.3-70B	$-2.97\pm 0.23$	$-2.05\pm 0.21$	$\mathbf{-0.25\pm 0.06}$	$-0.54\pm 0.11$
Phi-4	$-3.63\pm 0.21$	$-3.60\pm 0.22$	$\mathbf{-0.61\pm 0.08}$	$-2.23\pm 0.20$

Table 3: Comparison of BAS across four confidence elicitation methods on SimpleQA. Top-

k

consistently yields the highest BAS across a subset of five representative models.

	BAS $\uparrow$		ECE (%) $\downarrow$
Model	Before	After	Before	After
GPT-4o-mini	$-3.86\pm 0.21$	$\mathbf{\phantom{-}0.001\pm 0.001}$	$88.6\pm 1.4$	$\mathbf{3.6\pm 1.3}$
GPT-oss-120B	$-0.66\pm 0.04$	$\mathbf{-0.000\pm 0.006}$	$64.2\pm 1.8$	$\mathbf{6.0\pm 1.4}$
Grok-3-Mini	$-0.88\pm 0.07$	$\mathbf{\phantom{-}0.026\pm 0.008}$	$60.7\pm 2.7$	$\mathbf{1.7\pm 1.7}$
Llama-3.3-70B	$-2.97\pm 0.23$	$\mathbf{\phantom{-}0.024\pm 0.007}$	$68.1\pm 2.6$	$\mathbf{1.5\pm 1.7}$
Phi-4	$-3.63\pm 0.21$	$\mathbf{\phantom{-}0.004\pm 0.001}$	$84.1\pm 1.8$	$\mathbf{1.8\pm 1.4}$

Table 4: Effect of post-hoc calibration via isotonic regression on SimpleQA. Calibration substantially reduces ECE and improves BAS across a subset of five representative models.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Risk Profile	Weighting $w(t)$	SimpleQA	AIME	MedQA
DeepSeek-R1
General Purpose	Uniform: $w(t)=1$	-0.4847	0.4492	0.7640
Risk-Aware	Linear: $w(t)=2t$	-0.7904	0.3706	0.6688
Safety-Critical	Quadratic: $w(t)=3t^{2}$	-0.9291	0.3299	0.5989
GPT-4o
General Purpose	Uniform: $w(t)=1$	-1.1125	-5.0629	0.7030

Appendix

Table A1: Comparison of BAS for frontier models across tasks when varying the risk-prior

w(t)

. Higher scores indicate better alignment with expressed uncertainty. The standard

\mathrm{BAS}

(Uniform:

w(t)=1

) serves as the baseline, while Linear and Quadratic weights simulate increasingly safety-critical environments.

Figure A1: Relationship between BAS and the standard reliability metrics (ECE, AURC) across all model-dataset pairs. While BAS is correlated with ECE and AURC, several points deviate substantially from the trend, indicating that models with similar confidence calibration or ranking can exhibit highly different decision-level reliability.

Figure A2: Distribution of predicted confidence values for Llama 3.3 and Mistral (M) on AIME and SimpleQA, separated by correctness. While both models exhibit high-confidence predictions, Llama 3.3 assigns extremely high confidence to a larger number of incorrect answers. These rare but highly overconfident errors are strongly penalized by BAS but contribute only modestly to ECE, explaining the observed discrepancies between the metrics.

Figure A3: Relationship between BAS and the proper scoring rule log loss across all model-dataset pairs. The two metrics are highly correlated in practice, reflecting their shared sensitivity to high-confidence errors and the clear tendency of modern LLMs to exhibit overconfidence rather than underconfidence. However, BAS differs structurally by imposing an asymmetric penalty that strongly prioritizes avoiding overconfident errors, enabling it to distinguish between models that trade off overconfidence with underconfidence.

Figure A4: Comparison of confidence elicitation methods on SimpleQA across BAS, accuracy, ECE, and AURC. Results shown for the same subset of representative models as in Table 3 . Top-

k

consistently improves BAS by reducing overconfident errors, while accuracy and AURC remain largely unchanged across methods. ECE improves when moving away from direct elicitation, indicating better calibration.

Val set size	BAS	ECE (%) $\downarrow$
1000	$-0.000\pm 0.006$	$6.0\pm 1.4$
500	$0.002\pm 0.004$	$6.0\pm 1.4$
250	$0.000\pm 0.004$	$6.7\pm 1.4$
100	$-0.005\pm 0.004$	$9.3\pm 1.5$
50	$-0.000\pm 0.003$	$6.3\pm 1.5$
25	$-0.016\pm 0.005$	$13.7\pm 1.6$

Appendix

Table A2: Post-hoc calibration (isotonic regression) ablation results for GPT-oss-120B on SimpleQA. Validation set: 10 - 1,000 samples, test set: 1,000 samples.

Figure A5: Effect of post-hoc calibration on model confidence. Raw model confidence values are transformed using isotonic regression to better align with empirical accuracy, leading to improved decision-theoretic evaluation under BAS.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection
Feb 18, 2026Xie Xiaohu, Liu Xiaohu, Yao BenjaminLanguage Model Error Detection Confidence Estimation in Language Models
Alexa AI, Amazon, Seattle, WA, USA
When Linguistic and Internal Confidence Diverge in Large Language Models
Aug 28, 2026Hefan Zhang, Bingquan Zhang, Ming Cheng +3Confidence Estimation in Language Models Verbalized Confidence Estimation
Dartmouth College · Oakland University
Calibration as a First-Class Criterion in LLM Evaluation
Sep 22, 2026Mario Sanz-Guerrero, Katharina von der WenseLLM Evaluation Confidence Estimation in Language Models
Johannes Gutenberg University Mainz, Germany · University of Colorado Boulder, USA

Feb 18, 2026J/K move · Enter open · S save

Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection

Xie Xiaohu, Liu Xiaohu, Yao Benjamin

Alexa AI, Amazon, Seattle, WA, USA

Language Model Error Detection Confidence Estimation in Language Models LLM Reliability Language Model Calibration

Open paper AskPDF arXiv

As large language models (LLMs) are increasingly deployed in critical decision-making systems, the lack of reliable methods to measure their uncertainty presents a fundamental trustworthiness risk. We introduce a normalized confidence score based on output anchor token probabilities: classification labels for structured tasks and self-evaluation responses (Yes/No) for open-ended generation. This enables direct detection of errors and hallucinations with minimal overhead and without external validation. We make three key contributions. First, we propose a normalized confidence score and self-evaluation framework that exposes reliable confidence estimates for error detection across seven diverse benchmark tasks and five LLMs of varying architectures and sizes. Second, our theoretical analysis reveals that supervised fine-tuning (SFT) yields well-calibrated confidence through maximum-likelihood estimation, whereas reinforcement learning methods (PPO, GRPO) and DPO induce overconfidence via reward exploitation. Third, we propose post-RL SFT with self-distillation to restore confidence reliability in RL-trained models. Empirical results demonstrated that SFT improved average confidence-correctness AUROC from 0.806 to 0.879 and reduced calibration error from 0.163 to 0.034 on Qwen3-4B, while GRPO and DPO degraded confidence reliability. We demonstrated practical value through adaptive retrieval-augmented generation (RAG) that selectively retrieves context when the model lacks confidence, using only 58% of retrieval operations to recover 95% of the maximum achievable accuracy gain on TriviaQA

Abstract

LLM Evaluation

Explore similar work

Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection

When Linguistic and Internal Confidence Diverge in Large Language Models

Calibration as a First-Class Criterion in LLM Evaluation