Entropy Alone is Insufficient for Safe Selective Prediction in LLMs

Edward Phillips; Fredrik K. Gustafsson; Sean Wu; Anshul Thakur; David A. Clifton

cs.CLMar 22, 2026

Entropy Alone is Insufficient for Safe Selective Prediction in LLMs

Authors: Edward Phillips, Fredrik K. Gustafsson, Sean Wu, Anshul Thakur, David A. Clifton

Organizations: Department of Engineering Science, University of Oxford · Oxford Suzhou Centre for Advanced Research, University of Oxford, Suzhou

Abstract

Selective prediction systems can mitigate harms resulting from language model hallucinations by abstaining from answering in high-risk cases. Uncertainty quantification techniques are often employed to identify such cases, but are rarely evaluated in the context of the wider selective prediction policy and its ability to operate at low target error rates. We identify a model-dependent failure mode of entropy-based uncertainty methods that leads to unreliable abstention behaviour, and address it by combining entropy scores with a correctness probe signal. We find that across three QA benchmarks (TriviaQA, BioASQ, MedicalQA) and four model families, the combined score generally improves both the risk--coverage trade-off and calibration performance relative to entropy-only baselines. Our results highlight the importance of deployment-facing evaluation of uncertainty methods, using metrics that directly reflect whether a system can be trusted to operate at a stated risk level.

Figures & tables

Figure 1: Performance of the PC probe and combined method relative to entropy-only baselines, averaged across all four evaluated models. Positive values denote an improvement in the target metric (i.e., higher AUROC/AUPRC or lower E-AURC/TCE). Abbreviations: NLL (Sequence NLL), SE (Semantic Entropy), SEP (Semantic Entropy Probe). The combined method consistently outperforms both the entropy baselines and the standalone PC probe across most configurations, with the notable exception of SE on the MedicalQA dataset.

Figure 2: SE score vs. PC probe logit for Qwen on TriviaQA. Many answers, both correct and hallucinated, collapse to

\mathrm{SE}=0

, yet receive distinct PC probe scores.

Model	SE AUROC	PC AUROC	$\rho_{s}$
Llama 3.2 3B	$\mathbf{0.884}$	$0.813$	0.51
Qwen3 4B	$0.844$	$\mathbf{0.888}$	0.54
Gemma 3 4B	$\mathbf{0.752}$	$\mathbf{0.752}$	0.34
Ministral 8B	$\mathbf{0.823}$	$0.750$	0.50

Table 1: Hallucination detection AUROC for SE and PC probe on TriviaQA, with Spearman correlation

\rho_{s}

between their risk scores. The best method varies across models, with only moderate correlation between scores.

Figure 3: Ministral 8B selective prediction on BioASQ. Left: risk–coverage curves; the shaded region marks a high-trust regime (

\alpha\leq 0.15

). Entropy based methods fail to enter this regime at non-trivial coverage. Right: realized hallucination rate vs. target

\alpha

; a perfect system lies on the diagonal. Entropy-based methods diverge sharply at strict targets; combined methods do not.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Method	AUROC $\uparrow$	AUPRC $\uparrow$	E-AURC $\downarrow$	TCE $\downarrow$
TriviaQA	Sequence NLL	$0.646_{11.5}$	$0.544_{3.4}$	$0.217_{9.9}$	$0.189_{11.5}$
	Semantic Entropy	$\mathbf{0.826_{5.5}}$	$\mathbf{0.732_{6.7}}$	$\mathbf{0.094_{5.5}}$	$0.084_{8.7}$
	SE Probe	$0.745_{2.5}$	$0.643_{9.9}$	$0.122_{1.4}$	$0.083_{4.3}$
	PC Probe	$0.801_{6.5}$	$0.712_{12.9}$	$0.103_{4.1}$	$\mathbf{0.061_{1.8}}$
BioASQ	Sequence NLL	$0.662_{1.4}$	$0.694_{4.8}$	$0.245_{2.6}$	$0.278_{12.9}$
	Semantic Entropy	$\mathbf{0.760_{4.2}}$	$\mathbf{0.771_{5.4}}$	$\mathbf{0.164_{3.3}}$	$\mathbf{0.109_{6.0}}$

Appendix

Table A1: Performance of base hallucination detection methods across datasets, averaged over all models.

Method	TriviaQA		BioASQ		MedicalQA
	AUROC	AUPRC	AUROC	AUPRC	AUROC	AUPRC
Llama 3.2 3B	Acc: 64.2%		Acc: 48.3%		Acc: 22.0%
Sequence NLL	$0.664_{4.5}$	$0.531_{7.2}$	$0.656_{4.0}$	$0.633_{5.5}$	$\mathbf{0.690_{5.5}}$	$0.853_{4.3}$
Semantic Entropy	$\mathbf{0.884_{2.7}}$	$0.767_{5.8}$	$\mathbf{0.737_{4.2}}$	$0.699_{5.8}$	$0.688_{5.6}$	$0.866_{3.4}$
SE Probe	$0.723_{4.4}$	$0.606_{6.8}$	$0.697_{4.3}$	$0.721_{5.4}$	$0.626_{5.4}$	$0.864_{3.2}$
PC Probe	$0.813_{3.5}$	$0.691_{6.6}$	$0.669_{3.6}$	$0.683_{5.4}$	$0.642_{5.3}$	$0.868_{3.3}$

Appendix

Table A2: Detailed hallucination detection classification metrics (AUROC and AUPRC) for all evaluated models and datasets. Results are reported as

\text{Mean}_{\text{Std}}

across

200

bootstrap iterations. The best performing method per column is highlighted in bold, and Acc denotes the base model accuracy at full coverage.

Llama 3.2 3B
Method	TriviaQA		BioASQ		MedicalQA
	E-AURC $\downarrow$	TCE $\downarrow$	E-AURC $\downarrow$	TCE $\downarrow$	E-AURC $\downarrow$	TCE $\downarrow$
Sequence NLL	$0.194_{4.2}$	$0.181_{8.1}$	$0.244_{3.9}$	$0.198_{7.6}$	$0.225_{3.4}$	$0.437_{7.7}$
Semantic Entropy	$\mathbf{0.045_{1.3}}$	$\mathbf{0.041_{1.2}}$	$\mathbf{0.163_{3.2}}$	$\mathbf{0.073_{1.2}}$	$\mathbf{0.210_{3.7}}$	$\mathbf{0.332_{9.7}}$
SE Probe	$0.119_{2.5}$	$0.073_{4.8}$	$0.214_{3.9}$	$0.294_{15.1}$	$0.266_{3.6}$	$0.357_{1.3}$
PC Probe	$0.087_{2.2}$	$0.073_{4.3}$	$0.237_{3.3}$	$0.263_{16.8}$	$0.242_{3.6}$	$0.403_{4.1}$

Appendix

Table A3: Detailed selective prediction metrics: Full-coverage Excess Area Under the Risk–Coverage Curve (E-AURC) and Target Calibration Error (TCE) for all evaluated models and datasets. TCE is averaged over operational target risk levels

\alpha\in[0.05,0.3]

. Lower values indicate better performance, with the best method per column highlighted in bold. In cases where coverage is zero over the whole operational range, we report TCE as the model base hallucination rate.

Model	NLL		SE		SE Probe
	LR	MLP	LR	MLP	LR	MLP
Llama 3.2 3B ( TriviaQA )	$0.826_{3.4}$	$\mathbf{0.831_{3.3}}$	$0.863_{2.9}$	$\mathbf{0.864_{2.8}}$	$\mathbf{0.836_{3.2}}$	$\mathbf{0.836_{3.2}}$
Llama 3.2 3B ( BioASQ )	$\mathbf{0.674_{3.6}}$	$0.674_{3.6}$	$0.702_{3.5}$	$\mathbf{0.707_{3.5}}$	$\mathbf{0.693_{3.6}}$	$0.693_{3.6}$
Llama 3.2 3B ( MedicalQA )	$\mathbf{0.649_{5.2}}$	$0.560_{6.2}$	$\mathbf{0.666_{5.2}}$	$0.639_{5.3}$	$\mathbf{0.654_{5.4}}$	$0.650_{5.5}$
Qwen3 4B ( TriviaQA )	$0.888_{2.6}$	$\mathbf{0.891_{2.5}}$	$0.910_{2.1}$	$\mathbf{0.911_{2.1}}$	$\mathbf{0.894_{2.4}}$	$0.889_{2.5}$
Qwen3 4B ( BioASQ )	$0.711_{4.3}$	$\mathbf{0.721_{4.3}}$	$\mathbf{0.722_{4.1}}$	$0.721_{4.2}$	$\mathbf{0.707_{4.3}}$	$0.694_{4.4}$

Appendix

Table A4: Ablation study comparing the classification performance (AUROC) of Logistic Regression (LR) and Multi-Layer Perceptron (MLP) combiners for fusing correctness and entropy signals.

Panel of Figure 9 — Figure A1: Risk–coverage curves for all four model families on TriviaQA. The shaded region marks the high-trust regime ( $\alpha\leq 0.15$ ). The relative ranking of methods varies substantially across models, motivating model-aware evaluation of uncertainty quantification approaches.

Explore similar work

Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guarantees
Jul 5, 2026Sijin Dong, Hiroyuki ShinnouSelective Prediction Confidence Estimation in Language Models
1Ibaraki University, Japan.
LLMs Uncertainty Quantification via Adaptive Conformal Semantic Entropy
May 5, 2026Hamed Karimi, Vaishali Meyappan, Reza SamaviSelective Prediction Semantic Entropy
Toronto Metropolitan University, Toronto, Ontario, Canada · Vector Institute, Toronto, Ontario, Canada
Strong yet Controllable LLM Abstention without Learning to Abstain
Oct 12, 2026Junwei Zhang, Yifei Wang, Wanli Yang +6LLM Abstention LLM Hallucination Mitigation
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · Beijing Institute of Technology

Jul 5, 2026J/K move · Enter open · S save

Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guarantees

Sijin Dong, Hiroyuki Shinnou

1Ibaraki University, Japan.

Selective Prediction Confidence Estimation in Language Models Question Answering LLM Reliability LLM Uncertainty Estimation Uncertainty Calibration

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly deployed in question answering (QA) systems, yet they may generate hallucinated or misaligned responses without reliable confidence estimates. Uncertainty quantification (UQ) offers a natural basis for selective answering, where a system answers only when its prediction is deemed reliable and abstains otherwise. However, existing uncertainty scores for LLMs are often heuristic: a threshold chosen on such scores does not, by itself, provide statistical guarantees on the error rate among accepted answers. We propose CIC, a confidence-interval-based calibration framework that converts arbitrary uncertainty scores into risk-controlled selective answering rules. Given a held-out calibration set, CIC evaluates each generated response using an application-specific alignment criterion and associates it with an uncertainty score and a binary error label. For each candidate uncertainty threshold, CIC estimates the acceptance-conditioned error rate and constructs a high-probability upper confidence bound using either Hoeffding-style or Clopper-Pearson confidence intervals. It then selects the largest threshold whose upper bound is below a user-specified risk level

α

, thereby maximizing the answering rate subject to a finite-sample reliability constraint. Under exchangeability, CIC guarantees with probability at least

1-δ

that the selected threshold, if non-null, controls the error rate among accepted answers at level

α

. We evaluate CIC on both closed-ended and open-ended QA benchmarks across seven LLMs and multiple uncertainty estimators. Experimental results show that CIC consistently achieves valid risk control while retaining strong answering efficiency, providing a practical and statistically grounded mechanism for deploying LLMs in reliability-sensitive QA workflows.

Abstract

Explore similar work

Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guarantees

LLMs Uncertainty Quantification via Adaptive Conformal Semantic Entropy

Strong yet Controllable LLM Abstention without Learning to Abstain