Explainable Uncertainty Estimation for Reliable Medical AI

Li Rong Wang; Jamie Duell; Xinran Xu; Thomas C. Henderson; Yu Yue Hew; Pik Wan Erica Chiang; Xiao Wei Alstar Ang; Bingwen Eugene Fan

cs.LGAug 28, 2026

Explainable Uncertainty Estimation for Reliable Medical AI

Authors: Li Rong Wang, Jamie Duell, Xinran Xu, Thomas C. Henderson, Yu Yue Hew, Pik Wan Erica Chiang, Xiao Wei Alstar Ang, Bingwen Eugene Fan,

Organizations: College of Computing and Data Science (CCDS) Nanyang Technological University (NTU) A*STAR Centre for Frontier AI Research Singapore · School Of Computing and Digital Technologies Sheffield Hallam University South Yorkshire, England · CCDS NTU Singapore · School of Computing University of Utah United States ·

Abstract

Artificial intelligence has strong potential to support clinical decision-making, yet its adoption in healthcare remains limited due to a lack of trust. Uncertainty estimation can signal unreliable predictions, and explainable AI (XAI) can clarify how predictions are made but existing methods treat them separately, providing no feature-level insight into why a prediction is uncertain or which tests to prioritize to reduce it. To address this gap, we propose explainable uncertainty estimation, which unifies uncertainty estimation and XAI to both quantify uncertainty and explain feature-level contributions. We introduce the Expected Gradients Reconstruction Uncertainty Estimate (egRUE), which incorporates prediction explanations into its uncertainty computation and decomposes uncertainty into feature-wise contributions. We prove theoretical properties of egRUE and show through experiments that it improves reliability and interpretability compared to existing methods. A user study with medical experts further demonstrates that egRUE's explanations improve calibrated trust over uncertainty scores alone, increasing confidence in correct predictions and reducing confidence in incorrect ones. By combining prediction uncertainty with feature-level explanations, egRUE strengthens decision-making support in safety-critical healthcare settings, clarifying both when predictions may be unreliable and which features drive that uncertainty.

Figures & tables

Fig. 1 : System architecture for RUE and egRUE.

UE	Lung Cancer				Colorectal Cancer
UE	Corr ( $\uparrow$ )	AUROC ( $\uparrow$ )	AURC ( $\downarrow$ )	$\sigma$ -Risk ( $\downarrow$ )	Correlation ( $\uparrow$ )	AUROC ( $\uparrow$ )	AURC ( $\downarrow$ )	$\sigma$ -Risk ( $\downarrow$ )
egRUE	0.688 $\pm$ 0.151	0.835 $\pm$ 0.101	0.015 $\pm$ 0.006	0.008 $\pm$ 0.001	0.602 $\pm$ 0.047	0.812 $\pm$ 0.013	0.044 $\pm$ 0.015	0.016 $\pm$ 0.004
Entropy	0.676 $\pm$ 0.121	0.814 $\pm$ 0.078	0.024 $\pm$ 0.023	0.026 $\pm$ 0.014	0.703 $\pm$ 0.035	0.812 $\pm$ 0.019	0.045 $\pm$ 0.020	0.000 $\pm$ 0.000
MCD	0.300 $\pm$ 0.244	0.665 $\pm$ 0.133	0.039 $\pm$ 0.034	0.000 $\pm$ 0.000	-0.531 $\pm$ 0.301	0.307 $\pm$ 0.217	0.222 $\pm$ 0.113	0.441 $\pm$ 0.324
DE	0.179 $\pm$ 0.671	0.617 $\pm$ 0.320	0.064 $\pm$ 0.088	0.065 $\pm$ 0.094	-0.262 $\pm$ 0.314	0.531 $\pm$ 0.240	0.121 $\pm$ 0.105	0.164 $\pm$ 0.217
PN Alea	0.486 $\pm$ 0.112	0.830 $\pm$ 0.066	0.036 $\pm$ 0.025	0.010 $\pm$ 0.002	0.386 $\pm$ 0.141	0.741 $\pm$ 0.137	0.031 $\pm$ 0.012	0.020 $\pm$ 0.004

TABLE I : Performance comparison of uncertainty estimates on four datasets. The best-performing uncertainty estimates for each metric have been bolded , and the next best have been underlined .

UE	OCTMNIST ( $\uparrow$ )
UE	OCTDL-IN	OCTDL-OUT	ChestMNIST
egRUE	0.961 $\pm$ 0.015	0.977 $\pm$ 0.007	1.000 $\pm$ 0.000
Entropy	0.674 $\pm$ 0.031	0.386 $\pm$ 0.066	0.826 $\pm$ 0.060
MCD	0.661 $\pm$ 0.034	0.392 $\pm$ 0.064	0.844 $\pm$ 0.060
DE	0.739 $\pm$ 0.009	0.497 $\pm$ 0.029	0.900 $\pm$ 0.028
PN Alea	0.625 $\pm$ 0.063	0.492 $\pm$ 0.025	0.500 $\pm$ 0.107

TABLE II : Out-of-distribution (OOD) detection AUROC on image datasets. OOD datasets are ordered by increasing dissimilarity from the training set. Best uncertainty estimates are bolded ; second-best are underlined .

Panel of Figure 4 — Fig. 3 : Comparison of Expected Gradients (EG), RUE, and egRUE for mispredicted “Diabetic Macular Edema (DME)” (OCTMNIST) and “Immature Granulocytes” (BloodMNIST) images. Green highlights mark regions driving incorrect predictions (EG) or predictive uncertainty (RUE, egRUE). ( 3(a) ) EG highlights most of the retina and some artefacts without pinpointing any specific region of uncertainty. RUE significantly overestimates uncertainty regions with even more artefacts. egRUE highlights the edema (triangular region) typical of DME cases. ( 3(b) ) EG highlights most of the central cell, RUE highlights all cells in the image, while egRUE focuses on the nucleus, whose shape is atypical of a neutrophil, driving model uncertainty.

Fig. 4 : Results of ROAR experiments on the Colorectal Cancer dataset. egRUE yields a more stable, higher AUC than RUE after removing uncertain features, indicating its uncertainty explanations better reflect true uncertainties.

Data	UE	Corr ( $\uparrow$ )	AUROC ( $\uparrow$ )	AURC ( $\downarrow$ )	$\sigma$ -Risk ( $\downarrow$ )
Lung Cancer	egRUE	0.688 $\pm$ 0.151	0.835 $\pm$ 0.101	0.015 $\pm$ 0.006	0.008 $\pm$ 0.001
	RUE	0.589 $\pm$ 0.115	0.796 $\pm$ 0.092	0.021 $\pm$ 0.011	0.019 $\pm$ 0.003
Colorectal Cancer	egRUE	0.602 $\pm$ 0.047	0.812 $\pm$ 0.013	0.044 $\pm$ 0.015	0.016 $\pm$ 0.004
	RUE	0.542 $\pm$ 0.059	0.794 $\pm$ 0.026	0.049 $\pm$ 0.021	0.018 $\pm$ 0.003
OCT MNIST	egRUE	0.439 $\pm$ 0.039	0.786 $\pm$ 0.021	0.120 $\pm$ 0.025	0.020 $\pm$ 0.016
	RUE	0.395 $\pm$ 0.059	0.771 $\pm$ 0.034	0.129 $\pm$ 0.028	0.051 $\pm$ 0.047

TABLE III : Performance comparison of egRUE and RUE on four datasets. The best-performing uncertainty estimates for each metric have been bolded .

	egRUE	igRUE	gradCamRUE	guidedBackpropRUE
BloodMNIST	0.148	0.144	0.133	0.138

TABLE IV : Area under the ROAD curve (

\uparrow

) for each attribution method, paired with RUE.

	Entropy	DEC	egRUE	PN	DE	MCD	BNN
Time / s	0.00132	0.00139	0.00200	0.00407	0.01026	0.01249	0.01259

TABLE V : Instance inference time (s) of uncertainty estimates.

Fig. 5 : User Study Results: Change in respondents’ confidence after the addition of uncertainty explanations.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 647 papers

Apr 2, 2026

Enhancing the Reliability of Medical AI through Expert-guided Uncertainty Modeling

Artificial intelligence (AI) systems accelerate medical workflows and improve diagnostic accuracy in healthcare, serving as second-opinion systems. However, the unpredictability of AI errors poses a significant challenge, particularly in healthcare contexts, where mistakes can have severe consequences. A widely adopted safeguard is to pair predictions with uncertainty estimation, enabling human experts to focus on high-risk cases while streamlining routine verification. Current uncertainty estimation methods, however, remain limited, particularly in quantifying aleatoric uncertainty, which arises from data ambiguity and noise. To address this, we propose a novel approach that leverages disagreement in expert responses to generate targets for training machine learning models. These targets are used in conjunction with standard data labels to estimate two components of uncertainty separately, as given by the law of total variance, via a two-ensemble approach, as well as its lightweight variant. We validate our method on binary image classification, binary and multi-class image segmentation, and multiple-choice question answering. Our experiments demonstrate that incorporating expert knowledge can enhance uncertainty estimation quality by

9\%

50\%

depending on the task, making this source of information invaluable for the construction of risk-aware AI systems in healthcare applications.

Aleksei Khalin, Ekaterina Zaychenkova, Aleksandr Yugay +4

Kharkevich Institute for Information Transmission Problems of Russian Academy of Sciences, Moscow, Russia · Moscow Insitute of Physics and Technology, Dolgoprudny, Russia · Skolkovo Institute of Science and Technology, Moscow, Russia +2

Aleatoric Uncertainty Uncertainty Quantification Segmentation Uncertainty Estimation Uncertainty Decomposition

Ask PDF

Mar 27, 2026

Concerning Uncertainty -- A Systematic Survey of Uncertainty-Aware XAI

This paper surveys uncertainty-aware explainable artificial intelligence (UAXAI), examining how uncertainty is incorporated into explanatory pipelines and how such methods are evaluated. Across the literature, three recurring approaches to uncertainty quantification emerge (Bayesian, Monte Carlo, and Conformal methods), alongside distinct strategies for integrating uncertainty into explanations: assessing trustworthiness, constraining models or explanations, and explicitly communicating uncertainty. Evaluation practices remain fragmented and largely model centered, with limited attention to users and inconsistent reporting of reliability properties (e.g., calibration, coverage, explanation stability). Recent work leans towards calibration, distribution free techniques and recognizes explainer variability as a central concern. We argue that progress in UAXAI requires unified evaluation principles that link uncertainty propagation, robustness, and human decision-making, and highlight counterfactual and calibration approaches as promising avenues for aligning interpretability with reliability.

Helena Löfström, Tuwe Löfström, Anders Hjort +1

Jönköping International Business School, Gjuterigatan 5, 553 18 Jönköping, Sweden · Jönköping School of Engineering, Gjuterigatan 5, 553 18 Jönköping, Sweden · Eiendomsverdi AS, Nedre Slottsgate 8, 0157 Oslo, Norway +2

Explainable Artificial Intelligence Uncertainty Quantification Explainability Evaluation Uncertainty Propagation

Ask PDF

Jul 22, 2026

Bayesian uncertainty estimation improves clinical decision making in medical AI agents

Machine learning models for medical image analysis typically lack a reliable measure of confidence, limiting their use in ambiguous or atypical cases. Here we show that Monte Carlo dropout, applied to a multi-task chest-radiograph classifier (eight thoracic findings, 137,593 training images), provides an epistemic uncertainty signal that tracks generalisation across training-set scales and flags confident yet error-prone predictions. Adding this signal to the point prediction raised error-detection AUROC from 0.74 to 0.77 (

Δ

AUROC +0.023, 95% CI [+0.014, +0.033]). In a controlled 2x2 factorial experiment, a clinical-decision-support agent exploited this uncertainty only when it was delivered as a binary error-risk flag rather than as raw scores, cutting confident misdiagnoses on unreliable findings from 8.5% to 2.7%. Epistemic uncertainty estimation thus carries decision-relevant information beyond point predictions, but its value for downstream agents depends on how it is communicated.

Frederik Hauke, Patrick Wienholt, Christiane Kuhl +4

Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany. · Department of Medical Oncology, National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany. · Else Kröner Fresenius Center for Digital Health, TU Dresden, Dresden, Germany. +1

Clinical Decision Support Clinical Decision-Making Monte Carlo Dropout Epistemic Uncertainty

Ask PDF

Abstract

Uncertainty Quantification

Explore similar work

Enhancing the Reliability of Medical AI through Expert-guided Uncertainty Modeling

Concerning Uncertainty -- A Systematic Survey of Uncertainty-Aware XAI

Bayesian uncertainty estimation improves clinical decision making in medical AI agents