Explainable Uncertainty Estimation for Reliable Medical AI

Li Rong Wang; Jamie Duell; Xinran Xu; Thomas C. Henderson; Yu Yue Hew; Pik Wan Erica Chiang; Xiao Wei Alstar Ang; Bingwen Eugene Fan

cs.LGAug 28, 2026

Explainable Uncertainty Estimation for Reliable Medical AI

Authors: Li Rong Wang, Jamie Duell, Xinran Xu, Thomas C. Henderson, Yu Yue Hew, Pik Wan Erica Chiang, Xiao Wei Alstar Ang, Bingwen Eugene Fan,

Organizations: College of Computing and Data Science (CCDS) Nanyang Technological University (NTU) A*STAR Centre for Frontier AI Research Singapore · School Of Computing and Digital Technologies Sheffield Hallam University South Yorkshire, England · CCDS NTU Singapore · School of Computing University of Utah United States ·

Abstract

Artificial intelligence has strong potential to support clinical decision-making, yet its adoption in healthcare remains limited due to a lack of trust. Uncertainty estimation can signal unreliable predictions, and explainable AI (XAI) can clarify how predictions are made but existing methods treat them separately, providing no feature-level insight into why a prediction is uncertain or which tests to prioritize to reduce it. To address this gap, we propose explainable uncertainty estimation, which unifies uncertainty estimation and XAI to both quantify uncertainty and explain feature-level contributions. We introduce the Expected Gradients Reconstruction Uncertainty Estimate (egRUE), which incorporates prediction explanations into its uncertainty computation and decomposes uncertainty into feature-wise contributions. We prove theoretical properties of egRUE and show through experiments that it improves reliability and interpretability compared to existing methods. A user study with medical experts further demonstrates that egRUE's explanations improve calibrated trust over uncertainty scores alone, increasing confidence in correct predictions and reducing confidence in incorrect ones. By combining prediction uncertainty with feature-level explanations, egRUE strengthens decision-making support in safety-critical healthcare settings, clarifying both when predictions may be unreliable and which features drive that uncertainty.

Figures & tables

Fig. 1 : System architecture for RUE and egRUE.

UE	Lung Cancer				Colorectal Cancer
UE	Corr ( $\uparrow$ )	AUROC ( $\uparrow$ )	AURC ( $\downarrow$ )	$\sigma$ -Risk ( $\downarrow$ )	Correlation ( $\uparrow$ )	AUROC ( $\uparrow$ )	AURC ( $\downarrow$ )	$\sigma$ -Risk ( $\downarrow$ )
egRUE	0.688 $\pm$ 0.151	0.835 $\pm$ 0.101	0.015 $\pm$ 0.006	0.008 $\pm$ 0.001	0.602 $\pm$ 0.047	0.812 $\pm$ 0.013	0.044 $\pm$ 0.015	0.016 $\pm$ 0.004
Entropy	0.676 $\pm$ 0.121	0.814 $\pm$ 0.078	0.024 $\pm$ 0.023	0.026 $\pm$ 0.014	0.703 $\pm$ 0.035	0.812 $\pm$ 0.019	0.045 $\pm$ 0.020	0.000 $\pm$ 0.000
MCD	0.300 $\pm$ 0.244	0.665 $\pm$ 0.133	0.039 $\pm$ 0.034	0.000 $\pm$ 0.000	-0.531 $\pm$ 0.301	0.307 $\pm$ 0.217	0.222 $\pm$ 0.113	0.441 $\pm$ 0.324
DE	0.179 $\pm$ 0.671	0.617 $\pm$ 0.320	0.064 $\pm$ 0.088	0.065 $\pm$ 0.094	-0.262 $\pm$ 0.314	0.531 $\pm$ 0.240	0.121 $\pm$ 0.105	0.164 $\pm$ 0.217
PN Alea	0.486 $\pm$ 0.112	0.830 $\pm$ 0.066	0.036 $\pm$ 0.025	0.010 $\pm$ 0.002	0.386 $\pm$ 0.141	0.741 $\pm$ 0.137	0.031 $\pm$ 0.012	0.020 $\pm$ 0.004

TABLE I : Performance comparison of uncertainty estimates on four datasets. The best-performing uncertainty estimates for each metric have been bolded , and the next best have been underlined .

UE	OCTMNIST ( $\uparrow$ )
UE	OCTDL-IN	OCTDL-OUT	ChestMNIST
egRUE	0.961 $\pm$ 0.015	0.977 $\pm$ 0.007	1.000 $\pm$ 0.000
Entropy	0.674 $\pm$ 0.031	0.386 $\pm$ 0.066	0.826 $\pm$ 0.060
MCD	0.661 $\pm$ 0.034	0.392 $\pm$ 0.064	0.844 $\pm$ 0.060
DE	0.739 $\pm$ 0.009	0.497 $\pm$ 0.029	0.900 $\pm$ 0.028
PN Alea	0.625 $\pm$ 0.063	0.492 $\pm$ 0.025	0.500 $\pm$ 0.107

TABLE II : Out-of-distribution (OOD) detection AUROC on image datasets. OOD datasets are ordered by increasing dissimilarity from the training set. Best uncertainty estimates are bolded ; second-best are underlined .

Panel of Figure 4 — Fig. 3 : Comparison of Expected Gradients (EG), RUE, and egRUE for mispredicted “Diabetic Macular Edema (DME)” (OCTMNIST) and “Immature Granulocytes” (BloodMNIST) images. Green highlights mark regions driving incorrect predictions (EG) or predictive uncertainty (RUE, egRUE). ( 3(a) ) EG highlights most of the retina and some artefacts without pinpointing any specific region of uncertainty. RUE significantly overestimates uncertainty regions with even more artefacts. egRUE highlights the edema (triangular region) typical of DME cases. ( 3(b) ) EG highlights most of the central cell, RUE highlights all cells in the image, while egRUE focuses on the nucleus, whose shape is atypical of a neutrophil, driving model uncertainty.

Fig. 4 : Results of ROAR experiments on the Colorectal Cancer dataset. egRUE yields a more stable, higher AUC than RUE after removing uncertain features, indicating its uncertainty explanations better reflect true uncertainties.

Data	UE	Corr ( $\uparrow$ )	AUROC ( $\uparrow$ )	AURC ( $\downarrow$ )	$\sigma$ -Risk ( $\downarrow$ )
Lung Cancer	egRUE	0.688 $\pm$ 0.151	0.835 $\pm$ 0.101	0.015 $\pm$ 0.006	0.008 $\pm$ 0.001
	RUE	0.589 $\pm$ 0.115	0.796 $\pm$ 0.092	0.021 $\pm$ 0.011	0.019 $\pm$ 0.003
Colorectal Cancer	egRUE	0.602 $\pm$ 0.047	0.812 $\pm$ 0.013	0.044 $\pm$ 0.015	0.016 $\pm$ 0.004
	RUE	0.542 $\pm$ 0.059	0.794 $\pm$ 0.026	0.049 $\pm$ 0.021	0.018 $\pm$ 0.003
OCT MNIST	egRUE	0.439 $\pm$ 0.039	0.786 $\pm$ 0.021	0.120 $\pm$ 0.025	0.020 $\pm$ 0.016
	RUE	0.395 $\pm$ 0.059	0.771 $\pm$ 0.034	0.129 $\pm$ 0.028	0.051 $\pm$ 0.047

TABLE III : Performance comparison of egRUE and RUE on four datasets. The best-performing uncertainty estimates for each metric have been bolded .

	egRUE	igRUE	gradCamRUE	guidedBackpropRUE
BloodMNIST	0.148	0.144	0.133	0.138

TABLE IV : Area under the ROAD curve (

\uparrow

) for each attribution method, paired with RUE.

	Entropy	DEC	egRUE	PN	DE	MCD	BNN
Time / s	0.00132	0.00139	0.00200	0.00407	0.01026	0.01249	0.01259

TABLE V : Instance inference time (s) of uncertainty estimates.

Fig. 5 : User Study Results: Change in respondents’ confidence after the addition of uncertainty explanations.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 647 papers

Enhancing the Reliability of Medical AI through Expert-guided Uncertainty Modeling
Apr 2, 2026Aleksei Khalin, Ekaterina Zaychenkova, Aleksandr Yugay +4Aleatoric Uncertainty Uncertainty Quantification
Kharkevich Institute for Information Transmission Problems of Russian Academy of Sciences, Moscow, Russia · Moscow Insitute of Physics and Technology, Dolgoprudny, Russia · Skolkovo Institute of Science and Technology, Moscow, Russia +2
Concerning Uncertainty -- A Systematic Survey of Uncertainty-Aware XAI
Mar 27, 2026Helena Löfström, Tuwe Löfström, Anders Hjort +1Explainable Artificial Intelligence Uncertainty Quantification
Jönköping International Business School, Gjuterigatan 5, 553 18 Jönköping, Sweden · Jönköping School of Engineering, Gjuterigatan 5, 553 18 Jönköping, Sweden · Eiendomsverdi AS, Nedre Slottsgate 8, 0157 Oslo, Norway +2
Bayesian uncertainty estimation improves clinical decision making in medical AI agents
Jul 22, 2026Frederik Hauke, Patrick Wienholt, Christiane Kuhl +4Clinical Decision Support Clinical Decision-Making
Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany. · Department of Medical Oncology, National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany. · Else Kröner Fresenius Center for Digital Health, TU Dresden, Dresden, Germany. +1

Apr 2, 2026J/K move · Enter open · S save

Enhancing the Reliability of Medical AI through Expert-guided Uncertainty Modeling

Aleksei Khalin, Ekaterina Zaychenkova, Aleksandr Yugay, Andrey Goncharov+3

Kharkevich Institute for Information Transmission Problems of Russian Academy of Sciences, Moscow, Russia · Moscow Insitute of Physics and Technology, Dolgoprudny, Russia · Skolkovo Institute of Science and Technology, Moscow, Russia+2

Aleatoric Uncertainty Uncertainty Quantification Segmentation Uncertainty Estimation Uncertainty Decomposition

Open paper AskPDF arXiv

Artificial intelligence (AI) systems accelerate medical workflows and improve diagnostic accuracy in healthcare, serving as second-opinion systems. However, the unpredictability of AI errors poses a significant challenge, particularly in healthcare contexts, where mistakes can have severe consequences. A widely adopted safeguard is to pair predictions with uncertainty estimation, enabling human experts to focus on high-risk cases while streamlining routine verification. Current uncertainty estimation methods, however, remain limited, particularly in quantifying aleatoric uncertainty, which arises from data ambiguity and noise. To address this, we propose a novel approach that leverages disagreement in expert responses to generate targets for training machine learning models. These targets are used in conjunction with standard data labels to estimate two components of uncertainty separately, as given by the law of total variance, via a two-ensemble approach, as well as its lightweight variant. We validate our method on binary image classification, binary and multi-class image segmentation, and multiple-choice question answering. Our experiments demonstrate that incorporating expert knowledge can enhance uncertainty estimation quality by

9\%

50\%

depending on the task, making this source of information invaluable for the construction of risk-aware AI systems in healthcare applications.

Abstract

Uncertainty Quantification

Explore similar work

Enhancing the Reliability of Medical AI through Expert-guided Uncertainty Modeling

Concerning Uncertainty -- A Systematic Survey of Uncertainty-Aware XAI

Bayesian uncertainty estimation improves clinical decision making in medical AI agents