Enhancing the Reliability of Medical AI through Expert-guided Uncertainty Modeling

Aleksei Khalin; Ekaterina Zaychenkova; Aleksandr Yugay; Andrey Goncharov; Sergey Korchagin; Alexey Zaytsev; Egor Ershov

cs.LGApr 2, 2026

Enhancing the Reliability of Medical AI through Expert-guided Uncertainty Modeling

Authors: Aleksei Khalin, Ekaterina Zaychenkova, Aleksandr Yugay, Andrey Goncharov, Sergey Korchagin, Alexey Zaytsev, Egor Ershov

Organizations: Kharkevich Institute for Information Transmission Problems of Russian Academy of Sciences, Moscow, Russia · Moscow Insitute of Physics and Technology, Dolgoprudny, Russia · Skolkovo Institute of Science and Technology, Moscow, Russia · Sber, Moscow, Russia ·

Abstract

Artificial intelligence (AI) systems accelerate medical workflows and improve diagnostic accuracy in healthcare, serving as second-opinion systems. However, the unpredictability of AI errors poses a significant challenge, particularly in healthcare contexts, where mistakes can have severe consequences. A widely adopted safeguard is to pair predictions with uncertainty estimation, enabling human experts to focus on high-risk cases while streamlining routine verification. Current uncertainty estimation methods, however, remain limited, particularly in quantifying aleatoric uncertainty, which arises from data ambiguity and noise. To address this, we propose a novel approach that leverages disagreement in expert responses to generate targets for training machine learning models. These targets are used in conjunction with standard data labels to estimate two components of uncertainty separately, as given by the law of total variance, via a two-ensemble approach, as well as its lightweight variant. We validate our method on binary image classification, binary and multi-class image segmentation, and multiple-choice question answering. Our experiments demonstrate that incorporating expert knowledge can enhance uncertainty estimation quality by $9\%$ to $50\%$ depending on the task, making this source of information invaluable for the construction of risk-aware AI systems in healthcare applications.

Figures & tables

Figure 1: Integration of expert knowledge into aleatoric uncertainty estimation provides total UE quality enhancement.

Figure 2: Overview of the proposed method. First, an ensemble of models is trained to predict ground truth labels from the dataset. Next, a different ensemble is created via fine-tuning on “soft” labels acquired from expert annotations. The two ensembles are then used to estimate epistemic and aleatoric uncertainty, respectively, given by equations ( 2 ) and ( 3 ). Finally, we evaluate the performance of algorithms with different uncertainty thresholds and construct rejection curves.

UE Method	Uses experts	AAC, $\times 10^{-3}$ , $\downarrow$
UE Method	Uses experts	Image class.	Binary seg.	Multiclass seg.	Multiple choice QA
Single model		$5.11$	$8.27$	$30.07$	$116.56$
Prediction variance		$4.89$ $({\color[rgb]{0,0.043,0}-4.3\%})$	$2.93$ $({\color[rgb]{0,0.646,0}-64.6\%})$	$9.72$ $({\color[rgb]{0,0.677,0}-67.7\%})$	$110.93$ $({\color[rgb]{0,0.048,0}-4.8\%})$
Total variance, base		$4.28$ $({\color[rgb]{0,0.02,0}-16.2\%})$	$2.63$ $({\color[rgb]{0,0.682,0}-68.2\%})$	$12.54$ $({\color[rgb]{0,0.583,0}-58.3\%})$	$110.32$ $({\color[rgb]{0,0.054,0}-5.4\%})$
Total variance, CAE	✓, training	$2.18$ $({\color[rgb]{0,0.573,0}-57.3\%})$	$2.44$ $({\color[rgb]{0,0.705,0}-70.5\%})$	$6.45$ $({\color[rgb]{0,0.786,0}-78.6\%})$	$100.85$ $({\color[rgb]{0,0.135,0}-13.5\%})$
Total variance, experts	✓, inference	$2.37$ $({\color[rgb]{0,0.536,0}-53.6\%})$	$1.38$ $({\color[rgb]{0,0.833,0}-83.3\%})$	$6.27$ $({\color[rgb]{0,0.791,0}-79.1\%})$	$66.11$ $({\color[rgb]{0,0.433,0}-43.3\%})$

Table 1: Comparison of uncertainty estimation methods using rejection curves. AAC stands for the Area Above Curve and is normalized with respect to the optimal rejection-accuracy curve for a given model. The value in the bracket shows improvement over the baseline single-model method.

UE Method	Uses experts	AUROC, $\uparrow$
UE Method	Uses experts	Image classification	Multiple choice QA
Single model		$0.889\pm 0.033$	$0.693\pm 0.022$
Prediction variance		$0.888\pm 0.008$	$0.702\pm 0.011$
Total variance, base		$0.900\pm 0.008$	$0.709\pm 0.012$
Total variance, CAE	✓, training	$\mathbf{0.942\pm 0.008}$	$0.732\pm 0.014$
Total variance, experts	✓, inference	$\mathbf{0.942\pm 0.008}$	$\mathbf{0.818\pm 0.007}$

Table 2: Comparison of uncertainty estimation methods in error detection, where predicted uncertainty score is used as confidence in classification of model responses into errors and correct answers. AUROC stands for the Area Under the Receiver-Operator Curve. The best solution for each task is in bold and the second best solution is underlined.

Panel of Figure 3 — Figure 3: Rejection curves for proposed methods across four different machine learning tasks. Shaded areas represent the standard deviation between ensemble models. The higher the curve, the better.

Panel of Figure 4 — Figure 4: (a) The pairwise consistency between experts estimated as the Cohen’s kappa coefficient. Values on the diagonal are experts’ accuracies. (b) Uncertainty estimation quality by the law of total variance where aleatoric uncertainty is estimated from different sources. Experts, best stands for using the best $k$ experts based on accuracy, and Experts, average is computed by averaging AAC for all possible sets of $k$ experts.

Figure 5: Relation of uncertainty estimation quality AAC on the multiple choice question answering task to the sizes of the base ensemble and the CAE. Lower values are better.

Figure 6: Accuracy that can be achieved with the Total Variance with CAE and a single model as the uncertainty estimation method for different rejection rates.

UE Method	Weighting coefficient, $\alpha$
UE Method	0.0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
Image classification	2.58	2.47	2.38	2.30	2.23	2.18	2.15	2.14	2.17	2.24	4.89
Image segmentation	19.21	8.09	7.55	7.10	6.73	6.45	6.24	6.10	6.04	6.06	9.72
Multiple-choice QA	103.12	103.00	102.20	101.75	101.33	100.85	100.67	100.58	101.05	102.46	111.27

Table 3: Dependency of AAC,

\times 10^{-3}

, on the weighting coefficient of epistemic uncertainty component for the Total Variance approach with CAE.

Figure 7: Accuracy-rejection curves compared between total variance methods using a base ensemble, an ensemble trained with a mixed loss, and a CAE on the BloodyWell dataset.

Figure 8: Accuracy-rejection curves compared between total variance methods using a base ensemble, an LLM as an oracle and experts on the PubMedQA dataset.

Aleatoric Uncertainty

Aleatoric uncertainty is irreducible variability in outcomes, arising from noisy measurements, ambiguous inputs, or stochastic processes.

See all 44 papers

Explainable Uncertainty Estimation for Reliable Medical AI
Aug 28, 2026Li Rong Wang, Jamie Duell, Xinran Xu +6Uncertainty Quantification Explainable Artificial Intelligence
College of Computing and Data Science (CCDS) Nanyang Technological University (NTU) A*STAR Centre for Frontier AI Research Singapore · School Of Computing and Digital Technologies Sheffield Hallam University South Yorkshire, England · CCDS NTU Singapore +5
Bayesian uncertainty estimation improves clinical decision making in medical AI agents
Jul 22, 2026Frederik Hauke, Patrick Wienholt, Christiane Kuhl +4Clinical Decision Support Clinical Decision-Making
Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany. · Department of Medical Oncology, National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany. · Else Kröner Fresenius Center for Digital Health, TU Dresden, Dresden, Germany. +1
The Illusion of AI Expertise Under Uncertainty: A Probabilistic Paradigm to Interpret Results Under Ground Truth Variation
Jan 9, 2026Aparna Elangovan, Lei Xu, Mahsa Elyasi +8LLM Evaluation Generative AI Evaluation
Independent Researcher · Amazon · Gazi University +6

Aug 28, 2026J/K move · Enter open · S save

Explainable Uncertainty Estimation for Reliable Medical AI

Li Rong Wang, Jamie Duell, Xinran Xu, Thomas C. Henderson+5

College of Computing and Data Science (CCDS) Nanyang Technological University (NTU) A*STAR Centre for Frontier AI Research Singapore · School Of Computing and Digital Technologies Sheffield Hallam University South Yorkshire, England · CCDS NTU Singapore+5

Uncertainty Quantification Explainable Artificial Intelligence Uncertainty Decomposition Trust in AI

Open paper AskPDF arXiv

Artificial intelligence has strong potential to support clinical decision-making, yet its adoption in healthcare remains limited due to a lack of trust. Uncertainty estimation can signal unreliable predictions, and explainable AI (XAI) can clarify how predictions are made but existing methods treat them separately, providing no feature-level insight into why a prediction is uncertain or which tests to prioritize to reduce it. To address this gap, we propose explainable uncertainty estimation, which unifies uncertainty estimation and XAI to both quantify uncertainty and explain feature-level contributions. We introduce the Expected Gradients Reconstruction Uncertainty Estimate (egRUE), which incorporates prediction explanations into its uncertainty computation and decomposes uncertainty into feature-wise contributions. We prove theoretical properties of egRUE and show through experiments that it improves reliability and interpretability compared to existing methods. A user study with medical experts further demonstrates that egRUE's explanations improve calibrated trust over uncertainty scores alone, increasing confidence in correct predictions and reducing confidence in incorrect ones. By combining prediction uncertainty with feature-level explanations, egRUE strengthens decision-making support in safety-critical healthcare settings, clarifying both when predictions may be unreliable and which features drive that uncertainty.

Abstract

Aleatoric Uncertainty

Explore similar work

Explainable Uncertainty Estimation for Reliable Medical AI

Bayesian uncertainty estimation improves clinical decision making in medical AI agents

The Illusion of AI Expertise Under Uncertainty: A Probabilistic Paradigm to Interpret Results Under Ground Truth Variation