Extending ROC Analysis to Uncertainty-Aware Risk Prediction with an Interval-Based AUC (iAUC)

Yuqi Li; Matthew M. Engelhard

cs.LGFeb 4, 2026

Extending ROC Analysis to Uncertainty-Aware Risk Prediction with an Interval-Based AUC (iAUC)

Authors: Yuqi Li, Matthew M. Engelhard

Organizations: Department of Biostatistics & Bioinformatics, Duke University

Abstract

In high-stakes risk prediction, interval-valued predictions provide a natural way to represent predictive uncertainty. However, standard evaluation tools such as the receiver operating characteristic (ROC) curve and the area under the curve (AUC) are designed for point-valued predictions and do not capture how uncertainty affects discrimination performance. To address this gap, we develop an uncertainty-aware ROC framework and corresponding interval-based AUC (iAUC) metrics for evaluating the performance of interval-valued risk predictions. The framework constructs two ROC-style curves with associated areas $\mathrm{AUC}_L$ and $\mathrm{AUC}_U$ , and shows that these quantities induce a three-region decomposition of positive-negative rankings into confidently correct, ambiguous, and confidently incorrect orderings. Under valid class-conditional coverage, $\mathrm{AUC}_L$ and $\mathrm{AUC}_U$ , together with the pairwise miscoverage rate, provide theoretical bounds on the optimal AUC, linking interval coverage to achievable discrimination. The framework is compatible with a range of methods for quantifying epistemic uncertainty in risk prediction, including Bayesian, bootstrap, and ensemble-based procedures, and provides a practical way to compare predictive models and interval-generation methods. Experiments on clinical benchmark data and MIMIC-IV mortality prediction show that models with similar point AUC can have different uncertainty-aware ranking profiles, and that the proposed decomposition can support risk-aware model selection and threshold-based abstention.

Figures & tables

Figure 1: Interval-based comparisons. Left : patient-level interval predictions relative to a decision threshold, where intervals entirely above or below the threshold support definitive threshold-based decisions and intervals spanning the threshold indicate ambiguity. Right : positive-negative interval comparisons used for AUC-style ranking, yielding correct, incorrect, or ambiguous orderings.

Figure 2: Geometric interpretation of interval-based ROC analysis. (a) The lower curve plots

\mathrm{TPR}_{L}

versus

\mathrm{FPR}_{U}

, and the upper curve plots

\mathrm{TPR}_{U}

versus

\mathrm{FPR}_{L}

, separating the ROC plane into correct, incorrect, and ambiguous ordering regions. (b) At a fixed threshold

t

, the two curve points form a right triangle whose legs measure positive-class (AC) and negative-class (BC) threshold ambiguity.

Figure 3: Three-region decomposition on the Pima Indians Diabetes benchmark across nominal interval levels. Each panel shows a representative model-generator pair.

Model	Generator	Point AUC	$P(I_{1}>I_{0})$	$P(\mathrm{overlap})$	$P(I_{1}<I_{0})$
LR	Bootstrap	0.821	0.591	0.352	0.057
RF	Deep Ensemble	0.810	0.728	0.147	0.125
HistGBM	Bootstrap	0.826	0.305	0.685	0.009
HistGBM	Deep Ensemble	0.824	0.566	0.385	0.049
EBM	Bootstrap	0.830	0.265	0.728	0.007
EBM	Deep Ensemble	0.831	0.675	0.249	0.076

Table 1: Selected Pima Indians Diabetes model-generator comparisons at 90% nominal interval level.

Panel of Figure 4 — Figure 4: Additional empirical analyses. (a) MIMIC-IV mortality prediction at the 90% nominal interval level. Each bar decomposes positive-negative pairs into confidently correct (blue), ambiguous (gray), and confidently incorrect (red) rankings. (b) Synthetic validation of the coverage-based bound on the optimal AUC. The shaded region denotes $[\operatorname{AUC}_{L}-p_{\mathrm{pair}},\operatorname{AUC}_{U}+p_{\mathrm{pair}}]$ .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Quantity	Value	Difference
$\operatorname{AUC}_{L}$ vs. $P(I_{1}>I_{0})$	0.6074 / 0.6072	0.0002
$\operatorname{AUC}_{U}$ vs. $1-P(I_{1}<I_{0})$	0.9427 / 0.9439	0.0013
$P(I_{1}>I_{0})$	0.6072	—
$P(\mathrm{overlap})$	0.3367	—
$P(I_{1}<I_{0})$	0.0561	—
Sum of probabilities	1.0000	—

Appendix

Table 2: Numerical validation of AUC–probability equivalence at the 90% nominal interval level.

Figure 5: Full three-region decomposition on the Pima Indians Diabetes benchmark across nominal interval levels. Rows correspond to base models and columns correspond to interval-generators.

Model	Generator	Point AUC	$P(I_{1}>I_{0})$	$P(\mathrm{overlap})$	$P(I_{1}<I_{0})$	$\mathrm{AUC}_{L}$	$\mathrm{AUC}_{U}$	Sum
LR	Bootstrap	0.821	0.591	0.352	0.057	0.591	0.943	1.000
LR	Bayesian Bootstrap	0.822	0.604	0.337	0.059	0.604	0.941	1.000
LR	Naive	0.821	0.573	0.377	0.051	0.573	0.950	1.000
RF	Bootstrap	0.812	0.493	0.471	0.036	0.493	0.964	1.000
RF	Deep Ensemble	0.810	0.728	0.147	0.125	0.728	0.875	1.000
RF	Bayesian Bootstrap	0.811	0.674	0.234	0.092	0.674	0.907	1.000

Appendix

Table 3: Full Pima Indians Diabetes decomposition at the 90% nominal interval level.

Model	AUC	$\operatorname{AUC}_{L}$	$\operatorname{AUC}_{U}$	$P(\mathrm{overlap})$	$P(I_{1}<I_{0})$
LR	0.772	0.678	0.848	0.170	0.152
RF	0.814	0.637	0.925	0.288	0.075
HistGB	0.835	0.567	0.962	0.395	0.038
MLP	0.827	0.405	0.985	0.581	0.015

Appendix

Table 4: MIMIC-IV model comparison at the 90% nominal interval level.

$\epsilon$	Eligible models	Selected model	Selected $\operatorname{AUC}_{L}$
0.05	HistGB, MLP	HistGB	0.567
0.10	RF, HistGB, MLP	RF	0.637
0.15	RF, HistGB, MLP	RF	0.637

Appendix

Table 5: Risk-aware model selection for MIMIC-IV using

\max_{m}P_{m}(I_{1}>I_{0})

subject to

P_{m}(I_{1}<I_{0})\leq\epsilon

Model	Threshold	Point TPR	Point FPR	Abstain	Retained	Pos. abstain	Neg. abstain	Retained TPR/FPR
LR	Train prev.	0.664	0.263	0.137	0.863	0.155	0.135	0.683 / 0.228
RF	Train prev.	0.722	0.261	0.248	0.752	0.265	0.246	0.777 / 0.195
HistGB	Train prev.	0.741	0.241	0.317	0.683	0.405	0.306	0.801 / 0.141
MLP	Train prev.	0.757	0.274	0.476	0.524	0.606	0.461	0.824 / 0.093
LR	$t=0.20$	0.343	0.078	0.057	0.943	0.120	0.049	0.333 / 0.058
RF	$t=0.20$	0.437	0.082	0.134	0.866	0.291	0.115	0.410 / 0.044

Appendix

Table 6: MIMIC-IV threshold-abstention results at the 90% nominal interval level.

Figure 6: Synthetic validation of theoretical bounds on the optimal AUC

\operatorname{AUC}^{*}

. The shaded region denotes the theoretical range

[\operatorname{AUC}_{L}-p_{\mathrm{pair}},\operatorname{AUC}_{U}+p_{\mathrm{pair}}]

, which contains the Bayes-optimal AUC across evaluated miscoverage rates.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 649 papers

Local Discrimination of Binary Classifiers via Localized AUC Functional
Oct 22, 2026Wenye Ren, Anna Győrffy-Kerekes, Tancredi Dorsi +1Binary Classification Clinical Risk Prediction
Department of Computer Science, ETH Zurich, Switzerland · Department of Computer Science, Tsinghua University, China · Center for Learning Systems, Max Planck Institute, Tübingen, Germany +2
A Ground-Truth Framework for Uncertainty Disentanglement with Posterior Risk
Aug 6, 2026Frieder Wizgall, Georg Tirpitz, Moritz Seiler +2Uncertainty Quantification Aleatoric Uncertainty
University of Tübingen, Tübingen, Germany · Hertie Institute for AI in Brain Health, University of Tübingen, Tübingen, Germany · Bernstein Center for Computational Neuroscience Berlin, Berlin, Germany +2
CURA: Clinical Uncertainty Risk Alignment for Language Model-Based Risk Prediction
Apr 16, 2026Sizhe Wang, Ziqi Xu, Claire Najjuuko +2Language Model Calibration LLM Uncertainty Estimation
Washington University in St. Louis

Oct 22, 2026J/K move · Enter open · S save

Local Discrimination of Binary Classifiers via Localized AUC Functional

Wenye Ren, Anna Győrffy-Kerekes, Tancredi Dorsi, Olga Demler

Department of Computer Science, ETH Zurich, Switzerland · Department of Computer Science, Tsinghua University, China · Center for Learning Systems, Max Planck Institute, Tübingen, Germany+2

Binary Classification Clinical Risk Prediction

Open paper AskPDF arXiv

Risk prediction models are frequently evaluated with AUC, which means that during evaluation we are blind to performance heterogeneity, which could lead to bad decision-making in clinics. In this work, we aim to change that by identifying patient cohorts for whom the risk prediction model might not be an accurate predictor. To separate these cohorts, we use a version of the weighted AUC, which we call Localized AUC or LAUC. Using this metric, we build a method that we coin LOCATOR. This method is able to identify cohorts of patients for whom the risk prediction model is inaccurate. We test this method both in real-world data and in simulations. We also build a statistical background for the use of Localized AUC for our purposes.

Abstract

Uncertainty Quantification

Explore similar work

Local Discrimination of Binary Classifiers via Localized AUC Functional

A Ground-Truth Framework for Uncertainty Disentanglement with Posterior Risk

CURA: Clinical Uncertainty Risk Alignment for Language Model-Based Risk Prediction