Local Discrimination of Binary Classifiers via Localized AUC Functional

Wenye Ren; Anna Győrffy-Kerekes; Tancredi Dorsi; Olga Demler

cs.LGOct 22, 2026

Local Discrimination of Binary Classifiers via Localized AUC Functional

Authors: Wenye Ren, Anna Győrffy-Kerekes, Tancredi Dorsi, Olga Demler

Organizations: Department of Computer Science, ETH Zurich, Switzerland · Department of Computer Science, Tsinghua University, China · Center for Learning Systems, Max Planck Institute, Tübingen, Germany · Department of Mathematics, ETH Zurich, Switzerland ·

Abstract

Risk prediction models are frequently evaluated with AUC, which means that during evaluation we are blind to performance heterogeneity, which could lead to bad decision-making in clinics. In this work, we aim to change that by identifying patient cohorts for whom the risk prediction model might not be an accurate predictor. To separate these cohorts, we use a version of the weighted AUC, which we call Localized AUC or LAUC. Using this metric, we build a method that we coin LOCATOR. This method is able to identify cohorts of patients for whom the risk prediction model is inaccurate. We test this method both in real-world data and in simulations. We also build a statistical background for the use of Localized AUC for our purposes.

Figures & tables

Figure 1: The comparison of predicted $\logit(\lauc(x_{i}))$ values between model $g_{M}$ ( Model M in legend ) and model $g_{N}$ ( Model N in legend ) in a simulated data set. According to the simulation design in section 4 , model $f_{M}$ should give more precise prediction when $x_{3}<0.5$ than model $f_{N}$ , while model $f_{N}$ should give more precise prediction when $x_{3}\geq 0.5$ . $g_{M}$ and $g_{N}$ are XGBoost models trained based on the weighted $\logit(\lauc(x_{i}))$ of $f_{M}$ and $f_{N}$ respectively. From the figure it is clear that model $g_{M}$ has larger $\logit(\lauc(x_{i}))$ than model $g_{N}$ when $x_{3}<0.5$ while model $g_{N}$ has larger $\logit(\lauc(x_{i}))$ than model $g_{M}$ when $x_{3}\geq 0.5$ , which aligns with our expectation.

Figure 2: ROC curves comparison of low- and high-performance subgroups from the negative-control simulation 4.2 . The XGBoost model used to identify the high- and low-performance subgroups is based on pure noise. The confidence bands of the two ROC curves overlap, as expected.

Figure 3: ROC curves comparison of low- and high-performance subgroups of the simulation as in section 4.3 . The test set is split into low- and high-performance subgroups according to the median of the predictions of the trained XGBoost model. The confidence level is set to be 95%. The high-performance subgroup has significantly higher ROC curve than low-performance subgroup, and the confidence bands are separable. The standard (unweighted) AUC value for low-performance subgroup is AUC=0.69, with CI=0.66-0.71, which is lower than the standard AUC value for high-performance subgroup AUC=0.88, with CI=0.86-0.90. The figure gives another validation for our method in identifying performance subgroups of the certain given model.

Figure 4: The distribution of $x_{1}$ of low- and high-performance subgroups, identified by the XGBoost model trained in the simulation. The low-performance subgroup is in the middle of two high-performance subgroups, which aligns with the simulation set up $x_{1}\sim\mathcal{N}(A\cdot y,1)$ with two normal distributions overlapping around $x_{1}=\frac{A}{2}=0.595116$ . So at the middle point it is hard to decide whether the data point comes from the event data $y=1$ or non-event data $y=0$ .

Figure 5: The three plots show the capability of different XGBoost models with maximum delta (AUC i difference), 70th percentile delta, and median delta. The two ROC curves in a high- and a low-performance subgroup were identified by splitting above and below the median of XGBoost predictions of $\logit(\lauc(x_{i}))$ . Subgroup $0\%-50\%$ corresponds to low-performance subgroup and Subgroup $50\%-100\%$ corresponds to high-performance subgroup. The two ROC curves show separation, with AUC values shown in the legend significantly different as expected. With confidence level of $95\%$ , the confidence bands are shown in the figure and the confidence intervals of AUC values are shown in the legend.

Figure 6: AUC with respect to the fraction assigned to the low-and high-performance subgroup in the NHANES data.

Subgroup	Total Number	CVD death number	mean classical AUC value
Low perforamance	2663	190	0.780
High performance	2663	37	0.880

Table 1: Total sizes, CVD death event numbers and mean AUC values of high and low performance subgroups from test data set

Figure 7: External Validaton in UK Biobank. The two ROC curves of a high- and a low-performance subgroup were identified by splitting above and below the median of XGBoost predictions of $\logit(\lauc(x_{i}))$ Subgroup $0\%-50\%$ corresponds to low-performance subgroup and Subgroup $50\%-100\%$ corresponds to high-performance subgroup. The two ROC curves show separation, with AUC values shown in the legend significantly different as expected. With confidence level of $95\%$ , the confidence bands are shown in the figure and the confidence intervals of AUC values are shown in the legend.

Figure 8: SHAP summary plot. the most critical variable is age, followed by sex, bp_tx(now taking prescribed medicine), smoking and statin medication use.

Figure 9: Attrition of NHANES data. Initially there are $62,160$ respondents from the NHANES 1999–2010 cycles (A–F). We restricted the sample to adults aged 30–79 years (the age range for the PREVENT equations), leaving $23,920$ individuals. After excluding pregnant women ( $n=466$ ) and individuals with self-reported prior cardiovascular disease ( $n=2,272$ ), $21,182$ remained. We further excluded participants with missing data on any of the 11 PREVENT predictor variables ( $n=3,151$ ), resulting in $18,031$ individuals with complete data. Of these, $259$ were excluded because the preventr scoring algorithm flagged out‑of‑range input values, leaving $17,772$ valid risk scores. Finally, we restricted to participants eligible for mortality linkage, yielding the final analytical cohort of $17,748$ individuals.

Figure 10: Absence of UMAP clusters in NHANES datset. UMAP visualization of the distribution of age, sex (male=1, female=0), education (Less Than 9th Grade=1, 9-11th Grade (Includes 12th grade with no diploma)=2, High School Grad/GED or Equivalent=3, Some College or AA degree=4, College Graduate or above=5, Refused=7, Don’t Know=9), race/ethnicity(Mexican American=1, Other Hispanic=2, Non-Hispanic White=3, Non-Hispanic Black=4, Other Race - Including Multi-Racial=5) and the ratio of poverty to income in training data set. From the figure it can be seen that there isn’t obvious cluster of subgroups.

Figure 11: Three ROC curves of three models split using $33\%$ and $67\%$ percentiles of XGBoost predictions $\logit(\lauc(x_{i}))$ . Subgroup $0\%-33\%$ corresponds to low-performance subgroup, Subgroup $33\%-67\%$ corresponds to medium-performance subgroup, and Subgroup $67\%-100\%$ corresponds to high-performance subgroup. The ROC curves of the three subgroups shows increasing performance when climbing from low- to high-performance subgroup. The AUC values shown in the legend also validate this. With confidence level of $95\%$ , the confidence band is shown in the figure and the confidence intervals of AUC values are shown in the legend.

Figure 12: Harmonization summary of NHANES and UK Biobank variable definitions

Figure 13: The histogram of standard AUC values of low- and high-performance subgroups in the test set of NHANES data with 100 seeds (0-99). The whole method is implemented: train the XGBoost model in the training set, then identify low- and high-performance subgroups in the test set. Then calculate the unweighted AUC in low- and high-performance subgroups respectively. Repeat the procedure for 100 seeds from 0 to 99 and plot the histogram using all AUC values. The figure shows that low-performance subgroup has lower AUC than high-performance subgroup. The mean AUC of low-performance subgroup is 0.744, with standard variation 0.0157. The mean AUC of high-performance subgroup is 0.822, with standard vatiation 0.0394.

Figure 14: UMAP visualization of the distribution of age, sex (male=1, female=0), education (Less Than 9th Grade=1, 9-11th Grade (Includes 12th grade with no diploma)=2, High School Grad/GED or Equivalent=3, Some College or AA degree=4, College Graduate or above=5, Refused=7, Don’t Know=9), race/ethnicity(Mexican American=1, Other Hispanic=2, Non-Hispanic White=3, Non-Hispanic Black=4, Other Race - Including Multi-Racial=5) and the ratio of poverty to income in testing data set.

Figure 15: The histogram of the calculated weighted AUC in test set. The chosen seed is 64, corresponding to the maximum AUC-delta parsimonious model. Mean AUC: 0.8260, Median AUC: 0.8282

Figure 16: The histogram of the calculated weighted AUC in train set. The chosen seed is 64, corresponding to the maximum AUC-delta parsimonious model. Mean AUC: 0.8321, Median AUC: 0.8359

Figure 17: The histogram of the Gower distance in test set. The chosen seed is 64, corresponding to the maximum AUC-delta parsimonious model. The histograms show that the distribution of Gower distance is close to Poisson or Gaussian distribution, which helps show that both the training set and testing set are non-biased and are not having preference towards specific subgroups which share certain similarities.

		Grouped by cvd_death
		Missing	Overall	False	True	P-Value
n			17748	16992	756
Age (years), mean (SD)		0	51.5 (13.5)	50.9 (13.2)	65.3 (11.0)	$<$ 0.001
Sex, n (%)	female		8989 (50.6)	8671 (51.0)	318 (42.1)	$<$ 0.001
	male		8759 (49.4)	8321 (49.0)	438 (57.9)
Race / Ethnicity, n (%)	Mexican American		3682 (20.7)	3557 (20.9)	125 (16.5)	$<$ 0.001

Table 2: Baseline Demographic and Clinical Characteristics

		Grouped by Outcome
		Missing	Overall	High-Performance Subgroup	Low-Performance Subgroup	P-Value
n			3550	1775	1775
Hypertension Meds, n (%)	False		2698 (76.0)	1608 (90.6)	1090 (61.4)	$<$ 0.001
	True		852 (24.0)	167 (9.4)	685 (38.6)
Statin Use, n (%)	False		3099 (87.3)	1692 (95.3)	1407 (79.3)	$<$ 0.001
	True		451 (12.7)	83 (4.7)	368 (20.7)

Table 3: Baseline Demographic and Clinical Characteristics

Binary Classification

Binary classification assigns examples to one of two classes while balancing accuracy, calibration, class imbalance, and the costs of different errors.

See all 66 papers

Extending ROC Analysis to Uncertainty-Aware Risk Prediction with an Interval-Based AUC (iAUC)
Feb 4, 2026Yuqi Li, Matthew M. EngelhardUncertainty Quantification Clinical Risk Prediction
Department of Biostatistics & Bioinformatics, Duke University
Geographically Regularized AUC-Maximizing Personalized Federated Learning
Sep 8, 2026Mayu Hiraishi, Kensuke Tanioka, Toshio ShimokawaClinical Risk Prediction Federated Learning
Department of Medicine · Wakayama Medical University · Wakayama, Japan +3
LOCUS: A Distribution-Free Loss-Quantile Score for Risk-Aware Predictions
Mar 2, 2026Matheus Barreto, Mário de Castro, Thiago R. Ramos +2Uncertainty Calibration
Department of Statistics, Federal University of S˜ao Carlos, S˜ao Carlos, S˜ao Paulo, Brazil · Institute of Mathematics and Computer Science, University of S˜ao Paulo, S˜ao Carlos, S˜ao Paulo, Brazil · School of Forest, Fisheries, and Geomatics Sciences, University of Florida, Gainesville, Florida, United States of America

Feb 4, 2026J/K move · Enter open · S save

Extending ROC Analysis to Uncertainty-Aware Risk Prediction with an Interval-Based AUC (iAUC)

Yuqi Li, Matthew M. Engelhard

Department of Biostatistics & Bioinformatics, Duke University

Uncertainty Quantification Clinical Risk Prediction

Open paper AskPDF arXiv

In high-stakes risk prediction, interval-valued predictions provide a natural way to represent predictive uncertainty. However, standard evaluation tools such as the receiver operating characteristic (ROC) curve and the area under the curve (AUC) are designed for point-valued predictions and do not capture how uncertainty affects discrimination performance. To address this gap, we develop an uncertainty-aware ROC framework and corresponding interval-based AUC (iAUC) metrics for evaluating the performance of interval-valued risk predictions. The framework constructs two ROC-style curves with associated areas

\mathrm{AUC}_L

and

\mathrm{AUC}_U

, and shows that these quantities induce a three-region decomposition of positive-negative rankings into confidently correct, ambiguous, and confidently incorrect orderings. Under valid class-conditional coverage,

\mathrm{AUC}_L

and

\mathrm{AUC}_U

, together with the pairwise miscoverage rate, provide theoretical bounds on the optimal AUC, linking interval coverage to achievable discrimination. The framework is compatible with a range of methods for quantifying epistemic uncertainty in risk prediction, including Bayesian, bootstrap, and ensemble-based procedures, and provides a practical way to compare predictive models and interval-generation methods. Experiments on clinical benchmark data and MIMIC-IV mortality prediction show that models with similar point AUC can have different uncertainty-aware ranking profiles, and that the proposed decomposition can support risk-aware model selection and threshold-based abstention.

Abstract

Binary Classification

Explore similar work

Extending ROC Analysis to Uncertainty-Aware Risk Prediction with an Interval-Based AUC (iAUC)

Geographically Regularized AUC-Maximizing Personalized Federated Learning

LOCUS: A Distribution-Free Loss-Quantile Score for Risk-Aware Predictions