Where Is a Prediction Model Miscalibrated, and for Whom? Local Audits in Outcome-Learned Pockets

Bernardo Modenesi; Taoyi Chen; Sima Najafzadehkhoei; Lucia Wang; Ameya Diwan; Beatrice Knudsen

cs.LGNov 2, 2026

Where Is a Prediction Model Miscalibrated, and for Whom? Local Audits in Outcome-Learned Pockets

Authors: Bernardo Modenesi, Taoyi Chen, Sima Najafzadehkhoei, Lucia Wang, Ameya Diwan, Beatrice Knudsen

Organizations: University of Utah, USA · Ohio State University, USA · rpotential.ai, USA · Google, USA

Abstract

Overall and group-level averages, and ranking metrics such as the AUC, can hide sets of observations for which a prediction model systematically over- or under-predicts. We present an audit that asks where a model is miscalibrated (in which sets of similar observations it over- or under-predicts on average) and for whom (whether its average prediction error differs between demographic groups, such as by race or sex, within the same set). It groups observations into pockets using random-forest similarity, which weighs each feature by its importance for predicting the outcome, with a ridge-regression grouping alongside. Pockets are learned on one sample and tested on another, with simultaneous confidence intervals that allow for correlation within units such as hospitals; the audit needs only features, outcomes and fixed predictions. In 17 simulation scenarios with an error planted for one group, the combined audit located it better than the strongest alternative grouping in 15, with false-alarm rates near or below 5%; with a local error planted for everyone, it won in 12 of 15 scenarios at the smaller error size and lost when the error was unrelated to the outcome. In public synthetic data from the care-management algorithm studied by Obermeyer et al., it found no detectable overall error but under-prediction of illness in the sickest pockets and over-prediction elsewhere, and distinguished Black patients' concentration where the score under-predicted for everyone from a Black-White gap within pockets. For APACHE IVa, an exploratory analysis located its previously reported over-prediction of ICU mortality in a mostly ventilated pocket and a low-risk pocket, with none detected in a moderate-risk one. Two chest X-ray classifiers showed offsetting local errors behind no detectable overall error. Code: https://github.com/modenesi/pocket-audit

Figures & tables

Panel of Figure 1 — Figure 1. The audit at a glance. Whole units (e.g., hospitals) are split once into development, discovery and audit roles. Two pocket maps are learned from the outcome on discovery data, without the group attribute or any model’s errors as inputs (group counts and predictions enter only the support check), and frozen: forest pockets $C_{F}$ and ridge pockets $C_{R}$ . Audit observations are assigned to a pocket in each map (in the forest map, weak or ambiguous assignments abstain). For each pocket the audit estimates the mean prediction error $\hat{\mu}_{c}$ and its between-group gap $\hat{\delta}_{c}$ ; each map’s tested pockets receive simultaneous intervals (Section 4 ), and an observation inherits its pocket’s flag in either map. The similarity forest $g$ is distinct from the assessed models $f$ (any model or score). A flow diagram: data are split by unit into development, discovery and audit roles; discovery produces two frozen pocket maps; audit observations are assigned to pockets in each map, each pocket's mean error and between-group gap are estimated with model predictions, and the report gives flags with simultaneous intervals, errors ruled out and untested pockets.

Null design	Single-map $\alpha$	Replicates	Forest	Combined
Between-group gap $\delta$
Continuous, $n=10{,}000$ , 7 specifications	.05	700–900	.050–.059 [.036, .076]	.046–.057 [.033, .074]
Continuous, $n=20{,}000$ , linear risk	.05	700	.059 [.043, .078]	.049 [.035, .067]
Continuous, $n=20{,}000$ , nonlinear risk	.05	700	.054 [.040, .074]	.047 [.034, .065]
Binary, $n=50{,}000$ , independent observations	.025	450	.016 [.008, .032]	.020 [.011, .038]
Binary, $n=50{,}000$ , 150 dependent units	.0125	450	.013 [.006, .029]	.013 [.006, .029]

Table 1. False flags when no error exists: the share of replicates in which any pocket (or the overall comparison) is flagged, with 95% Wilson (binomial) intervals (rows of seven specifications: ranges, and the envelope of their intervals). The

\delta

rows audit between-group gaps; the

\mu

rows audit local errors alone, in the localization simulations. “Forest” is audited at the single-map level; “Combined” audits each map at half of it. Binary levels are simulation-calibrated for

\delta

(Section 4 ); in the

\delta

150-unit design, 50 of 450 forest-map replicates had nothing testable and count as unflagged. No row shows clear inflation above 5% except

\mu

with 150 dependent units; the 30-replicate small check is imprecise.

Figure 2. Localization of planted between-group gaps ( $\delta$ ) in 17 simulation scenarios. (a) Observation-level $F_{1}$ for each grouping, with the number of replicates; bold marks the best non-oracle method in the row; outlined columns are our groupings. “Forest” is the spectral partition of the forest similarity in the continuous rows and the SBM partition of the forest graph elsewhere; “Combined” is the forest + ridge audit; “Leiden” partitions the forest graph with Leiden; “Ridge” is ridge-weighted K-means. Where a scenario ran in two batches, the displayed batch was selected retrospectively by a fixed rule, the batch in which the strongest method not built on the forest scored highest; † : shown from the other batch and left out of panel (b); –: not run. (b) Paired differences with 95% replicate-bootstrap intervals against the strongest available alternative grouping (mostly risk bins; named per row in the supplement): dots, forest; diamonds, combined audit. Intervals entirely right of zero favor our audit (green), entirely left favor the alternative (red); they are pointwise, and the 17 comparisons are not simultaneous. A heat map of F1 scores, 17 scenario rows by ten method columns, with our four groupings outlined, and a dot plot of paired differences against the strongest alternative grouping, with confidence intervals.

Continuous outcome, $n=10{,}000$
	Smaller shift			Larger shift ( $\delta$ designs)			Null
Scenario	Comb.	Error-t.	Best other	Comb.	Error-t.	Best other	flags, %
Linear risk	.73	.19	.55 (risk bins)	.71	.56	.64	3.6
+ scales, categories, U-shaped risk	.56	.18	.10 (risk bins)	.60	.57	.30	3.8
Nonlinear; region on top linear feature	.80	.20	.52 (risk bins)	.78	.57	.67	3.8
Nonlinear; standardized	.53	.18	.10 (risk bins)	.51	.57	.30	4.6

Table 2. Localizing a planted local miscalibration

\mu

: observation-level

F_{1}

, scored over all observations (Figure 2 scores focal-group observations), of the combined audit, the error-targeted grouping and the best other alternative, chosen separately at each shift (named at the smaller one; CEM-3 is CEM top-3); shifts of 0.2 SD or 5 points (smaller) and 0.4 SD or 10 points (the

\delta

designs). Bold: the largest unrounded score of the three. Null flags: replicates with no planted error in which the combined audit flagged any

\mu

; 9.1% (150 dependent units) exceeds the 5% target, and dashes mark irrelevant-feature rows, whose null runs are those of the matching risk-feature design. Binary one-tail and masking rows reuse maps frozen under the

\delta

design (30–40 replicates); others use 150–500. Two intermediate continuous specifications, which behave like linear risk, and all other groupings are in the supplement.

Figure 3. Overall errors conceal opposing local errors; the health-label model has smaller errors and gaps. (a) Local calibration error

\mu_{c}

of three predictions of the number of chronic conditions, by forest pocket of the care-management data. (b) The Black–White gap

\delta_{c}

in the same pockets. (c) Exploratory local calibration error of APACHE IVa and LightGBM ICU mortality predictions, by tree-cut pocket. Bars are simultaneous intervals (care management: forest map at

\alpha/2=0.025

; ICU:

\alpha=0.0125

); filled markers are flagged (open: interval includes zero); percentages give each pocket’s share of all audit observations (about 2% abstained and belong to no pocket; ICU: APACHE-valid stays). Pocket names summarize the typical features of discovery members, not exact membership rules. The neurological pocket could not be tested because one hospital held more than 20% of its stays. Three forest plots. Panel a: the commercial score and cost-label model under-predict chronic conditions by about 0.8 in the hypertension pocket and over-predict by 0.2 to 0.7 in four other pockets, while the health-label model is near zero everywhere. Panel b: Black minus White gaps are negative for the cost-based predictions, largest in the hypertension pocket, and small for the health-label model. Panel c: APACHE IVa over-predicts mortality overall, in ventilated and in low-risk stays, but not in a moderate pocket; LightGBM is near zero.

Model Calibration

Model calibration aligns predicted confidence or model outputs with observed outcomes and measurements to support more reliable decisions.

See all 229 papers

Oct 22, 2026

Local Discrimination of Binary Classifiers via Localized AUC Functional

Risk prediction models are frequently evaluated with AUC, which means that during evaluation we are blind to performance heterogeneity, which could lead to bad decision-making in clinics. In this work, we aim to change that by identifying patient cohorts for whom the risk prediction model might not be an accurate predictor. To separate these cohorts, we use a version of the weighted AUC, which we call Localized AUC or LAUC. Using this metric, we build a method that we coin LOCATOR. This method is able to identify cohorts of patients for whom the risk prediction model is inaccurate. We test this method both in real-world data and in simulations. We also build a statistical background for the use of Localized AUC for our purposes.

Wenye Ren, Anna Győrffy-Kerekes, Tancredi Dorsi +1

Department of Computer Science, ETH Zurich, Switzerland · Department of Computer Science, Tsinghua University, China · Center for Learning Systems, Max Planck Institute, Tübingen, Germany +2

Binary Classification Clinical Risk Prediction

Ask PDF

Jul 30, 2026

KAISEN: Reproducible Subgroup Fairness Auditing for Clinical Risk Models

Clinical risk models routinely achieve strong aggregate performance while producing materially different error rates across patient subgroups. Audit pipelines have been proposed to catch this, but their components are rarely stress-tested, so it is unclear which parts of an audit can be trusted and under what conditions. We present KAISEN, a five-phase audit pipeline covering subgroup stratification, disparity measurement, mechanism diagnostics, post-hoc mitigation, and drift monitoring, evaluated to the point of failure on a synthetic benchmark of 16 disease tasks, 15 social-determinant axes from Healthy People 2030, and three prespecified intersections. Four findings follow. (i) Significance tracks each axis's gap against its own minimum detectable effect: rank correlation between significance count and raw equalized-odds difference (EOD) across the 15 axes is rho = 0.56, rising to rho = 0.78 once EOD is standardized by that floor. (ii) Per-group threshold optimization reduces EOD in 48 of 48 held-out runs (paired delta = -0.285, 95% CI [-0.313, -0.252]), while group-wise Platt scaling -- the better calibrator -- behaves as a coin flip on EOD (19 of 48 runs improved, 95% CI [0.26, 0.55]) with mean effect near zero, so what an audit should report is the variance, not the average. (iii) The mechanism diagnostic classifies 144 of 144 controlled cases correctly but recovers none of 48 model-driven cases under proxy misspecification, with no signal that it failed. (iv) CUSUM failures and false alarms track cohort realization far more than disease: at the reference threshold, all 27 false alarms and 7 of 8 missed shifts come from different seeds (chi-squared p = 0.002), so a threshold tuned on one cohort fails to transfer. All results are synthetic with known ground truth and do not establish clinical validity. Code, artifacts, and scripts reproducing every number are released.

Sparsh Roy, Samuel Girmachew, Nishita Chavan

Massachusetts Institute of Technology · Hopewell Valley Central High School · East Brunswick High School

Distribution Shift Robustness Algorithmic Fairness Clinical Risk Prediction

Ask PDF

May 7, 2026

Correcting heterogeneous diagnostic bias when developing clinical prediction models using causal hidden Markov models

In routine care, individuals identified a priori as high-risk are usually tested for conditions more frequently. Protected attributes, such as sex or ethnicity may also determine testing frequency. Such heterogeneous detection rates across a population induce label error. This causes systematic model error for specific groups and biases performance metrics during validation. This paper proposes a method to correct for such bias in prediction models due to differential diagnostic delay. We use a causal inference framework to define our target estimand: an individual's diagnosis probability in a counterfactual scenario where their diagnosis rate matches that of a reference group. We model the longitudinal process as a hidden Markov model, in which confirmatory test results are emissions from a latent progressive disease stage. We validate our approach in simulated data and apply it to a case study of chronic kidney disease prediction using electronic health records. In simulations, our method reduces prediction bias and improves calibration-in-the-large, correcting the Observed:Expected ratio in the underdiagnosed group from 1.34 (standard deviation: 0.09) in a model developed without any correction for underdiagnosis bias to 1.02 (0.09). Violations of assumptions in the simulation affected the estimation of model parameters, but the proposed approach nonetheless remained better calibrated than the standard model. In the clinical case study, we identify diabetes as the main driver of observability, with an odds ratio of 10.36 (95% confidence interval, 9.80 - 11.02) in 6-month urine albumin-creatinine ratio testing rate. Using our approach to predict the counterfactual diagnostic rate in patients without diabetes, we improved the Observed:Expected ratio of a developed clinical prediction model from 1.55 (1.51 - 1.59) to 1.01 (0.98 - 1.04).

Jose Benitez-Aurioles, Ricardo Silva, Brian McMillan +1

Division of Informatics, Imaging & Data Sciences, University of Manchester, Manchester, United Kingdom · Department of Statistical Science, UCL, London, United Kingdom · Division of Population Health, Health Services Research & Primary Care, University of Manchester, Manchester, United Kingdom

Counterfactual Prediction Model Calibration Clinical Prediction Causal Inference Hidden Markov Models

Ask PDF

Abstract

Model Calibration

Explore similar work

Local Discrimination of Binary Classifiers via Localized AUC Functional

KAISEN: Reproducible Subgroup Fairness Auditing for Clinical Risk Models

Correcting heterogeneous diagnostic bias when developing clinical prediction models using causal hidden Markov models