Missingness-Aware Conformal Prediction Under Cross-Hospital Distribution Shift

Liang You; Dongwen Ou; Hengyu Shi; Siyuan Dai

cs.LGSep 25, 2026

Missingness-Aware Conformal Prediction Under Cross-Hospital Distribution Shift

Authors: Liang You, Dongwen Ou, Hengyu Shi, Siyuan Dai

Organizations: University of Pittsburgh · Duke University · Xiamen University

Abstract

Clinical measurements are recorded for some patients but not others, at rates that differ across hospitals, and marginal conformal coverage does not ensure coverage within groups defined by missingness. We propose a missingness-aware conformal calibration procedure for mortality prediction under cross-hospital distribution shift. It selects a measurement on an independent sample, groups patients by whether that measurement is recorded, and applies Mondrian calibration within each group, so no calibration outcome is reused. We evaluate the procedure across hospitals in eICU and across care units within one MIMIC-IV hospital, using three predictors. Relative to pooled calibration, it reduces the average worst-group coverage gap on its selected groups in all six settings, with a median reduction of 1.9 percentage points; paired site-bootstrap intervals exclude zero in five. These gains do not extend uniformly. Calibration by predicted risk achieves smaller gaps on a broader panel of missingness groups, and when eICU hospitals are evaluated separately, the gain shrinks for all three predictors and reverses in sign for one. We explain this discrepancy with a hospital-level decomposition. Pooling reweights hospitals through a covariance between group shares and coverage errors, and lets errors of opposite sign cancel: weighting explains the reversal, and cancellation accounts for most of the attenuation for the other two predictors. Constructed population distributions show that pooled and within-hospital evaluations can rank calibration methods oppositely even without sampling noise. Pooled improvement alone therefore cannot establish better coverage within hospitals, even when the calibration groups are fixed.

Figures & tables

Data	Model	Pooled	Missing	Risk (2)	Risk (3)	Label	Gain [95% interval], pp
eICU	LR	0.040	0.021	0.018	0.014	0.040	1.91 [1.73, 3.17]
eICU	XGB	0.029	0.026	0.012	0.009	0.030	0.34 [-0.08, 1.07]
eICU	MLP	0.029	0.017	0.017	0.014	0.029	1.17 [1.09, 1.94]
MIMIC-IV	LR	0.083	0.052	0.053	0.038	0.086	3.13 [1.76, 4.52]
MIMIC-IV	XGB	0.060	0.041	0.027	0.026	0.090	1.83 [1.33, 2.46]
MIMIC-IV	MLP	0.058	0.029	0.039	0.038	0.059	2.87 [1.83, 3.97]

Table 1: Mean selected gap on full test samples. Risk (2/3): two/three predicted-risk groups; Label: outcome-only calibration. Gain is pooled minus missingness in percentage points. Paired 95% basic bootstrap intervals condition on fitted models and assignments; MIMIC-IV uses only six units. Ten eICU and 18 MIMIC-IV assignments share predictions and evaluation groups.

Figure 1: Coverage differences on the selected groups and the audit panel.

\Delta

gap is the gap of missingness-aware calibration minus that of the indicated comparator; negative values favor missingness. Pooled: pooled calibration; Risk: risk terciles. Small points are individual assignments (ten eICU, 18 MIMIC-IV); large markers and bars show means

\pm

one SD. Assignments overlap, so bars are not confidence intervals.

Model	Group weights	Patient/pre	Patient/post	Equal/pre	Equal/post
LR	-0.0208	-0.0178	-0.0072	-0.0159	-0.0046
XGB	-0.0052	+0.0019	+0.0020	+0.0028	+0.0023
MLP	-0.0124	-0.0077	-0.0007	-0.0066	-0.0004

Table 2: Missingness minus pooled gap in eICU on matched hospitals, averaged across assignments; negative values favor missingness. Pre: signed errors aggregated before the absolute maximum (

A_{w}

). Post: each hospital’s gap computed first (

B_{w}

). Group weights: ordinary patient pooling within each group. All columns use the same hospitals within an assignment.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

$w_{0}$	Shift	Rule	$A$	$D$	$B$	$D-A$
0.5	0.00	Pooled	0.0190	0.1190	0.1190	0.1000
0.5	0.00	Mask	0.0000	0.1000	0.1000	0.1000
0.5	0.12	Pooled	0.0736	0.1736	0.1736	0.1000
0.5	0.12	Mask	0.0545	0.1545	0.1545	0.1000
0.8	0.00	Pooled	0.0524	0.1076	0.1076	0.0552
0.8	0.00	Mask	0.0600	0.1000	0.1000	0.0400

Appendix

Table 3: Exact population aggregation quantities at nominal coverage 0.9.

w_{0}

is the first hospital’s deployment weight. Shift applies only to the second hospital’s second mask state. These are population values, not estimates from simulated samples.

Model	Weights	$\Delta A$	$\Delta D$	$\Delta B$	$\Delta(D-A)$	$\Delta(B-D)$
LR	Patient	-0.01778	-0.00643	-0.00720	+0.01135	-0.00076
LR	Equal	-0.01586	-0.00464	-0.00462	+0.01121	+0.00002
XGB	Patient	+0.00192	+0.00203	+0.00205	+0.00011	+0.00002
XGB	Equal	+0.00282	+0.00259	+0.00235	-0.00022	-0.00024
MLP	Patient	-0.00768	-0.00137	-0.00066	+0.00631	+0.00071
MLP	Equal	-0.00658	-0.00095	-0.00039	+0.00563	+0.00056

Appendix

Table 4: Paired missingness-minus-pooling differences in the clinical hospital decomposition. Means over ten eICU assignments, with the same complete hospitals and weights for both methods.

Mask state	Pooled calibration	Missingness calibration	Difference
0	-0.00362	-0.00393	-0.00032
1	+0.01382	+0.01478	+0.00096

Appendix

Table 5: Mean signed covariance terms for eICU XGBoost over ten assignments. State 0 is observed and state 1 is missing. Difference is missingness minus pooled calibration; entries are in coverage-probability units.

Data	Model	Gain (pp)	95% lower	95% upper	Adjusted lower	Adjusted upper
eICU	LR	1.91	1.73	3.17	1.55	3.46
eICU	XGB	0.34	-0.08	1.07	-0.22	1.33
eICU	MLP	1.17	1.09	1.94	1.00	2.16
MIMIC-IV	LR	3.13	1.76	4.52	1.33	5.01
MIMIC-IV	XGB	1.83	1.33	2.46	1.30	2.66
MIMIC-IV	MLP	2.87	1.83	3.97	1.61	4.40

Appendix

Table 6: Paired improvement in pooled selected gap (pooled minus missingness), in percentage points. Basic intervals reflect bootstrap quantiles around the observed improvement. Adjusted endpoints use quantiles at

0.05/12

and

1-0.05/12

for six comparisons.

Data	Model	All groups	$n\geq 500$	$n\geq 1000$	Standardized
eICU	LR	+0.069	+0.069	+0.069	+18.133
eICU	XGB	+0.097	+0.097	+0.097	+24.858
eICU	MLP	+0.034	+0.034	+0.034	+9.493
MIMIC-IV	LR	+0.066	+0.066	+0.054	+11.046
MIMIC-IV	XGB	+0.037	+0.040	+0.041	+7.255
MIMIC-IV	MLP	+0.016	+0.017	+0.016	+2.757

Appendix

Table 7: Missingness minus risk-tercile panel error. Positive entries favor risk calibration. The first three columns are raw absolute gaps with the stated minimum group size; the last uses the nominal standardized error, in different units. Means across assignments.

Data	Model	Rule	Coverage	Size	Selected	Panel
eICU	LR	Pooled	0.896	1.221	0.040	0.116
eICU	LR	Missingness	0.897	1.224	0.021	0.113
eICU	LR	Risk (2)	0.900	1.340	0.018	0.064
eICU	LR	Risk (3)	0.900	1.212	0.014	0.044
eICU	LR	Label	0.896	1.253	0.040	0.117
eICU	LR	Risk + label	0.901	1.669	0.008	0.034

Appendix

Table 9

Data	Model	Mean changed (%)	Assignment range (%)
eICU	LR	4.52	0.98–11.09
eICU	XGB	1.03	0.01–3.91
eICU	MLP	1.52	0.66–3.07
MIMIC-IV	LR	9.38	5.40–17.57
MIMIC-IV	XGB	4.97	1.24–12.80
MIMIC-IV	MLP	7.08	3.18–13.29

Appendix

Table 9: Test patients whose prediction sets differ between missingness and pooled calibration. Means and ranges are across ten eICU or 18 MIMIC-IV assignments.

Data	Model	Rule	$U$	$O$	Panel gap	$C_{0}$	$C_{1}$
eICU	LR	Pooled	0.019	0.026	0.116	0.897	0.888
eICU	LR	Missingness	0.014	0.010	0.113	0.898	0.885
eICU	LR	Risk (2)	0.009	0.012	0.064	0.896	0.937
eICU	LR	Risk (3)	0.006	0.008	0.044	0.903	0.868
eICU	LR	Label	0.019	0.026	0.117	0.895	0.906
eICU	XGB	Pooled	0.017	0.016	0.118	0.971	0.155

Appendix

Table 11

Data	Model	Rule	$\varnothing$	$\{0\}$	$\{1\}$	$\{0,1\}$
eICU	LR	Pooled	0.000	0.625	0.153	0.221
eICU	LR	Missingness	0.000	0.624	0.152	0.224
eICU	LR	Risk (2)	0.043	0.473	0.100	0.383
eICU	LR	Risk (3)	0.054	0.603	0.076	0.267
eICU	LR	Label	0.000	0.592	0.155	0.253
eICU	XGB	Pooled	0.048	0.935	0.017	0.000

Appendix

Table 12

Data	Rule	Selected gap	Panel gap	$U$	$C$	Size
eICU	Missingness	0.031	0.125	0.125	0.893	0.952
eICU	Risk	0.016	0.039	0.032	0.899	1.026
eICU	Conditional	0.036	0.055	0.022	0.900	1.016
eICU	Conditional + mask	0.025	0.051	0.031	0.899	1.015
MIMIC-IV	Missingness	0.045	0.087	0.068	0.906	1.024
MIMIC-IV	Risk	0.029	0.052	0.025	0.906	1.110

Appendix

Table 12: Conditional conformal prediction with and without the selected mask in its basis, XGBoost, identical 2,000-patient test samples. Gaps and shortfall use nominal coverage 0.9.

Data	Sample	Population	Cells	$n$ range	$\mu$ range	$N_{\rm empty}$
eICU	calibration	All	10	10542–12247	0.071–0.101	0
eICU	calibration	Mask	20	783–11139	0.064–0.126	0
eICU	calibration	Site + mask	172	0–4333	0.016–0.266	7
eICU	test	All	10	20097–22316	0.081–0.109	0
eICU	test	Mask	20	2045–19316	0.065–0.123	0
eICU	test	Site + mask	342	0–3607	0.000–0.438	9

Appendix

Table 13: Population denominators and mortality ranges, including repeated assignment-specific cells. Mortality ranges exclude empty cells; denominator ranges retain them.

N_{\rm empty}

counts cells with zero patients. Mask refers to the two selected states. Counts are shared by all predictors and include repeated evaluation of a hospital across assignments.

Data	Model	Population	$C$	$C_{0}$	$C_{1}$	Bound
eICU	LR	Mask	0.896	0.898	0.874	0.033
eICU	LR	Site + mask	0.899	0.899	0.857	0.153
eICU	MLP	Mask	0.900	0.945	0.448	0.018
eICU	MLP	Site + mask	0.897	0.943	0.445	0.082
eICU	XGB	Mask	0.895	0.970	0.157	0.015
eICU	XGB	Site + mask	0.892	0.969	0.163	0.052

Appendix

Table 14: Outcome coverage within the selected mask states at

\alpha=0.1

, missingness calibration. Unweighted means over nonempty cells;

C_{y}

excludes cells without label

y

. Bound averages

\max\{0,1-(1-C)/\mu\}

over cells with deaths, using each cell’s actual coverage and mortality; it is descriptive, not a guarantee for each cell.

Data	Model	$\alpha$	$C$	$C_{0}$	$C_{1}$	Size	$U$
eICU	LR	0.05	0.948	0.948	0.950	1.433	0.071
eICU	LR	0.10	0.897	0.898	0.885	1.224	0.112
eICU	LR	0.20	0.797	0.800	0.770	0.985	0.163
eICU	MLP	0.05	0.948	0.981	0.631	1.209	0.044
eICU	MLP	0.10	0.898	0.944	0.448	1.038	0.079
eICU	MLP	0.20	0.797	0.851	0.277	0.867	0.124

Appendix

Table 15: Missingness calibration across coverage levels, recalibrated on the original calibration patients.

U

is the largest shortfall below

1-\alpha

across the measurement panel. Means across the ten eICU or 18 MIMIC-IV assignments.

Figure 2: Death coverage under missingness-aware calibration. Top: varying the error allowance

\alpha

. Bottom: multiplying predicted odds at fixed

\alpha=0.1

and recalibrating; rankings and AUROC are unchanged. Faint points are individual assignments (ten eICU, 18 MIMIC-IV); lines show means; dashed lines mark nominal coverage. Assignments overlap, so no independence-based interval is implied.

Data	Model	Odds factor	$C$	$C_{0}$	$C_{1}$	Size	$U$
eICU	LR	0.25	0.896	0.939	0.473	0.985	0.104
eICU	LR	1.00	0.897	0.898	0.885	1.224	0.112
eICU	LR	4.00	0.896	0.885	0.998	1.751	0.124
eICU	MLP	0.25	0.898	0.950	0.391	1.018	0.082
eICU	MLP	1.00	0.898	0.944	0.448	1.038	0.079
eICU	MLP	4.00	0.898	0.938	0.510	1.064	0.077

Appendix

Table 16: Multiplying predicted odds and recalibrating missingness groups at

\alpha=0.1

. Overall and outcome coverage, mean set size, and panel shortfall are assignment means.

Data	Risk	Label	Risk only	Label only	Joint	Joint range
eICU	1	0	0.905	1.000	0.900	0.887–0.914
eICU	1	1	0.000	0.000	0.916	0.828–1.000
eICU	2	0	0.928	1.000	0.901	0.894–0.913
eICU	2	1	0.000	0.308	0.921	0.873–0.954
eICU	3	0	0.997	0.638	0.895	0.873–0.921
eICU	3	1	0.589	1.000	0.906	0.883–0.924

Appendix

Table 17: Coverage in the six risk–outcome cells for XGBoost. Risk tercile increases from 1 to 3; labels 0 and 1 denote survival and death. Means and joint-rule ranges are over ten eICU or 18 MIMIC-IV assignments; ranges describe variation across assignments and are not confidence intervals.

Data	Model	Rule	$C$	$C_{1}$	$U$	Size
eICU	LR	Label	0.896	0.906	0.116	1.253
eICU	LR	Risk + label	0.901	0.904	0.034	1.669
eICU	LR	Risk (3)	0.900	0.868	0.043	1.212
eICU	MLP	Label	0.899	0.907	0.065	1.500
eICU	MLP	Risk + label	0.900	0.906	0.026	1.750
eICU	MLP	Risk (3)	0.898	0.587	0.045	1.121

Appendix

Table 18: Single-target and joint calibration at

\alpha=0.1

, full test samples.

C

is overall coverage,

C_{1}

death coverage, and

U

panel shortfall.

Data	Risk	Label	$n_{\rm cal}$	Rank	$n_{\rm test}$	Max rank
eICU	1	0	3042–4418	2739–3978	6331–7797	0
eICU	1	1	18–35	18–33	23–60	1
eICU	2	0	3100–3835	2791–3453	5854–7621	0
eICU	2	1	81–158	74–144	171–302	0
eICU	3	0	2245–3443	2022–3100	4966–6595	0
eICU	3	1	622–1088	561–981	1372–2051	0

Appendix

Table 19: Joint calibration denominators, quantile ranks, and test denominators, XGBoost. Entries are ranges across assignments. Max rank counts assignments in which the quantile rank equals the calibration count and therefore selects the maximum score. No test cell is empty.

Data	Model	Rule	$\varnothing$	$\{0\}$	$\{1\}$	$\{0,1\}$
eICU	LR	Label	0.000	0.592	0.155	0.253
eICU	LR	Risk + label	0.000	0.202	0.129	0.669
eICU	LR	Risk (3)	0.054	0.603	0.076	0.267
eICU	MLP	Label	0.000	0.363	0.136	0.500
eICU	MLP	Risk + label	0.000	0.128	0.122	0.750
eICU	MLP	Risk (3)	0.042	0.747	0.046	0.164

Appendix

Table 20: Four output-set types for the target-alignment comparisons, computed within each assignment and then averaged. Entries are mean fractions of full test populations at

\alpha=0.1

Data	Rule	Coverage	Size	$U$	Site $U$	$C_{1}$
eICU	Missingness	0.895	0.952	0.116	0.194	0.161
eICU	Risk bins	0.900	1.026	0.019	0.073	0.512
eICU	Learned risk	0.898	1.010	0.040	0.098	0.457
eICU	+ Mask	0.898	1.010	0.040	0.098	0.457
eICU	Gated	0.898	1.010	0.040	0.098	0.457
eICU	Label	0.897	1.246	0.117	0.169	0.903

Appendix

Table 21: Learned threshold comparison, assignment means.

U

is the largest undercoverage on the full measurement panel; Site

U

first computes this maximum within each nonempty test site and then averages sites within an assignment.

C_{1}

is coverage among deaths. Missingness and Risk bins are the original missingness and risk-tercile methods. All rows use identical test patients within an assignment.

Conformal Prediction

Conformal prediction wraps predictive models with calibrated sets or intervals that provide statistical coverage guarantees.

See all 352 papers

May 14, 2026

On the Burden of Achieving Fairness in Conformal Prediction

Conformal prediction is often calibrated with a single pooled threshold, but this can hide cross-group heterogeneity in score distributions and distort group-wise coverage. We study this phenomenon through the population score distributions underlying split conformal calibration. First, we derive a conservation law and lower bound showing that pooled calibration incurs irreducible group-wise coverage distortion at a scale set by cross-group quantile heterogeneity. Second, we demonstrate that the two leading fairness definitions for conformal prediction, Equalized Coverage and Equalized Set Size, are fundamentally in tension. Third, we quantify the cost of moving between policies which treat groups separately or pool them. Experiments on synthetic and real data confirm the same bidirectional trade-off after finite-sample calibration. Our results show that, for the policy families studied here, calibration choice does not remove cross-group heterogeneity; it determines whether the resulting distortion appears in the coverage or size dimension, providing a principled lens for analyzing fairness-oriented calibration choices in practice.

Ziang Gao, Pengqi Liu, Archer Yi Yang +3

McGill University Montreal, Canada · TD Insurance Montreal, Canada · Layer 6 AI Toronto, Canada

Fair Conformal Prediction Conformal Prediction Group Fairness

Ask PDF

Feb 23, 2026

Making Conformal Predictors Robust in Healthcare Settings: a Case Study on EEG Classification

Quantifying uncertainty in clinical predictions is critical for high-stakes diagnosis tasks. Conformal prediction offers a principled approach by providing prediction sets with theoretical coverage guarantees. However, in practice, patient distribution shifts violate the i.i.d. assumptions underlying standard conformal methods, leading to poor coverage in healthcare settings. In this work, we evaluate several conformal prediction approaches on EEG seizure classification, a task with known distribution shift challenges and label uncertainty. We demonstrate that personalized calibration strategies can improve coverage by over 20 percentage points while maintaining comparable prediction set sizes. Our implementation is available via PyHealth, an open-source healthcare AI framework: https://github.com/sunlabuiuc/PyHealth.

Arjun Chatterjee, Sayeed Sajjad Razin, John Wu +3

University of Illinois Urbana-Champaign, Urbana, IL 61801, USA · PyHealth · Bangladesh University of Engineering and Technology

Fair Conformal Prediction Conformal Prediction Clinical Prediction EEG Seizure Detection

Ask PDF

Sep 27, 2026

Sharp training-conditional coverage for conformal prediction under covariate shift

Weighted split conformal prediction reweights calibration scores by the likelihood ratio between the test and training covariate distributions and guarantees marginal coverage under covariate shift. We study its coverage conditional on the calibration data. An elementary argument, based on a single concentration inequality at a fixed population quantile, gives explicit training-conditional bounds without unspecified constants, and shows that the relevant scale is not the supremum of the likelihood ratio but a variance proxy built from the chi-squared divergence of the shift and from the average of the ratio over the part of the test population, of probability equal to the miscoverage level, where it is largest. A two-point lower bound shows that the root-m rate and the chi-squared contribution are intrinsic to the shift. Run at an explicitly inflated level, the weighted quantile becomes a deterministic PAC prediction set. We compare it with randomized rejection sampling and with importance-weighted learn-then-test and, through a certified choice of a clipping level for the likelihood ratio, map the regime in which each gives the narrower valid set. The analysis extends to estimated likelihood ratios and to tail functionals estimated from an unlabeled source sample, which yields a fully finite-sample certificate.

Mehrdad Pournaderi

Mofid Securities

Weighted Conformal Prediction Set Prediction Conformal Prediction Covariate Shift

Ask PDF

Abstract

Conformal Prediction

Explore similar work

On the Burden of Achieving Fairness in Conformal Prediction

Making Conformal Predictors Robust in Healthcare Settings: a Case Study on EEG Classification

Sharp training-conditional coverage for conformal prediction under covariate shift