Uncertainty quantification is essential for deploying machine learning models in high-stakes domains such as scientific discovery and healthcare. Conformal Prediction (CP) provides finite-sample coverage guarantees under exchangeability, an assumption often violated in practice due to distribution shift. Under covariate shift, restoring validity requires importance weighting, yet accurate density-ratio estimation becomes unstable when training and test distributions exhibit limited support overlap. We propose KMM-CP, a conformal prediction framework based on Kernel Mean Matching (KMM) for covariate-shift correction. We show that KMM directly controls the bias-variance components governing conformal coverage error by minimizing RKHS moment discrepancy under explicit weight constraints, and establish asymptotic coverage guarantees under mild conditions. We then introduce a selective extension that identifies regions of reliable support overlap and restricts conformal correction to this subset, further improving stability in low-overlap regimes. Experiments on molecular property prediction benchmarks with realistic distribution shifts show that KMM-CP reduces coverage gap by over 50% compared to existing approaches. The code is available at https://github.com/siddharthal/KMM-CP.
Figures & tables
Figure 1: Synthetic experiment: (a) Covariate shift with partial support overlap, the target includes regions poorly supported by the source. (b) Classifier-based density-ratio assigns heavy-tailed weights to low-support regions, reducing effective sample size (ESS). (c) Standard KMM enforces bounded weights and matches moments over the full domain, improving stability but still correcting unsupported regions. (d) Selective KMM jointly optimizes source weights and target selection variables, filtering unsupported regions and stabilizing weights. (e) Classifier weighting reduces ESS, KMM lowers moment discrepancy (MMD), and selective KMM achieves both stable ESS and improved moment matching.
∑i=1nw~i1{Si≤t}≥1−αconf
Algorithm 1 Split Conformal Prediction with (Selective) KMM
Figure 2: Calibration curves under covariate shift (global calibration). The dashed line denotes nominal coverage. Uniform CP undercovers systematically, density-ratio baselines partially correct this, KMM improves alignment, and SKMM most closely tracks the target across datasets.
Dataset
Random
Shifted
AMES
0.0121
0.0844
BBB-Martins
0.0346
0.4579
HIV
0.0044
0.1830
Tox21
0.0249
0.4366
hERG-Karim
0.0254
0.0384
Table 1: Mean absolute deviation (MAD) of uniform conformal prediction under random (no-shift) and shifted splits.
Uniform
KDE
CoDrug KDE
Logistic
KMM
SKMM
Dataset
G
M
G
M
G
M
G
M
G
M
G
M
AMES
0.166
0.084
0.137
0.046
0.093
0.090
0.102
0.096
0.078
0.079
0.065
0.072
Tox21
0.409
0.437
0.323
0.331
0.103
0.110
0.073
0.089
0.053
0.057
0.015
0.020
BBB-Martins
0.424
0.458
0.364
0.393
0.329
0.381
0.396
0.443
0.319
0.353
0.213
0.270
hERG
0.034
0.038
0.096
0.094
0.031
0.030
0.020
0.022
0.028
0.036
0.009
0.019
HIV
0.185
0.183
0.291
0.283
0.145
0.151
0.297
0.269
0.087
0.098
0.013
0.022
Table 2: Mean absolute deviation (MAD) from nominal coverage under covariate shift for global (G) and Mondrian (M) calibration. Lower is better. Best per column is bold ; second-best is gray bold . SKMM attains the lowest MAD under global calibration and the best overall mean rank across both settings.
Figure 3: Bias–variance proxy ( MMD+1/ESS ) under covariate shift. MAD in brackets.
Conformal predictive systems (CPS) output calibrated bands of CDFs under exchangeability. We extend generalized CPS to non-exchangeable settings by encoding distributional shifts through observation-specific permutation weights. This yields shift-aware predictive systems that remain valid whenever the test point is, conditionally on the unordered sample, a weighted draw from the observed atoms. Since such weights are typically estimated, we introduce weight-uncertainty boxes and construct robust CPS envelopes with finite-sample or asymptotic confidence guarantees. We derive efficient computation for conformity-measure CPS, conformal binning, and conformal isotonic distributional regression. Experiments under covariate shift and feedback-driven biomolecular design show calibrated predictive bands that widen under stronger shifts and tighten as sample size increases.
Jef Jonkers, Johanna Ziegel
IDLab Department of Electronics and Information Systems Ghent University Ghent, Belgium · Seminar for Statistics ETH Zurich Zurich, Switzerland
Backward conformal prediction (BCP) fixes the size of a prediction set and estimates its miscoverage, but assumes calibration and test data are exchangeable. We extend BCP to covariate shift through importance weighting, preserving its guarantees when the density ratio is known. We show that BCP's tight miscoverage report depends only on the relative weight assigned to missed calibration examples, explaining why a better domain classifier can sometimes produce a worse estimate. Motivated by the fact that BCP depends only on nonconformity scores, we propose estimating density ratios on sorted score vectors rather than high-dimensional inputs. We prove that this approach recovers the same report asymptotically under an interpretable condition and characterize the resulting bias otherwise. Experiments with simulated and benchmark distribution shifts show that the proposed method improves BCP's miscoverage estimates, while score-based weighting consistently yields more accurate estimates than embedding-based weighting.
Conformal prediction (CP) offers distribution-free marginal coverage guarantees under an exchangeability assumption, but these guarantees can fail if the data distribution shifts. We analyze the use of pseudo-calibration as a tool to counter this performance loss under a bounded label-conditional covariate shift model. Using tools from domain adaptation, we derive a lower bound on target coverage in terms of the source-domain loss of the classifier and a Wasserstein measure of the shift. Using this result, we provide a method to design pseudo-calibrated sets that inflate the conformal threshold by a slack parameter to keep target coverage above a prescribed level. Finally, we propose a source-tuned pseudo-calibration algorithm that interpolates between hard pseudo-labels and randomized labels as a function of classifier uncertainty. Numerical experiments show that our bounds qualitatively track pseudo-calibration behavior and that the source-tuned scheme mitigates coverage degradation under distribution shift while maintaining nontrivial prediction set sizes.
Farbod Siahkali, Ashwin Verma, Vijay Gupta
Elmore Family School of Electrical and Computer Engineering, at Purdue University, West Lafayette, IN 47907 USA