Backward conformal prediction (BCP) fixes the size of a prediction set and estimates its miscoverage, but assumes calibration and test data are exchangeable. We extend BCP to covariate shift through importance weighting, preserving its guarantees when the density ratio is known. We show that BCP's tight miscoverage report depends only on the relative weight assigned to missed calibration examples, explaining why a better domain classifier can sometimes produce a worse estimate. Motivated by the fact that BCP depends only on nonconformity scores, we propose estimating density ratios on sorted score vectors rather than high-dimensional inputs. We prove that this approach recovers the same report asymptotically under an interpretable condition and characterize the resulting bias otherwise. Experiments with simulated and benchmark distribution shifts show that the proposed method improves BCP's miscoverage estimates, while score-based weighting consistently yields more accurate estimates than embedding-based weighting.
Figures & tables
weights
hardness β=1
hardness β=2
easy β=1
easy β=2
benign β=1
benign β=2
true miscoverage
0.274
0.406
0.023
0.010
0.072
0.036
identity report
true weights
+0.169
+0.343
+0.014
+0.014
+0.012
+0.023
features
+0.013
+0.106
+0.028
+0.027
+0.009
+0.025
features + Platt
− 0.064
+0.046
+0.059
+0.042
+0.025
+0.034
sorted scores
+0.267
+0.340
+0.014
+0.017
+0.016
+0.021
Table 1: Dry Bean, T=1 : exact miscoverage of the size- 1 set under the tilted pool and gap (report − miscoverage) of the identity and tight reports, by the weights used, and of ATC-NE. “Features”: gradient boosting on the 16 features; “+ Platt”: the same classifier calibrated by Platt scaling on grouped out-of-fold predictions; “sorted scores”: gradient boosting on G(x) . Means over 80 draws; the largest standard error in the table is 0.012. T=2 in Appendix D .
Figure 1: CIFAR-10-C at T=2 : the tight report against the true miscoverage of the size- T set, one vertical segment per setting, both axes logarithmic. Hollow blue markers: mean over the 60 draws with embedding weights; filled red markers: with score weights. Dotted line: no weights, the miss rate on the clean calibration images, the same in every setting. Squares: walls, cells with nearly separable clean and corrupted embeddings (AUC at least 0.95 ). Points below the dashed diagonal understate the miscoverage. T=1 and T=3 are in Appendix D .
true
tight report
identity report
model
T
miscoverage
none
emb.
score
ATC-NE
valid
EfficientNet-B0
1
no wall
23.6%
14.69
8.15
4.74
3.12
9/12
walls
77.0%
68.13
52.34
42.98
29.59
0/3
2
no wall
12.2%
9.20
6.65
4.78
3.49
10/12
walls
63.8%
60.80
53.65
48.67
41.87
0/3
3
no wall
7.0%
5.67
4.28
3.48
2.57
11/12
Table 2: CIFAR-10-C by model, T and overlap (walls: held-out embedding AUC at least 0.95 ); each row averages over the settings of its group. Tight report with no, embedding and score weights, and ATC-NE: mean absolute gap ∣report−truth∣ in percentage points over the settings and their 60 draws; without weights the tight report is the corrected estimator of ST-BCP. Identity report: settings in which it is valid, with score weights. Appendix D gives the signed gaps by group, and per-setting gaps of the weighted reports and ATC-NE with standard errors.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
tight report
model
T
none
emb.
score
ATC-NE
EfficientNet-B0
1
no wall
− 14.64
− 7.86
− 3.05
− 2.59
walls
− 68.13
− 52.33
− 42.98
− 29.59
2
no wall
− 9.15
− 6.36
− 3.43
− 2.79
walls
− 60.80
− 53.65
− 48.67
− 41.87
3
no wall
− 5.64
− 4.03
− 2.27
− 1.88
Appendix
Table 3: CIFAR-10-C by model, T and overlap, as Table 2 : mean gap (report − truth) in percentage points over the settings of a group and their 60 draws, for the tight report with no, embedding and score weights and for ATC-NE.
settings without a wall (embedding AUC <0.95 )
walls (AUC ≥0.95 )
model
T
#cells
identity valid
WSCP
#cells
identity valid
WSCP
score
emb.
none
valid
∣C∣>T
score
emb.
none
valid
1
12
9
2
0
7
2%
3
0
0
0
0
EfficientNet-B0
2
12
10
7
3
1
25%
3
0
0
0
0
3
12
11
9
6
2
30%
3
0
0
0
0
1
9
0
0
0
8
1%
6
0
2
0
1
Appendix
Table 4: CIFAR-10-C: number of cells in which each method is valid (mean over 60 draws of the report minus the true miscoverage is ≥0 ; for WSCP, mean miscoverage ≤ its level α ), for the identity report with score weights (“score”), embedding weights (“emb.”) and no weights (“none”). WSCP uses score weights; ∣C∣>T is the mean share of test images whose WSCP set has more than T labels.
Figure 2: CIFAR-10-C at T=1 (top) and T=3 (bottom), with the layout of Figure 1 .
weights
hardness β=1
hardness β=2
easy β=1
easy β=2
benign β=1
benign β=2
true miscoverage
0.064
0.144
0.005
0.003
0.012
0.007
identity report
true weights
+0.170
+0.316
+0.025
+0.018
+0.043
+0.040
features
+0.094
+0.152
+0.035
+0.028
+0.045
+0.044
features + Platt
+0.055
+0.110
+0.051
+0.036
+0.050
+0.046
sorted scores
+0.233
+0.287
+0.026
+0.021
+0.046
+0.037
Appendix
Table 5: Dry Bean, T=2 , same layout as Table 1 . Largest standard error 0.012.
Table 12: Non-degeneracy on CIFAR-10-C. Score and embedding weights: number of the 900 draws of a row ( 15 cells, 60 draws each) with a degenerate fold, and the largest change of a setting’s mean tight report relative to the weighted miss rate, in percentage points. Last two columns: the limit of the tight report when the calibration data come from the corrupted images themselves, without weights, which is also its limit under exact weights; number of the 15 cells in which this limit is below the miscoverage, and the largest such shortfall in percentage points.
no walls
walls
score weights
domain classifier
embedding
score
embedding
score
closer in
Gradient boosting (main text)
4.20
2.93
20.85
17.84
12 of 12 groups
Logistic regression
5.40
3.37
23.13
17.83
12 of 12 groups
Multilayer perceptron
4.77
3.45
22.14
17.98
12 of 12 groups
Gradient boosting, early stopping
4.27
3.53
20.94
18.89
12 of 12 groups
Appendix
Table 13: CIFAR-10-C: mean absolute gap of the tight report, in percentage points, with embedding and with score weights from four domain classifiers fitted on the same draws. Means over the settings without walls or at the walls and their 60 draws. Last column: groups of Table 2 in which score weights have the smaller mean absolute gap.
CIFAR-10-C, tight report (percentage points)
Dry Bean, hardness, T=1
score weights, no walls
embedding, no walls
walls, mean gap
score
feature weights
score weights
clip of q^
mean gap
SD
∣gap∣
mean gap
∣gap∣
score
emb.
closer in
β=1
β=2
β=1
β=2
[0.01,0.99] (main text)
− 1.72
2.20
2.93
− 3.79
4.20
− 17.6
− 19.6
12 of 12
− 0.081
− 0.101
+0.042
− 0.006
[0.001,0.999]
− 0.93
3.60
3.18
− 3.75
4.23
− 16.9
− 18.6
12 of 12
− 0.081
− 0.104
+0.085
+0.032
[10−6,1−10−6]
− 0.85
3.89
3.26
− 3.75
4.23
− 16.9
− 18.0
12 of 12
− 0.081
− 0.104
+0.093
+0.021
Appendix
Table 14: Sensitivity to the clip of the domain classifier’s probability q^ . The classifiers are fitted once and only the clip changes. CIFAR-10-C: tight report over the settings without walls or at the walls and their 60 draws; SD is the standard deviation of the gap over the draws of a setting, averaged over the settings; “closer in” counts the groups of Table 2 in which score weights have the smaller mean absolute gap. Dry Bean: gap of the tight report as in Table 1 , means over 80 draws.
level of the main text
level of the previous draw
T
settings
valid
excess
valid
excess
CIFAR-10-C, EfficientNet-B0
1
12
7
+2.24
7
+2.16
2
12
1
+5.29
1
+5.35
3
12
2
+3.68
1
+3.71
CIFAR-10-C, WRN-40-2 (AugMix)
1
9
8
− 0.54
8
− 0.37
2
9
4
+0.30
3
+0.35
Appendix
Table 15: WSCP with its level chosen as in the main text and with the level chosen in the previous draw. CIFAR-10-C: score weights, settings without walls. Dry Bean: feature weights. Valid: settings in which the mean miscoverage is at most the mean level. Excess: miscoverage minus level in percentage points, averaged over the settings of a row and their draws ( 60 on CIFAR-10-C, 80 on Dry Bean; the first draw of a setting has no previous level).
tight report
identity
gradient boosting
score weights from
binned
report
model
T
miscoverage
no weights
embedding
score
LR
MLP
GB-ES
prediction
ATC-NE
(score)
EfficientNet-B0
1
18.05%
− 8.99
− 5.02
− 0.68
− 3.40
− 3.35
− 3.73
− 3.55
− 0.66
+0.90
2
7.00%
− 3.85
− 2.03
+0.77
− 1.57
− 1.38
− 1.26
− 1.53
+1.05
+4.21
3
4.00%
− 2.59
− 1.65
+0.02
− 1.52
− 1.49
− 1.25
− 1.46
+0.56
+4.78
WRN-40-2 (AugMix)
1
11.30%
− 6.55
− 3.26
− 0.57
− 2.21
− 2.71
− 2.85
− 2.55
− 1.21
− 1.61
Appendix
Table 16: CIFAR-10.1 as the test distribution, with calibration on clean CIFAR-10 test images. Miscoverage: miss rate of the size- T set over the 2,000 CIFAR-10.1 images. Other columns: gap (report − miscoverage) in percentage points, means over 60 draws. The tight report is given without weights, with embedding and score weights from the gradient boosting of the main text, and with score weights from logistic regression (LR), the perceptron (MLP) and early-stopped gradient boosting (GB-ES). The binned prediction uses all images and has no draws. The identity report uses the score weights of the main text. The largest standard error over draws is 0.64.
identity gap
covariance
model
T
truth
clipped
unclipped
tight gap
E[α~]
E[1{Y∈/C^}/α~]
clipped
unclipped
EfficientNet-B0
1
8.91%
− 1.29
+1.05
− 0.03
7.66%
0.67
+3.80
+3.19
2
3.05%
+1.59
+3.02
+0.03
4.63%
0.43
+1.06
− 0.14
3
1.40%
+2.41
+3.67
− 0.01
3.81%
0.26
+0.41
+0.52
WRN-40-2 (AugMix)
1
4.93%
− 1.73
− 1.73
− 0.09
3.21%
0.71
+2.65
+2.65
2
1.24%
+0.95
+0.95
0.00
2.19%
0.30
+0.59
+0.59
Appendix
Table 17: No shift: clean CIFAR-10, no weights, n=1,000 , means over 60 draws. Truth: miscoverage of the size- T set with the clipped score. Gaps (report − miscoverage under the same score) and covariances in percentage points.
identity report
tight report
valid settings
mean gap
mean ∣gap∣
T
settings
floor
no floor
floor
no floor
floor
no floor
no walls
1
12
9
11
+2.55
+4.51
4.74
4.74
2
12
10
11
+2.71
+3.91
4.78
4.78
3
12
11
11
+4.29
+5.27
3.48
3.49
walls
1
3
0
0
− 22.06
− 22.01
42.98
42.98
Appendix
Table 18: EfficientNet-B0 on CIFAR-10-C with the floor 10−12 on the probabilities in the score and without it, with the same score weights. Each report is compared with the miscoverage under the same score. Gaps in percentage points, averaged over the settings of a row and their 60 draws.
shift
weights
ECE
log-loss
rank corr.
MSE
identity gap ( T=1 )
tight gap ( T=1 )
tight gap ( T=2 )
hardness β=1
features
0.054
0.576
0.68
3.17
+0.013
− 0.081
− 0.027
hardness β=1
+ Platt (1 model)
0.041
0.565
0.68
2.27
− 0.064
− 0.121
− 0.037
hardness β=1
+ Platt (5 models)
0.046
0.561
0.71
2.19
− 0.065
− 0.121
− 0.037
hardness β=1
+ isotonic (5 models)
0.038
0.561
0.70
2.81
− 0.030
− 0.104
− 0.031
hardness β=1
features, early stopping
0.043
0.560
0.73
2.20
− 0.042
− 0.110
− 0.036
hardness β=1
logistic regression
0.061
0.600
0.58
2.82
− 0.127
− 0.165
− 0.046
Appendix
Table 19: Dry Bean: calibrated domain classifiers. Means over 80 draws; the largest standard error of a gap is 0.010.
Weighted split conformal prediction reweights calibration scores by the likelihood ratio between the test and training covariate distributions and guarantees marginal coverage under covariate shift. We study its coverage conditional on the calibration data. An elementary argument, based on a single concentration inequality at a fixed population quantile, gives explicit training-conditional bounds without unspecified constants, and shows that the relevant scale is not the supremum of the likelihood ratio but a variance proxy built from the chi-squared divergence of the shift and from the average of the ratio over the part of the test population, of probability equal to the miscoverage level, where it is largest. A two-point lower bound shows that the root-m rate and the chi-squared contribution are intrinsic to the shift. Run at an explicitly inflated level, the weighted quantile becomes a deterministic PAC prediction set. We compare it with randomized rejection sampling and with importance-weighted learn-then-test and, through a certified choice of a clipping level for the likelihood ratio, map the regime in which each gives the narrower valid set. The analysis extends to estimated likelihood ratios and to tail functionals estimated from an unlabeled source sample, which yields a fully finite-sample certificate.
Conformal prediction (CP) offers distribution-free marginal coverage guarantees under an exchangeability assumption, but these guarantees can fail if the data distribution shifts. We analyze the use of pseudo-calibration as a tool to counter this performance loss under a bounded label-conditional covariate shift model. Using tools from domain adaptation, we derive a lower bound on target coverage in terms of the source-domain loss of the classifier and a Wasserstein measure of the shift. Using this result, we provide a method to design pseudo-calibrated sets that inflate the conformal threshold by a slack parameter to keep target coverage above a prescribed level. Finally, we propose a source-tuned pseudo-calibration algorithm that interpolates between hard pseudo-labels and randomized labels as a function of classifier uncertainty. Numerical experiments show that our bounds qualitatively track pseudo-calibration behavior and that the source-tuned scheme mitigates coverage degradation under distribution shift while maintaining nontrivial prediction set sizes.
Farbod Siahkali, Ashwin Verma, Vijay Gupta
Elmore Family School of Electrical and Computer Engineering, at Purdue University, West Lafayette, IN 47907 USA
Conformal prediction provides distribution-free uncertainty quantification under exchangeability. However, this assumption is violated by label shift, where the marginal distribution of labels changes while the conditional distribution of inputs given labels remains stable. Under such shifts, standard conformal procedures no longer maintain their intended coverage behavior. Existing approaches address this via importance weighting. They pair the reweighting with residual-based nonconformity scores that ignore predictive uncertainty. The resulting intervals have uniform width. Bayesian conformal methods produce adaptive intervals by leveraging predictive distributions. They evaluate conformity under the source predictive, which is misaligned with the target domain under label shift. We propose the Label-Shift-Adjusted Bayesian Score (LSA score), a nonconformity score derived from a posterior predictive tilting identity. This identity shows that the target predictive is an importance-weighted transformation of the source predictive. We use it to derive a direct correction to the Bayesian score. We evaluate the method on molecular property prediction under controlled label shift. The LSA score consistently yields shorter intervals than residual-based and source-based Bayesian scores. Coverage in the target domain remains comparable. Under stronger shift, all methods incur some coverage loss due to pseudo-label-based density-ratio estimation. The LSA score is defined for any source predictive with a tractable log-density. We instantiate it with Bayesian Ridge Regression, where the correction admits a closed form.
Hyeonsu Lee, Juyeon Kim, Erkhembayar Jadamba +2
MOGAM Institute for Biomedical Research, Korea · CROID Research and aSSIST University, Korea