Conformal prediction (CP) offers distribution-free marginal coverage guarantees under an exchangeability assumption, but these guarantees can fail if the data distribution shifts. We analyze the use of pseudo-calibration as a tool to counter this performance loss under a bounded label-conditional covariate shift model. Using tools from domain adaptation, we derive a lower bound on target coverage in terms of the source-domain loss of the classifier and a Wasserstein measure of the shift. Using this result, we provide a method to design pseudo-calibrated sets that inflate the conformal threshold by a slack parameter to keep target coverage above a prescribed level. Finally, we propose a source-tuned pseudo-calibration algorithm that interpolates between hard pseudo-labels and randomized labels as a function of classifier uncertainty. Numerical experiments show that our bounds qualitatively track pseudo-calibration behavior and that the source-tuned scheme mitigates coverage degradation under distribution shift while maintaining nontrivial prediction set sizes.
Figures & tables
Dataset
σ
Source Cal
Hard Pseudo Cal
Source-tuned Cal
ECP
WQLCP
Target Cal
Cov (%)
ESS
Cov (%)
ESS
Cov (%)
ESS
Cov (%)
ESS
Cov (%)
ESS
Cov (%)
ESS
MNIST
0.7
72.12
0.73
78.36
0.80
90.14
0.97
86.37
0.90
77.25
0.79
79.76
0.81
1.6
37.21
0.45
51.34
0.80
75.72
2.65
53.67
0.90
51.86
0.82
80.39
3.18
2.0
23.92
0.41
33.41
0.80
52.54
2.27
35.20
0.89
34.87
0.88
80.28
5.52
CIFAR-10
0.3
75.93
0.91
70.02
0.79
90.08
1.46
70.52
0.80
70.12
0.80
80.18
1.02
0.9
51.53
0.86
49.46
0.80
90.88
3.84
51.14
0.84
51.10
0.84
79.83
2.34
TABLE I : Coverage and ESS versus representative shift levels σ at nominal 1−α=0.8 , averaged over 5 runs. Boldface marks the least conservative non-oracle method meeting the nominal coverage target, or the closest method to 1−α if none meets it.
Fig. 1 : Coverage on Qσ vs. shift σ for various methods. Dashed curves show the coverage bounds from Theorem 1 . Shaded bands indicate one std across five independent runs.
Fig. 2 : Coverage (solid, left axis) and ESS (dashed, right axis) on Qσ vs. shift for various methods: hard pseudo-calibration qQ~σ,α , τ -adjusted qQ~σ,α+τ(σ) , and oracle qQσ,α . The black curve is the hinge-loss lower bound.
Fig. 3 : Coverage difference of hard pseudo-calibration and source calibration. Positive values indicate that pseudo-calibration has smaller coverage loss.
Fig. 4 : Empirical coverage on Qσ versus the nominal miscoverage level α for representative shifted settings. The dashed line denotes the nominal target 1−α . Across the tested α values, source-tuned pseudo-calibration remains less prone to undercoverage than hard pseudo-calibration, but is often conservative for smaller α .
Fig. 5 : Expected set size (ESS) on Qσ versus the nominal miscoverage level α for representative shifted settings. The improved coverage of source-tuned pseudo-calibration is accompanied by larger ESS, especially for smaller α , illustrating the same coverage–set-size tradeoff seen in the main text.
Fig. 6 : Empirical coverage versus corruption severity on public corruption benchmarks at α=0.2 . Source-tuned pseudo-calibration remains less prone to undercoverage than hard pseudo-calibration across the tested severities, while staying closest to oracle calibration.
Fig. 7 : Expected set size versus corruption severity on public corruption benchmarks at α=0.2 . The improved coverage of source-tuned pseudo-calibration is accompanied by larger expected set size, especially on CIFAR-10-C at higher severity.