Machine learning predictors are rarely deployed on data that match the data they were calibrated on. Conformal prediction and its risk-control generalization promise distribution-free guarantees at deployment, but only under exchangeability or a covariate shift whose likelihood ratio is known exactly -- and the estimated-ratio case, the one that occurs in practice, is explicitly open, out of reach of the Bayesian construction available under exchangeability. We reach it from a different starting point: pushing a Dirichlet process prior on the calibration distribution through the likelihood ratio yields the exact Bayesian posterior over the deployed risk. That posterior is exact given the weight function, which at deployment is itself estimated, so we further prove finite-sample upper and lower bounds on the risk the selected threshold actually incurs, in terms of the weight-estimation error and the conditional shift. On synthetic and real covariate shifts, in regression and classification, the realized failure rate then stays close to target where shift-blind selection fails, and what deployment receives is a posterior over the risk of every threshold rather than a single number.
Figures & tables
Figure 1: WBCP overview. (a) Calibration vs. test covariate distribution, with the likelihood ratio wi=dP~X/dPX(Xi) shading the calibration points. (b) The core step: the same Bayesian-bootstrap masses, reweighted by wi , give the exact posterior under shift (Proposition 1 )—no Dirichlet is assumed. (c–d) Existing approaches give either a posterior or shift-robustness, not both. (e–f) WBCP gives both: a posterior over the deployed risk, hence a threshold selected at a target failure rate, whose spread σpost tracks neff .
Figure 2: The WBCP pipeline , drawn for the miscoverage loss. (a) Covariate shift enters through the likelihood ratio wi . (b) The law of the masses, Eq. ( 3 ), which depends on the weights alone: median mass at each calibration score with its central 50% and 90% credible bands. (c) Sampling masses from that law gives one risk curve Lw+(λ) per draw, each crossing the target α at that draw’s threshold λ(m) (ticks on the α line). (d) The crossings over the M draws—for this single calibration set—form the threshold posterior: a β fraction of its mass lies at or below λHPDβ , and the gray tail 1−β is the tolerated failure credibility. The trailing test atom of Eq. ( 6 ) is omitted for clarity.
Table 4: Fashion-MNIST, entropy tilt toward hard inputs, estimated weights ( 2 tilt strengths ×200 splits). Mean size is the mean number of classes in the deployed prediction set (of 10 ).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Distribution of each rule’s selected threshold across the 10,000 trials of the synthetic regression experiment ( γ=1 , estimated weights; main text Table 1 ). The population risk under the tilted test law is a known decreasing function of λ , so the deployed risk exceeds α exactly when the selection falls below λ∗(α) (shaded region); the annotated frequency is the relative frequency reported in the table. λHPD0.95 denotes the HPD selection under each Bayesian arm’s own posterior—equal-weight in (a), weighted in (d). The blind rules select almost entirely inside the violation region; W-CRC straddles λ∗(α) , violating in individual trials while its mean risk sits at the target; WBCP places all but 5.0% of its selections on the safe side, at visibly larger thresholds—the price of certification.
Figure 4: Validity and width against shift strength on the synthetic regression benchmark (estimated weights, 10,000 trials per γ ). (a) WBCP’s violation rate stays at the nominal 5% across the entire sweep while the blind rules climb toward 100% ; W-CRC’s expectation semantics hold its mean at target with roughly 40% of individual trials violating. (b) The shortest valid length 2λ∗(α,γ) —below it, the population risk exceeds α for any rule—rises with the shift. The blind rules’ lengths stay flat and fall below it near γ≈0.5 , exactly where their violation rates take off. W-CRC’s mean length exceeds the valid length, yet its selection distribution straddles λ∗(α) trial by trial (Figure 3 ): mean control pins the center of the distribution at the boundary, not its quantiles. WBCP shifts the whole distribution above the boundary, which is both why it violates in only 5% of trials and why its mean length is largest. At γ=0 the two Bayesian arms coincide.
Weighted split conformal prediction reweights calibration scores by the likelihood ratio between the test and training covariate distributions and guarantees marginal coverage under covariate shift. We study its coverage conditional on the calibration data. An elementary argument, based on a single concentration inequality at a fixed population quantile, gives explicit training-conditional bounds without unspecified constants, and shows that the relevant scale is not the supremum of the likelihood ratio but a variance proxy built from the chi-squared divergence of the shift and from the average of the ratio over the part of the test population, of probability equal to the miscoverage level, where it is largest. A two-point lower bound shows that the root-m rate and the chi-squared contribution are intrinsic to the shift. Run at an explicitly inflated level, the weighted quantile becomes a deterministic PAC prediction set. We compare it with randomized rejection sampling and with importance-weighted learn-then-test and, through a certified choice of a clipping level for the likelihood ratio, map the regime in which each gives the narrower valid set. The analysis extends to estimated likelihood ratios and to tail functionals estimated from an unlabeled source sample, which yields a fully finite-sample certificate.
Backward conformal prediction (BCP) fixes the size of a prediction set and estimates its miscoverage, but assumes calibration and test data are exchangeable. We extend BCP to covariate shift through importance weighting, preserving its guarantees when the density ratio is known. We show that BCP's tight miscoverage report depends only on the relative weight assigned to missed calibration examples, explaining why a better domain classifier can sometimes produce a worse estimate. Motivated by the fact that BCP depends only on nonconformity scores, we propose estimating density ratios on sorted score vectors rather than high-dimensional inputs. We prove that this approach recovers the same report asymptotically under an interpretable condition and characterize the resulting bias otherwise. Experiments with simulated and benchmark distribution shifts show that the proposed method improves BCP's miscoverage estimates, while score-based weighting consistently yields more accurate estimates than embedding-based weighting.
Conformal Bayes combines Bayesian posterior predictives with conformal calibration to produce prediction sets that are both statistically valid and geometrically efficient. We study conformal Bayes under label shift from a unified perspective, identifying two complementary approaches that restore nominal target-domain coverage through importance-weighted conformal calibration but operate through independent mechanisms. Post-hoc calibration tilts the posterior predictive toward the target domain and corrects the conformal threshold via an importance-weighted quantile, leaving the parameter posterior unchanged. In-training adaptation tilts the parameter posterior itself to the target domain, producing a corrected predictive whose highest predictive density region serves as the highest predictive density (HPD)-based prediction set under the fitted target predictive; efficiency is model-dependent and does not imply finite-sample conditional optimality. Two controlled experiments isolate the regime-dependence of each strategy: in the low-dimensional, well-estimated regime StrategyA produces the narrowest valid intervals, while in the high-dimensional, underdetermined regime StrategyB achieves up to 43% width reduction at unchanged coverage, under the stated source-sampling and label-shift assumptions.
Seungjin Choi
1CROID Research and aSSIST University, Seoul, Korea.