Machine learning predictors are rarely deployed on data that match the data they were calibrated on. Conformal prediction and its risk-control generalization promise distribution-free guarantees at deployment, but only under exchangeability or a covariate shift whose likelihood ratio is known exactly -- and the estimated-ratio case, the one that occurs in practice, is explicitly open, out of reach of the Bayesian construction available under exchangeability. We reach it from a different starting point: pushing a Dirichlet process prior on the calibration distribution through the likelihood ratio yields the exact Bayesian posterior over the deployed risk. That posterior is exact given the weight function, which at deployment is itself estimated, so we further prove finite-sample upper and lower bounds on the risk the selected threshold actually incurs, in terms of the weight-estimation error and the conditional shift. On synthetic and real covariate shifts, in regression and classification, the realized failure rate then stays close to target where shift-blind selection fails, and what deployment receives is a posterior over the risk of every threshold rather than a single number.
Figures & tables
Figure 1: WBCP overview. (a) Calibration vs. test covariate distribution, with the likelihood ratio wi=dP~X/dPX(Xi) shading the calibration points. (b) The core step: the same Bayesian-bootstrap masses, reweighted by wi , give the exact posterior under shift (Proposition 1 )—no Dirichlet is assumed. (c–d) Existing approaches give either a posterior or shift-robustness, not both. (e–f) WBCP gives both: a posterior over the deployed risk, hence a threshold selected at a target failure rate, whose spread σpost tracks neff .
Figure 2: The WBCP pipeline , drawn for the miscoverage loss. (a) Covariate shift enters through the likelihood ratio wi . (b) The law of the masses, Eq. ( 3 ), which depends on the weights alone: median mass at each calibration score with its central 50% and 90% credible bands. (c) Sampling masses from that law gives one risk curve Lw+(λ) per draw, each crossing the target α at that draw’s threshold λ(m) (ticks on the α line). (d) The crossings over the M draws—for this single calibration set—form the threshold posterior: a β fraction of its mass lies at or below λHPDβ , and the gray tail 1−β is the tolerated failure credibility. The trailing test atom of Eq. ( 6 ) is omitted for clarity.
Table 4: Fashion-MNIST, entropy tilt toward hard inputs, estimated weights ( 2 tilt strengths ×200 splits). Mean size is the mean number of classes in the deployed prediction set (of 10 ).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Distribution of each rule’s selected threshold across the 10,000 trials of the synthetic regression experiment ( γ=1 , estimated weights; main text Table 1 ). The population risk under the tilted test law is a known decreasing function of λ , so the deployed risk exceeds α exactly when the selection falls below λ∗(α) (shaded region); the annotated frequency is the relative frequency reported in the table. λHPD0.95 denotes the HPD selection under each Bayesian arm’s own posterior—equal-weight in (a), weighted in (d). The blind rules select almost entirely inside the violation region; W-CRC straddles λ∗(α) , violating in individual trials while its mean risk sits at the target; WBCP places all but 5.0% of its selections on the safe side, at visibly larger thresholds—the price of certification.
Figure 4: Validity and width against shift strength on the synthetic regression benchmark (estimated weights, 10,000 trials per γ ). (a) WBCP’s violation rate stays at the nominal 5% across the entire sweep while the blind rules climb toward 100% ; W-CRC’s expectation semantics hold its mean at target with roughly 40% of individual trials violating. (b) The shortest valid length 2λ∗(α,γ) —below it, the population risk exceeds α for any rule—rises with the shift. The blind rules’ lengths stay flat and fall below it near γ≈0.5 , exactly where their violation rates take off. W-CRC’s mean length exceeds the valid length, yet its selection distribution straddles λ∗(α) trial by trial (Figure 3 ): mean control pins the center of the distribution at the boundary, not its quantiles. WBCP shifts the whole distribution above the boundary, which is both why it violates in only 5% of trials and why its mean length is largest. At γ=0 the two Bayesian arms coincide.