Conformal Bayes under Continuous Label Shift: Sensitivity Analysis and the Limits of Exact Validity

Seungjin Choi

stat.MLAug 24, 2026

Conformal Bayes under Continuous Label Shift: Sensitivity Analysis and the Limits of Exact Validity

Authors: Seungjin Choi

Organizations: CROID Research and aSSIST University, Seoul, Korea

Abstract

Conformal Bayes combines Bayesian posterior predictive scores with conformal calibration, but under continuous label shift both the score and calibration weight depend on the unknown response-marginal density ratio. Existing methods typically estimate one shift parameter from pseudo-labels or predictive samples and plug it into calibration. We instead propose Joint Tilt-Sensitivity Conformal Bayes (JTS-CB), which performs sensitivity analysis over a prespecified set of plausible tilts; its split-conformal realization is JTS-SCB. Each tilt jointly determines the Bayesian conformal score and conformal importance weight. JTS-SCB forms a bounded sensitivity envelope over candidate tilts, but its calibration-only construction does not inherit the exact finite-sample weighted-conformal guarantee. We therefore study a separate candidate-weighted exact counterpart and show that its usefulness depends sharply on tail behavior. For scalar linear exponential tilts, any nonzero candidate tilt makes the exact set unbounded. More generally, tail-growing density ratios produce the same pathology, whereas quadratic tilts with a negative coefficient on $y^2$ have vanishing tail weights and admit bounded exact inference on the original target. Ratio clipping provides a complementary bounded exact construction for a surrogate target when tails grow. Experiments show that strong plug-in predictive sampling can match the oracle when the shift is well identified, while sensitivity analysis is most useful for richer, weakly identified, or systematically biased shift models, at the cost of wider prediction sets.

Figures & tables

Panel of Figure 1 — Figure 1: Illustration of exponential-family response-marginal tilting from a standard Gaussian source. (a) The source marginal is $p_{s}(y)=\mathcal{N}(0,1)$ . (b) The linear tilt $w(y)\propto\exp(\beta y)$ with $\beta=1$ shifts the mean while preserving the variance. (c) The quadratic tilt $w(y)\propto\exp(\beta_{1}y+\beta_{2}y^{2})$ with $(\beta_{1},\beta_{2})=(0.5,0.15)$ changes both location and spread. The linear model is nested at $\beta_{2}=0$ .

Panel of Figure 2 — Figure 2: Schematic view of practical JTS-SCB. A candidate tilt $\bm{\beta}$ jointly determines the Bayesian conformal score and the conformal importance weight. The inner step computes the weighted empirical quantile (equivalently, the smallest minimizer of a weighted quantile-regression problem) and returns the calibration threshold for that fixed tilt. The sensitivity aggregation then takes the largest threshold over the uncertainty set. JTS-SCB therefore provides a sensitivity-analysis alternative to estimating one target tilt from pseudo-labels or predictive samples.

Panel of Figure 3 — Figure 3: Two complementary ways to handle continuous label shift. Pseudo-label SCB estimates a single target tilt from pseudo-labels and calibrates once; JTS-SCB instead computes one sensitivity threshold over an analyst-specified tilt uncertainty set and uses it to form the practical prediction set. The right panel separates this method from two theoretical counterparts used to analyze validity: the exact candidate-weighted construction and its clipped surrogate version.

\widehat{\bm{\beta}}^{\,*}\in\operatorname*{arg\,max}_{\bm{\beta}\in\mathcal{B}}\widehat{q}_{\bm{\beta}},\qquad\widehat{\theta}_{\rm JTS}=\widehat{q}_{\widehat{\bm{\beta}}^{\,*}}.

Algorithm 1 JTS-SCB: coupled tilt-sensitivity weighted-quantile calibration

Candidate tilt set	Coverage	Grid width	Tail diagnostic	True width
Linear truth $\beta^{\star}=0.9$ : candidate interval $[0,B]$
$B=0.00$	.865	1.85	–	finite
$B=0.10$	.866	1.90	$y_{W}\approx 35.1$	$\infty$
$B=0.30$	.902	20.09	$y_{W}\approx 11.9$	$\infty$
$B=0.60$	.919	25.83	$y_{W}\approx 6.2$	$\infty$
$B=0.90$	.952	27.70	$y_{W}\approx 4.5$	$\infty$

Table 1: Exact candidate-weighted validity benchmarks, high-SNR regime, nominal

0.90

, averaged over

60

trials. Linear rows use truth

\beta^{\star}=0.9

and candidate interval

[0,B]

; their numerical width is measured on

[-20,30]

, while Proposition 4 gives the true infinite width for every

B>0

. The final row uses quadratic truth

\bm{\beta}^{\star}=(0.30,-0.10)

and an

11\times 11

grid over

\mathcal{B}_{Q}^{-}=[0,0.30]\times[-0.15,-0.05]

; numerical width is measured on

[-8,8]

, with no accepted points at either grid boundary in any trial, and Corollary 3 certifies finite true width.

$M$	$2$	$4$	$M^{\star}\!\approx\!5.8$	$8$	$12$	$20$	$\infty$
width	2.36	2.26	2.30	2.28	2.33	2.29	38.5
cov (true)	.961	.941	.945	.951	.945	.951	.942

Table 2: Clipping the density ratio to

[1/M,M]

\beta^{\star}=1.2

, high-SNR,

\beta^{\star}

budget. Proposition 5 guarantees boundedness for

M\leq M^{\star}=\sqrt{\alpha n/(1-\alpha)}\approx 5.8

. The larger tested values

M=8,12,20

also show no tail divergence in the numerical evaluation, illustrating that the sufficient worst-case threshold is conservative in this experiment; the unclipped object (

M=\infty

) diverges (width grid-capped). “cov” is under the true exponential target.

25

trials.

Method	$\beta^{\star}\!=\!0.3$	$0.6$	$0.9$	$1.2$	$1.5$
Coverage
Source-CB	.899	.879	.864	.830	.798
Oracle-WT	.900	.895	.900	.886	.887
Robust-IW	.906	.894	.902	.894	.878
PL-point	.901	.894	.900	.888	.897
Exact counterpart ‡	.915	.924	.948	.957	.979

Table 3: Coverage and mean width vs. shift, high-SNR regime (

v_{s}^{2}=1.3

R^{2}=0.77

), nominal

0.90

, one-sided

1\times

budget aligned with the positive target tilt. Standard errors

\leq.01

. † uncertified diagnostic. ‡ the exact union is unbounded for every column (Prop. 4 ); the width block therefore reports its true Lebesgue width

\infty

. Finite-window tail diagnostics are reported separately in Table 1 .

Method	$\beta^{\star}\!=\!0.3$	$0.6$	$0.9$	$1.2$	$1.5$
Source-CB	.897	.865	.807	.751	.662
Oracle-WT	.905	.903	.902	.893	.873
PL-point	.901	.883	.854	.825	.773
JTS $\mathcal{C}^{\rm prac}_{\rm JTS}$	.935	.948	.958	.965	.965
JTS $\mathcal{C}^{*}$	.913	.905	.892	.887	.822
$\hat{\beta}$ (PL)	.087	.177	.272	.358	.431

Table 4: Moderate-SNR regime (

v_{s}^{2}=1.0

R^{2}=0.30

), coverage, nominal

0.90

. Pseudo-label under-correction is severe here; JTS

\mathcal{C}^{\rm prac}_{\rm JTS}

stays conservative.

$\beta^{\star}$ ( $\hat{\beta}_{\rm PL\text{-}tilt}$ )	Oracle	PL-tilt	JTS $\mathcal{C}^{\rm prac}_{\rm JTS}$
$0.6$ ( $0.59$ )	.891 / 1.81	.891 / 1.82	.935 / 2.08
$0.9$ ( $0.94$ )	.894 / 1.84	.891 / 1.83	.946 / 2.24
$1.2$ ( $1.26$ )	.892 / 1.81	.887 / 1.82	.954 / 2.34
$1.5$ ( $1.58$ )	.885 / 1.80	.877 / 1.77	.961 / 2.49

Table 5: Strong pseudo-labels (PL-tilt) vs. JTS, high-SNR,

n_{\rm tg}=300

, nominal

0.90

, coverage/width. PL-tilt matches the oracle; JTS-SCB is wider at no coverage benefit.

Predictive model	$\hat{\beta}_{\rm PL}$	PL-tilt	JTS $\mathcal{C}^{\rm prac}_{\rm JTS}$
Well-specified	1.05	.886 / 1.82	.944 / 2.24
Over-regularized	1.03	.898 / 1.85	.951 / 2.28
Mean-biased ( $+0.5$ )	1.57	.835 / 2.48	.972 / 2.86

Table 6: Model misspecification,

\beta^{\star}=1.0

n_{\rm tg}=300

, coverage/width. A biased predictive mean breaks PL-tilt; JTS-SCB holds by widening.

Condition	PL-point	PL-boot	JTS $\mathcal{C}^{\rm prac}_{\rm JTS}$
Well-specified, $n_{\rm tg}=300$	.891 / 1.81	.909 / 1.92	.946 / 2.25
Well-specified, $n_{\rm tg}=10$	.880 / 1.78	.947 / 2.24	.951 / 2.33
Biased predictive $+0.5$	.832 / 2.47	.855 / 2.58	.972 / 2.86
Biased predictive $-0.5$	.841 / 2.41	.861 / 2.52	.921 / 2.91

Table 7: Uncertainty-propagated pseudo-labels (PL-boot) vs. point plug-in and JTS, coverage/width. PL-boot matches JTS-SCB at less width under variance unreliability (small

n_{\rm tg}

) but, like any variance-hedge, fails under systematic bias, where JTS-SCB’s zero-anchored band still covers.

Multiplier	$0.25\times$	$0.50\times$	$0.75\times$	$1.0\times$	$1.5\times$	$2.0\times$	$3.0\times$
$\mathcal{C}^{\rm prac}_{\rm JTS}$ coverage	.877	.908	.934	.957	.976	.988	.995
$\mathcal{C}^{\rm prac}_{\rm JTS}$ width	1.97	2.11	2.20	2.37	2.65	2.88	3.42
$\mathcal{C}^{*}$ coverage	.854	.877	.881	.900	.896	.903	.861
$\mathcal{C}^{*}$ width	1.88	1.92	1.92	1.99	2.09	2.13	2.30

Table 8: Budget sensitivity at

\beta^{\star}=1.2

, high-SNR regime. Multiplier is the

\beta

-bound divided by

\beta^{\star}

. The practical JTS-SCB sets are pathwise nested as the budget expands (Prop. 2 ), so their marginal coverage is nondecreasing;

\mathcal{C}^{*}

is an uncertified heuristic.

Method	Coverage	Width
Oracle-QT	.894	1.925
PL-point-Q	.880	1.881
SPS-Q	.890	1.914
TPS-Q	.894	1.945
JTS-Q	.933	2.206

Table 9: Quadratic exponential tilt under a correctly specified Gaussian predictive model,

(\beta_{1}^{\star},\beta_{2}^{\star})=(0.3,0.15)

n_{\rm tg}=300

, nominal coverage

0.90

30

trials.

$n_{\rm tg}$	Oracle-QT	PL-point-Q	SPS-Q	TPS-Q	JTS-Q
300	.907 / 3.439	.809 / 2.984	.902 / 3.256	.906 / 3.485	.976 / 4.300
30	.901 / 3.439	.784 / 2.944	.884 / 3.208	.863 / 3.367	.977 / 4.307
10	.904 / 3.439	.762 / 2.920	.848 / 3.168	.847 / 3.562	.979 / 4.308

Table 10: Quadratic tilt with bimodal residuals: pseudo-label hierarchy as target information decreases. Entries are coverage/mean width, nominal

0.90

30

trials.

$n_{\rm tg}$	mean $\widehat{\beta}_{1}$	mean $\widehat{\beta}_{2}$	log-weight RMSE
300	.269	.152	.169
100	.310	.137	.227
30	.324	.131	.343
10	.917	$-.049$	1.571

Table 11: Quadratic tilted-predictive-sampling diagnostics in the bimodal-residual experiment. Truth is

(\beta_{1}^{\star},\beta_{2}^{\star})=(0.3,0.15)

Predictive model	Oracle-QT	TPS-quadratic	JTS-Q
Well specified	.887 / 3.317	.885 / 3.305	.975 / 4.259
Mean bias $+0.5$	.893 / 4.464	.842 / 4.454	.977 / 5.230
Mean bias $-0.5$	.895 / 3.686	.882 / 4.106	.966 / 4.839

Table 12: Quadratic tilt under systematic predictive-mean bias, bimodal-residual model,

n_{\rm tg}=300

, nominal

0.90

30

trials. Entries are coverage/mean width.

Method	$n_{\rm tg}=30$	$n_{\rm tg}=300$ , mean bias $+0.5$
Oracle-WT	.899 / 3.250	.902 / 4.144
PL-point-Q	.791 / 2.848	.822 / 3.328
SPS-Q	.891 / 3.168	.886 / 3.931
TPS-L	.864 / 3.054	.797 / 3.691
TPS-Q	.865 / 3.234	.823 / 4.205
JTS-Q	.975 / 4.221	.976 / 5.200

Table 13: One encompassing quadratic family when the true target tilt is randomly linear or quadratic. Entries are pooled coverage/mean width across the linear- and quadratic-truth trials, nominal

0.90

, from the primary

30

-trial mixed-truth run. The same JTS-Q box and the same universal TPS-Q procedure are used on every trial.

Method	Coverage	Width
Oracle-WT	.902	3.292
Beta-specific practical union	.967	4.061
Common-threshold JTS-Q envelope	.974	4.234

Table 14: Decomposition of JTS-Q conservatism in the mixed linear-or-quadratic experiment with

n_{\rm tg}=30

. Entries are averages over an independent

40

-trial diagnostic run; Table 13 uses the primary

30

-trial run, so repeated Oracle-WT and JTS-Q summaries differ slightly by Monte Carlo variation. Both JTS rows are practical calibration-only constructions and carry no exact finite-sample certificate.

Conditional-noise scale $c$	$1.00$	$1.25$	$1.50$	$2.00$
$\mathcal{C}^{\rm prac}_{\rm JTS}$ coverage	.947	.857	.745	.516
Symmetric bimodal $\delta$	$0.00$	$0.50$	$1.00$	$1.50$
$\mathcal{C}^{\rm prac}_{\rm JTS}$ coverage	.948	.938	.925	.898
Asymmetric bimodal $\delta$	$0.50$	$1.00$	$1.50$	$2.00$
$\mathcal{C}^{\rm prac}_{\rm JTS}$ coverage	.948	.945	.932	.904

Table 15: Shift-model stress tests, high-SNR regime, nominal

0.90

. In the conditional-noise block,

c=1

is the correctly specified linear-shift baseline with

\beta^{\star}=1.0

, while

c>1

breaks label shift. The bimodal blocks are separate response-marginal constructions;

\delta=0

in the symmetric block is a no-shift baseline. Reported entries are practical JTS-SCB coverage.

Conformal Prediction

Conformal prediction wraps predictive models with calibrated sets or intervals that provide statistical coverage guarantees.

See all 352 papers

Conformal Bayes under Label Shift: Post-Hoc Calibration vs. In-Training Adaptation
Jun 10, 2026Seungjin ChoiWeighted Conformal Prediction Label Shift
1CROID Research and aSSIST University, Seoul, Korea.
Split Conformal Prediction with Label-Shift-Adjusted Bayesian Scores
Sep 11, 2026Hyeonsu Lee, Juyeon Kim, Erkhembayar Jadamba +2Label Shift Conformal Prediction
MOGAM Institute for Biomedical Research, Korea · CROID Research and aSSIST University, Korea
Weighted Bayesian Conformal Prediction
Apr 7, 2026Xiayin Lou, Peng LuoConformal Prediction Weighted Conformal Prediction
Technical University of Munich · Massachusetts Institute of Technology

Jun 10, 2026J/K move · Enter open · S save

Conformal Bayes under Label Shift: Post-Hoc Calibration vs. In-Training Adaptation

Seungjin Choi

1CROID Research and aSSIST University, Seoul, Korea.

Weighted Conformal Prediction Label Shift Post-Hoc Calibration Bayesian Inference Conformal Prediction

Open paper AskPDF arXiv

Conformal Bayes combines Bayesian posterior predictives with conformal calibration to produce prediction sets that are both statistically valid and geometrically efficient. We study conformal Bayes under label shift from a unified perspective, identifying two complementary approaches that restore nominal target-domain coverage through importance-weighted conformal calibration but operate through independent mechanisms. Post-hoc calibration tilts the posterior predictive toward the target domain and corrects the conformal threshold via an importance-weighted quantile, leaving the parameter posterior unchanged. In-training adaptation tilts the parameter posterior itself to the target domain, producing a corrected predictive whose highest predictive density region serves as the highest predictive density (HPD)-based prediction set under the fitted target predictive; efficiency is model-dependent and does not imply finite-sample conditional optimality. Two controlled experiments isolate the regime-dependence of each strategy: in the low-dimensional, well-estimated regime Strategy~~A produces the narrowest valid intervals, while in the high-dimensional, underdetermined regime Strategy~~B achieves up to

43\%

width reduction at unchanged coverage, under the stated source-sampling and label-shift assumptions.

Abstract

Conformal Prediction

Explore similar work

Conformal Bayes under Label Shift: Post-Hoc Calibration vs. In-Training Adaptation

Split Conformal Prediction with Label-Shift-Adjusted Bayesian Scores

Weighted Bayesian Conformal Prediction