Backward Conformal Prediction under Covariate Shift

Amin Meghrazi; Yuanyuan Li

cs.LGOct 12, 2026

Backward Conformal Prediction under Covariate Shift

Authors: Amin Meghrazi, Yuanyuan Li

Organizations: The Ohio State University · Munich RE

Abstract

Backward conformal prediction (BCP) fixes the size of a prediction set and estimates its miscoverage, but assumes calibration and test data are exchangeable. We extend BCP to covariate shift through importance weighting, preserving its guarantees when the density ratio is known. We show that BCP's tight miscoverage report depends only on the relative weight assigned to missed calibration examples, explaining why a better domain classifier can sometimes produce a worse estimate. Motivated by the fact that BCP depends only on nonconformity scores, we propose estimating density ratios on sorted score vectors rather than high-dimensional inputs. We prove that this approach recovers the same report asymptotically under an interpretable condition and characterize the resulting bias otherwise. Experiments with simulated and benchmark distribution shifts show that the proposed method improves BCP's miscoverage estimates, while score-based weighting consistently yields more accurate estimates than embedding-based weighting.

Figures & tables

identity report
weights	hardness $\beta=1$	hardness $\beta=2$	easy $\beta=1$	easy $\beta=2$	benign $\beta=1$	benign $\beta=2$
true miscoverage	0.274	0.406	0.023	0.010	0.072	0.036
true weights	+0.169	+0.343	+0.014	+0.014	+0.012	+0.023
features	+0.013	+0.106	+0.028	+0.027	+0.009	+0.025
features + Platt	$-$ 0.064	+0.046	+0.059	+0.042	+0.025	+0.034
sorted scores	+0.267	+0.340	+0.014	+0.017	+0.016	+0.021

Table 1: Dry Bean,

T=1

: exact miscoverage of the size-

1

set under the tilted pool and gap (report

-

miscoverage) of the identity and tight reports, by the weights used, and of ATC-NE. “Features”: gradient boosting on the

16

features; “+ Platt”: the same classifier calibrated by Platt scaling on grouped out-of-fold predictions; “sorted scores”: gradient boosting on

G(x)

. Means over

80

draws; the largest standard error in the table is 0.012.

T=2

in Appendix D .

Figure 1: CIFAR-10-C at

T=2

: the tight report against the true miscoverage of the size-

T

set, one vertical segment per setting, both axes logarithmic. Hollow blue markers: mean over the

60

draws with embedding weights; filled red markers: with score weights. Dotted line: no weights, the miss rate on the clean calibration images, the same in every setting. Squares: walls, cells with nearly separable clean and corrupted embeddings (AUC at least

0.95

). Points below the dashed diagonal understate the miscoverage.

T=1

and

T=3

are in Appendix D .

			true	tight report				identity report
model	$T$		miscoverage	none	emb.	score	ATC-NE	valid
EfficientNet-B0	1	no wall	23.6%	14.69	8.15	4.74	3.12	9/12
		walls	77.0%	68.13	52.34	42.98	29.59	0/3
	2	no wall	12.2%	9.20	6.65	4.78	3.49	10/12
		walls	63.8%	60.80	53.65	48.67	41.87	0/3
	3	no wall	7.0%	5.67	4.28	3.48	2.57	11/12

Table 2: CIFAR-10-C by model,

T

and overlap (walls: held-out embedding AUC at least

0.95

); each row averages over the settings of its group. Tight report with no, embedding and score weights, and ATC-NE: mean absolute gap

|\text{report}-\text{truth}|

in percentage points over the settings and their

60

draws; without weights the tight report is the corrected estimator of ST-BCP. Identity report: settings in which it is valid, with score weights. Appendix D gives the signed gaps by group, and per-setting gaps of the weighted reports and ATC-NE with standard errors.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

model	$T$		none	emb.	score	ATC-NE
			tight report
EfficientNet-B0	1	no wall	$-$ 14.64	$-$ 7.86	$-$ 3.05	$-$ 2.59
		walls	$-$ 68.13	$-$ 52.33	$-$ 42.98	$-$ 29.59
	2	no wall	$-$ 9.15	$-$ 6.36	$-$ 3.43	$-$ 2.79
		walls	$-$ 60.80	$-$ 53.65	$-$ 48.67	$-$ 41.87
	3	no wall	$-$ 5.64	$-$ 4.03	$-$ 2.27	$-$ 1.88

Appendix

Table 3: CIFAR-10-C by model,

T

and overlap, as Table 2 : mean gap (report

-

truth) in percentage points over the settings of a group and their

60

draws, for the tight report with no, embedding and score weights and for ATC-NE.

		settings without a wall (embedding AUC $<0.95$ )						walls (AUC $\geq 0.95$ )
model	$T$	#cells	identity valid			WSCP		#cells	identity valid			WSCP
			score	emb.	none	valid	$\|C\|>T$		score	emb.	none	valid
	1	12	9	2	0	7	2%	3	0	0	0	0
EfficientNet-B0	2	12	10	7	3	1	25%	3	0	0	0	0
	3	12	11	9	6	2	30%	3	0	0	0	0
	1	9	0	0	0	8	1%	6	0	2	0	1

Appendix

Table 4: CIFAR-10-C: number of cells in which each method is valid (mean over

60

draws of the report minus the true miscoverage is

\geq 0

; for WSCP, mean miscoverage

\leq

its level

\alpha

), for the identity report with score weights (“score”), embedding weights (“emb.”) and no weights (“none”). WSCP uses score weights;

|C|>T

is the mean share of test images whose WSCP set has more than

T

labels.

Figure 2: CIFAR-10-C at

T=1

(top) and

T=3

(bottom), with the layout of Figure 1 .

identity report
weights	hardness $\beta=1$	hardness $\beta=2$	easy $\beta=1$	easy $\beta=2$	benign $\beta=1$	benign $\beta=2$
true miscoverage	0.064	0.144	0.005	0.003	0.012	0.007
true weights	+0.170	+0.316	+0.025	+0.018	+0.043	+0.040
features	+0.094	+0.152	+0.035	+0.028	+0.045	+0.044
features + Platt	+0.055	+0.110	+0.051	+0.036	+0.050	+0.046
sorted scores	+0.233	+0.287	+0.026	+0.021	+0.046	+0.037

Appendix

Table 5: Dry Bean,

T=2

, same layout as Table 1 . Largest standard error 0.012.

cell	AUC	truth	score weights: gap			emb. weights: gap		ATC-NE	WSCP (score weights)
			identity	transf.	tight	identity	tight	gap	$\alpha$	miscov.	$\|C\|>T$
Gaussian noise 1	0.909	0.329	$-$ 0.013 (0.007)	$-$ 0.125 (0.004)	$-$ 0.083 (0.005)	$-$ 0.159 (0.007)	$-$ 0.171 (0.006)	$-$ 0.073 (0.003)	0.256	0.327	2%
Gaussian noise 3	0.985 ∗	0.727	$-$ 0.230 (0.008)	$-$ 0.407 (0.006)	$-$ 0.397 (0.006)	$-$ 0.477 (0.012)	$-$ 0.525 (0.010)	$-$ 0.336 (0.004)	0.343	0.728	3%
Gaussian noise 5	0.993 ∗	0.809	$-$ 0.318 (0.007)	$-$ 0.497 (0.005)	$-$ 0.493 (0.004)	$-$ 0.541 (0.013)	$-$ 0.603 (0.010)	$-$ 0.438 (0.004)	0.328	0.808	3%
Defocus blur 1	0.513	0.091	+0.001 (0.003)	$-$ 0.032 (0.001)	+0.008 (0.002)	+0.003 (0.009)	$-$ 0.000 (0.002)	$-$ 0.001 (0.001)	0.102	0.092	1%
Defocus blur 3	0.769	0.177	+0.032 (0.006)	$-$ 0.046 (0.003)	$-$ 0.001 (0.003)	$-$ 0.031 (0.004)	$-$ 0.037 (0.003)	$-$ 0.014 (0.002)	0.182	0.177	1%

Appendix

Table 6: CIFAR-10-C, EfficientNet-B0,

T=1

cell	AUC	truth	score weights: gap			emb. weights: gap		ATC-NE	WSCP (score weights)
			identity	transf.	tight	identity	tight	gap	$\alpha$	miscov.	$\|C\|>T$
Gaussian noise 1	0.909	0.179	$-$ 0.000 (0.004)	$-$ 0.104 (0.003)	$-$ 0.074 (0.005)	$-$ 0.076 (0.004)	$-$ 0.122 (0.003)	$-$ 0.067 (0.003)	0.043	0.141	26%
Gaussian noise 3	0.985 ∗	0.577	$-$ 0.295 (0.007)	$-$ 0.457 (0.004)	$-$ 0.430 (0.006)	$-$ 0.421 (0.007)	$-$ 0.504 (0.006)	$-$ 0.398 (0.005)	0.087	0.567	27%
Gaussian noise 5	0.993 ∗	0.690	$-$ 0.437 (0.005)	$-$ 0.593 (0.003)	$-$ 0.572 (0.004)	$-$ 0.522 (0.007)	$-$ 0.621 (0.007)	$-$ 0.547 (0.004)	0.076	0.682	25%
Defocus blur 1	0.513	0.032	+0.023 (0.002)	$-$ 0.006 (0.001)	+0.003 (0.001)	+0.026 (0.006)	+0.001 (0.002)	+0.001 (0.001)	0.010	0.010	23%
Defocus blur 3	0.769	0.079	+0.038 (0.004)	$-$ 0.026 (0.003)	$-$ 0.011 (0.003)	+0.003 (0.002)	$-$ 0.032 (0.002)	$-$ 0.013 (0.002)	0.025	0.040	25%

Appendix

Table 7: CIFAR-10-C, EfficientNet-B0,

T=2

cell	AUC	truth	score weights: gap			emb. weights: gap		ATC-NE	WSCP (score weights)
			identity	transf.	tight	identity	tight	gap	$\alpha$	miscov.	$\|C\|>T$
Gaussian noise 1	0.909	0.105	+0.028 (0.003)	$-$ 0.067 (0.002)	$-$ 0.053 (0.003)	$-$ 0.023 (0.003)	$-$ 0.075 (0.002)	$-$ 0.048 (0.002)	0.016	0.075	32%
Gaussian noise 3	0.985 ∗	0.431	$-$ 0.228 (0.005)	$-$ 0.370 (0.003)	$-$ 0.356 (0.005)	$-$ 0.305 (0.005)	$-$ 0.394 (0.004)	$-$ 0.340 (0.004)	0.035	0.442	31%
Gaussian noise 5	0.993 ∗	0.546	$-$ 0.370 (0.003)	$-$ 0.500 (0.002)	$-$ 0.493 (0.003)	$-$ 0.408 (0.006)	$-$ 0.513 (0.005)	$-$ 0.482 (0.003)	0.030	0.547	31%
Defocus blur 1	0.513	0.015	+0.029 (0.001)	$-$ 0.002 (0.001)	+0.001 (0.001)	+0.033 (0.005)	+0.001 (0.002)	+0.000 (0.001)	0.006	0.005	28%
Defocus blur 3	0.769	0.036	+0.053 (0.003)	$-$ 0.009 (0.002)	$-$ 0.003 (0.002)	+0.029 (0.002)	$-$ 0.014 (0.001)	$-$ 0.003 (0.001)	0.011	0.018	30%

Appendix

Table 8: CIFAR-10-C, EfficientNet-B0,

T=3

cell	AUC	truth	score weights: gap			emb. weights: gap		ATC-NE	WSCP (score weights)
			identity	transf.	tight	identity	tight	gap	$\alpha$	miscov.	$\|C\|>T$
Gaussian noise 1	0.927	0.086	$-$ 0.026 (0.003)	$-$ 0.045 (0.002)	$-$ 0.011 (0.002)	$-$ 0.017 (0.003)	$-$ 0.003 (0.003)	$-$ 0.005 (0.001)	0.079	0.085	1%
Gaussian noise 3	0.993 ∗	0.204	$-$ 0.081 (0.005)	$-$ 0.124 (0.003)	$-$ 0.080 (0.004)	$-$ 0.058 (0.008)	$-$ 0.079 (0.009)	$-$ 0.058 (0.002)	0.136	0.203	1%
Gaussian noise 5	0.997 ∗	0.288	$-$ 0.135 (0.006)	$-$ 0.187 (0.004)	$-$ 0.145 (0.005)	$-$ 0.082 (0.012)	$-$ 0.143 (0.013)	$-$ 0.109 (0.002)	0.156	0.286	2%
Defocus blur 1	0.529	0.049	$-$ 0.013 (0.001)	$-$ 0.023 (0.001)	+0.005 (0.002)	$-$ 0.018 (0.001)	$-$ 0.001 (0.001)	+0.001 (0.001)	0.055	0.049	0%
Defocus blur 3	0.834	0.054	$-$ 0.014 (0.001)	$-$ 0.026 (0.001)	+0.003 (0.002)	$-$ 0.010 (0.001)	+0.008 (0.002)	+0.001 (0.001)	0.058	0.055	0%

Appendix

Table 9: CIFAR-10-C, WideResNet-40-2 (AugMix),

T=1

cell	AUC	truth	score weights: gap			emb. weights: gap		ATC-NE	WSCP (score weights)
			identity	transf.	tight	identity	tight	gap	$\alpha$	miscov.	$\|C\|>T$
Gaussian noise 1	0.927	0.029	+0.010 (0.001)	$-$ 0.016 (0.001)	$-$ 0.008 (0.001)	+0.021 (0.002)	$-$ 0.001 (0.002)	$-$ 0.005 (0.001)	0.006	0.010	20%
Gaussian noise 3	0.993 ∗	0.093	$-$ 0.016 (0.003)	$-$ 0.064 (0.002)	$-$ 0.053 (0.003)	+0.019 (0.005)	$-$ 0.041 (0.007)	$-$ 0.043 (0.002)	0.020	0.075	20%
Gaussian noise 5	0.997 ∗	0.151	$-$ 0.056 (0.004)	$-$ 0.115 (0.002)	$-$ 0.103 (0.003)	+0.017 (0.010)	$-$ 0.092 (0.008)	$-$ 0.088 (0.002)	0.028	0.144	20%
Defocus blur 1	0.529	0.013	+0.012 (0.001)	$-$ 0.005 (0.000)	+0.000 (0.001)	+0.009 (0.000)	$-$ 0.002 (0.000)	+0.001 (0.001)	0.004	0.003	18%
Defocus blur 3	0.834	0.017	+0.009 (0.001)	$-$ 0.008 (0.000)	$-$ 0.004 (0.001)	+0.013 (0.001)	$-$ 0.004 (0.001)	$-$ 0.003 (0.001)	0.004	0.004	18%

Appendix

Table 10: CIFAR-10-C, WideResNet-40-2 (AugMix),

T=2

cell	AUC	truth	score weights: gap			emb. weights: gap		ATC-NE	WSCP (score weights)
			identity	transf.	tight	identity	tight	gap	$\alpha$	miscov.	$\|C\|>T$
Gaussian noise 1	0.927	0.012	+0.021 (0.001)	$-$ 0.004 (0.000)	$-$ 0.002 (0.001)	+0.032 (0.002)	$-$ 0.002 (0.002)	$-$ 0.000 (0.001)	0.003	0.005	24%
Gaussian noise 3	0.993 ∗	0.051	+0.011 (0.002)	$-$ 0.033 (0.001)	$-$ 0.030 (0.002)	+0.050 (0.004)	$-$ 0.035 (0.004)	$-$ 0.027 (0.002)	0.009	0.042	25%
Gaussian noise 5	0.997 ∗	0.091	$-$ 0.015 (0.003)	$-$ 0.069 (0.001)	$-$ 0.067 (0.002)	+0.063 (0.008)	$-$ 0.067 (0.006)	$-$ 0.061 (0.002)	0.013	0.094	25%
Defocus blur 1	0.529	0.005	+0.016 (0.001)	$-$ 0.000 (0.000)	+0.000 (0.000)	+0.014 (0.000)	$-$ 0.000 (0.000)	+0.001 (0.001)	0.002	0.002	21%
Defocus blur 3	0.834	0.007	+0.016 (0.001)	$-$ 0.002 (0.000)	$-$ 0.001 (0.000)	+0.020 (0.001)	$-$ 0.002 (0.000)	$-$ 0.001 (0.001)	0.002	0.002	22%

Appendix

Table 11: CIFAR-10-C, WideResNet-40-2 (AugMix),

T=3

model	$T$	draws	largest change	draws	largest change	cells	largest shortfall
		score weights		embedding weights		corrupted calibration data
EfficientNet-B0	1	33	0.01	62	0.99	7	14.96
	2	0	0.00	13	0.00	4	5.31
	3	0	0.00	0	0.00	3	0.83
WRN-40-2 (AugMix)	1	0	0.00	26	0.02	1	0.16
	2	0	0.00	0	0.00	0	0.00

Appendix

Table 12: Non-degeneracy on CIFAR-10-C. Score and embedding weights: number of the

900

draws of a row (

15

cells,

60

draws each) with a degenerate fold, and the largest change of a setting’s mean tight report relative to the weighted miss rate, in percentage points. Last two columns: the limit of the tight report when the calibration data come from the corrupted images themselves, without weights, which is also its limit under exact weights; number of the

15

cells in which this limit is below the miscoverage, and the largest such shortfall in percentage points.

	no walls		walls		score weights
domain classifier	embedding	score	embedding	score	closer in
Gradient boosting (main text)	4.20	2.93	20.85	17.84	12 of 12 groups
Logistic regression	5.40	3.37	23.13	17.83	12 of 12 groups
Multilayer perceptron	4.77	3.45	22.14	17.98	12 of 12 groups
Gradient boosting, early stopping	4.27	3.53	20.94	18.89	12 of 12 groups

Appendix

Table 13: CIFAR-10-C: mean absolute gap of the tight report, in percentage points, with embedding and with score weights from four domain classifiers fitted on the same draws. Means over the settings without walls or at the walls and their

60

draws. Last column: groups of Table 2 in which score weights have the smaller mean absolute gap.

	CIFAR-10-C, tight report (percentage points)								Dry Bean, hardness, $T=1$
	score weights, no walls			embedding, no walls		walls, mean gap		score	feature weights		score weights
clip of $\hat{q}$	mean gap	SD	$\|\mathrm{gap}\|$	mean gap	$\|\mathrm{gap}\|$	score	emb.	closer in	$\beta=1$	$\beta=2$	$\beta=1$	$\beta=2$
$[0.01,\,0.99]$ (main text)	$-$ 1.72	2.20	2.93	$-$ 3.79	4.20	$-$ 17.6	$-$ 19.6	12 of 12	$-$ 0.081	$-$ 0.101	+0.042	$-$ 0.006
$[0.001,\,0.999]$	$-$ 0.93	3.60	3.18	$-$ 3.75	4.23	$-$ 16.9	$-$ 18.6	12 of 12	$-$ 0.081	$-$ 0.104	+0.085	+0.032
$[10^{-6},\,1-10^{-6}]$	$-$ 0.85	3.89	3.26	$-$ 3.75	4.23	$-$ 16.9	$-$ 18.0	12 of 12	$-$ 0.081	$-$ 0.104	+0.093	+0.021

Appendix

Table 14: Sensitivity to the clip of the domain classifier’s probability

\hat{q}

. The classifiers are fitted once and only the clip changes. CIFAR-10-C: tight report over the settings without walls or at the walls and their

60

draws; SD is the standard deviation of the gap over the draws of a setting, averaged over the settings; “closer in” counts the groups of Table 2 in which score weights have the smaller mean absolute gap. Dry Bean: gap of the tight report as in Table 1 , means over

80

draws.

			level of the main text		level of the previous draw
	$T$	settings	valid	excess	valid	excess
CIFAR-10-C, EfficientNet-B0	1	12	7	+2.24	7	+2.16
	2	12	1	+5.29	1	+5.35
	3	12	2	+3.68	1	+3.71
CIFAR-10-C, WRN-40-2 (AugMix)	1	9	8	$-$ 0.54	8	$-$ 0.37
	2	9	4	+0.30	3	+0.35

Appendix

Table 15: WSCP with its level chosen as in the main text and with the level chosen in the previous draw. CIFAR-10-C: score weights, settings without walls. Dry Bean: feature weights. Valid: settings in which the mean miscoverage is at most the mean level. Excess: miscoverage minus level in percentage points, averaged over the settings of a row and their draws (

60

on CIFAR-10-C,

80

on Dry Bean; the first draw of a setting has no previous level).

			tight report								identity
				gradient boosting		score weights from			binned		report
model	$T$	miscoverage	no weights	embedding	score	LR	MLP	GB-ES	prediction	ATC-NE	(score)
EfficientNet-B0	1	18.05%	$-$ 8.99	$-$ 5.02	$-$ 0.68	$-$ 3.40	$-$ 3.35	$-$ 3.73	$-$ 3.55	$-$ 0.66	+0.90
	2	7.00%	$-$ 3.85	$-$ 2.03	+0.77	$-$ 1.57	$-$ 1.38	$-$ 1.26	$-$ 1.53	+1.05	+4.21
	3	4.00%	$-$ 2.59	$-$ 1.65	+0.02	$-$ 1.52	$-$ 1.49	$-$ 1.25	$-$ 1.46	+0.56	+4.78
WRN-40-2 (AugMix)	1	11.30%	$-$ 6.55	$-$ 3.26	$-$ 0.57	$-$ 2.21	$-$ 2.71	$-$ 2.85	$-$ 2.55	$-$ 1.21	$-$ 1.61

Appendix

Table 16: CIFAR-10.1 as the test distribution, with calibration on clean CIFAR-10 test images. Miscoverage: miss rate of the size-

T

set over the

2{,}000

CIFAR-10.1 images. Other columns: gap (report

-

miscoverage) in percentage points, means over

60

draws. The tight report is given without weights, with embedding and score weights from the gradient boosting of the main text, and with score weights from logistic regression (LR), the perceptron (MLP) and early-stopped gradient boosting (GB-ES). The binned prediction uses all images and has no draws. The identity report uses the score weights of the main text. The largest standard error over draws is 0.64.

			identity gap					covariance
model	$T$	truth	clipped	unclipped	tight gap	$\mathbb{E}[\tilde{\alpha}]$	$\mathbb{E}[\mathbf{1}\{Y\notin\hat{C}\}/\tilde{\alpha}]$	clipped	unclipped
EfficientNet-B0	1	8.91%	$-$ 1.29	+1.05	$-$ 0.03	7.66%	0.67	+3.80	+3.19
	2	3.05%	+1.59	+3.02	+0.03	4.63%	0.43	+1.06	$-$ 0.14
	3	1.40%	+2.41	+3.67	$-$ 0.01	3.81%	0.26	+0.41	+0.52
WRN-40-2 (AugMix)	1	4.93%	$-$ 1.73	$-$ 1.73	$-$ 0.09	3.21%	0.71	+2.65	+2.65
	2	1.24%	+0.95	+0.95	0.00	2.19%	0.30	+0.59	+0.59

Appendix

Table 17: No shift: clean CIFAR-10, no weights,

n=1{,}000

, means over

60

draws. Truth: miscoverage of the size-

T

set with the clipped score. Gaps (report

-

miscoverage under the same score) and covariances in percentage points.

			identity report				tight report
			valid settings		mean gap		mean $\|\mathrm{gap}\|$
	$T$	settings	floor	no floor	floor	no floor	floor	no floor
no walls	1	12	9	11	+2.55	+4.51	4.74	4.74
	2	12	10	11	+2.71	+3.91	4.78	4.78
	3	12	11	11	+4.29	+5.27	3.48	3.49
walls	1	3	0	0	$-$ 22.06	$-$ 22.01	42.98	42.98

Appendix

Table 18: EfficientNet-B0 on CIFAR-10-C with the floor

10^{-12}

on the probabilities in the score and without it, with the same score weights. Each report is compared with the miscoverage under the same score. Gaps in percentage points, averaged over the settings of a row and their

60

draws.

shift	weights	ECE	log-loss	rank corr.	MSE	identity gap ( $T{=}1$ )	tight gap ( $T{=}1$ )	tight gap ( $T{=}2$ )
hardness $\beta=1$	features	0.054	0.576	0.68	3.17	+0.013	$-$ 0.081	$-$ 0.027
hardness $\beta=1$	+ Platt (1 model)	0.041	0.565	0.68	2.27	$-$ 0.064	$-$ 0.121	$-$ 0.037
hardness $\beta=1$	+ Platt (5 models)	0.046	0.561	0.71	2.19	$-$ 0.065	$-$ 0.121	$-$ 0.037
hardness $\beta=1$	+ isotonic (5 models)	0.038	0.561	0.70	2.81	$-$ 0.030	$-$ 0.104	$-$ 0.031
hardness $\beta=1$	features, early stopping	0.043	0.560	0.73	2.20	$-$ 0.042	$-$ 0.110	$-$ 0.036
hardness $\beta=1$	logistic regression	0.061	0.600	0.58	2.82	$-$ 0.127	$-$ 0.165	$-$ 0.046

Appendix

Table 19: Dry Bean: calibrated domain classifiers. Means over

80

draws; the largest standard error of a gap is 0.010.

Explore similar work

Sharp training-conditional coverage for conformal prediction under covariate shift
Sep 27, 2026Mehrdad PournaderiWeighted Conformal Prediction Set Prediction
Mofid Securities
Coverage Guarantees for Pseudo-Calibrated Conformal Prediction under Distribution Shift
Feb 16, 2026Farbod Siahkali, Ashwin Verma, Vijay GuptaConformal Prediction Distribution Shift Robustness
Elmore Family School of Electrical and Computer Engineering, at Purdue University, West Lafayette, IN 47907 USA
Split Conformal Prediction with Label-Shift-Adjusted Bayesian Scores
Sep 11, 2026Hyeonsu Lee, Juyeon Kim, Erkhembayar Jadamba +2Label Shift Conformal Prediction
MOGAM Institute for Biomedical Research, Korea · CROID Research and aSSIST University, Korea

Sep 27, 2026J/K move · Enter open · S save

Sharp training-conditional coverage for conformal prediction under covariate shift

Mehrdad Pournaderi

Mofid Securities

Weighted Conformal Prediction Set Prediction Conformal Prediction Covariate Shift

Open paper AskPDF arXiv

Weighted split conformal prediction reweights calibration scores by the likelihood ratio between the test and training covariate distributions and guarantees marginal coverage under covariate shift. We study its coverage conditional on the calibration data. An elementary argument, based on a single concentration inequality at a fixed population quantile, gives explicit training-conditional bounds without unspecified constants, and shows that the relevant scale is not the supremum of the likelihood ratio but a variance proxy built from the chi-squared divergence of the shift and from the average of the ratio over the part of the test population, of probability equal to the miscoverage level, where it is largest. A two-point lower bound shows that the root-m rate and the chi-squared contribution are intrinsic to the shift. Run at an explicitly inflated level, the weighted quantile becomes a deterministic PAC prediction set. We compare it with randomized rejection sampling and with importance-weighted learn-then-test and, through a certified choice of a clipping level for the likelihood ratio, map the regime in which each gives the narrower valid set. The analysis extends to estimated likelihood ratios and to tail functionals estimated from an unlabeled source sample, which yields a fully finite-sample certificate.

Abstract

Explore similar work

Sharp training-conditional coverage for conformal prediction under covariate shift

Coverage Guarantees for Pseudo-Calibrated Conformal Prediction under Distribution Shift

Split Conformal Prediction with Label-Shift-Adjusted Bayesian Scores