Predicted Class Correction: Conformal Thresholds for Classes with No Calibration Data

One Octadion; Novanto Yudistira; Yuta Nakashima

cs.LGOct 18, 2026

Predicted Class Correction: Conformal Thresholds for Classes with No Calibration Data

Authors: One Octadion, Novanto Yudistira, Yuta Nakashima

Organizations: Universitas Brawijaya, Malang, Indonesia · SANKEN, The University of Osaka, Osaka, Japan

Abstract

Class-conditional conformal prediction calibrates a separate score threshold for every class and therefore requires labelled calibration examples of every class. In long-tailed deployments these are exactly what is missing: rare classes are left with few or no held-out examples, and existing methods are either undefined for them or fall back to the shared marginal threshold. We ask whether the per-class correction can instead be predicted from descriptors of the class that require no labels at all. We propose Predicted Class Correction (PCC), which regresses the offset between the per-class and the marginal conformal quantile on geometric descriptors of the classifier head, shrinks the prediction by a scalar selected on the labelled classes, and refits a single marginal offset. The descriptors depend only on the model's weights, so a class with zero calibration examples still receives its own threshold, and since only one scalar is calibrated, split conformal's marginal guarantee is retained. On ImageNet, PCC improves worst-class coverage on such classes by 0.058 [0.006, 0.110] at matched average set size, 42% of an oracle that sees the test labels. Of eight baselines on the same splits, six change the statistic by exactly zero and the best reaches 0.016 with an interval containing zero, so PCC is the only one distinguishable from no improvement; the gain persists with only 100 labelled classes out of 1000. A second finding concerns evaluation: worst-class coverage becomes unmeasurable before it becomes unachievable. Below roughly 35 test examples per class no improvement can be shown even for an oracle, so a null result there reflects the evaluation set, not the method. Restricting evaluation above this depth recovers a significant improvement on Pl@ntNet-300K, where the effect had appeared absent.

Figures & tables

Figure 1: A class with no calibration example still receives a threshold of its own. ImageNet class $754$ is held out, so $n_{y}=0$ : every classwise method either leaves it undefined or hands it the shared marginal threshold. Its descriptor $\varphi(y)$ is read from the classifier head, whose three nearest rows to it are other audio equipment, and the fitted regression turns that geometry into an offset. The threshold moves from $\hat{q}=0.9657$ to $0.9812$ and coverage of the class from $0.658$ to $0.750$ , against $0.803$ for an oracle given the test labels. The class shown is the held-out class with the lowest coverage under the marginal threshold, fixed by that rule before any image was looked at.

Method	$\Delta$ worst-class	Undefined
Standard	$+0.000\ [\ \ 0.000,\ 0.000]$	$0/300$
Classwise	$+0.000\ [\ \ 0.000,\ 0.000]$	$\mathbf{300/300}$
Clustered	$+0.000\ [\ \ 0.000,\ 0.000]$	$0/300$
Rc3p	$+0.000\ [\ \ 0.000,\ 0.000]$	$\mathbf{300/300}$
Fuzzy , quantile	$+0.000\ [\ \ 0.000,\ 0.000]$	$0/300$
Fuzzy , rarity	$+0.000\ [-0.000,+0.001]$	$0/300$

Table 1: Classes with zero calibration examples. Standard [ 45 ] , Classwise [ 38 ] , Clustered [ 11 ] , Rc3p [ 41 ] , and Fuzzy and Interp-Q [ 12 ] . Change in worst-class coverage over the marginal threshold, at matched average set size (

1.221

for every row), on

300

held-out ImageNet classes, five splits at the protocol of Sec. 4 . PCC is in the split variant of Sec. 3.5 , so the row carrying the headline is the one Proposition 2 covers; the reuse variant gives the same figure to four decimals ( Sec. P ), as does solving the size match on calibration rows rather than evaluation rows ( Sec. V ). PCC is the only method whose interval excludes zero. “Undefined” counts classes a method could not give a threshold to, before the pre-registered fallback. Baseline numbers come from their released code, and for Fuzzy CP the kernel bandwidth is swept with the best kept, which favours the baseline; the winning bandwidth is not stable across splits.

Ablation	$\Delta$ vs. full	$d_{z}$	Holm
$\lambda=0$ (no correction)	$+0.032$	$1.30$	✓
$\lambda=1$ (unshrunk)	$+0.584$	$3.99$	✓
distance descriptors only	$+0.039$	$1.49$	✓
prevalence descriptor only	$+0.032$	$1.30$	✓
without prevalence	$+0.011$	$0.33$	—
$n^{\star}$ rule replaced	$+0.000$	—	—

Table 2: Ablations on ImageNet at

\alpha=0.10

, ten splits. Positive means the full method is better.

d_{z}

is Cohen’s paired effect size [ 8 ] ; the last column marks Holm-corrected significance at

0.05

[ 24 ] . The last two rows are identical to the full method at matched set size by construction ( Sec. I ).

Figure 2: (a) Evaluation examples available per class, sorted, for the three datasets. The dashed line is the

30

-example threshold below which a per-class coverage estimate is withheld ( Tab. S2 );

83\%

of Pl@ntNet-300K classes and

97\%

of iNaturalist-2018 classes fall below it, against none of ImageNet’s. (b) Capping evaluation examples per class inside one dataset with everything else fixed: below roughly

35

the oracle ceiling itself falls to zero, so no method, including one that observes the test labels, can be shown to improve worst-class coverage. The settings where PCC appeared to fail sit at

35

(additional backbones),

3

(Pl@ntNet-300K) and

2

(iNaturalist-2018). (c) Sweeping how many classes have no calibration data at all; the

x

-values correspond to

900

700

500

300

and

100

labelled classes out of

1000

. Bars are

95\%

confidence intervals over five splits.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Method	$q_{y}$ built from	If $n_{y}=0$
Standard [ 45 ]	shared quantile	shared
Classwise [ 38 ]	own quantile	$\infty$
Clustered [ 11 ]	cluster quantile	shared
Rc3p [ 41 ]	quantile + rank	$\infty$
Fuzzy [ 12 ]	kernel weights	shared
Pas [ 12 ]	shared quantile	shared

Appendix

Table S1: Methods that give a class its own conformal threshold. All of them target at least marginal coverage; the columns that separate them are what happens to a class with no labelled calibration examples, and how specific to that class the threshold it receives can be. Three methods do produce a defined threshold there without per-class labels: Pas reshapes the score using prevalence, Interp-Q caps the undefined classwise quantile at one before blending, and Tacp penalises the score of head classes, which gives one degree of freedom per prevalence group instead of one per class. Behaviours were obtained by running the authors’ released implementations on the same held-out classes, except for Tacp , whose authors release no code; Tab. 1 reports the coverage that results.

Metric	Definition
$\hat{c}_{y}$	$\frac{1}{\|J_{y}\|}\sum_{i\in J_{y}}\mathbf{1}\{Y_{i}\in\mathcal{C}(X_{i})\}$
Worst-class (A)	$\min_{y}\hat{c}_{y}$
Bin-worst (B)	$\min_{b}\hat{c}^{(b)}$ , prevalence bins
$\Delta$ worst	worst-class minus marginal
Below target	$\frac{1}{\|\mathcal{Y}\|}\sum_{y}\mathbf{1}\{\hat{c}_{y}<1-\alpha\}$
SSCV	$\max_{b}\|\hat{c}^{(b)}-(1-\alpha)\|$ , strata

Appendix

Table S2: Evaluation metrics.

\hat{c}_{y}

is the empirical coverage of class

y

, and regimes A and B are the per-class and prevalence-binned cases.

J_{y}

is the set of test indices with label

y

and

N

the number of test points. Bins and strata are pooled groups, used where a per-class estimate is too noisy to report: prevalence bins require

200

test rows each and set-size strata

30

. The empty-set rate is included because an average set size below one is only possible when a substantial share of test points receive nothing, a regime in which coverage and set size both merely look low.

Configuration	Shrunk	Unshrunk	$\lambda_{\text{oracle}}$
C_comp_competitors_s1	$+0.0487$	$-0.6571$	$0.10$
H_heldout_ho0p5_s2	$+0.1928$	$-0.4605$	$0.70$
C_comp_competitors_s2	$+0.1569$	$-0.4902$	$0.70$
H_heldout_ho0p7_s2	$+0.0667$	$-0.5733$	$0.70$
H_heldout_ho0p9_s2	$+0.1579$	$-0.4605$	$0.70$

Appendix

Table S3: The five configurations where the two ceiling definitions differ most. A quantity that can sit

0.65

below the marginal threshold it is meant to bound is not a ceiling; the shrunk version cannot be exceeded by PCC by construction, since PCC optimises the same

\lambda

over a worse

\delta

Axis	Setting	$\Delta$ worst-class	% ceiling
Score	LAC [ 38 ]	$\mathbf{+0.058}\ [+0.006,+0.110]$	$42$
	PAS [ 12 ]	$+0.066\ [-0.017,+0.149]$	$28$
	SAPS [ 25 ]	$+0.031\ [-0.019,+0.081]$	$24$
	APS [ 36 ]	$+0.022\ [+0.004,+0.040]$	$15$
	RAPS [ 1 ]	$+0.010\ [-0.009,+0.029]$	$10$
$\alpha$	$0.10$	$\mathbf{+0.058}\ [+0.006,+0.110]$	$42$

Appendix

Table S4: Score function, miscoverage level and calibration depth on held-out ImageNet classes, five splits, ten for calibration depth. The ceiling is flat across the depth block,

0.128

0.132

, so the rise there comes from the method and not from a wider ceiling.

Thresholds as emitted	Marginal cov.	Set size
Marginal threshold $\hat{q}$	$0.9013$	$1.21$
Pcc , no recalibration	$0.8963$	$2.00$
Pcc , with recalibration	$0.8989$	$2.31$
Target	$0.9000$	—

Appendix

Table S5: The unmatched view, ten ImageNet splits,

\alpha=0.10

. Recalibration moves marginal coverage towards the target and costs set size. The matched comparison used everywhere else cancels this shift by construction.

Backbone	$\varphi$	$\Delta$ worst-class
ConvNeXt-T [ 30 ]	own	$+0.000\ [-0.000,+0.000]$
	fixed	$+0.011\ [-0.017,+0.040]$
ResNet-50 [ 21 ]	own	$+0.017\ [-0.005,+0.040]$
	fixed	$+0.006\ [-0.005,+0.017]$
ViT-B/16 [ 13 ]	own	$-0.029\ [-0.072,+0.015]$
	fixed	$-0.011\ [-0.066,+0.043]$

Appendix

Table S6: Both descriptor arms on three backbones, five splits. “own” uses each model’s own classifier head, “fixed” the ResNet-50 head also used for ImageNet. Top-1 accuracies of all three matched published values to four decimal places, which we used as a correctness check on the weight extraction.

Dataset	Thr.	Kept	$\Delta$ worst-class
Pl@ntNet	$\geq 35$	$151$	$+0.020\ [-0.011,+0.052]$
Pl@ntNet	$\geq 75$	$98$	$\mathbf{+0.019}\ [+0.010,+0.028]$
iNat-2018	$\geq 35$	$237$	$+0.000$
iNat-2018	$\geq 75$	$75$	$+0.000$

Appendix

Table S7: Restricting evaluation to classes whose slice can support a per-class estimate. Five splits. Pl@ntNet recovers a significant improvement; iNaturalist does not, and there

\lambda=0

means the method declines to act rather than acting and failing.

$\lambda$	$0$	$0.1$	$0.3$	$0.5$	$1.0$
worst	$\mathbf{0.699}$	$0.466$	$0.385$	$0.350$	$0.319$

Appendix

Table S8: Shrinkage curve on the labelled classes of iNaturalist-2018, mean over five splits, at the

\geq 75

threshold. Every positive

\lambda

is worse than none, so the selection step returns

\lambda=0

. A flat curve would have meant

\hat{\delta}

was degenerate; a decreasing one means it is informative and points the wrong way.

Corruption	Sev.	$\Delta$	Cov.	Size	Empty
brightness	3	$+0.0200$	$0.841$	$1.345$	$0.063$
brightness	5	$+0.0067$	$0.746$	$1.362$	$0.103$
contrast	3	$-0.0133$	$0.600$	$1.297$	$0.171$
contrast	5	$+0.0000$	$0.053$	$0.317$	$0.757$
defocus_blur	3	$+0.0067$	$0.500$	$1.146$	$0.268$
defocus_blur	5	$+0.0000$	$0.229$	$0.763$	$0.497$

Appendix

Table S9: ImageNet-C, per corruption and severity, three splits.

\Delta

worst is the change in worst-class coverage on held-out classes at matched set size. Pooled over all

90

conditions the improvement is

+0.003

[+0.001,+0.005]

Figure S1 : The mechanism, on the primary ImageNet configuration. Every labelled class contributes an observed offset

\delta_{y}

and a predicted one

g_{\theta}(\varphi(y))

; held-out classes appear as ticks on the horizontal axis, because the observed coordinate does not exist for them. The regression recovers the direction of the correction and understates its magnitude, and the classes that decide the minimum lie in the dense band near

\delta_{y}=0

, where the cost is prediction error rather than shrinkage towards the mean. That error is the quantity Prop. 1 bounds, though Appendix Z finds that bound is not what selects

\hat{\lambda}

Figure S2 : Class-conditional coverage of every class, sorted, on the primary ImageNet configuration. Rank rather than percentile on the horizontal axis, because worst-class coverage is decided by the leftmost few classes and a linear axis compresses them to nothing. The correction rotates the curve: the tail rises and the classes that had coverage to spare give some back, with the crossing near rank

150

300

on the held-out classes. The oracle performs the same rotation further, which is the sense in which it is a ceiling rather than a different kind of method.

Improved $11.3\%$ , worsened $43.3\%$
Quintile	Classes	Med. prev.	Mean $\Delta\hat{c}_{y}$
1, frequent	$60$	$1182$	$-0.0078$
2	$60$	$1174$	$-0.0033$
3	$60$	$1169$	$-0.0033$
4	$60$	$1163$	$-0.0039$
5, rare	$60$	$1147$	$-0.0024$

Appendix

Table S10: Change in per-class coverage on the

300

held-out ImageNet classes, by prevalence quintile, at matched average set size. The correction concentrates its effect on the failing classes and pays for it with small reductions spread across the rest.

Score	Worst-class	Set size	Macro	$\Delta$ Pcc
Softmax (LAC)	$0.534$	$1.256$	$0.900$	$-0.005$
Average over augment.	$0.485$	$1.228$	$0.898$	$+0.002$
Learned aggregation	$0.534$	$1.233$	$0.899$	$+0.001$
Uncertainty-modulated	$0.518$	$1.230$	$0.900$	$-0.003$

Appendix

Table S11: Four score functions on the same backbone under the marginal threshold, held-out classes,

\alpha=0.10

, three splits. Changing the score leaves worst-class coverage within

0.049

and does not order the scores the way marginal coverage would. The last column reports what Pcc adds on top of each, in a configuration whose

25

evaluation examples per class put it below the depth at which the statistic can move ( Sec. 4.3 ).

Quantity	Rows	Classes
$\hat{q}$ (1)	calib.	labelled
$\delta_{y}$ (3)	calib.	labelled, $n_{y}\geq n_{\mathrm{cal}}$
$g_{\theta}$	calib.	labelled, $n_{y}\geq n_{\mathrm{cal}}$
$\lambda$ (5)	calib.	labelled
$n_{\mathrm{cal}}$	calib.	labelled
$c$ (9)	calib.	labelled

Appendix

Table S12: What each estimated quantity may see; the equation numbers are those of the main paper. Evaluation rows and held-out classes are used only for reporting. The default configuration lets

g_{\theta}

\lambda

and

c

share the same calibration rows, which is convenient but not what Prop. 2 assumes; Pcc -split divides those rows in two and is the variant the proposition covers.

Variant	Marg. cov.	Deficit	$\Delta$ worst
Reuse	$0.8994$	$-0.0018$	$+0.0588$
Split $25\%$	$\mathbf{0.9011}$	$\mathbf{-0.0000}$	$+0.0636$
Split $50\%$	$0.9003$	$-0.0007$	$+0.0585$
Target	$0.9000$	—	—

Appendix

Table S13: Pcc -split against reuse, ten paired splits on ImageNet at

\alpha=0.10

. “Deficit” is marginal coverage minus that of the marginal threshold measured on the same evaluation rows, so it isolates the optimism from reusing calibration rows rather than the finite-sample slack common to both.

\Delta

worst-class is the matched-size statistic used throughout the paper.

Rows	Cal.	Eval.	$\Delta$ worst-class	Ceil.
$250$ k	$177$	$\ 76$	$+0.058\ [+0.006,+0.110]$	$0.140$
$500$ k	$354$	$152$	$+0.050\ [+0.011,+0.089]$	$0.105$
$750$ k	$532$	$228$	$+0.076\ [+0.034,+0.118]$	$0.116$

Appendix

Table S14: The row budget swept over a factor of three, five splits each, everything else at the primary configuration. No point estimate falls outside the intervals of the other two. The

250

k row reproduces the corresponding entry of Fig. 2 b to four decimal places, which is the check that the sweep shares the configuration of the rest of the paper.

Descriptors	$\Delta$ worst-class	CV err.
Cosine, all	$+0.058\ [+0.006,+0.110]$	$0.036$
Euclidean	$+0.045\ [+0.009,+0.081]$	$0.040$
No head bias	$+0.058\ [+0.006,+0.110]$	$0.036$
$k\in\{1,2,5\}$	$+0.058\ [+0.006,+0.110]$	$0.036$
$k\in\{1,10,20,100\}$	$+0.063\ [+0.016,+0.111]$	$0.036$

Appendix

Table S15: Descriptor choices, five splits each on held-out ImageNet classes. “CV error” is the cross-validated mean squared error of

g_{\theta}

on the labelled classes. The neighbour count held out as the distance reference follows the smallest

k

requested, so each arm withholds its own feature and none is advantaged.

Data	Head	$\Delta$ worst-class	$\bar{\lambda}$	$\hat{\lambda}{=}0$
Image Net	linear	$\mathbf{+0.058}\ [+0.006,+0.110]$	$0.10$	$0/5$
	quadratic	$+0.058\ [-0.011,+0.128]$	$0.20$	$0/5$
	kernel	$+0.026\ [+0.008,+0.045]$	$0.32$	$0/5$
	$\gamma{=}0.3$	$+0.046\ [+0.004,+0.088]$	$0.23$	$0/3$
	$\gamma{=}3$	$-0.022\ [-0.065,+0.021]$	$0.27$	$0/3$
Pl@nt Net	linear	$\mathbf{+0.019}\ [+0.010,+0.028]$	$0.22$	$0/5$

Appendix

Table S16: Replacing the ridge

g_{\theta}

with a nonlinear head, everything else at the configuration of the corresponding table. Five splits, except the two indented RBF-width rows, which are three and carry their own ceiling of

0.153

. Unindented kernel rows use

\gamma=1

. Oracle ceilings are

0.140

on ImageNet,

0.086

on Pl@ntNet-300K under the

75

-example restriction of Tab. S7 , and

0.018

on iNaturalist-2018. The last column counts splits in which the shrinkage step declined the prediction entirely.

Data	$r$	$\sigma=\sqrt{r^{2}-1}$	$\rho$ with abundance
ImageNet, control	$1.04$	$0.28$	$-0.09$
iNaturalist-2018	$1.07$	$0.39$	$-0.36$
Pl@ntNet-300K	$\mathbf{1.61}$	$\mathbf{1.26}$	$\mathbf{-0.88}$

Appendix

Table S17: Head-row geometry against class abundance.

r

is the ratio of median

\|w_{y}-\bar{w}\|

between the bottom and top deciles of calibration depth, and

\rho

is the Spearman correlation of

\|w_{y}\|

with log abundance. ImageNet is the control: its deciles differ by a factor of

1.0

in abundance, so it has almost no variation to track, and it shows none.

$\sigma$	ratio	$\Delta$ worst-class	CVaR 5%
$0.00$	$1.00$	$\mathbf{+0.058}\ [+0.006,+0.110]$	$0.732$
$0.39$	$1.07$	$+0.049\ [+0.000,+0.098]$	$0.725$
$0.60$	$1.17$	$+0.043\ [-0.014,+0.099]$	$0.718$
$1.26$	$1.61$	$-0.014\ [-0.072,+0.043]$	$0.672$
$2.00$	$2.24$	$-0.069\ [-0.127,-0.012]$	$0.625$

Appendix

Table S18: Held-out head rows perturbed outward by

\sigma

, five splits each, everything else at the primary configuration.

\sigma=0.39

and

\sigma=1.26

are the magnitudes Tab. S17 measures for iNaturalist-2018 and Pl@ntNet-300K. The oracle ceiling is

0.140

throughout, since the oracle reads the test labels and not

\varphi

. CVaR 5% is the mean coverage of the worst five per cent of classes and falls monotonically with the minimum, so the tail moves as a whole.

Phase	Content	Configs	Median
C	baseline comparison	$3$	$481$ s
D	ImageNet-C	$90$	$21$ s
E	additional backbones	$30$	$21$ s
F	evaluation-depth sweep	$25$	$235$ s
G	long-tailed datasets	$20$	$63$ s
H	held-out fraction sweep	$25$	$131$ s

Appendix

Table S19: Wall-clock cost per configuration, one CPU core, measured from the run reports. Phase D and E are fast because their evaluation slices are small; phase F is slow because each of its five depth caps is run at ten splits.

Quantity	Value
Shrinkage grid $\Lambda$	$0,\ .05,\ .1,\ .2,\ .3,\ .5,\ .7,\ 1$
Ridge penalty	$1.0$ , fixed, never tuned
Descriptor scaling	standardised on fit classes
$n_{\mathrm{cal}}$	$25$ , pre-registered
Target size $T$	avg. size of $\hat{q}$
$\alpha$	$0.10$ unless stated

Appendix

Table S20: Every hyperparameter and protocol constant. Values are the code’s defaults, which are what produced every number in the paper; the sweeps state their own departures. The ridge penalty is fixed rather than cross-validated, because the fit has at most

K

points and nine features and we would rather report a fixed choice than a tuned one.

Statistic	Marginal	Pcc	Oracle	$\Delta$ Pcc
Worst class	$0.6579$	$0.6912$	$0.7922$	$+0.0333$
$5$ th percentile	$0.7646$	$0.7662$	$0.8421$	$+0.0016$
$10$ th percentile	$0.7992$	$0.8155$	$0.8571$	$+0.0163$
CVaR 5%	$0.7313$	$0.7381$	$0.8230$	$+0.0068$
CVaR 10%	$0.7565$	$0.7630$	$0.8387$	$+0.0066$
Mean over classes	$0.9039$	$0.8998$	$0.8812$	$-0.0041$

Appendix

Table S21: Lower-tail statistics of per-class coverage on the held-out classes, seed

0

of the primary configuration under the reuse variant, all arms at matched average set size. CVaR q is the mean coverage of the worst

q

of classes. This is a single split and therefore carries no interval: it is a diagnostic, and Tab. 1 is the five-split statement.

$\lambda$	$\\|\bar{\delta}-\bar{\delta}^{\star}\\|_{\infty}$	$\Delta\sigma$	$L_{\mathrm{eff}}$	$L_{\mathrm{med}}$	worst-class
$0.00$	$0.9421$	$+0.008$	$19.1$	$0.54$	$0.5825$
$0.05$	$0.9234$	$+0.011$	$10.3$	$0.49$	$0.6197$
$0.10$	$0.9006$	$+0.009$	$16.0$	$0.51$	$\mathbf{0.6405}$
$0.20$	$0.8416$	$-0.014$	$16.8$	$0.47$	$0.6136$
$0.30$	$0.7776$	$-0.042$	$34.7$	$0.47$	$0.5700$
$0.50$	$0.6627$	$-0.099$	$16.2$	$0.43$	$0.4855$

Appendix

Table S22: Both sides of Eq. 7 on the held-out label space, five splits at the primary configuration.

\Delta\sigma

is the differential size-matching shift. The

\lambda=0

and

\lambda=0.1

rows reproduce Tab. 1 : their difference is

+0.0580

. The labelled label space, where

\hat{\lambda}

is selected, behaves the same way: the distance is smallest at

\lambda=0.7

and worst-class coverage peaks at

0.1

Conformal Prediction

Conformal prediction wraps predictive models with calibrated sets or intervals that provide statistical coverage guarantees.

See all 352 papers

Jul 20, 2026

The Label Complexity of Useful Class-Conditional Prediction Sets under Distribution Shift

Prediction sets can make deployed classifiers safer by returning several plausible labels when a single prediction is uncertain. Their value depends on classwise reliability: average coverage can meet its target while rare or difficult classes fail repeatedly. This concern is sharper after distribution shift, when calibration labels come from a source environment but reliability is needed on the target. We ask what labeled source data and unlabeled target inputs reveal about class-conditional prediction sets, and when target labels are necessary. Under unrestricted joint shift, two target laws can produce the same observable data while requiring different classwise thresholds; any label-free rule covering both must enlarge its sets on one law. We give a labeled target audit that estimates the missing quantiles with a simultaneous guarantee. Probability-scale error is invariant to increasing score transformations, and threshold recovery follows under local regularity. At fixed confidence, achieving threshold tolerance

\varepsilon

with fixed, nonadaptive class-stratified labeled pairs has total complexity

Θ(K\varepsilon^{-2}\log K)

, or

Θ(\varepsilon^{-2}\log K)

labels per class under equal allocation. Class imbalance creates a separate acquisition cost; for foreground class probabilities of order

1/K

, the mixed-stream label complexity is also

Θ(K\varepsilon^{-2}\log K)

at fixed confidence. Experiments on action-recognition and image shifts show that marginal coverage can conceal severe class failures and that source classwise calibration depends on the shift. The results connect the information available at deployment to the target labels needed for useful class-conditional prediction.

Weijia Han, Lisha Qu, Tianxin Zhou +2

University of Washington

Distribution Shift Robustness Conformal Prediction

Ask PDF

Jun 26, 2026

Conformal Prediction with Macro-Coverage Guarantees

Prediction sets should have high coverage to be useful, but some coverage notions are more practically relevant than others. In the classification setting, class-conditional coverage requires that the prediction set (i.e., the set of candidate labels for a new test point) must achieve the target accuracy level within each class, which may be challenging to satisfy when many classes are rare and have few calibration points. At the other extreme, marginal coverage requires only that coverage holds on average over the distribution of all classes, which can lead to low-probability labels being essentially ignored. To find a middle ground, recent work has introduced macro-coverage, defined as the unweighted average of class-conditional coverages. Macro-coverage offers a compromise between marginal coverage and class-conditional coverage that is particularly appropriate for long-tailed settings. In this work, we show that label-weighted conformal prediction can be used to produce prediction sets with a finite-sample macro-coverage guarantee, and more generally a guarantee on a family of generalized macro-coverage objectives that aggregate coverage at the level of arbitrary class groupings and take a weighted average. We further characterize the form of the smallest prediction sets satisfying a given generalized macro-coverage objective and propose a corresponding conformal score function. We validate our theoretical results on two large-scale image classification datasets.

Aabesh Bhattacharyya, Tiffany Ding, Rina Foygel Barber

Department of Statistics, University of Chicago · Department of Statistics, UC Berkeley

Set Prediction Conformal Prediction

Ask PDF

May 21, 2025

Adaptive Cumulative Mass Calibration with Conformal Prediction

Reliable probability estimates by classifiers are essential in high-risk applications. In practice, however, predicted probabilities are often miscalibrated, and many existing post-hoc calibration methods typically lack guarantees that a specific notion of calibration is achieved after the correction procedure is applied. We introduce a set-based perspective on calibration through the notion of cumulative mass calibration and the corresponding error measures. We propose a new calibration procedure based on conformal prediction that forms cumulative probabilities with guaranteed marginal coverage. We introduce an adaptive temperature scaling algorithm, with the temperature tuned for each input to satisfy the conformal coverage constraint. As we show, this procedure can be efficiently implemented. Across image classification tasks, particularly in settings with many classes, our method improves newly introduced calibration error measures (CMCE and

α

-CMCE) and standard metrics (such as ECE, cw-ECE, MCE) over the existing baselines.

Daniil Kazantsev, Eric Moulines, Maxim Panov +2

Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates · École pour l’informatique et les techniques avancées (EPITA), Paris, France

Post-Hoc Calibration Conformal Prediction

Ask PDF

Abstract

Conformal Prediction

Explore similar work

The Label Complexity of Useful Class-Conditional Prediction Sets under Distribution Shift

Conformal Prediction with Macro-Coverage Guarantees

Adaptive Cumulative Mass Calibration with Conformal Prediction