Calibration Count Reuse: Validity Does Not Determine Efficiency

Rudra Chopra

stat.MESep 21, 2026

Calibration Count Reuse: Validity Does Not Determine Efficiency

Authors: Rudra Chopra

Abstract

Calibration count reuse raises separate validity and efficiency questions. We give a validity criterion for general count-dependent nonconformity scores: transferring one count from another class to the scored class must not improve its conformity. A leave-self-out full conformal reference proves the criterion without requiring normalization or preservation of same-class score order. For a common separable transformation, universal exchangeable validity is equivalent to being nondecreasing in the count, provided $Kα\geq 1$ ; normalized multiplicative weights obey the complementary nonincreasing condition. Additive penalties are covered under the stated information restrictions. Efficiency has no parallel ordering: two iid constructions make the same valid rule improve or worsen expected size at unchanged coverage. An expanded 55-rule study finds no resolved advantage from selected live-count rules over uniform weights. Image studies identify undercoverage under iid resampling, including at numerical convergence. Separately, execution of the released Conf-OT pipeline on its DTD and Aircraft benchmark subsets produces near-nominal median coverage under fixed stratified counts. The native results are reported separately from the iid analyses, without treating a benchmark observation as a universal guarantee. The findings separate validity, classifier confidence, numerical convergence, and population-specific efficiency.

Figures & tables

Rows and true class	Base $(A,B)$	Counts $(5,4)$	Add candidate $B$
		Weights $(6,5)$	Weights $(6,6)$
Five calibration $A$ rows	$(5/2,1)$	$3/4$	$5/7$
Four calibration $B$ rows	$(1,4)$	$10/13$	$4/5$
Target $B$ row	$(1,3)$	$5/7$	$3/4$
Calibration scores above target		$9$	$4$
Decision for candidate $B$		Exclude	Include

Figure 1: The mechanism, in exact fractions. Displayed probabilities are for each row’s true label. The features, labels, and base scores never change; only the candidate’s count is added. Augmentation moves the target above all five class

A

scores while preserving its order relative to class

B

scores. This is a samplewise rank flip, not a claim of zero marginal coverage for this two class law. In particular, it does not invoke the later

K\alpha\geq 1

converse.

Law, $p=4/5$	Coverage, both (%)	$\mathbb{E}\|C_{0}\|$	$\mathbb{E}\|C_{f}\|$
Helpful	97.31564544	1.1463129088	1.0255169536
Harmful	99.99995904	0.9999995904	1.5905575936

Table 1: Exact iid efficiency witnesses at nominal

90\%

. Coverage is identical within each row, not equal between the two laws. Differences come exclusively from false labels. Values are exact rational probabilities, not empirical estimates.

Encoder	Data	$n/K=2$	$n/K=6$	$n/K=20$
CLIP B/32	CIFAR10	88.38 [86.78, 89.99]	89.30 [87.93, 90.66]	89.78 [88.50, 91.06]
CLIP B/32	Food101	88.53 [87.20, 89.87]	89.39 [88.11, 90.67]	89.78 [88.52, 91.03]
CLIP B/32	Aircraft	87.10 [85.72, 88.48]	88.87 [87.58, 90.16]	89.69 [88.43, 90.95]
CLIP L/14	CIFAR10	88.34 [86.75, 89.93]	89.32 [87.96, 90.68]	89.82 [88.54, 91.10]
CLIP L/14	Food101	88.22 [86.87, 89.58]	89.33 [88.05, 90.61]	89.77 [88.51, 91.02]
CLIP L/14	Aircraft	86.95 [85.56, 88.34]	88.76 [87.47, 90.06]	89.65 [88.39, 90.91]

Table 2: All

27

primary native temperature, one cycle conditions at nominal

90\%

. Values are mean coverage percentages with simultaneous

95\%

intervals from the fixed family of

648

endpoints. Each cell integrates all target roles in

2,048

independent bags. Interval endpoints are rounded outward. Eight upper bounds lie below

90\%

; all eight occur at

n=2K

Figure 2: Coverage at native model temperatures. Later analysis on the known pools at

n=2K

, with a fixed native temperature and no artificial score sharpening. Each point uses 1,024 independent target episodes; bars use the separate simultaneous allocation. This figure does not replace the frozen one cycle study. All nine pairs and both empirical and augmented limits are shown.

Base	Nominal (%)	Uniform	Selected	Frozen match	Lower/equal/higher
Raw	90	7.0563	6.6677	6.5475	12/4/11
Column	90	5.7211	5.7728	5.7141	2/19/6
Raw	95	10.4201	9.8754	9.6776	15/0/12
Column	95	8.1824	8.2324	8.1522	4/16/7

Table 3: Expanded search: equal-cell mean set sizes across 27 cache/budget cells. Each selected rule was chosen using only the separate fitting pool. The last column counts its evaluation size relative to uniform, not significant wins. The matching frozen rule is selected by the same independent choice, not by evaluation results.

		Nominal 90%		Nominal 95%
Data	Score	Unadapted	Conf-OT	Unadapted	Conf-OT
DTD	LAC	89.95	89.48	95.27	94.33
DTD	APS	90.25	89.89	95.45	94.62
DTD	RAPS	90.25	89.95	95.27	94.62
Aircraft	LAC	89.76	89.97	95.12	94.82
Aircraft	APS	89.91	89.85	94.97	95.29

Table 4: Completed native Conf-OT subset. Median coverage (%) across 20 released stratified splits per cell. Unadapted is the repository’s none branch. These are descriptive medians, not simultaneous confidence bounds. Full sizes and split ranges are in Appendix K .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Claim	Quantification	Evidence type
Count-transfer sufficiency	Every $n$ and admissible exchangeable law; fixed feature marks and maps satisfying ( 4 )	Leave-self-out reference containment, main text.
Common-map converses	Fixed $K,\alpha$ , map; $K\alpha\geq 1$ ; separable or normalized specialization	Exact balanced construction, main text; normalized continuous detail below.
Iid fixed-catalog collapse	Specified iid law and predictor at a fixed $n$ ; sequence sharpens for the vanishing limit	Binomial calculation, main text and Appendix C .
Noisy converged collapse	Fixed feature law, $K$ , $a$ , noise; score boost grows as stated	Deterministic quota lemma and a count tail, Appendix D .
Growing catalog limits	$K\to\infty$ , $n/K\to\lambda$ ; fixed contrast and bounded intensities	Concentration, contraction, and dependent quantile argument, Appendix E .
Native model findings	Conditional on each fixed image pool, under declared repeated sampling	Complete frozen or explicitly later confidence families, Appendix G .

Appendix

Table 5: Scope and supporting evidence for each result. Pointwise identities, distributional theorems, numerical enclosures, and confidence intervals have distinct roles.

Encoder	Data	$n/K$	Empirical limit	Augmented limit
CLIP B/32	CIFAR10	2	75.10 [69.60, 80.09]	88.77 [84.52, 92.23]
CLIP B/32	CIFAR10	6	85.64 [81.02, 89.55]	90.82 [86.88, 93.94]
CLIP B/32	CIFAR10	20	87.60 [83.20, 91.24]	89.26 [85.08, 92.64]
CLIP B/32	Food101	2	82.81 [77.89, 87.07]	90.14 [86.09, 93.37]
CLIP B/32	Food101	6	86.82 [82.32, 90.57]	89.75 [85.64, 93.05]
CLIP B/32	Food101	20	89.65 [85.53, 92.97]	90.04 [85.97, 93.29]

Appendix

Table 6: Every exact-limit condition at nominal 90% in the post-confirmation sensitivity. Each cell uses

1,024

independent target episodes. Coverage and simultaneous

95\%

intervals are percentages; all intervals use the separate

432

-endpoint allocation and round outward. This table does not replace the original frozen one-cycle results.

Encoder	Data	$n/K$	Empirical limit	Augmented limit
CLIP B/32	CIFAR10	2	85.55 [80.91, 89.47]	94.24 [90.92, 96.66]
CLIP B/32	CIFAR10	6	92.87 [89.28, 95.59]	95.21 [92.12, 97.39]
CLIP B/32	CIFAR10	20	93.46 [89.98, 96.05]	94.24 [90.92, 96.66]
CLIP B/32	Food101	2	90.33 [86.31, 93.54]	95.21 [92.12, 97.39]
CLIP B/32	Food101	6	92.97 [89.39, 95.67]	94.92 [91.76, 97.17]
CLIP B/32	Food101	20	94.63 [91.40, 96.95]	95.21 [92.12, 97.39]

Appendix

Table 7: Every exact-limit condition at nominal 95% in the post-confirmation sensitivity. Each cell uses

1,024

independent target episodes. Coverage and simultaneous

95\%

intervals are percentages; all intervals use the separate

432

-endpoint allocation and round outward. This table does not replace the original frozen one-cycle results.

Method	Coverage (%)	Mean labels	Catalog (%)
Unadapted LAC	89.88	7.037	10.06
Randomized APS	89.61	7.657	11.71
Randomized RAPS	89.63	8.049	12.05
Split ridge	90.12	23.778	37.74
Full conformal ridge	90.24	18.470	29.10
Empirical prior, $T=1$ $\dagger$	89.01	5.342	8.56

Appendix

Table 8: Complete prediction sets on the three confirmation datasets, native temperature and nominal

90\%

. Each value equally averages the

27

cache/sample-size cell means, with

128

complete episodes per cell. These descriptive mixed-task means are not an efficiency significance test. Catalog fraction is averaged separately.

\dagger

: no general coverage guarantee. All

13

methods use matched total calibration/fitting label allowances.

	Empirical count reuse		Full augmentation
$\gamma$	Coverage (%)	Mean labels	Coverage (%)	Mean labels
-1	91.13	6.349	90.26	5.853
-0.5	90.77	5.965	90.32	5.728
0	90.37	5.641	90.37	5.641
0.5	89.87	5.418	90.39	5.634
1	89.28	5.245	90.40	5.662

Appendix

Table 9: New known-pool directional study, column-normalized base, nominal

90\%

. Each entry equally averages all

27

cache/budget cell means, each using

128

full episodes. All exponents are shown; these are descriptive comparisons, not an efficiency significance test. Decreasing weights preserve marginal validity but are not smaller than the fixed uniform baseline here.

	Empirical count reuse		Full augmentation
$\gamma$	Coverage (%)	Mean labels	Coverage (%)	Mean labels
-1	91.08	7.455	90.26	7.036
-0.5	90.66	7.176	90.26	6.975
0	90.26	6.959	90.26	6.959
0.5	89.74	6.786	90.27	6.979
1	89.21	6.650	90.29	7.031

Appendix

Table 10: New known-pool directional study, raw base, nominal

90\%

. Each entry equally averages all

27

cache/budget cell means, each using

128

Target subset	$N$	Empirical (%)	Augmented (%)
All	1024	64.26 [58.05, 70.15]	88.87 [84.44, 92.44]
Correct	972	67.28 [61.00, 73.17]	91.77 [87.68, 94.89]
Correct, score $\geq.9$	775	76.52 [69.97, 82.30]	97.16 [93.96, 98.97]
Correct, score $\geq.99$	198	91.92 [81.49, 97.60]	100.00 [94.85, 100.00]

Appendix

Table 11: Retrospective target strata for CLIP L/14, CIFAR10,

n=20

. Clopper–Pearson intervals ( Clopper and Pearson, 1934 ) use the separate 864-cell allocation, so the all-target interval differs from the original convergence interval. Conditional coverage is diagnostic, not a promised guarantee; all four strata and every other cell remain in the supplied CSV. Endpoints are rounded outward and verified by exact binomial tails. In the highest confidence stratum, the point estimate is above nominal; its 198 targets and wide interval do not establish either a deficit or nominal equivalence.

Check	Count
Exact candidate membership comparisons	6,480
Guard equals ordinary union augmented	2,160
Guarded whole-bag guarantee checks	108
Exact converse roles	2,100
Exact quota-constant substitutions	60
Explicit quantifier boundary fixtures	3

Appendix

Table 12: Internal computational checks. All declared fixtures passed.

Data	Score	Method	Median	Mean	Range	Size
DTD	LAC	Unadapted	89.95	89.96	88.06–91.37	10.866
DTD	LAC	Conf-OT	89.48	89.83	88.42–91.61	8.632
DTD	APS	Unadapted	90.25	90.15	88.53–92.43	12.523
DTD	APS	Conf-OT	89.89	89.89	88.18–91.84	10.026
DTD	RAPS	Unadapted	90.25	90.11	88.30–92.20	12.243
DTD	RAPS	Conf-OT	89.95	89.93	88.18–91.84	9.869

Appendix

Table 13: All native benchmark cells at nominal 90%. Coverage entries are percentages; the range is the minimum and maximum observed split value, not a confidence interval. Size is the median split mean number of labels. Each row has 20 repetitions.

Data	Score	Method	Median	Mean	Range	Size
DTD	LAC	Unadapted	95.27	95.20	92.20–96.57	17.291
DTD	LAC	Conf-OT	94.33	94.65	93.38–96.81	12.471
DTD	APS	Unadapted	95.45	95.18	91.73–96.22	18.823
DTD	APS	Conf-OT	94.62	94.78	93.03–96.22	13.752
DTD	RAPS	Unadapted	95.27	95.19	91.96–96.45	17.599
DTD	RAPS	Conf-OT	94.62	94.65	93.14–96.34	13.522

Appendix

Table 14: All native benchmark cells at nominal 95%. Coverage entries are percentages; the range is the minimum and maximum observed split value, not a confidence interval. Size is the median split mean number of labels. Each row has 20 repetitions.

Conformal Prediction

Conformal prediction wraps predictive models with calibrated sets or intervals that provide statistical coverage guarantees.

See all 352 papers

Aug 23, 2026

Replicable Conformal Prediction

Two analysts who calibrate the same predictive model on independent samples will deploy different prediction sets every time, because the calibration threshold inherits the randomness of the data. Wherever deployments must be audited, cached, or approved across sites, this instability is costly: no one can verify that two calibrations produced the same object. We ask two questions: when can independent calibrations yield the identical classifier, and what must that agreement cost? Perfect agreement is impossible, since a procedure that almost always returns one fixed answer cannot remain valid for every distribution, and exact agreement through shared randomness forces the procedure to ignore its data. Sharing a single random seed and rounding the calibrated threshold up to a coarse shared grid resolves the tension: the deployed classifier becomes identical across analysts with any desired probability, coverage guarantees survive, and the price is a quantified increase in set size and calibration data. Matching lower bounds show that no threshold method can pay less, and the method's one tuning constant vanishes asymptotically. Without any shared seed, a fixed grid still confines all analysts to two adjacent classifiers, and no method does better. Replicability also blocks gaming: selecting the most favorable of many recalibrations barely moves a replicable classifier, while the same selection silently undercovers standard conformal prediction. Experiments on real ImageNet outputs, a four-hospital site split, and four language-model families match the theory, including the measured sample-cost frontier.

Marios Papamichalis, Regina Ruane, Theofanis Papamichalis

Human Nature Lab, Yale University, New Haven, CT 06511, USA. · Department of Statistics and Data Science, The Wharton School, University of Pennsylvania, Philadelphia, PA, USA. · Department of Economics, Yale University, New Haven, CT, USA.

Conformal Prediction ML Reproducibility Set Prediction

Ask PDF

May 14, 2026

On the Burden of Achieving Fairness in Conformal Prediction

Conformal prediction is often calibrated with a single pooled threshold, but this can hide cross-group heterogeneity in score distributions and distort group-wise coverage. We study this phenomenon through the population score distributions underlying split conformal calibration. First, we derive a conservation law and lower bound showing that pooled calibration incurs irreducible group-wise coverage distortion at a scale set by cross-group quantile heterogeneity. Second, we demonstrate that the two leading fairness definitions for conformal prediction, Equalized Coverage and Equalized Set Size, are fundamentally in tension. Third, we quantify the cost of moving between policies which treat groups separately or pool them. Experiments on synthetic and real data confirm the same bidirectional trade-off after finite-sample calibration. Our results show that, for the policy families studied here, calibration choice does not remove cross-group heterogeneity; it determines whether the resulting distortion appears in the coverage or size dimension, providing a principled lens for analyzing fairness-oriented calibration choices in practice.

Ziang Gao, Pengqi Liu, Archer Yi Yang +3

McGill University Montreal, Canada · TD Insurance Montreal, Canada · Layer 6 AI Toronto, Canada

Fair Conformal Prediction Conformal Prediction Group Fairness

Ask PDF

Feb 20, 2026

Conformal Tradeoffs: Operational Profiles Beyond Coverage

Conformal prediction gives exact finite-sample coverage guarantees under exchangeability, but deployed systems are judged by more than coverage alone. For a fixed calibrated rule reused over a finite operational window, stakeholders also care about deployment-facing quantities such as commitment frequency, deferral, and decisive error exposure. These are not determined by coverage: calibration choices with similar coverage can still induce materially different operational profiles. We study this characterization gap in a scoped setting: binary split conformal prediction under exchangeability with a fixed deployed rule. We introduce the Small-Sample Beta Correction (SSBC) which gives finite-sample coverage semantics for the deployed rule: it inverts the Beta/Beta--Binomial law governing calibration-conditional coverage to map a user request

(α^\star,δ)

to the least conservative calibration grid point with calibration-conditional PAC semantics for the realized deployed rule. Calibrate-and-Audit then fixes the rule by calibration and uses an independent audit split to estimate the induced region--class label table, a reusable summary from which deployment-facing Key Performance Indicators (KPIs) follow by projection. Under this design, fixed operational rates admit exact finite-sample Binomial inference, while Beta--Binomial envelopes serve as practical predictive summaries for future windows. The induced partition also exposes regime boundaries, Pareto-relevant tradeoffs, and inverse-pricing questions for fixed downstream conventions. Simulations validate the SSBC semantics and compare audit-based summaries with leave-one-out planning proxies; molecular toxicity data provide an audit-based empirical example, and a solubility case study illustrates scenario planning once coverage semantics are fixed.

Petrus H. Zwart

Center for Advanced Mathematics in Energy Research Applications, Berkeley Synchrotron Infrared Structural Biology Program, & Molecular Biophysics and Integrated Bioimaging Division, 1 Cyclotron Road, Berkeley, CA 94720, USA

Conformal Prediction Selective Prediction

Ask PDF

Abstract

Conformal Prediction

Explore similar work

Replicable Conformal Prediction

On the Burden of Achieving Fairness in Conformal Prediction

Conformal Tradeoffs: Operational Profiles Beyond Coverage