Conformal Tradeoffs: Operational Profiles Beyond Coverage

Petrus H. Zwart

stat.MEFeb 20, 2026

Conformal Tradeoffs: Operational Profiles Beyond Coverage

Authors: Petrus H. Zwart

Organizations: Center for Advanced Mathematics in Energy Research Applications, Berkeley Synchrotron Infrared Structural Biology Program, & Molecular Biophysics and Integrated Bioimaging Division, 1 Cyclotron Road, Berkeley, CA 94720, USA

Abstract

Conformal prediction gives exact finite-sample coverage guarantees under exchangeability, but deployed systems are judged by more than coverage alone. For a fixed calibrated rule reused over a finite operational window, stakeholders also care about deployment-facing quantities such as commitment frequency, deferral, and decisive error exposure. These are not determined by coverage: calibration choices with similar coverage can still induce materially different operational profiles. We study this characterization gap in a scoped setting: binary split conformal prediction under exchangeability with a fixed deployed rule. We introduce the Small-Sample Beta Correction (SSBC) which gives finite-sample coverage semantics for the deployed rule: it inverts the Beta/Beta--Binomial law governing calibration-conditional coverage to map a user request $(α^\star,δ)$ to the least conservative calibration grid point with calibration-conditional PAC semantics for the realized deployed rule. Calibrate-and-Audit then fixes the rule by calibration and uses an independent audit split to estimate the induced region--class label table, a reusable summary from which deployment-facing Key Performance Indicators (KPIs) follow by projection. Under this design, fixed operational rates admit exact finite-sample Binomial inference, while Beta--Binomial envelopes serve as practical predictive summaries for future windows. The induced partition also exposes regime boundaries, Pareto-relevant tradeoffs, and inverse-pricing questions for fixed downstream conventions. Simulations validate the SSBC semantics and compare audit-based summaries with leave-one-out planning proxies; molecular toxicity data provide an audit-based empirical example, and a solubility case study illustrates scenario planning once coverage semantics are fixed.

Figures & tables

Figure 1: Operational view of a calibrated conformal rule. Calibration fixes a region partition; auditing the resulting region–class table yields a reusable operational summary; sweeping calibration settings traces feasible trade-offs.

Figure 2: Thresholds induce a finite region partition. In the binary probability-normalized setting, the score support determines which regions can carry mass under a fixed pair of thresholds.

Figure 3: Deployment policies as projections on a fixed region structure. Different region-based policies act on the same calibrated partition but induce different reported outputs.

Figure 4: Operational rate envelopes and score geometry. (A) Singleton rate and (B) singleton error for two class prevalences and two class 1 generating distributions, with class 0 drawn from $\mathrm{Beta}(2,7)$ . Red rectangles denote the two-sample Beta–Binomial future-window summary and the dashed vertical line its center; blue and orange intervals show leave-one-out envelopes under two inflation levels. (C1–C4) Histograms of predicted class 1 probability by true class make explicit how score geometry shapes singleton mass, singleton error, and the asymmetry of the envelopes.

Operational quantity	Class	LOO PE $\mathcal{D}_{1}$	LOO 95% PI $\mathcal{D}_{1}$	PE $\mathcal{D}_{2}$	BB 95% PI $\mathcal{D}_{2}$
Singleton rate	Class 0	$0.662$	$[0.604,0.718]$	$0.651$	$[0.615,0.686]$
	Class 1	$0.130$	$[0.092,0.173]$	$0.117$	$[0.094,0.142]$
Doublet rate	Class 0	$0.163$	$[0.121,0.211]$	$0.201$	$[0.172,0.231]$
	Class 1	$0.045$	$[0.023,0.074]$	$0.031$	$[0.020,0.046]$
Wrong-singleton rate	Class 0	$0.073$	$[0.044,0.107]$	$0.070$	$[0.052,0.090]$
	Class 1	$0.012$	$[0.002,0.030]$	$0.011$	$[0.005,0.020]$

Table 1: Representative Tox21 endpoint: SR-MMP. Joint rates are normalized by the endpoint test-set size. The

\mathcal{D}_{1}

LOO columns provide planning summaries (Point Estimate PE and 95% Prediction Interval PI) from calibration data, while the

\mathcal{D}_{2}

columns report the independent audit reference and its predictive interval.

Figure 5: Solubility planning and inverse pricing. Left: attainable planning regimes induced by sweeping SSBC settings on a restricted deployment scenario. Right: cost-ratio regions for which a fixed downstream convention is decision-optimal relative to the conformal interface.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Data splits and window sizes
Symbol	Meaning	Scope / Remarks
$\mathcal{D}_{\mathrm{cal}}$ , $n_{\mathrm{cal}}$	Calibration dataset and size	Exchangeable sample for conformal thresholds.
$\mathcal{D}_{\mathrm{audit}}$ , $n_{\mathrm{audit}}$	Audit dataset and size	Exchangeable sample for region–class label and operational rate estimation.
$m$	Future window size	Number of future cases for realized rates and predictive envelopes.
Region–policy–audit
$R_{\tau}(x)$ , $R_{\tau(\theta)}$	Region map	Partition of score space; $R_{\tau}:\mathcal{X}\to\{0,1\}^{K}$ ; fixed by thresholds $\tau(\theta)$ .

Appendix

Table 2: Key notation used across calibration, SSBC, and operational analysis.

Figure 6: Semantic interpretation of nominal coverage requests under finite calibration. Each panel visualizes the effective calibration level $\alpha_{\mathrm{adj}}$ selected by SSBC as a function of the user-specified miscoverage level $\alpha$ and confidence level $\delta$ , for fixed $n_{\mathrm{cal}}$ . Color encodes the deployed level $\alpha_{\mathrm{adj}}$ , while contours indicate iso-semantic sets: distinct nominal requests that induce the same deployed calibration grid point. The feasibility boundary reflects the finite-sample constraint $\alpha\gtrsim 1-\delta^{1/n_{\mathrm{cal}}}$ .

$n_{\mathrm{cal}}$	Method	$u$	$\alpha_{\mathrm{grid}}$	Obs	Beta	BetaBinom	$\alpha_{\mathrm{cont}}$
50	None	5	0.0980	0.3963	0.4312	0.3964	–
50	SSBC	2	0.0392	0.0476	0.0338	0.0472	–
50	DKWM	1	0.0196	0.0096	0.0052	0.0095	$-0.0731$
75	None	7	0.0921	0.3454	0.3673	0.3464	–
75	SSBC	4	0.0526	0.0769	0.0504	0.0768	–
75	DKWM	1	0.0132	0.0016	0.0004	0.0017	$-0.0413$

Appendix

Table 3: Calibration-conditional coverage violation rates with theory. Target miscoverage

\alpha^{\star}=0.10

, confidence

\delta=0.10

\alpha_{\mathrm{grid}}

is the grid point selected on the conformal grid,

\alpha_{\mathrm{cont}}

is the requested value under the DKWM correction, and

m_{\mathrm{cal}}

is the calibration-window size. Results are based on

10^{6}

calibration draws and a finite deployment window of size

m_{\mathrm{infer}}=100

. The Obs column reports

\mathbb{P}(\widehat{C}_{m}<1-\alpha^{\star})

. Beta reports

\mathbb{P}(p_{\mathrm{cov}}<1-\alpha^{\star})

under

p_{\mathrm{cov}}\sim\mathrm{Beta}(k,u)

. BetaBinom reports

\mathbb{P}(\widehat{C}_{m}<1-\alpha^{\star})

under the induced Beta–Binomial law.

Endpoint	Total Samples	Positive Rate	Calib. Positives	Calib. Negatives
NR-AR	7265	4.3%	77	1739
NR-AR-LBD	6758	3.5%	59	1630
NR-AhR	6549	11.7%	192	1445
NR-Aromatase	5821	5.2%	75	1380
NR-ER	6193	12.8%	198	1350
NR-ER-LBD	6955	5.0%	87	1651

Appendix

Table 4: Tox21 dataset composition and effective average class-conditional calibration sizes under the experimental protocol.

Method	Mean Coverage	Violation Rate	Avg. Set Size	Singleton Rate
Standard Split Conformal	0.917	0.305	1.41	0.52
DKWM Correction	0.986	0.005	1.78	0.22
SSBC	0.951	0.068	1.54	0.40

Appendix

Table 5: Aggregated Tox21 coverage and set statistics. Results are averaged over twelve endpoints and 100 random splits.

Operational quantity	Class	$\mathcal{D}_{1}$	LOO 95% PI $\mathcal{D}_{1}$	$\mathcal{D}_{2}$	BB 95% PI $\mathcal{D}_{2}$
Singleton rate	Class 0	$0.163$	$[0.125,0.205]$	$0.133$	$[0.112,0.157]$
	Class 1	$0.026$	$[0.011,0.047]$	$0.029$	$[0.019,0.041]$
Doublet rate	Class 0	$0.796$	$[0.751,0.838]$	$0.818$	$[0.792,0.843]$
	Class 1	$0.015$	$[0.005,0.031]$	$0.020$	$[0.012,0.030]$
Wrong-singleton rate	Class 0	$0.086$	$[0.058,0.119]$	$0.060$	$[0.045,0.077]$
	Class 1	$0.002$	$[0.000,0.010]$	$0.001$	$[0.000,0.004]$

Appendix

Table 6: NR-AR endpoint. Operational performance summary for the NR-AR endpoint. All reported quantities are joint probabilities normalized by the total test-set size. Rows report the singleton rate, doublet rate, and wrong-singleton rate (

P(Y=c,\,|S|=1,\,\hat{y}\neq Y)

) by true class. "

\mathcal{D}_{1}

" and "LOO 95% PI

\mathcal{D}_{1}

" denote leave-one-out point estimates and their beta–binomial planning intervals computed on the calibration data. "

\mathcal{D}_{2}

" reports empirical rates on the audit set. "BB 95% PI

\mathcal{D}_{2}

" gives Beta–Binomial predictive summaries for the corresponding

\mathcal{D}_{2}

quantities over a future window of the same size.

	Loss-minimizing regime		High-decisiveness regime
$\alpha_{0}$	0.125		0.150
$\delta_{0}$	0.150		0.150
$\alpha_{1}$	0.050		0.150
$\delta_{1}$	0.075		0.100
	Rate	Interval	Rate	Interval
Loss rate	3	[0, 35]	12	[0, 58]

Appendix

Table 7: Two representative Pareto-optimal operating regimes for the solubility scenario, illustrating contrasting planning postures. Rates are expected counts per 1000 molecules, with planning envelopes in brackets.

Explore similar work

Conformal Prediction via Transported Beta Laws
May 18, 2026Thiago R. Ramos, Helton Graziadei, Luben M. C. CabezasDistribution Shift Robustness Conformal Prediction
Federal University of S˜ao Carlos · Federal University of S˜ao Carlos, University of S˜ao Paulo, Inria and Universit´e Grenoble Alpes
Optimal Spatio-Temporal Decoupling for Bayesian Conformal Prediction
May 1, 2026Yu-Hsueh Fang, Chia-Yen LeeConformal Prediction Temporal Distribution Shift
Department of Information Management, National Taiwan University, No. 1, Sec. 4, Roosevelt Rd., Da’an Dist., Taipei, 106216, Taiwan.
Isotonic Conformal Prediction
Jul 18, 2026Daniel Bensimon, Sean Xiang Yu, Eric D. Kolaczyk +1Conformal Prediction Uncertainty Calibration
Department of Mathematics and Statistics, McGill University; Mila · Eli Lilly and Company

May 18, 2026J/K move · Enter open · S save

Conformal Prediction via Transported Beta Laws

Thiago R. Ramos, Helton Graziadei, Luben M. C. Cabezas

Federal University of S˜ao Carlos · Federal University of S˜ao Carlos, University of S˜ao Paulo, Inria and Universit´e Grenoble Alpes

Distribution Shift Robustness Conformal Prediction

Open paper AskPDF arXiv

Split conformal prediction provides finite-sample marginal coverage under exchangeability, but this guarantee averages over the random calibration sample. We study instead the law of the calibration-conditional coverage induced by a realized conformal threshold. In the continuous i.i.d. setting this law is exactly

Beta(k,n+1-k)

, so the usual marginal guarantee corresponds to its mean. We take this beta law as a finite-sample reference object and quantify departures from it using Wasserstein distances on

[0,1]

. The framework yields direct bounds on marginal coverage gaps and on bad-calibration probabilities, and separates different sources of non-i.i.d. behavior according to how they deform the beta reference: test-side shift acts through a transport map on the coverage scale, while calibration dependence changes the order-statistic law itself. We instantiate the framework in scale-shift, clustered, and stationary mixing settings, where the induced deformations can be characterized explicitly or through Berry-Esseen approximations. Simulations on dependent processes confirm that the first-order approximation tracks the empirical Wasserstein distance even at moderate sample sizes.

Abstract

Explore similar work

Conformal Prediction via Transported Beta Laws

Optimal Spatio-Temporal Decoupling for Bayesian Conformal Prediction

Isotonic Conformal Prediction