Conformal Tradeoffs: Operational Profiles Beyond Coverage

Petrus H. Zwart

stat.MEFeb 20, 2026

Conformal Tradeoffs: Operational Profiles Beyond Coverage

Authors: Petrus H. Zwart

Organizations: Center for Advanced Mathematics in Energy Research Applications, Berkeley Synchrotron Infrared Structural Biology Program, & Molecular Biophysics and Integrated Bioimaging Division, 1 Cyclotron Road, Berkeley, CA 94720, USA

Abstract

Conformal prediction gives exact finite-sample coverage guarantees under exchangeability, but deployed systems are judged by more than coverage alone. For a fixed calibrated rule reused over a finite operational window, stakeholders also care about deployment-facing quantities such as commitment frequency, deferral, and decisive error exposure. These are not determined by coverage: calibration choices with similar coverage can still induce materially different operational profiles. We study this characterization gap in a scoped setting: binary split conformal prediction under exchangeability with a fixed deployed rule. We introduce the Small-Sample Beta Correction (SSBC) which gives finite-sample coverage semantics for the deployed rule: it inverts the Beta/Beta--Binomial law governing calibration-conditional coverage to map a user request $(α^\star,δ)$ to the least conservative calibration grid point with calibration-conditional PAC semantics for the realized deployed rule. Calibrate-and-Audit then fixes the rule by calibration and uses an independent audit split to estimate the induced region--class label table, a reusable summary from which deployment-facing Key Performance Indicators (KPIs) follow by projection. Under this design, fixed operational rates admit exact finite-sample Binomial inference, while Beta--Binomial envelopes serve as practical predictive summaries for future windows. The induced partition also exposes regime boundaries, Pareto-relevant tradeoffs, and inverse-pricing questions for fixed downstream conventions. Simulations validate the SSBC semantics and compare audit-based summaries with leave-one-out planning proxies; molecular toxicity data provide an audit-based empirical example, and a solubility case study illustrates scenario planning once coverage semantics are fixed.

Figures & tables

Figure 1: Operational view of a calibrated conformal rule. Calibration fixes a region partition; auditing the resulting region–class table yields a reusable operational summary; sweeping calibration settings traces feasible trade-offs.

Figure 2: Thresholds induce a finite region partition. In the binary probability-normalized setting, the score support determines which regions can carry mass under a fixed pair of thresholds.

Figure 3: Deployment policies as projections on a fixed region structure. Different region-based policies act on the same calibrated partition but induce different reported outputs.

Figure 4: Operational rate envelopes and score geometry. (A) Singleton rate and (B) singleton error for two class prevalences and two class 1 generating distributions, with class 0 drawn from $\mathrm{Beta}(2,7)$ . Red rectangles denote the two-sample Beta–Binomial future-window summary and the dashed vertical line its center; blue and orange intervals show leave-one-out envelopes under two inflation levels. (C1–C4) Histograms of predicted class 1 probability by true class make explicit how score geometry shapes singleton mass, singleton error, and the asymmetry of the envelopes.

Operational quantity	Class	LOO PE $\mathcal{D}_{1}$	LOO 95% PI $\mathcal{D}_{1}$	PE $\mathcal{D}_{2}$	BB 95% PI $\mathcal{D}_{2}$
Singleton rate	Class 0	$0.662$	$[0.604,0.718]$	$0.651$	$[0.615,0.686]$
	Class 1	$0.130$	$[0.092,0.173]$	$0.117$	$[0.094,0.142]$
Doublet rate	Class 0	$0.163$	$[0.121,0.211]$	$0.201$	$[0.172,0.231]$
	Class 1	$0.045$	$[0.023,0.074]$	$0.031$	$[0.020,0.046]$
Wrong-singleton rate	Class 0	$0.073$	$[0.044,0.107]$	$0.070$	$[0.052,0.090]$
	Class 1	$0.012$	$[0.002,0.030]$	$0.011$	$[0.005,0.020]$

Table 1: Representative Tox21 endpoint: SR-MMP. Joint rates are normalized by the endpoint test-set size. The

\mathcal{D}_{1}

LOO columns provide planning summaries (Point Estimate PE and 95% Prediction Interval PI) from calibration data, while the

\mathcal{D}_{2}

columns report the independent audit reference and its predictive interval.

Figure 5: Solubility planning and inverse pricing. Left: attainable planning regimes induced by sweeping SSBC settings on a restricted deployment scenario. Right: cost-ratio regions for which a fixed downstream convention is decision-optimal relative to the conformal interface.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Data splits and window sizes
Symbol	Meaning	Scope / Remarks
$\mathcal{D}_{\mathrm{cal}}$ , $n_{\mathrm{cal}}$	Calibration dataset and size	Exchangeable sample for conformal thresholds.
$\mathcal{D}_{\mathrm{audit}}$ , $n_{\mathrm{audit}}$	Audit dataset and size	Exchangeable sample for region–class label and operational rate estimation.
$m$	Future window size	Number of future cases for realized rates and predictive envelopes.
Region–policy–audit
$R_{\tau}(x)$ , $R_{\tau(\theta)}$	Region map	Partition of score space; $R_{\tau}:\mathcal{X}\to\{0,1\}^{K}$ ; fixed by thresholds $\tau(\theta)$ .

Appendix

Table 2: Key notation used across calibration, SSBC, and operational analysis.

Figure 6: Semantic interpretation of nominal coverage requests under finite calibration. Each panel visualizes the effective calibration level $\alpha_{\mathrm{adj}}$ selected by SSBC as a function of the user-specified miscoverage level $\alpha$ and confidence level $\delta$ , for fixed $n_{\mathrm{cal}}$ . Color encodes the deployed level $\alpha_{\mathrm{adj}}$ , while contours indicate iso-semantic sets: distinct nominal requests that induce the same deployed calibration grid point. The feasibility boundary reflects the finite-sample constraint $\alpha\gtrsim 1-\delta^{1/n_{\mathrm{cal}}}$ .

$n_{\mathrm{cal}}$	Method	$u$	$\alpha_{\mathrm{grid}}$	Obs	Beta	BetaBinom	$\alpha_{\mathrm{cont}}$
50	None	5	0.0980	0.3963	0.4312	0.3964	–
50	SSBC	2	0.0392	0.0476	0.0338	0.0472	–
50	DKWM	1	0.0196	0.0096	0.0052	0.0095	$-0.0731$
75	None	7	0.0921	0.3454	0.3673	0.3464	–
75	SSBC	4	0.0526	0.0769	0.0504	0.0768	–
75	DKWM	1	0.0132	0.0016	0.0004	0.0017	$-0.0413$

Appendix

Table 3: Calibration-conditional coverage violation rates with theory. Target miscoverage

\alpha^{\star}=0.10

, confidence

\delta=0.10

\alpha_{\mathrm{grid}}

is the grid point selected on the conformal grid,

\alpha_{\mathrm{cont}}

is the requested value under the DKWM correction, and

m_{\mathrm{cal}}

is the calibration-window size. Results are based on

10^{6}

calibration draws and a finite deployment window of size

m_{\mathrm{infer}}=100

. The Obs column reports

\mathbb{P}(\widehat{C}_{m}<1-\alpha^{\star})

. Beta reports

\mathbb{P}(p_{\mathrm{cov}}<1-\alpha^{\star})

under

p_{\mathrm{cov}}\sim\mathrm{Beta}(k,u)

. BetaBinom reports

\mathbb{P}(\widehat{C}_{m}<1-\alpha^{\star})

under the induced Beta–Binomial law.

Endpoint	Total Samples	Positive Rate	Calib. Positives	Calib. Negatives
NR-AR	7265	4.3%	77	1739
NR-AR-LBD	6758	3.5%	59	1630
NR-AhR	6549	11.7%	192	1445
NR-Aromatase	5821	5.2%	75	1380
NR-ER	6193	12.8%	198	1350
NR-ER-LBD	6955	5.0%	87	1651

Appendix

Table 4: Tox21 dataset composition and effective average class-conditional calibration sizes under the experimental protocol.

Method	Mean Coverage	Violation Rate	Avg. Set Size	Singleton Rate
Standard Split Conformal	0.917	0.305	1.41	0.52
DKWM Correction	0.986	0.005	1.78	0.22
SSBC	0.951	0.068	1.54	0.40

Appendix

Table 5: Aggregated Tox21 coverage and set statistics. Results are averaged over twelve endpoints and 100 random splits.

Operational quantity	Class	$\mathcal{D}_{1}$	LOO 95% PI $\mathcal{D}_{1}$	$\mathcal{D}_{2}$	BB 95% PI $\mathcal{D}_{2}$
Singleton rate	Class 0	$0.163$	$[0.125,0.205]$	$0.133$	$[0.112,0.157]$
	Class 1	$0.026$	$[0.011,0.047]$	$0.029$	$[0.019,0.041]$
Doublet rate	Class 0	$0.796$	$[0.751,0.838]$	$0.818$	$[0.792,0.843]$
	Class 1	$0.015$	$[0.005,0.031]$	$0.020$	$[0.012,0.030]$
Wrong-singleton rate	Class 0	$0.086$	$[0.058,0.119]$	$0.060$	$[0.045,0.077]$
	Class 1	$0.002$	$[0.000,0.010]$	$0.001$	$[0.000,0.004]$

Appendix

Table 6: NR-AR endpoint. Operational performance summary for the NR-AR endpoint. All reported quantities are joint probabilities normalized by the total test-set size. Rows report the singleton rate, doublet rate, and wrong-singleton rate (

P(Y=c,\,|S|=1,\,\hat{y}\neq Y)

) by true class. "

\mathcal{D}_{1}

" and "LOO 95% PI

\mathcal{D}_{1}

" denote leave-one-out point estimates and their beta–binomial planning intervals computed on the calibration data. "

\mathcal{D}_{2}

" reports empirical rates on the audit set. "BB 95% PI

\mathcal{D}_{2}

" gives Beta–Binomial predictive summaries for the corresponding

\mathcal{D}_{2}

quantities over a future window of the same size.

	Loss-minimizing regime		High-decisiveness regime
$\alpha_{0}$	0.125		0.150
$\delta_{0}$	0.150		0.150
$\alpha_{1}$	0.050		0.150
$\delta_{1}$	0.075		0.100
	Rate	Interval	Rate	Interval
Loss rate	3	[0, 35]	12	[0, 58]

Appendix

Table 7: Two representative Pareto-optimal operating regimes for the solubility scenario, illustrating contrasting planning postures. Rates are expected counts per 1000 molecules, with planning envelopes in brackets.

Explore similar work

May 18, 2026

Conformal Prediction via Transported Beta Laws

Split conformal prediction provides finite-sample marginal coverage under exchangeability, but this guarantee averages over the random calibration sample. We study instead the law of the calibration-conditional coverage induced by a realized conformal threshold. In the continuous i.i.d. setting this law is exactly

Beta(k,n+1-k)

, so the usual marginal guarantee corresponds to its mean. We take this beta law as a finite-sample reference object and quantify departures from it using Wasserstein distances on

[0,1]

. The framework yields direct bounds on marginal coverage gaps and on bad-calibration probabilities, and separates different sources of non-i.i.d. behavior according to how they deform the beta reference: test-side shift acts through a transport map on the coverage scale, while calibration dependence changes the order-statistic law itself. We instantiate the framework in scale-shift, clustered, and stationary mixing settings, where the induced deformations can be characterized explicitly or through Berry-Esseen approximations. Simulations on dependent processes confirm that the first-order approximation tracks the empirical Wasserstein distance even at moderate sample sizes.

Thiago R. Ramos, Helton Graziadei, Luben M. C. Cabezas

Federal University of S˜ao Carlos · Federal University of S˜ao Carlos, University of S˜ao Paulo, Inria and Universit´e Grenoble Alpes

Distribution Shift Robustness Conformal Prediction

Ask PDF

May 1, 2026

Optimal Spatio-Temporal Decoupling for Bayesian Conformal Prediction

Online conformal prediction must balance fast adaptation to distribution shift against stable coverage: feedback-driven methods react quickly but become volatile, while strongly discounted Bayesian methods lag and inflate intervals at tight coverage. We introduce State-Adaptive Bayesian Conformal Prediction (SA-BCP), which forms the predictive quantile as a gated convex combination of long-term temporal inertia and local spatial evidence from a kernel density estimate, controlled by a single interpretable evidence threshold

K

. We establish three results: (i) asymptotic marginal validity of the resulting intervals up to a gate-controlled bias that vanishes as spatial evidence accumulates (exact under recurrent states); (ii) a closed-form expression for the MSE-optimal threshold,

K^*_{\mathrm{MSE}}=α(1-α)/M^{\mathcal{T}}

, trading the coverage-indicator (Bernoulli) variance against the temporal structural bias

M^{\mathcal{T}}

; and (iii) a rolling-origin procedure for selecting

K

online -- consistent under stationarity, with

O(\sqrt{T\log N})

regret against the best fixed

K

and, for a segmented variant, a sublinear dynamic-regret bound under sublinearly many (

B_T=o(T)

) threshold shifts. Across four financial-volatility and weather datasets, three target coverage levels, and eight baselines, SA-BCP attains at-or-above-nominal coverage in most settings while producing substantially sharper intervals -- up to roughly

3\times

lower Winkler score than discounted Bayesian CP at the tightest coverage -- and a coverage-matched audit confirms these efficiency gains are not an artifact of under-coverage. We disclose our principal limitation: a volatility-specialized CF-GARCH competitor remains more efficient on its home volatility-base series, though it does not transfer across domains.

Yu-Hsueh Fang, Chia-Yen Lee

Department of Information Management, National Taiwan University, No. 1, Sec. 4, Roosevelt Rd., Da’an Dist., Taipei, 106216, Taiwan.

Conformal Prediction Temporal Distribution Shift Regression Prediction Intervals

Ask PDF

Jul 18, 2026

Isotonic Conformal Prediction

A point prediction that is well calibrated on average can still be systematically biased conditional on its own value, undermining its use in downstream decision-making. We consider two objectives for reliable uncertainty quantification: self-calibration, requiring a point prediction to be unbiased conditional on its own value, and prediction-conditional validity, requiring a prediction interval to attain nominal coverage conditional on the prediction. Self-Calibrating Conformal Prediction (SC-CP) attains both objectives exactly in finite samples, but requires refitting its calibrator for every candidate outcome, which is computationally prohibitive for continuous outcomes. We propose Isotonic Conformal Prediction (ICP), a framework that decouples calibration from prediction-set construction by fitting a single isotonic recalibration map and constructing prediction intervals within strata of similar recalibrated predictions. Within this framework we develop two procedures. Split Isotonic Conformal Prediction (SICP) attains prediction-conditional validity in finite samples and self-calibration asymptotically, at the computational cost of split conformal prediction. Transductive Isotonic Conformal Prediction (TICP) attains both objectives exactly in finite samples through a per-test-point inner loop that avoids refitting the isotonic calibrator. On synthetic heteroscedastic regression problems and a real-world healthcare-utilization dataset, both procedures match the coverage of SC-CP at substantially lower computational cost.

Daniel Bensimon, Sean Xiang Yu, Eric D. Kolaczyk +1

Department of Mathematics and Statistics, McGill University; Mila · Eli Lilly and Company

Conformal Prediction Uncertainty Calibration Isotonic Regression Regression Prediction Intervals

Ask PDF