Abstention Spends Its Budget on Local Label Variance More Than on Novelty: Auditing Conformal Calibration and Selective Prediction under Molecular Scaffold Shift

Arnav Srivastava; Krishna Kassetty

cs.LGNov 1, 2026

Abstention Spends Its Budget on Local Label Variance More Than on Novelty: Auditing Conformal Calibration and Selective Prediction under Molecular Scaffold Shift

Authors: Arnav Srivastava, Krishna Kassetty

Organizations: New York University · PhonePe

Abstract

Selective prediction with split conformal calibration promises a post-hoc fix for molecular property prediction: abstain on the least reliable compounds, ideally out-of-distribution chemistry, and the retained set becomes trustworthy without retraining. We audit this recipe for a gradient-boosted fingerprint model on lipophilicity (regression) and BBBP (classification) under a Bemis-Murcko scaffold split, holding the fitted model and evaluation molecules fixed while varying only the calibration fold (18 replicates). (i) On lipophilicity, coverage depends on calibration provenance: calibrating on held-out training-scaffold molecules falls below nominal (PICP 0.841 at a 0.90 target), while novel-scaffold calibration recovers it (0.904); on BBBP, coverage stays above nominal in every regime. (ii) For split conformalized quantile regression (CQR) the conformal correction is a constant, so conformalization does not change what is abstained on. (iii) The abstention score (CQR width; effectively ensemble variance on BBBP) tracks local label variance more than chemical novelty: controlling for local label variance largely explains away the novelty association (partial Spearman 0.067 and 0.047). (iv) At a fixed assay budget, gating on interval width or ensemble variance never improves hit precision over ranking by point prediction, at the pre-specified operating point or across 20 exploratory settings; a plain novelty gate shows small, uncorrected gains at the smallest budgets. (v) In an exploratory analysis, exact conformal selection exceeds its false discovery rate target under training-scaffold calibration (0.303 at q=0.2) and is consistent with it under novel-scaffold calibration, contributing a data-driven batch size rather than a better ranking. Conformalization earns its keep through guarantees whose validity hinges on calibration provenance, not by choosing what to abstain on.

Figures & tables

Lipophilicity (regression; CQR)
Split / calibration	PICP	MPIW	RMSE
random (control)	${0.905}$ [0.897, 0.912]	$2.81$	$0.658$
scaffold, A (train-side)	$\mathbf{0.841}$ [0.831, 0.849]	$2.67$	$0.784$
scaffold, B (novel-scaffold)	${0.904}$ [0.894, 0.913]	$3.08$	$0.784$

Table 1: Coverage and calibration under scaffold shift (

\alpha{=}0.10

; mean over 18 replicates, 95% bootstrap interval). Lipophilicity falls below the

0.90

target only under training-scaffold calibration (A). BBBP coverage is inflated by the non-empty-set rule of our APS implementation (unforced values in Appendix C ). Coverage depends on the calibration regime; ECE, Brier and NLL are properties of the fitted model and are shared across regimes.

Figure 1: Coverage depends on calibration provenance. (a) Realized vs. nominal coverage on lipophilicity under the scaffold split (9 replicates; the base quantile models are fixed at levels

0.05/0.95

and only the conformal level varies): training-scaffold calibration (A) falls below nominal at every level, novel-scaffold calibration (B) tracks it. (b) Coverage at

\alpha{=}0.10

by split and regime (95% bootstrap intervals); BBBP values include the non-empty-set rule.

Figure 2: The abstention score follows local label variance more than novelty (regime A). (a,b) Enrichment of the abstained set relative to random abstention (1.0) for molecular novelty, local label variance and realized error across abstention budgets; on BBBP (b) the score is effectively ensemble variance. (c) Raw and partial Spearman correlation of the score with molecular novelty and with local label variance; controlling for variance largely removes the novelty association, not vice versa. Error bars: 95% bootstrap intervals over replicates.

Policy (select top- $B$ by point prediction among retained)	Precision	Paired uplift vs. naive
point prediction, no UQ ( naive )	$0.615$ [0.601, 0.628]	—
abstain: CQR width (conformal)	$0.591$ [0.575, 0.607]	$-0.024$ [-0.035, -0.015]
abstain: quantile width (uncalibrated)	$0.591$ [0.575, 0.607]	$-0.024$ [-0.035, -0.015]
abstain: shift-weighted CQR width	$0.591$ [0.575, 0.607]	$-0.024$ [-0.035, -0.015]
abstain: ensemble variance	$0.590$ [0.577, 0.603]	$-0.025$ [-0.034, -0.015]
abstain: novelty ( $1-\max$ Tanimoto)	$0.607$ [0.593, 0.622]	$-0.008$ [-0.017, +0.001]

Table 2: Fixed-budget hit finding on lipophilicity (regime A,

B{=}15\%

of eval , gates drop the top

20\%

by their score; mean and paired 95% bootstrap interval over 18 replicates; hit base rate

0.218

). The three width rows are identical by construction (Remark 1 ).

Figure 3: Uncertainty gates never beat point-prediction ranking; an exploratory novelty gate helps only at the smallest budgets. (a) Paired precision uplift over naive at the primary operating point (95% bootstrap intervals; ranking by the conformal lower bound, $-0.171$ , is omitted for scale). (b–d) Mean paired uplift across assay budgets (rows) and abstained fractions (columns) for the CQR-width, novelty and ensemble-variance gates; bold values have 95% intervals that exclude zero (not corrected for multiple comparisons).

Calibration	$q$	FDR (mean FDP)	Empty	# sel. mean (median)	Prec. (pooled)	Top- $k$ by $\hat{\mu}$	Overlap
A (train-side)	$0.1$	$0.298$ [0.180, 0.448]	$28\%$	$16.1$ ( $12.5$ )	$0.707$	$0.714$	$0.71$
A (train-side)	$0.2$	$0.303$ [0.243, 0.355]	$6\%$	$44.8$ ( $46.5$ )	$0.677$	$0.677$	$0.89$
A (train-side)	$0.3$	$0.358$ [0.335, 0.380]	$0\%$	$91.8$ ( $91.0$ )	$0.639$	$0.639$	$0.99$
B (novel-scaffold)	$0.1$	$0.139$ [0.000, 0.306]	$67\%$	$0.7$ ( $0.0$ )	$0.538$	$0.769$	$0.28$
B (novel-scaffold)	$0.2$	$0.165$ [0.082, 0.252]	$33\%$	$15.9$ ( $5.5$ )	$0.661$	$0.657$	$0.83$
B (novel-scaffold)	$0.3$	$0.302$ [0.230, 0.368]	$17\%$	$72.2$ ( $61.0$ )	$0.600$	$0.603$	$1.00$

Table 3: Exact conformal selection [ 9 ] on lipophilicity with

p

-values from Eq. ( 1 ). FDR is the replicate mean of FDP with FDP

\,:=0

for empty selections (95% bootstrap interval); precision is pooled over all selected compounds; “Top-

k

\hat{\mu}

” is the pooled precision of point-prediction ranking at the same batch sizes; “Overlap” is the mean fraction of selected compounds among that top-

k

(18 replicates).

Figure 4: FDR control depends on calibration provenance. (a) Realized FDR of exact conformal selection against the target level

q

; the dashed line is FDR

=q

. (b) Mean number of true hits selected. 95% bootstrap intervals over replicates.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Statistic (regime A)	Lipophilicity	BBBP
$\rho$ (width, molecular novelty)	$+0.184$ [+0.173, +0.194]	$+0.158$ [+0.139, +0.176]
$\rho$ (width, scaffold novelty)	$+0.165$ [+0.154, +0.175]	$+0.190$ [+0.174, +0.206]
$\rho$ (width, local label variance)	$+0.309$ [+0.295, +0.323]	$+0.528$ [+0.514, +0.543]
$\rho$ (width, $\|y-\hat{y}\|$ or error)	$+0.157$ [+0.147, +0.167]	$+0.377$ [+0.363, +0.390]
partial $\rho$ (width, novelty $\mid$ variance)	$+0.067$ [+0.055, +0.077]	$+0.047$ [+0.026, +0.068]
partial $\rho$ (width, variance $\mid$ novelty)	$+0.254$ [+0.238, +0.270]	$+0.497$ [+0.483, +0.512]

Appendix

Table 4: Spearman and partial Spearman (rank-residualized) correlations among the abstention score (width), the per-molecule covariates and realized error in regime A (mean and 95% bootstrap interval over 18 replicates). For BBBP the score is effectively ensemble variance and the error is misclassification.

	Coverage		Cov. (minority)		Empty sets	eval positive
Split / calibration	forced	unforced	forced	unforced	(unforced)	rate
random (control)	$0.930$	$0.906$	$0.787$	$0.771$	$2.6\%$	$0.76$
scaffold, A (train-side)	$0.968$	$0.945$	$0.909$	$0.891$	$2.3\%$	$0.82$
scaffold, B (novel-scaffold)	$0.952$	$0.901$	$0.866$	$0.834$	$5.3\%$	$0.82$

Appendix

Table 5: BBBP coverage at

\alpha{=}0.10

with (“forced”) and without (“unforced”) the non-empty-set rule (mean over 18 replicates), the fraction of empty unforced sets, and the positive-class rate of eval .

Calibration	Quantile	Coverage	Cov. (minority)	Cov. (majority)	$\|\mathcal{C}\|$
A (train-side)	marginal	$0.968$	$0.909$	$0.981$	$1.17$
A (train-side)	class-conditional	$0.961$	$0.969$	$0.960$	$1.34$
B (novel-scaffold)	marginal	$0.951$	$0.865$	$0.970$	$1.10$
B (novel-scaffold)	class-conditional	$0.962$	$0.942$	$0.967$	$1.19$

Appendix

Table 6: Marginal vs. class-conditional APS on BBBP (nominal 0.90; mean over 18 replicates; includes the non-empty-set rule). Class-conditional calibration raises minority coverage at the cost of larger sets. APS is randomized: the regime-A marginal values coincide with Table 1 , while the regime-B values use different random draws and differ from it in the third decimal.

Selective Prediction

Selective prediction lets models answer only when sufficiently reliable, trading coverage for lower error through abstention, referral, or extra evidence gathering.

See all 299 papers

A Quiet Failure in Calibrated Virtual Screening: Marginal Conformal Prediction Under-Covers the Minority Class, and a Class-Conditional Fix Recovers It
Jul 7, 2026Muhammadjon Tursunbadalov, Mustafojon TursunbadalovClass-Imbalanced Learning Conformal Prediction
School of Science and Technology, Champions College Prep, United States
Split Conformal Prediction with Label-Shift-Adjusted Bayesian Scores
Sep 11, 2026Hyeonsu Lee, Juyeon Kim, Erkhembayar Jadamba +2Label Shift Conformal Prediction
MOGAM Institute for Biomedical Research, Korea · CROID Research and aSSIST University, Korea
Benchmarking non-conformity score functions in conformal prediction
May 24, 2026Sol Erika BomanSet Prediction Conformal Prediction
Department of Medical Epidemiology and Biostatistics, Karolinska Institutet · Department of Molecular Medicine and Surgery, Karolinska Institutet

Jul 7, 2026J/K move · Enter open · S save

A Quiet Failure in Calibrated Virtual Screening: Marginal Conformal Prediction Under-Covers the Minority Class, and a Class-Conditional Fix Recovers It

Muhammadjon Tursunbadalov, Mustafojon Tursunbadalov

School of Science and Technology, Champions College Prep, United States

Class-Imbalanced Learning Conformal Prediction Virtual Screening Drug Discovery

Open paper AskPDF arXiv

Conformal prediction is being adopted in drug discovery to put an honest number on model reliability: pick an error rate alpha, and the method returns prediction sets containing the true label with probability at least 1 - alpha. We show this guarantee can be dangerous on imbalanced datasets. Across four datasets, standard (marginal) conformal prediction hits its global 90% coverage target while leaving the minority class badly exposed: realized minority coverage falls to 64.8% on blood-brain-barrier penetration and to 4.2% on clinical-trial toxicity, where the rare class is nearly abandoned. The failure is not tied to one model: a random forest, a graph network, and a frozen chemical language model all reproduce it (p < 0.001 in every case), with severity tracking baseline calibration on rare labels rather than architecture. A conservation identity explains the effect: the minority's shortfall equals the majority's surplus amplified by the imbalance ratio, predicting the measured gap to within one point and ordering severity across datasets. The failure survives realistic scaffold splits and a second conformal score, while aggregate accuracy and overall coverage stay reassuringly high, which is exactly why it is easy to miss. Class-conditional (Mondrian) conformal prediction closes the gap on every dataset, restoring minority coverage to target for a modest increase in prediction-set size. We localize the failures to generic molecular scaffolds - plain benzene and pyridine cores occurring in both classes - propose a one-number diagnostic, and show with a cost model that abstaining on affected compounds flips a screening campaign from net-negative to net-positive utility. Our contribution is demonstrating on real chemistry how severe and invisible this known conformal-theory gap becomes under imbalance, and laying out a practical protocol restoring per-class reliability.

Abstract

Selective Prediction

Explore similar work

A Quiet Failure in Calibrated Virtual Screening: Marginal Conformal Prediction Under-Covers the Minority Class, and a Class-Conditional Fix Recovers It

Split Conformal Prediction with Label-Shift-Adjusted Bayesian Scores

Benchmarking non-conformity score functions in conformal prediction