Abstention Spends Its Budget on Local Label Variance More Than on Novelty: Auditing Conformal Calibration and Selective Prediction under Molecular Scaffold Shift
Selective prediction with split conformal calibration promises a post-hoc fix for molecular property prediction: abstain on the least reliable compounds, ideally out-of-distribution chemistry, and the retained set becomes trustworthy without retraining. We audit this recipe for a gradient-boosted fingerprint model on lipophilicity (regression) and BBBP (classification) under a Bemis-Murcko scaffold split, holding the fitted model and evaluation molecules fixed while varying only the calibration fold (18 replicates). (i) On lipophilicity, coverage depends on calibration provenance: calibrating on held-out training-scaffold molecules falls below nominal (PICP 0.841 at a 0.90 target), while novel-scaffold calibration recovers it (0.904); on BBBP, coverage stays above nominal in every regime. (ii) For split conformalized quantile regression (CQR) the conformal correction is a constant, so conformalization does not change what is abstained on. (iii) The abstention score (CQR width; effectively ensemble variance on BBBP) tracks local label variance more than chemical novelty: controlling for local label variance largely explains away the novelty association (partial Spearman 0.067 and 0.047). (iv) At a fixed assay budget, gating on interval width or ensemble variance never improves hit precision over ranking by point prediction, at the pre-specified operating point or across 20 exploratory settings; a plain novelty gate shows small, uncorrected gains at the smallest budgets. (v) In an exploratory analysis, exact conformal selection exceeds its false discovery rate target under training-scaffold calibration (0.303 at q=0.2) and is consistent with it under novel-scaffold calibration, contributing a data-driven batch size rather than a better ranking. Conformalization earns its keep through guarantees whose validity hinges on calibration provenance, not by choosing what to abstain on.
Figures & tables
Lipophilicity (regression; CQR)
Split / calibration
PICP
MPIW
RMSE
random (control)
0.905 [0.897, 0.912]
2.81
0.658
scaffold, A (train-side)
0.841 [0.831, 0.849]
2.67
0.784
scaffold, B (novel-scaffold)
0.904 [0.894, 0.913]
3.08
0.784
Table 1: Coverage and calibration under scaffold shift ( α=0.10 ; mean over 18 replicates, 95% bootstrap interval). Lipophilicity falls below the 0.90 target only under training-scaffold calibration (A). BBBP coverage is inflated by the non-empty-set rule of our APS implementation (unforced values in Appendix C ). Coverage depends on the calibration regime; ECE, Brier and NLL are properties of the fitted model and are shared across regimes.
Figure 1: Coverage depends on calibration provenance. (a) Realized vs. nominal coverage on lipophilicity under the scaffold split (9 replicates; the base quantile models are fixed at levels 0.05/0.95 and only the conformal level varies): training-scaffold calibration (A) falls below nominal at every level, novel-scaffold calibration (B) tracks it. (b) Coverage at α=0.10 by split and regime (95% bootstrap intervals); BBBP values include the non-empty-set rule.
Figure 2: The abstention score follows local label variance more than novelty (regime A). (a,b) Enrichment of the abstained set relative to random abstention (1.0) for molecular novelty, local label variance and realized error across abstention budgets; on BBBP (b) the score is effectively ensemble variance. (c) Raw and partial Spearman correlation of the score with molecular novelty and with local label variance; controlling for variance largely removes the novelty association, not vice versa. Error bars: 95% bootstrap intervals over replicates.
Policy (select top- B by point prediction among retained)
Precision
Paired uplift vs. naive
point prediction, no UQ ( naive )
0.615 [0.601, 0.628]
—
abstain: CQR width (conformal)
0.591 [0.575, 0.607]
−0.024 [-0.035, -0.015]
abstain: quantile width (uncalibrated)
0.591 [0.575, 0.607]
−0.024 [-0.035, -0.015]
abstain: shift-weighted CQR width
0.591 [0.575, 0.607]
−0.024 [-0.035, -0.015]
abstain: ensemble variance
0.590 [0.577, 0.603]
−0.025 [-0.034, -0.015]
abstain: novelty ( 1−max Tanimoto)
0.607 [0.593, 0.622]
−0.008 [-0.017, +0.001]
Table 2: Fixed-budget hit finding on lipophilicity (regime A, B=15% of eval , gates drop the top 20% by their score; mean and paired 95% bootstrap interval over 18 replicates; hit base rate 0.218 ). The three width rows are identical by construction (Remark 1 ).
Figure 3: Uncertainty gates never beat point-prediction ranking; an exploratory novelty gate helps only at the smallest budgets. (a) Paired precision uplift over naive at the primary operating point (95% bootstrap intervals; ranking by the conformal lower bound, −0.171 , is omitted for scale). (b–d) Mean paired uplift across assay budgets (rows) and abstained fractions (columns) for the CQR-width, novelty and ensemble-variance gates; bold values have 95% intervals that exclude zero (not corrected for multiple comparisons).
Calibration
q
FDR (mean FDP)
Empty
# sel. mean (median)
Prec. (pooled)
Top- k by μ^
Overlap
A (train-side)
0.1
0.298 [0.180, 0.448]
28%
16.1 ( 12.5 )
0.707
0.714
0.71
A (train-side)
0.2
0.303 [0.243, 0.355]
6%
44.8 ( 46.5 )
0.677
0.677
0.89
A (train-side)
0.3
0.358 [0.335, 0.380]
0%
91.8 ( 91.0 )
0.639
0.639
0.99
B (novel-scaffold)
0.1
0.139 [0.000, 0.306]
67%
0.7 ( 0.0 )
0.538
0.769
0.28
B (novel-scaffold)
0.2
0.165 [0.082, 0.252]
33%
15.9 ( 5.5 )
0.661
0.657
0.83
B (novel-scaffold)
0.3
0.302 [0.230, 0.368]
17%
72.2 ( 61.0 )
0.600
0.603
1.00
Table 3: Exact conformal selection [ 9 ] on lipophilicity with p -values from Eq. ( 1 ). FDR is the replicate mean of FDP with FDP :=0 for empty selections (95% bootstrap interval); precision is pooled over all selected compounds; “Top- k by μ^ ” is the pooled precision of point-prediction ranking at the same batch sizes; “Overlap” is the mean fraction of selected compounds among that top- k (18 replicates).
Figure 4: FDR control depends on calibration provenance. (a) Realized FDR of exact conformal selection against the target level q ; the dashed line is FDR =q . (b) Mean number of true hits selected. 95% bootstrap intervals over replicates.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Statistic (regime A)
Lipophilicity
BBBP
ρ (width, molecular novelty)
+0.184 [+0.173, +0.194]
+0.158 [+0.139, +0.176]
ρ (width, scaffold novelty)
+0.165 [+0.154, +0.175]
+0.190 [+0.174, +0.206]
ρ (width, local label variance)
+0.309 [+0.295, +0.323]
+0.528 [+0.514, +0.543]
ρ (width, ∣y−y^∣ or error)
+0.157 [+0.147, +0.167]
+0.377 [+0.363, +0.390]
partial ρ (width, novelty ∣ variance)
+0.067 [+0.055, +0.077]
+0.047 [+0.026, +0.068]
partial ρ (width, variance ∣ novelty)
+0.254 [+0.238, +0.270]
+0.497 [+0.483, +0.512]
Appendix
Table 4: Spearman and partial Spearman (rank-residualized) correlations among the abstention score (width), the per-molecule covariates and realized error in regime A (mean and 95% bootstrap interval over 18 replicates). For BBBP the score is effectively ensemble variance and the error is misclassification.
Coverage
Cov. (minority)
Empty sets
eval positive
Split / calibration
forced
unforced
forced
unforced
(unforced)
rate
random (control)
0.930
0.906
0.787
0.771
2.6%
0.76
scaffold, A (train-side)
0.968
0.945
0.909
0.891
2.3%
0.82
scaffold, B (novel-scaffold)
0.952
0.901
0.866
0.834
5.3%
0.82
Appendix
Table 5: BBBP coverage at α=0.10 with (“forced”) and without (“unforced”) the non-empty-set rule (mean over 18 replicates), the fraction of empty unforced sets, and the positive-class rate of eval .
Calibration
Quantile
Coverage
Cov. (minority)
Cov. (majority)
∣C∣
A (train-side)
marginal
0.968
0.909
0.981
1.17
A (train-side)
class-conditional
0.961
0.969
0.960
1.34
B (novel-scaffold)
marginal
0.951
0.865
0.970
1.10
B (novel-scaffold)
class-conditional
0.962
0.942
0.967
1.19
Appendix
Table 6: Marginal vs. class-conditional APS on BBBP (nominal 0.90; mean over 18 replicates; includes the non-empty-set rule). Class-conditional calibration raises minority coverage at the cost of larger sets. APS is randomized: the regime-A marginal values coincide with Table 1 , while the regime-B values use different random draws and differ from it in the third decimal.
Selective prediction lets models answer only when sufficiently reliable, trading coverage for lower error through abstention, referral, or extra evidence gathering.
Conformal prediction is being adopted in drug discovery to put an honest number on model reliability: pick an error rate alpha, and the method returns prediction sets containing the true label with probability at least 1 - alpha. We show this guarantee can be dangerous on imbalanced datasets. Across four datasets, standard (marginal) conformal prediction hits its global 90% coverage target while leaving the minority class badly exposed: realized minority coverage falls to 64.8% on blood-brain-barrier penetration and to 4.2% on clinical-trial toxicity, where the rare class is nearly abandoned. The failure is not tied to one model: a random forest, a graph network, and a frozen chemical language model all reproduce it (p < 0.001 in every case), with severity tracking baseline calibration on rare labels rather than architecture. A conservation identity explains the effect: the minority's shortfall equals the majority's surplus amplified by the imbalance ratio, predicting the measured gap to within one point and ordering severity across datasets. The failure survives realistic scaffold splits and a second conformal score, while aggregate accuracy and overall coverage stay reassuringly high, which is exactly why it is easy to miss. Class-conditional (Mondrian) conformal prediction closes the gap on every dataset, restoring minority coverage to target for a modest increase in prediction-set size. We localize the failures to generic molecular scaffolds - plain benzene and pyridine cores occurring in both classes - propose a one-number diagnostic, and show with a cost model that abstaining on affected compounds flips a screening campaign from net-negative to net-positive utility. Our contribution is demonstrating on real chemistry how severe and invisible this known conformal-theory gap becomes under imbalance, and laying out a practical protocol restoring per-class reliability.
Conformal prediction provides distribution-free uncertainty quantification under exchangeability. However, this assumption is violated by label shift, where the marginal distribution of labels changes while the conditional distribution of inputs given labels remains stable. Under such shifts, standard conformal procedures no longer maintain their intended coverage behavior. Existing approaches address this via importance weighting. They pair the reweighting with residual-based nonconformity scores that ignore predictive uncertainty. The resulting intervals have uniform width. Bayesian conformal methods produce adaptive intervals by leveraging predictive distributions. They evaluate conformity under the source predictive, which is misaligned with the target domain under label shift. We propose the Label-Shift-Adjusted Bayesian Score (LSA score), a nonconformity score derived from a posterior predictive tilting identity. This identity shows that the target predictive is an importance-weighted transformation of the source predictive. We use it to derive a direct correction to the Bayesian score. We evaluate the method on molecular property prediction under controlled label shift. The LSA score consistently yields shorter intervals than residual-based and source-based Bayesian scores. Coverage in the target domain remains comparable. Under stronger shift, all methods incur some coverage loss due to pseudo-label-based density-ratio estimation. The LSA score is defined for any source predictive with a tractable log-density. We instantiate it with Bayesian Ridge Regression, where the correction admits a closed form.
Hyeonsu Lee, Juyeon Kim, Erkhembayar Jadamba +2
MOGAM Institute for Biomedical Research, Korea · CROID Research and aSSIST University, Korea
Conformal prediction is a useful and versatile alternative to model calibration in machine learning classification. It replaces single-class prediction with prediction sets, guaranteeing that the a priori probability of the prediction sets containing the true class is larger than or equal to a pre-specified rate. The size and usefulness of the prediction sets relies heavily on the choice of the non-conformity score function. The scientific literature contains many examples of non-conformity score functions but there is an absence of studies examining their properties and effectiveness. In this paper, we give an overview of properties of non-conformity score functions. We give examples of non-conformity score functions in the existing literature and introduce original modifications. We introduce an original method of evaluating the prediction set sizes of conformal predictors and use it to provide a comparison between non-conformity score functions. We also examine efficacy of different non-conformity score functions for class-conditional conformal prediction in a setting with imbalanced classes.
Sol Erika Boman
Department of Medical Epidemiology and Biostatistics, Karolinska Institutet · Department of Molecular Medicine and Surgery, Karolinska Institutet