Optimized Conformal Selection: Powerful Selective Inference After Data-Dependent Score Construction

Tian Bai; Ying Jin

stat.MENov 27, 2024

Optimized Conformal Selection: Powerful Selective Inference After Data-Dependent Score Construction

Authors: Tian Bai, Ying Jin

Organizations: Department of Mathematics and Statistics, McGill University · Data Science Initiative and Department of Health Care Policy, Harvard University

Abstract

We study data-driven score construction in conformal selection, which uses conformal p-values and multiple testing to select "interesting" instances with large unobserved labels from a pool of unlabeled data, while controlling the FDR in finite sample. While existing solutions require the score function to be constructed independent of the calibration and test data, it is desirable to select or construct powerful conformity score in a data-driven manner and to leverage all labeled data for score training when presented with many score choices and limited labeled data. We propose Optimized Conformal Selection (OptCS), a general framework that allows valid multiple testing after flexible data-driven score construction. We introduce general conditions under which OptCS constructs valid conformal p-values despite substantial data reuse and handles complex p-value dependencies to maintain finite-sample FDR control. We instantiate this general recipe to propose three FDR-controlling procedures, each optimizing the models differently: (i) selecting a powerful score function among multiple candidates, (ii) training the score with all labeled data without sample splitting, and (iii) combining full-sample training and model selection. We demonstrate the efficacy of our methods via simulation studies and real applications in drug discovery and large language models.

Figures & tables

Figure 1: Overview of OptCS. (Optional) Split labeled data into preparatory and calibration sets. 1. P-value : construct p-values $\{p_{j}\}$ based on conformity scores from any data-dependent score construction process $\mathcal{V}(\cdot)$ obeying a mild permutation-equivariance condition. 2. Threshold : calibrate individual thresholds $\widehat{R}_{j}$ obeying a mild permutation-invariance condition. 3. Multiple testing : combine $\{p_{j}\}$ and $\{\widehat{R}_{j}\}$ to produce a selection set $\mathcal{S}$ with finite-sample, distribution-free FDR control.

Figure 2: Preview of numerical results. Left: Performance of OptCS-MSel which selects from pre-trained models. Middle: Performance of OptCS-Full which leverages full data for model fitting with a given model class. Right: Performance of OptCS-Full-MSel which combines model selection and full-sample training.

Method	Model selection	Full-data training	Training cost
SCS	no	no	1 fit
OptCS-MSel	yes	no	$K$ fits (if not pre-trained)
OptCS-Full	no	yes	$(n_{2}+m)$ fits
OptCS-Full-Non-LOO	no	yes	$1$ fit
OptCS-Full-MSel	yes	yes	$K(n_{2}+m)$ fits
OptCS-Full-MSel-Non-LOO	yes	yes	$K$ fits

Table 1: Comparison of OptCS procedures and baseline, where

K

is the number of model classes,

n_{2}

is the number of calibration data, and

m

is the number of test data.

Panel of Figure 3 — Figure 3: Visualization of a score-generating functional.

Figure 4: OptCS-MSel modifies the greedy approach by selecting a model $\widehat{k}_{j}$ in a way that is permutation invariant to calibration data and the $j$ -th test point, replacing SCS in the evaluation with a permutation-invariant $\mathcal{S}_{j}(\cdot)$ .

Figure 5: OptCS-Full-MSel uses all labeled data for model training (similar to OptCS-Full), model selection (similar to OptCS-MSel), and final multiple testing.

Figure 6: Realized FDR (a) and power (b) for varying nominal FDR levels under Liang’s settings for conformity score selection. Each column corresponds to one data generating process. The selection power differs significantly across individual conformity scores.

Figure 7: Realized FDR (a) and power (b) for varying nominal FDR levels under Jin’s settings. Each subplot corresponds to one data generating process. In these settings, the selection power with each individual conformity score is similar to each other.

Figure 8: Empirical FDR (a) and power (b) for OptCS-Full and SCS using SVR at various nominal FDR levels with

n=500

m=100

, averaged over

N=500

runs.

Figure 9: Empirical FDR (a) and power (b) for OptCS-Full-MSel and baselines in Liang’s settings with

n=500

and

m=100

, averaged over

N=500

runs.

Figure 10: Empirical FDR (a) and power (b) for OptCS-Full-MSel and baselines in Jin’s settings with

n=500

and

m=100

. Details are otherwise the same as Figure 9 .

Figure 11: Empirical FDR (a) and power (b) for five drug discovery tasks over

N=500

runs.

Figure 12: Empirical FDR (a) and power (b) of OptCS-Full and SCS for aligning large language models with various nominal levels, averaged over

N=500

runs.

Figure 13: Empirical FDR (setup 1 in (1a) and setup 2 in (2a)) and power (setup 1 in (1b) and setup 2 in (2b)) in LLM alignment under various nominal FDR levels.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	$X_{i}$	$\theta_{i}$	$\epsilon_{i}$
1	$\mathcal{N}(0,\mathbf{I}_{d})$	$\ind\{i\text{ mod }20=0\}$	$\sigma\cdot\mathcal{N}(0,1)$
2	$\mathcal{N}(0,\mathbf{I}_{d})$	$\ind\{i\text{ mod }20=0\}$	$\sigma\cdot t_{\nu}(1)$
3	$\mathcal{N}(0,\mathbf{I}_{d})$	$1/d$	$\sigma\cdot\mathcal{N}(0,1/d)$
4	$t_{\nu}(0,\mathbf{I}_{d})$	$\ind\{i\text{ mod }20=0\}$	$\sigma\cdot\mathcal{N}(0,1)$

Appendix

Table 2: Details of the four data generating processes used in Liang’s settings.

Setting	$X_{i}$	$\mu(\cdot)$	$\epsilon_{i}$
1	$\text{Unif}[-1,1]^{d}$	$x_{1}\ind\{x_{2}>0,x_{4}>0.5\}\cdot(0.25+x_{4})$ $+x_{1}\ind\{x_{2}\leq 0,x_{4}\leq-0.5\}\cdot(x_{4}-0.25)$	$\sigma\cdot\mathcal{N}(0,1)$
2	$\text{Unif}[-1,1]^{d}$	$2(x_{1}x_{2}+e^{x_{4}}-1)$	$\frac{3}{2}\sigma\cdot\mathcal{N}(0,1)$
3	$\text{Unif}[-1,1]^{d}$	$x_{1}\ind\{x_{2}>0,x_{4}>0.5\}\cdot(0.25+x_{4})$ $+x_{1}\ind\{x_{2}\leq 0,x_{4}\leq-0.5\}\cdot(x_{4}-0.25)$	$\sigma\cdot(5.5-\|\mu(x)\|)/2$
4	$\text{Unif}[-1,1]^{d}$	$2(x_{1}x_{2}+e^{x_{4}}-1)$	$\sigma\cdot(5.5-\|\mu(x)\|)/2$

Appendix

Table 3: Details of the four data generating processes used in Jin’s settings.

Figure 14: Additional results for Section 5.1 when varying sample sizes for Liang’s settings. Each subplot corresponds to one data generating process, with

x

-axis being

n=n_{\textrm{train}}=n_{\textrm{calib}}

, while fixing

q=0.3

Figure 15: Additional results for experiments in Section 5.1 when varying sample sizes for Jin’s settings. Details are otherwise the same as Figure 14 .

Figure 16: Power vs. Sample Size analysis for OptCS-MSel. (a) Realized FDR and (b) power of OptCS-MSel and sample splitting baseline Base_cal_split and non-model-selection baseline Base_random for varying target FDR levels

q\in\{0.2,0.25,\dots,0.5\}

(rows) and Liang’s settings (column), averaged over

N=200

runs. The

x

-axis is the number of calibration samples

n_{2}

. We fix

n_{1}=100

for consistent model training and

m=100

for evaluation.

Figure 17: Standard deviation of FDP (a) and power (b) in Liang’s settings. Details are otherwise the same as Figure 6 .

Figure 18: Standard deviation of FDP (a) and power (b) in Jin’s settings. Details are otherwise the same as Figure 7 .

Setting	$X_{i}$	$\mu(\cdot)$	$\epsilon_{i}$
1	$\text{Unif}[-1,1]^{d}$	$\frac{1}{2}\ind\{x_{1}x_{2}>0\}+\ind\{x_{1}x_{2}\leq 0\}+x_{4}$	$2\sigma\cdot\mathcal{N}(0,1)$
2	$\text{Unif}[-1,1]^{d}$	$2(x_{1}x_{2}+e^{x_{4}}-1)$	$\frac{3}{2}\sigma\cdot\mathcal{N}(0,1)$
3	$\text{Unif}[-1,1]^{d}$	$\frac{1}{2}\ind\{x_{1}x_{2}>0\}+\ind\{x_{1}x_{2}\leq 0\}+x_{4}$	$\sigma\cdot(5.5-\|\mu(x)\|)/2$
4	$\text{Unif}[-1,1]^{d}$	$2(x_{1}x_{2}+e^{x_{4}}-1)$	$\sigma\cdot(5.5-\|\mu(x)\|)/3$

Appendix

Table 4: Details of the four data generating processes used in Jin’s settings for Section 5.2 .

Figure 19: (a) Empirical FDR and (b) Power for OptCS-Full (over-sampling, red) and baselines in Jin’s settings across test sample sizes

m\in\{20,50,100,200,500\}

and target FDR levels

q\in\{0.1,0.15,0.2,0.25\}

, fixing

n_{\text{labeled}}=200

and model class as support vector machine.

Figure 20: (a) Empirical FDR and (b) Power for OptCS-Full-MSel (over-sampling, red) and baselines in Liang’s settings and model classes identical to Section 5.3 across test sample sizes

m\in\{20,50,100,200,500\}

and target FDR levels

q\in\{0.2,0.3,0.4,0.5\}

, fixing

n_{\text{labeled}}=200

Figure 21: Comparison of OptCS-Full and OptCS-Full-sep with split and oracle baselines across Jin’s settings while varying the fraction of positive samples

\gamma\in\{0.4,0.5,0.6,0.7,0.8\}

and the number of test samples

m\in\{20,50,100,200,500\}

, fixing the number of labeled data at

n_{\text{labeled}}=200

, FDR target level at

q=0.2

, and model class as random forests from scikit-learn . Each subplot corresponds to one setting and one value of

\gamma

, and within each subplot, the

x

-axis is

m

Figure 22: (a) Empirical FDR and (b) Power of OptCS-Full and OptCS-Full-sep with split and oracle baselines across Jin’s settings while varying the noise standard deviation

\sigma\in\{0.1,0.2,0.3,0.4,0.5\}

and the number of test samples

m\in\{20,50,100,200,500\}

, fixing the number of labeled data at

n_{\text{labeled}}=200

, FDR target level at

q=0.2

, and model class as random forests from scikit-learn . Each subplot corresponds to one setting and one value of

\sigma

, and within each subplot, the

x

-axis is the value of

m

Figure 23: Comparison of OptCS-Full-MSel and OptCS-Full-MSel-sep with split baseline across Liang’s settings while varying the fraction of positive samples

\gamma\in\{0.4,0.5,0.6,0.7\}

and the number of test samples

m\in\{20,50,100,200,500\}

, fixing the number of labeled data at

n_{\text{labeled}}=200

, FDR target level at

q=0.3

, with the same model classes as Section 5.3 . Each subplot corresponds to one setting and one value of

\gamma

, and within each subplot, the

x

-axis is

m

Figure 24: (a) Empirical FDR and (b) Power of OptCS-Full vs. conformal selection methods (ACS) and outlier detection methods (AdaDetect and IC-based methods) in Jin’s settings across various test sample sizes

m\in\{20,50,100,200,500\}

, fixing

n_{\text{labeled}}=200

and model class as SVM in scikit-learn .

Figure 25: (a) Empirical FDR and (b) Power of OptCS-Full vs. conformal selection methods (ACS) and outlier detection methods (AdaDetect and IC-based methods) in LLM application (with different model classes) across various test sample sizes

m\in\{20,50,100,200,500\}

, fixing

n_{\text{labeled}}=200

q=0.2

Figure 26: Comparing OptCS-Full-MSel against conformal selection methods (ACS) and outlier detection methods (IC variants) in Jin’s settings across various test sample sizes

m\in\{20,50,100,200,500\}

, fixing

n_{\text{labeled}}=200

and model classes as

\{

GB, RF, SVR

\}

in scikit-learn .

Figure 27: Comparing OptCS-Full-MSel against ACS and IC variants in LLM application across various test sample sizes

m\in\{20,50,100,200,500\}

and FDR target level

q\in\{0.05,0.1,0.15,0.2\}

, fixing

n_{\text{labeled}}=200

and model classes as

\{

GB, RF, SVR, LR

\}

in scikit-learn .

Figure 28: Empirical FDR and power of OptCS-Full, OptCS-Full-Non-LOO, and FCND novelty-detection baselines in Jin’s settings at target FDR level

q=0.2

, varying the test sample size

m

(rows) and labeled sample size

n

(x-axis), averaged over

N=200

runs. OptCS uses a random forest classifier from scikit-learn . FCND variants use the official fcnd implementation with the null-fraction correction.

Figure 29: Empirical FDR and power of OptCS-Full and OptCS-Full-Non-LOO versus FCND variants, in the LLM application across test sample sizes

m\in\{20,50,100,200,500\}

and model classes, fixing

n_{\mathrm{labeled}}=200

and

q=0.2

. FCND variants use the official fcnd implementation with the null-fraction correction.

Figure 30: Empirical (a) FDR and (b) power of the binary and positive-part encodings for OptCS-Full across the four Jin settings. Each column corresponds to a model class. Results are averaged over

N=100

independent runs with

n=200

m=100

, and

q=0.2

. The red dashed line marks the nominal FDR level.

Figure 31: Empirical (a) FDR and (b) power of the binary, positive-part, and mixed encodings for OptCS-Full-MSel across the four Liang’s settings. Results are averaged over

N=100

independent runs with

n=200

m=100

, and varying

q\in\{0.2,0.3,0.4,0.5\}

. The red dashed line marks the nominal FDR level.

$n_{\mathrm{label}}$	$n_{\mathrm{test}}$	split_121	MSel-LOO	MSel-Non-LOO
100	20	$0.0255\pm 0.0006$	$2.4013\pm 0.0298$	$0.0299\pm 0.0013$
100	50	$0.0266\pm 0.0001$	$3.5795\pm 0.0201$	$0.0365\pm 0.0009$
100	100	$0.0313\pm 0.0003$	$6.2689\pm 0.0358$	$0.0529\pm 0.0011$
100	200	$0.0416\pm 0.0001$	$14.5313\pm 0.0302$	$0.0837\pm 0.0005$
100	500	$0.0722\pm 0.0006$	$69.3850\pm 0.5157$	$0.2218\pm 0.0014$
200	20	$0.0301\pm 0.0004$	$6.1515\pm 0.0556$	$0.0355\pm 0.0005$

Appendix

Table 5: Mean runtime

\pm

standard error (seconds) across 100 seeds.

Conformal Prediction

Conformal prediction wraps predictive models with calibrated sets or intervals that provide statistical coverage guarantees.

See all 348 papers

Jul 25, 2026

Robust Conformalized Selection with Noisy Responses

Conformalized selection has been widely applied to select high-quality candidates from large datasets with rigorous uncertainty quantification, such as reliable labeling, drug discovery, and the alignment of large language models. Nevertheless, existing methods assume clean responses on calibration data, an assumption that rarely holds in practice. In this paper, we formulate the above tasks as selecting candidates with true predicted labels or with responses exceeding certain values. We demonstrate that existing conformal selection methods fail to control the false discovery rate (FDR) or suffer from severe power loss under contaminated calibration data. To that end, we propose Robust Conformalized Selection (RCS), a unified framework for selective classification with valid FDR control under general label contamination. The key insight of RCS lies in a novel statistical reduction: by separately conditioning on different classes, we translate the intractable label noise into a localized covariate shift problem, which then enables a covariate-adjusted empirical-Bayes-type estimate of the number of false selections. Statistical properties such as the asymptotic FDR control, power optimality, and robustness of RCS are established. We further develop an instantiation of RCS under randomized response model, and also apply RCS to the task of selecting candidates with large response values. Extensive experiments on both simulated and real-world datasets demonstrate the effectiveness of RCS.

Chengyao Yu, Hongxin Wei, Bingyi Jing

Department of Statistics and Data Science Southern University of Science and Technology Shenzhen 518055, China · School of Artificial Intelligence The Chinese University of Hong Kong, Shenzhen Shenzhen Loop Area Institute Shenzhen 518172, China

Selective Prediction FDR Control Learning with Noisy Labels

Ask PDF

Apr 13, 2026

Beyond Fixed False Discovery Rates: Post-Hoc Conformal Selection with E-Variables

Conformal selection (CS) uses calibration data to identify test inputs whose unobserved outcomes are likely to satisfy a pre-specified minimal quality requirement, while controlling the false discovery rate (FDR). Existing methods fix the target FDR level before observing data, which prevents the user from adapting the balance between number of selected test inputs and FDR to downstream needs and constraints based on the available data. For example, in genomics or neuroimaging, researchers often inspect the distribution of test statistics, and decide how aggressively to pursue candidates based on observed evidence strength and available follow-up resources. To address this limitation, we introduce post-hoc CS (PH-CS), which generates a path of candidate selection sets, each paired with a data-driven false discovery proportion (FDP) estimate. PH-CS lets the user select any operating point on this path by maximizing a user-specified utility, arbitrarily balancing selection size and FDR. Building on conformal e-variables and the e-Benjamini-Hochberg (e-BH) procedure, PH-CS is proved to provide a finite-sample post-hoc reliability guarantee whereby the ratio between estimated FDP level and true FDP is, on average, upper bounded by 1, so that the average estimated FDP is, to first order, a valid upper bound on the true FDR. PH-CS is extended to control quality defined in terms of a general risk. Experiments on synthetic and real-world datasets demonstrate that, unlike CS, PH-CS can consistently satisfy user-imposed utility constraints while producing reliable FDP estimates and maintaining competitive FDR control.

Meiyi Zhu, Osvaldo Simeone

Department of Engineering, King’s College London, WC2R 2LS, London, U.K. · Institute for Intelligent Networked Systems, Northeastern University London, E1 8PH London, U.K.

FDR Control E-Values Conformal Prediction

Ask PDF

Jun 21, 2026

Null-Calibrated Conformal Selection via Target-Membership Scores

Conformal selection aims to identify test candidates whose unknown responses fall in a target region while controlling the false discovery rate. Existing methods often inherit prediction-oriented nonconformity scores, such as residual or clipped residual scores, from conformal prediction. We argue that the natural score for selection is instead the target-membership probability. This score directly addresses the binary event being selected, and any monotone transform of it gives the Neyman--Pearson oracle ranking at a fixed null selection level. This distinction is irrelevant for mean-monotone targets, where conventional scores induce essentially the same ranking, but becomes important for interval-valued, variance-driven, multimodal, or multi-condition targets, where prediction-oriented scores can be misaligned with selection power. We study membership-score-based conformal selection and isolate one conformal calibration route, Null-Calibrated Conformal Selection (NCCS), which ranks test scores against confirmed non-target calibration examples. Under null exchangeability, NCCS yields finite-sample valid null p-values, which can be combined with BY under arbitrary dependence or with BH under standard positive-dependence conditions. Experiments support the score principle: membership scores match conventional scores on mean-monotone targets, substantially improve over mean-score selection on variance-driven targets, and, when calibrated by NCCS, trade power for finite-sample null validity in rare-target regimes where direct empirical-FDP thresholding can be anti-conservative.

Seungjin Choi

CROID Research and aSSIST University, Korea

FDR Control Conformal Prediction

Ask PDF

Abstract

Conformal Prediction

Explore similar work

Robust Conformalized Selection with Noisy Responses

Beyond Fixed False Discovery Rates: Post-Hoc Conformal Selection with E-Variables

Null-Calibrated Conformal Selection via Target-Membership Scores