Conformal Policy Selection: When to Deploy AI Decisions for Targeted Intervention

Junyu Cao; Ruijiang Gao; Maytal Saar-Tsechansky; Mingzhang Yin

stat.MEOct 16, 2026

Conformal Policy Selection: When to Deploy AI Decisions for Targeted Intervention

Authors: Junyu Cao, Ruijiang Gao, Maytal Saar-Tsechansky, Mingzhang Yin

Organizations: McCombs School of Business, The University of Texas at Austin, Austin, TX 78712 · Naveen Jindal School of Management, The University of Texas at Dallas, Richardson, TX 75080 · Warrington College of Business, University of Florida, Gainesville, FL 32611

Abstract

Artificial-intelligence (AI) policies are increasingly used to make consequential decisions in medicine, finance, education, and public services, yet deploying them responsibly requires knowing not only whether a policy helps on average, but for whom. Existing tools for off-policy evaluation and safe policy learning provide population-level guarantees, but some individuals may experience adverse treatment effects. We recast targeted deployment as a multiple-testing problem: for each candidate unit, test whether the outcome under the AI policy would exceed a user-specified threshold, and control the false discovery rate (FDR) among the selected set. Our method, Conformal Policy Selection, combines weighted conformal $p$ -values that account for the distributional shift from behavior to target policy with a two-step selection-and-pruning procedure. We prove finite-sample FDR control and establish a doubly robust asymptotic guarantee, where FDR remains controlled if either the outcome model or the propensity model is correctly specified. Synthetic experiments verify these guarantees across four misspecification regimes, and a semi-synthetic study built on real electronic-health-record covariates illustrates how the same FDR control can act as a deployment safety gate, bounding the harmful treatments that a globally trained policy would otherwise administer to a high-risk subpopulation where it misgeneralizes. The framework offers a principled method for safe AI deployment in high-stakes applications.

Figures & tables

Panel of Figure 1 — Figure 1: FDR and power with correctly specified scores and weights on synthetic data ( $n=500$ , $m=100$ , 400 replications). The FDR remains well below the nominal level $q$ (dashed line) across all target levels, while power increases steadily.

Figure 2: Doubly robust FDR control experiment. Each panel shows the empirical FDR across nominal levels

q

under different combinations of score and weight specification. Top left : both correct, FDR controlled. Top right : correct weights, misspecified (but consistently applied) scores, FDR controlled. Bottom left : wrong weights, correct scores, FDR controlled. Bottom right : both wrong, FDR not controlled. The dashed line indicates the nominal FDR level

q

. Results averaged over 400 replications with

n=500

calibration and

m=100

test observations.

Scenario	$q=0.15$	$q=0.25$	$q=0.55$	$q=0.95$
Both correct	0.005	0.013	0.048	0.149
Correct W, wrong S	0.055	0.083	0.225	0.502
Wrong W, correct S	0.005	0.013	0.048	0.150
Both wrong	0.287	0.300	0.295	0.297

Table 1: Mean empirical FDR at selected nominal levels

q

across the four doubly robust scenarios (WBH method, 400 replications).

Figure 3: Conformal selection as a deployment safety gate on the EHR diabetes cohort (age

\geq 60

, BMI

\geq 30

n=5{,}993

, semi-synthetic causal-forest ground truth,

20

replications). Left : the empirical FDR (blue) stays below the nominal level

q

(dashed) across the whole range, reaching the select-all floor (

0.50

, dotted) only as

q\to 1

, with power shown in orange. Right : the percentage of the AI policy’s intended (harmful) treatments that are actually deployed through the gate, which is zero for all

q\leq 0.85

even though the gate certifies

93\%

of the batch for deployment at

q=0.85

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: FDR and power on the LaLonde job training dataset (614 observations, 100 random splits) with threshold $c=4{,}000$ . Empirical FDR stays at or below the nominal level $q$ for $q\leq 0.65$ and plateaus near the baseline negative rate ( $\approx 0.23$ ) as $q$ approaches $1$ . Power increases smoothly with $q$ , reflecting the trade-off the user controls by choosing the FDR budget.

Figure 5: LaLonde job training dataset. Mean post-program earnings ( re78 ) of the selected versus non-selected groups as a function of the nominal FDR level

q

. At small

q

the selected group has markedly higher mean earnings, and as

q\to 0.9

almost all units are selected and the two curves converge.

Estimator	mean $\widehat{\tau}$	frac. $\widehat{\tau}>0$	max $\widehat{\tau}$
S-learner (random forest)	$-0.269$	$0.0\%$	$-0.052$
T-learner (random forest)	$-0.981$	$0.0\%$	$+0.000$
X-learner (random forest)	$-0.994$	$0.0\%$	$+0.000$
Causal Forest (DML)	$-0.570$	$0.0\%$	$-0.304$

Appendix

Table 2: Estimated treatment effect on three-year avoidance for four heterogeneous-effect estimators (age

\geq 60

, BMI

\geq 30

subgroup). Every estimator returns a negative effect for essentially all patients.

Propensity decile	$n_{\text{treated}}$	avoid	$n_{\text{control}}$	avoid
4	$15$	$13.3\%$	$584$	$72.1\%$
6	$30$	$3.3\%$	$570$	$68.4\%$
8	$58$	$5.2\%$	$541$	$65.4\%$
10	$250$	$1.6\%$	$350$	$57.7\%$

Appendix

Table 3: Three-year avoidance rate of treated vs. untreated patients within estimated-propensity deciles. Even in balanced (high-propensity) deciles the treated fare far worse, confirming the harm is not an overlap artifact.

$q$	FDR	Power	Frac. treated	Beneficial treated	Harmful deployed
$\leq 0.35$	$0.00$	$0.00$	$0.00$	$0$	$0$
$0.45$	$0.22$	$0.85$	$0.49$	$76.5$	$24.4$
$0.55$	$0.29$	$1.00$	$0.49$	$91.4$	$30.5$
$\geq 0.65$	$0.29$	$1.00$	$0.49$	$91.4$	$30.5$

Appendix

Table 4: Heterogeneous safety gate on real covariates, averaged over

20

repetitions. The policy treats about half of the cohort. The realized FDR stays below the nominal

q

Figure 6: Heterogeneous safety gate. Left, empirical FDR against the nominal level with the diagonal for reference. Middle, selection power. Right, the counts of beneficial treated units and harmful deployments as functions of

q

Policy Evaluation

Policy evaluation estimates the outcomes, value, risks, and fairness of decision rules before or after deployment.

See all 55 papers

Conformal Policy Learning with Distribution-Free Safety Guarantees
Sep 15, 2026Ying Jin, Naoki EgamiConformal Prediction Causal Inference
Department of Statistics and Data Science, University of Pennsylvania · Department of Political Science & Statistics and Data Science Center, Massachusetts Institute of Technology
Conformal Selective Prediction with General Risk Control
Mar 25, 2026Tian Bai, Ying JinSelective Prediction Conformal Risk Control
Department of Statistics, Stanford University · Department of Statistics and Data Science, University of Pennsylvania
Conformal Policy Control
Mar 2, 2026Drew Prinster, Clara Fannjiang, Ji Won Park +4Constrained RL RL Exploration
Prescient Design, Genentech, U.S.A. · Johns Hopkins University, Baltimore, U.S.A. · New York University, NYC, U.S.A. +1

Sep 15, 2026J/K move · Enter open · S save

Conformal Policy Learning with Distribution-Free Safety Guarantees

Ying Jin, Naoki Egami

Department of Statistics and Data Science, University of Pennsylvania · Department of Political Science & Statistics and Data Science Center, Massachusetts Institute of Technology

Conformal Prediction Causal Inference Counterfactual Learning

Open paper AskPDF arXiv

Policy learning aims to determine who should be treated based on individual characteristics. In high-stakes settings such as medicine and public policy where safety is a central concern, improving the average outcomes alone may not be sufficient: decision makers may also seek to protect individuals from harm, in line with the Hippocratic principle of ``do no harm.'' In this paper, we propose conformal policy learning (CPL), a policy learning procedure with a new distribution-free safety guarantee that controls the probability of assigning treatment to an individual who would be harmed relative to control. CPL views each treatment decision as testing a hypothesis of counterfactual harm and assigns treatment by thresholding conformal p-values. These p-values use observable proxies and selective calibration to address the challenge that the potential outcomes under comparison are never simultaneously observed. For randomized experiments, under standard exchangeability conditions, CPL provides finite-sample safety guarantee at a user-specified level, without imposing any outcome modeling assumptions. Moreover, when the outcome model is consistently estimated, CPL achieves asymptotically optimal welfare subject to the safety constraint. In observational studies, CPL with learn-then-balance weights achieves doubly robust safety guarantees. We evaluate CPL through extensive simulations and apply it to an empirical study of AI-powered interventions designed to reduce conspiracy beliefs.

Abstract

Policy Evaluation

Explore similar work

Conformal Policy Learning with Distribution-Free Safety Guarantees

Conformal Selective Prediction with General Risk Control

Conformal Policy Control