Synthetic-Powered Multiple Testing with FDR Control

Yonghoon Lee; Meshi Bashari; Edgar Dobriban; Yaniv Romano

stat.MEFeb 18, 2026

Synthetic-Powered Multiple Testing with FDR Control

Authors: Yonghoon Lee, Meshi Bashari, Edgar Dobriban, Yaniv Romano

Organizations: Department of Statistics and Data Science, The Wharton School, University of Pennsylvania, USA · Department of Electrical and Computer Engineering, Technion IIT, Israel · Department of Computer Science, Technion IIT, Israel

Abstract

Multiple hypothesis testing with false discovery rate (FDR) control is a fundamental problem in statistical inference, with broad applications in genomics, drug screening, and outlier detection. In many such settings, researchers may have access not only to real experimental observations but also to auxiliary or synthetic data -- from past, related experiments or generated by generative models -- that can provide additional evidence about the hypotheses of interest. We introduce SynthBH, a synthetic-powered multiple testing procedure that safely leverages such synthetic data. We prove that SynthBH guarantees finite-sample, distribution-free FDR control under a mild PRDS-type positive dependence condition, without requiring the pooled-data p-values to be valid under the null. The proposed method adapts to the (unknown) quality of the synthetic data: it enhances the sample efficiency and may boost the power when synthetic data are of high quality, while controlling the FDR at a user-specified level regardless of their quality. We demonstrate the empirical performance of SynthBH on tabular outlier detection benchmarks and on genomic analyses of drug-cancer sensitivity associations, and further study its properties through controlled experiments on simulated data.

Figures & tables

Figure 1: Comparison of conformal outlier detection methods on three tabular datasets. The target FDR level is

\alpha=10\%

and

\varepsilon=10\%

. Top: detection rate (empirical power). Bottom: false discovery proportion.

Panel of Figure 2 — Figure 2: Power as a function of the empirical FDR for BH (real) and our proposed method SynthBH across different outlier detection datasets: (a) Shuttle, (b) Credit-card, and (c) KDDCup99. The first row corresponds to a trimming proportion of 2%, and the second row corresponds to a trimming proportion of 0%, representing a scenario where synthetic data are not useful. Here, $\alpha\in[0,20\%]$ .

Figure 3: Comparison of multiple testing methods on GDSC genomic data for 100 tissue-feature-drug hypotheses. The target FDR level is

\alpha=10\%

and

\varepsilon=10\%

. Left: number of rejections. Right: average ground-truth score of the rejection set.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : Comparison of multiple hypothesis testing as a function of the real data sample size

n

. Left: average detection rate (power). Right: average false discovery proportion.

Panel of Figure 5 — Figure S2 : Comparison of multiple hypothesis testing as a function of the synthetic signal strength $\rho_{\text{synth}}$ . Results are shown for (a) high-quality synthetic data and (b) worst-case synthetic data (in terms of error). Left: average detection rate (power). Right: average false discovery proportion.

Panel of Figure 6 — Figure S3 : Comparison of multiple hypothesis testing as a function of $\varepsilon$ . Results are shown for (a) high-quality synthetic data and (b) worst-case synthetic data (in terms of error). Left: average detection rate (power). Right: average false discovery proportion.

Figure S4 : Comparison of conformal outlier detection methods on the Shuttle dataset as a function of the real data sample size

n

. Left: detection rate (empirical power). Right: false discovery proportion.

Figure S5 : Comparison of conformal outlier detection methods on the Shuttle dataset as a function of the trimming proportion

\rho

. Left: detection rate (empirical power). Right: false discovery proportion.

Panel of Figure 9 — Figure S6 : Power as a function of the empirical FDR for BH (real) and our proposed method SynthBH across different outlier detection datasets: (a) Shuttle, (b) Credit-card, and (c) KDDCup99. The first row corresponds to a trimming proportion of 2%, and the second row corresponds to a trimming proportion of 0%, representing a scenario where synthetic data are not useful. Here, $\alpha\in[0,20\%]$ . Results are shown for $\varepsilon=5\%$ .

Figure S7 : Comparison of multiple testing methods on GDSC genomic data for 100 tissue-feature-drug hypotheses. Left: number of rejections. Right: average ground-truth score of the rejection set. Results are shown for

\varepsilon=5\%

Tissue	Genomic feature	Drug
BRCA	gain_cnaPANCAN301_(CDK12,ERBB2,MED24)	Ibrutinib
BRCA	PIK3CA_mut	Taselisib
BRCA	gain_cnaPANCAN301_(CDK12,ERBB2,MED24)	Sapitinib
BRCA	gain_cnaPANCAN301_(CDK12,ERBB2,MED24)	Osimertinib
BRCA	PIK3CA_mut	Alpelisib
BRCA	gain_cnaPANCAN301_(CDK12,ERBB2,MED24)	Afatinib

Appendix

Table S1 : List of tissue-feature-drug hypotheses tested in the GDSC experiments. BRCA denotes breast invasive carcinoma and LUAD denotes lung adenocarcinoma.

Multiple Hypothesis Testing

Multiple hypothesis testing develops procedures to control false discoveries when many statistical claims are evaluated together.

See all 33 papers

Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference
May 20, 2026Ziang Song, Ying Jin, Emmanuel J. CandèsFDR Control Conformal Prediction
Department of Statistics, Stanford University · Department of Statistics and Data Science, University of Pennsylvania · Department of Mathematics, Stanford University
Beyond Fixed False Discovery Rates: Post-Hoc Conformal Selection with E-Variables
Apr 13, 2026Meiyi Zhu, Osvaldo SimeoneFDR Control E-Values
Department of Engineering, King’s College London, WC2R 2LS, London, U.K. · Institute for Intelligent Networked Systems, Northeastern University London, E1 8PH London, U.K.
Finite Resources False Discovery Rate Control in Structured Hypothesis Spaces
Jun 13, 2026Binyamin Perets, Shie MannorFDR Control Multiple Hypothesis Testing
Technion – Israel Institute of Technology · NVIDIA

May 20, 2026J/K move · Enter open · S save

Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference

Ziang Song, Ying Jin, Emmanuel J. Candès

Department of Statistics, Stanford University · Department of Statistics and Data Science, University of Pennsylvania · Department of Mathematics, Stanford University

FDR Control Conformal Prediction Multiple Hypothesis Testing

Open paper AskPDF arXiv

Modern applications of conformal inference to multiple testing problems, such as outlier detection and candidate selection, often involve selecting test samples whose conformal p-values fall below a threshold. The quality of such methods is often measured by the false discovery proportion (FDP), defined as the fraction of incorrect selections. Existing approaches typically control the expected value of the FDP, using methods such as the Benjamini-Hochberg procedure. This approach fails to provide high-probability bounds on the realized false discovery proportion and invalidates statistical guarantees if the rejection threshold is selected after inspecting the data. This paper establishes finite-sample, distribution-free upper bounds on the FDP that hold simultaneously over all possible rejection thresholds, enabling arbitrary post hoc selection of the threshold. Simultaneous validity is achieved by constructing a high-probability envelope for the empirical distribution function of null conformal p-values by sampling from their joint distribution. Furthermore, our framework allows practitioners to modulate the envelope's shape, thereby producing tight bounds in rejection regions of primary interest. We use this flexible approach to derive simultaneous FDP upper bounds for both outlier detection and conformal selection. We demonstrate through synthetic and real-data experiments that the resulting bounds are both valid and substantially less conservative than those derived from existing approaches.

Abstract

Multiple Hypothesis Testing

Explore similar work

Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference

Beyond Fixed False Discovery Rates: Post-Hoc Conformal Selection with E-Variables

Finite Resources False Discovery Rate Control in Structured Hypothesis Spaces