Almost Controlled: The FDR of Benjamini-Hochberg on Weighted Conformal p-Values
Organizations: Albert Einstein College of Medicine
Abstract
Weighted conformal p-values generalize conformal inference to cases with covariate shift. One way to use them is to pass them to the Benjamini-Hochberg (BH) procedure, either to decide which test units to select or to detect outliers. But weighted conformal p-values may not be positively dependent, so we want to know: does BH still control the false discovery rate (FDR) in finite samples? Based on experiments, it appears to, but we prove it doesn't. We give two counterexamples, certified in exact rational arithmetic, where the FDR exceeds its target level . We also identify how large the difference between the FDR and can be. With calibration points, test units and weight ratio , the relative excess is bounded by a function of , , and that is of order when is small. The bound holds for every configuration with and every . This order can be reached when multiple non-null test units share a common threshold: in a Poisson-limit model, that configuration's excess is to first order as and then . For some fixed distributions with this structure, the excess decreases only like as grows. If every test unit is null, the excess is at fixed , and in certain configurations it grows linearly with . To assist practitioners in judging how large the FDR can be, we turn that bound, which depends only on , , and , into a lowered BH level with proved finite-sample control, and give a much less conservative rule of thumb that covers every worst case we were able to find but provides no guarantee.
Figures & tables
| Proved bound (Theorem 6 , form ) | 0.075 | 0.258 | 0.669 | 3.55 |
| Rule of thumb (numerical) | 0.0076 | 0.024 | 0.054 | 0.17 |
FDR Control
False discovery rate (FDR) control limits the expected share of false findings among selected discoveries in large-scale testing.
See all 46 papers