stat.MENov 16, 2026

Almost Controlled: The FDR of Benjamini-Hochberg on Weighted Conformal p-Values

Authors: Ryan Lim

Organizations: Albert Einstein College of Medicine

Abstract

Weighted conformal p-values generalize conformal inference to cases with covariate shift. One way to use them is to pass them to the Benjamini-Hochberg (BH) procedure, either to decide which test units to select or to detect outliers. But weighted conformal p-values may not be positively dependent, so we want to know: does BH still control the false discovery rate (FDR) in finite samples? Based on experiments, it appears to, but we prove it doesn't. We give two counterexamples, certified in exact rational arithmetic, where the FDR exceeds its target level qq. We also identify how large the difference between the FDR and qq can be. With nn calibration points, mm test units and weight ratio BB, the relative excess FDR/q−1\mathrm{FDR}/q-1 is bounded by a function of nn, mm, BB and qq that is of order Bm/(nq)\sqrt{Bm/(nq)} when Bm/(nq)Bm/(nq) is small. The bound holds for every configuration with B≤nB\le n and every nn. This order can be reached when multiple non-null test units share a common threshold: in a Poisson-limit model, that configuration's excess is c(q)Bm/(nq)c(q)\sqrt{Bm/(nq)} to first order as B/n→0B/n\to0 and then m→∞m\to\infty. For some fixed distributions with this structure, the excess decreases only like n−1/2n^{-1/2} as nn grows. If every test unit is null, the excess is O(B/n)O(B/n) at fixed mm, and in certain configurations it grows linearly with mm. To assist practitioners in judging how large the FDR can be, we turn that bound, which depends only on nn, mm, qq and BB, into a lowered BH level with proved finite-sample control, and give a much less conservative rule of thumb that covers every worst case we were able to find but provides no guarantee.

Figures & tables

Explore similar work

CardsList
  1. Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference

    May 20, 2026Ziang Song, Ying Jin, Emmanuel J. CandèsFDR ControlConformal Prediction

  2. Beyond Fixed False Discovery Rates: Post-Hoc Conformal Selection with E-Variables

    Apr 13, 2026Meiyi Zhu, Osvaldo SimeoneFDR ControlE-Values

  3. The Benjamini--Hochberg Procedure Can Fail to Control the FDR for Correlated Two-Sided Gaussian Tests

    Jul 13, 2026Edgar DobribanFDR ControlMultiple Hypothesis Testing