Empirically Calibrated Conditional Independence Tests

Milleno Pan; Antoine de Mathelin; Wesley Tansey

stat.MEFeb 24, 2026

Empirically Calibrated Conditional Independence Tests

Authors: Milleno Pan, Antoine de Mathelin, Wesley Tansey

Organizations: Computational Oncology Memorial Sloan Kettering Cancer Center

Abstract

Conditional independence tests (CIT) are widely used for causal discovery and feature selection. Even with false discovery rate (FDR) control procedures, they often fail to provide frequentist guarantees in practice. We highlight two common failure modes: (i) in small samples, asymptotic guarantees for many CITs can be inaccurate and even correctly specified models fail to estimate the noise levels and control the error, and (ii) when sample sizes are large but models are misspecified, unaccounted dependencies skew the test's behavior and fail to return uniform p-values under the null. We propose Empirically Calibrated Conditional Independence Tests (ECCIT), a method that measures and corrects for miscalibration. For a chosen base CIT (e.g., GCM, HRT), ECCIT optimizes an adversary that selects features and response functions to maximize a miscalibration metric. ECCIT then fits a monotone calibration map that adjusts the base-test p-values in proportion to the observed miscalibration. Across empirical benchmarks on synthetic and real data, ECCIT achieves valid FDR with higher power than existing calibration strategies while remaining test agnostic.

Figures & tables

Input: Dataset

(X,Y)

; test

T:(X,Y)\mapsto(p_{1},\ldots,p_{m})

; adversary class

\mathcal{F}

with

f\in\mathcal{F}

and

\tilde{Y}=f(X,\varepsilon)

; calibration metric

M(\cdot,\alpha)

Fit the worst-case adversary

f^{\star}

as in Equation 2 .

Let

\mathcal{H}_{0}=\{j:\gamma_{j}=0\}

denote the null features under

f^{\star}

for

b=1,\ldots,B

Bootstrap

X^{(b)}

by resampling rows of

X

Sample

\tilde{Y}^{(b)}=f^{\star}(X^{(b)},\varepsilon)

Compute

p^{(b)}=T(X^{(b)},\tilde{Y}^{(b)})

Algorithm 1 Empirical calibration (adversarial approach).

Figure 1: Single Experiment Performance. Realized Type-I error and power versus nominal

\alpha

for raw and calibrated HRT on a correlated dataset.

Panel of Figure 2 — Figure 2: Miscalibration over sample size (log scaled) by features on a well-specified model for both miscalibration metrics. The red dotted line indicates the selected nominal threshold of $\alpha=0.2$ .

Figure 3: Valid Power Gain by Features. Calibrated with a nonlinear adversary. Performance evaluated with

10m

samples on a nonlinear response

Y

Figure 4: Valid Power Gain by Distribution. Calibrated with a nonlinear adversary. Performance evaluated on a nonlinear response

Y

Panel of Figure 5 — Figure 5: Valid Power and FDR Comparison on Gene Expression Data. Calibrated with FDP metric. Nonlinear response $Y$ .

Panel of Figure 6 — Figure 6: Single Experiment Performance. Realized Type-I error and power versus nominal $\alpha$ for raw and calibrated procedures in independent and correlated settings.

Panel of Figure 7 — Figure 7: GCM Performance. Calibrated with a nonlinear adversary with both metrics. Performance evaluated on a nonlinear ground truth response. Results show the change in power and FDR before and after calibration across different numbers of samples and features.

Panel of Figure 8 — Figure 8: HRT Performance. Calibrated with a nonlinear adversary with both metrics. Performance evaluated on a nonlinear ground truth response. Results show the change in power and FDR before and after calibration across different numbers of samples and features.

Panel of Figure 9 — Figure 9: GCM calibration across distributions. Comparisons between different miscalibration combinations on the adversary and the test. Type-I metric (top) and FDP metric (bottom).

Panel of Figure 10 — Figure 10: HRT calibration across distributions. Comparisons between different miscalibration combinations on the adversary and the test. Type-I metric (top) and FDP metric (bottom).

Method	$\alpha=0.05$	$\alpha=0.10$	$\alpha=0.15$	$\alpha=0.20$
GCM	0.165 / 0.086	0.127 / 0.141	0.125 / 0.181	0.145 / 0.227
Calibrated GCM	0.166 / 0.058	0.179 / 0.070	0.177 / 0.075	0.177 / 0.082
HRT	0.109 / 0.220	0.074 / 0.321	0.053 / 0.388	0.057 / 0.444
Calibrated HRT	0.161 / 0.093	0.146 / 0.137	0.137 / 0.159	0.150 / 0.176
CONTRA-HRT	0.010 / 0.489	0.010 / 0.497	0.010 / 0.502	0.027 / 0.520
CONTRA-FASTCRT	0.012 / 0.539	0.012 / 0.548	0.008 / 0.562	0.017 / 0.572

Table 1: Breast Cancer Dataset. Entries are valid power / realized FDR.

Method	$\alpha=0.05$	$\alpha=0.10$	$\alpha=0.15$	$\alpha=0.20$
GCM	0.372 / 0.132	0.292 / 0.203	0.255 / 0.244	0.207 / 0.281
Calibrated GCM	0.432 / 0.043	0.407 / 0.085	0.333 / 0.158	0.292 / 0.204
HRT	0.400 / 0.030	0.388 / 0.050	0.370 / 0.076	0.347 / 0.103
Calibrated HRT	0.400 / 0.030	0.388 / 0.050	0.370 / 0.076	0.347 / 0.103
CONTRA-HRT	0.122 / 0.324	0.122 / 0.333	0.092 / 0.375	0.087 / 0.388
CONTRA-FASTCRT	0.133 / 0.335	0.125 / 0.352	0.068 / 0.405	0.068 / 0.425

Table 2: Wine Dataset. Entries are valid power / realized FDR.

Conditional Independence Testing

Conditional independence testing asks whether two variables remain independent after accounting for other variables, supporting causal discovery and valid inference.

See all 19 papers

Conditional Independence Tests for Constraint-Based Causal Discovery: A Survey
Aug 11, 2026Pavel Averin, Theodoros Moysiadis, Ioannis KatakisCausal Discovery Constraint-Based Causal Discovery
Department of Computer Science School of Sciences and Engineering University of Nicosia 2417, Nicosia, Cyprus
Fast Nonparametric Conditional Independence Testing via Two-Stage Regression
Jun 16, 2026Eric V. StroblCausal Discovery Causal Structure Learning
Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA 15213, USA
Meta-Dependence in Conditional Independence Testing
Apr 17, 2025Bijan Mazaheri, Jiaqi Zhang, Caroline UhlerConditional Independence Testing
Thayer School of Engineering, Dartmouth College, Hanover, New Hampshire, USA · Schmidt Center, Broad Institute of MIT and Harvard, Cambridge, Massachusetts, USA · Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA

Aug 11, 2026J/K move · Enter open · S save

Conditional Independence Tests for Constraint-Based Causal Discovery: A Survey

Pavel Averin, Theodoros Moysiadis, Ioannis Katakis

Department of Computer Science School of Sciences and Engineering University of Nicosia 2417, Nicosia, Cyprus

Causal Discovery Constraint-Based Causal Discovery Causal Inference Conditional Independence Testing Statistical Hypothesis Testing

Open paper AskPDF arXiv

Conditional Independence (CI) tests are the statistical engine of constraint-based causal discovery: in algorithms such as PC (Peter-Clark) and FCI (Fast Causal Inference), skeleton pruning and key orientations follow directly from CI decisions. This survey reviews CI testing with emphasis on assumptions, robustness, and scalability in high-dimensional and mixed-type settings common in biomedical domains. The survey organizes widely used CI methods into six families: partial-correlation, contingency-table, regression, nearest-neighbor, kernel, and machine-learning-based. Special emphasis is provided on the robustness layers that address the limitations of these families. For each family, the survey examines when CI decisions reflect the data-generating distribution and when they fail. By this, we link test-level properties, including power decay with conditioning set size and asymmetric type I/II error consequences, to graph-level errors in skeleton recovery and v-structure orientation. The survey also compares adoption across major R and Python libraries and summarizes open challenges, including mixed-type CI testing without discretization, small-sample error control, and strategies for improving scalability of CI-testing.

Abstract

Conditional Independence Testing

Explore similar work

Conditional Independence Tests for Constraint-Based Causal Discovery: A Survey

Fast Nonparametric Conditional Independence Testing via Two-Stage Regression

Meta-Dependence in Conditional Independence Testing