stat.MLOct 27, 2026

Doubly Robust Deep Kernel Tests for Structured Outcomes

Authors: Robin Leman, Bariscan Bozkurt, Sofiane Tanji, Arthur Gretton, Houssam Zenati

Organizations: University College London · Gatsby Computational Neuroscience Unit, University College London · ICTEAM/INMA, UCLouvain

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We propose a test of whether two interventions induce the same outcome distribution when outcomes are structured and high-dimensional, such as images, text and graphs. We represent the interventional distributions as mean embeddings on learned deep features, such that the treatment effect is a doubly robust representation of the difference in mean embeddings. Our test, DR-KTE-Deep, optimises the learned features so as to maximise an estimated power criterion on this treatment effect representation. To motivate our learning objective, we show that the power criterion characterises the asymptotic signal-to-noise ratio of the test under fixed alternatives, and we derive a uniform finite-sample bound for its regularised estimate. We show that, with the learned kernel fixed, a held-out test is asymptotically valid under the null given suitable nuisance-estimation rates. Empirically, DR-KTE-Deep achieves substantially higher power than fixed and bandwidth-optimised kernel baselines across image, text and graph outcomes, while maintaining well-calibrated Type-I error.

Figures & tables

Figure 1: Semi-synthetic medical-image experiments adapted from MedMNIST ( Yang et al., 2021 ; Yang et al., 2023 ) . Each pair shows a treated potential outcome Y1Y^{1} and its ground-truth individual treatment effect (ITE).
Figure 2: Synthetic example where treatment assignment is driven by an indicator of disease severity XX . The potential-outcome distributions are Gaussians with the same mean, but the traditional-treatment distribution is isotropic while the experimental-treatment distribution is anisotropic. The deep kernel adapts to the geometry of the distributions, while the Gaussian kernel treats points uniformly throughout the outcome space.
ScenarioDR-KTEDR-KTE-OptDR-KTE-Deep
Breast-Tumor0.749±0.018/0.053±0.0030.749{\scriptstyle\pm 0.018}/0.053{\scriptstyle\pm 0.003}0.761±0.017/0.051±0.0030.761{\scriptstyle\pm 0.017}/0.051{\scriptstyle\pm 0.003}0.989±0.002/0.054±0.003\mathbf{0.989{\scriptstyle\pm 0.002}}/0.054{\scriptstyle\pm 0.003}
Chest-Cardiomegaly0.156±0.007/0.050±0.0030.156{\scriptstyle\pm 0.007}/0.050{\scriptstyle\pm 0.003}0.158±0.008/0.050±0.0030.158{\scriptstyle\pm 0.008}/0.050{\scriptstyle\pm 0.003}0.928±0.046/0.051±0.004\mathbf{0.928{\scriptstyle\pm 0.046}}/0.051{\scriptstyle\pm 0.004}
Chest-Effusion0.231±0.009/0.053±0.0040.231{\scriptstyle\pm 0.009}/0.053{\scriptstyle\pm 0.004}0.245±0.011/0.053±0.0040.245{\scriptstyle\pm 0.011}/0.053{\scriptstyle\pm 0.004}0.999±0.000/0.048±0.004\mathbf{0.999{\scriptstyle\pm 0.000}}/0.048{\scriptstyle\pm 0.004}
Chest-Mass0.436±0.018/0.050±0.0030.436{\scriptstyle\pm 0.018}/0.050{\scriptstyle\pm 0.003}0.461±0.019/0.050±0.0030.461{\scriptstyle\pm 0.019}/0.050{\scriptstyle\pm 0.003}1.000±0.000/0.056±0.004\mathbf{1.000{\scriptstyle\pm 0.000}}/0.056{\scriptstyle\pm 0.004}
Derma-Melanoma0.171±0.009/0.051±0.0020.171{\scriptstyle\pm 0.009}/0.051{\scriptstyle\pm 0.002}0.172±0.009/0.050±0.0020.172{\scriptstyle\pm 0.009}/0.050{\scriptstyle\pm 0.002}0.898±0.040/0.053±0.004\mathbf{0.898{\scriptstyle\pm 0.040}}/0.053{\scriptstyle\pm 0.004}
OCT-CNV0.907±0.014/0.052±0.0030.907{\scriptstyle\pm 0.014}/0.052{\scriptstyle\pm 0.003}0.917±0.014/0.051±0.0030.917{\scriptstyle\pm 0.014}/0.051{\scriptstyle\pm 0.003}1.000±0.000/0.053±0.003\mathbf{1.000{\scriptstyle\pm 0.000}}/0.053{\scriptstyle\pm 0.003}
Table 1: Power ( H1H_{1} ) and Type-I error ( H0H_{0} ) for medical-image outcomes at N=1500N=1500 and α=0.05\alpha=0.05 . Each cell reports H1/H0H_{1}/H_{0} , with means ±\pm one standard error across 20 independent repetitions.
Figure 3: Results for OCT-Drusen. (a) Type-I error under H0H_{0} , (b) average power when varying NN while keeping δ=0.4\delta=0.4 fixed, (c) average power when varying δ\delta while keeping N=1500N=1500 fixed. Shaded bands show ±\pm one standard error across 20 independent repetitions. The dotted grey line marks α=0.05\alpha=0.05 .
ScenarioDR-KTEDR-KTE-OptDR-KTE-Deep
Oncology Notes0.371±0.015/0.048±0.0030.371{\scriptstyle\pm 0.015}/0.048{\scriptstyle\pm 0.003}0.371±0.015/0.048±0.0030.371{\scriptstyle\pm 0.015}/0.048{\scriptstyle\pm 0.003}0.809±0.023/0.055±0.003\mathbf{0.809{\scriptstyle\pm 0.023}}/0.055{\scriptstyle\pm 0.003}
Movie Reviews0.255±0.012/0.050±0.0030.255{\scriptstyle\pm 0.012}/0.050{\scriptstyle\pm 0.003}0.254±0.012/0.050±0.0030.254{\scriptstyle\pm 0.012}/0.050{\scriptstyle\pm 0.003}0.449±0.026/0.053±0.004\mathbf{0.449{\scriptstyle\pm 0.026}}/0.053{\scriptstyle\pm 0.004}
Clinical Notes0.058±0.004/0.052±0.0030.058{\scriptstyle\pm 0.004}/0.052{\scriptstyle\pm 0.003}0.058±0.004/0.051±0.0030.058{\scriptstyle\pm 0.004}/0.051{\scriptstyle\pm 0.003}0.939±0.013/0.058±0.004\mathbf{0.939{\scriptstyle\pm 0.013}}/0.058{\scriptstyle\pm 0.004}
Table 2: Power ( H1H_{1} ) and Type-I error ( H0H_{0} ) for text outcomes, represented as a sequence of tokens, at N=1500N=1500 and α=0.05\alpha=0.05 . Each cell reports H1/H0H_{1}/H_{0} , with means ±\pm one standard error across 20 independent repetitions.
Figure 4: Results for Movie Reviews. (a) Type-I error under H0H_{0} , (b) average test power when varying NN while keeping δ=1.0\delta=1.0 fixed, (c) average test power when varying δ\delta while keeping N=1500N=1500 fixed. Shaded bands show ±\pm one standard error across 20 independent repetitions. The dotted grey line marks α=0.05\alpha=0.05 .
ScenarioDR-KTEDR-KTE-OptDR-KTE-Deep
LLM Code Repairs0.968±0.008/0.048±0.0040.968{\scriptstyle\pm 0.008}/0.048{\scriptstyle\pm 0.004}0.963±0.009/0.048±0.0040.963{\scriptstyle\pm 0.009}/0.048{\scriptstyle\pm 0.004}1.000±0.000/0.051±0.003\mathbf{1.000{\scriptstyle\pm 0.000}}/0.051{\scriptstyle\pm 0.003}
Contact Networks0.959±0.006/0.052±0.0030.959{\scriptstyle\pm 0.006}/0.052{\scriptstyle\pm 0.003}0.958±0.006/0.052±0.0030.958{\scriptstyle\pm 0.006}/0.052{\scriptstyle\pm 0.003}0.962±0.006/0.051±0.002\mathbf{0.962{\scriptstyle\pm 0.006}}/0.051{\scriptstyle\pm 0.002}
Table 3: Power ( H1H_{1} ) and Type-I error ( H0H_{0} ) for graph outcomes at N=1500N=1500 . Each cell reports H1/H0H_{1}/H_{0} , with means ±\pm one standard error across 20 outer repetitions.
Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

HyperparameterValue
Training variance regularisation λ\lambda10−410^{-4}
Validation and test variance regularisation00
CME regularisation λμ\lambda_{\mu}10−310^{-3}
EMA decay γ\gamma0.950.95
Minimum mixture weight wmin⁡w_{\min}10−910^{-9}
Learning-rate scheduleCosine decay to 1%1\% of initial rate
Appendix
Table 4: Shared hyperparameters for our experiments.
ExperimentEpochsEnsemble Size MMFeature LROther LR
Image outcomes50032×10−22\times 10^{-2}10−310^{-3}
Text outcomes (token sequences)1000310−310^{-3}10−410^{-4}
Graph outcomes100032×10−22\times 10^{-2}10−310^{-3}
Text outcomes (embedded)1000310−510^{-5}10−410^{-4}
Scalar outcomes100052×10−22\times 10^{-2}10−310^{-3}
Causal-Blob100052×10−22\times 10^{-2}10−210^{-2}
Appendix
Table 5: Experiment-specific hyperparameters.
Figure 5: Four-layer fully connected feature extractor for scalar outcomes, Causal-Blob, and text outcomes under an embedded representation.
Figure 6: Four-layer convolutional feature extractor for image outcomes.
Figure 7: Two-layer transformer feature extractor for text outcomes represented as a sequence of tokens.
Figure 8: Three-layer message-passing feature extractor for graph outcomes.
Figure 9: Results for OCT-DME. From left to right: (a) Type-I error under H0H_{0} ( δ=0\delta=0 ), (b) average test power when varying NN while keeping δ=0.4\delta=0.4 fixed, (c) average test power when varying δ\delta while keeping N=1500N=1500 fixed. Shaded bands show ±\pm one standard error across 20 independent outer repetitions. The dotted grey line marks α=0.05\alpha=0.05 .
Figure 10: Results for Derma-Melanoma. From left to right: (a) Type-I error under H0H_{0} ( δ=0\delta=0 ), (b) average test power when varying NN with δ=1.4\delta=1.4 fixed, (c) average test power when varying δ\delta with N=1500N=1500 fixed. Shaded bands show ±\pm one standard error across 20 independent outer repetitions. The dotted grey line marks α=0.05\alpha=0.05 .
Figure 11: Data-generating process for MNIST, RetinaMNIST and BUSI. The covariate XX and treatment AA generate a scalar pseudo-outcome Y~\tilde{Y} , which is discretised to select the label of the sampled image outcome YY .
Figure 12: MNIST (Scenario II, left) and RetinaMNIST (Scenario III, right). Panels (a) and (c) show label distributions under each treatment. Panels (b) and (d) show sampled image outcomes.
DatasetNNScenarioDR-KTEDR-KTE-OptDR-KTE-Deep
MNIST1500I0.046±0.0030.046{\scriptstyle\pm 0.003}0.046±0.0030.046{\scriptstyle\pm 0.003}0.051±0.0030.051{\scriptstyle\pm 0.003}
II0.731±0.0260.731{\scriptstyle\pm 0.026}0.731±0.0260.731{\scriptstyle\pm 0.026}0.870±0.024\mathbf{0.870{\scriptstyle\pm 0.024}}
III0.721±0.0240.721{\scriptstyle\pm 0.024}0.721±0.0240.721{\scriptstyle\pm 0.024}0.803±0.023\mathbf{0.803{\scriptstyle\pm 0.023}}
3000I0.049±0.0030.049{\scriptstyle\pm 0.003}0.049±0.0030.049{\scriptstyle\pm 0.003}0.049±0.0040.049{\scriptstyle\pm 0.004}
II0.948±0.0080.948{\scriptstyle\pm 0.008}0.948±0.0080.948{\scriptstyle\pm 0.008}0.986±0.002\mathbf{0.986{\scriptstyle\pm 0.002}}
III0.938±0.0100.938{\scriptstyle\pm 0.010}0.937±0.0100.937{\scriptstyle\pm 0.010}0.973±0.006\mathbf{0.973{\scriptstyle\pm 0.006}}
Appendix
Table 6: Average rejection rates for MNIST, RetinaMNIST, and BUSI image outcomes at α=0.05\alpha=0.05 . Scenario I reports Type-I error; Scenarios II and III report power. Effect strengths are δ=1.0\delta=1.0 and 1.51.5 for MNIST II and III, δ=2.0\delta=2.0 and 4.04.0 for RetinaMNIST II and III, and δ=4.0\delta=4.0 for BUSI III. Values are means ±\pm one standard error across 20 independent outer repetitions.
NNScenarioDR-KTEDR-KTE-OptDR-KTE-Deep
1500H0H_{0}0.051±0.0030.051{\scriptstyle\pm 0.003}0.052±0.0030.052{\scriptstyle\pm 0.003}0.050±0.0040.050{\scriptstyle\pm 0.004}
H1H_{1}0.225±0.0080.225{\scriptstyle\pm 0.008}0.229±0.0080.229{\scriptstyle\pm 0.008}0.380±0.050\mathbf{0.380{\scriptstyle\pm 0.050}}
3000H0H_{0}0.054±0.0040.054{\scriptstyle\pm 0.004}0.054±0.0040.054{\scriptstyle\pm 0.004}0.050±0.0030.050{\scriptstyle\pm 0.003}
H1H_{1}0.502±0.0120.502{\scriptstyle\pm 0.012}0.514±0.0120.514{\scriptstyle\pm 0.012}0.961±0.019\mathbf{0.961{\scriptstyle\pm 0.019}}
Appendix
Table 7: Type-I error ( H0H_{0} ) and power ( H1H_{1} ) for Fake-MNIST at α=0.05\alpha=0.05 . Values are means ±\pm one standard error across 20 independent outer repetitions.
 Power/Type-I error ( H1/H0H_{1}/H_{0} )
NNDR-KTEDR-KTE-OptDR-KTE-Deep
15000.322±0.014/0.049±0.0040.322{\scriptstyle\pm 0.014}/0.049{\scriptstyle\pm 0.004}0.351±0.016/0.051±0.0030.351{\scriptstyle\pm 0.016}/0.051{\scriptstyle\pm 0.003}0.382±0.024/0.050±0.004\mathbf{0.382}{\scriptstyle\pm 0.024}/0.050{\scriptstyle\pm 0.004}
30000.710±0.020/0.046±0.0030.710{\scriptstyle\pm 0.020}/0.046{\scriptstyle\pm 0.003}0.780±0.022/0.048±0.0030.780{\scriptstyle\pm 0.022}/0.048{\scriptstyle\pm 0.003}0.891±0.019/0.045±0.003\mathbf{0.891}{\scriptstyle\pm 0.019}/0.045{\scriptstyle\pm 0.003}
Appendix
Table 8: Power and Type-I error for Causal-Blob at α=0.05\alpha=0.05 . Each cell reports H1/H0H_{1}/H_{0} , with means ±\pm one standard error across 20 independent outer repetitions.
  Power/Type-I error ( H1/H0H_{1}/H_{0} )
NNScenarioDR-KTEDR-KTE-OptDR-KTE-Deep
1500II0.999±0.000/0.055±0.0040.999{\scriptstyle\pm 0.000}/0.055{\scriptstyle\pm 0.004}1.000±0.000/0.059±0.004\mathbf{1.000}{\scriptstyle\pm 0.000}/0.059{\scriptstyle\pm 0.004}0.995±0.002/0.055±0.0040.995{\scriptstyle\pm 0.002}/0.055{\scriptstyle\pm 0.004}
III0.611±0.028/0.055±0.0040.611{\scriptstyle\pm 0.028}/0.055{\scriptstyle\pm 0.004}0.653±0.027/0.059±0.004\mathbf{0.653}{\scriptstyle\pm 0.027}/0.059{\scriptstyle\pm 0.004}0.614±0.028/0.055±0.0040.614{\scriptstyle\pm 0.028}/0.055{\scriptstyle\pm 0.004}
IV0.980±0.007/0.055±0.0040.980{\scriptstyle\pm 0.007}/0.055{\scriptstyle\pm 0.004}0.985±0.005/0.059±0.004\mathbf{0.985}{\scriptstyle\pm 0.005}/0.059{\scriptstyle\pm 0.004}0.981±0.004/0.055±0.0040.981{\scriptstyle\pm 0.004}/0.055{\scriptstyle\pm 0.004}
3000II1.000±0.000/0.048±0.003\mathbf{1.000}{\scriptstyle\pm 0.000}/0.048{\scriptstyle\pm 0.003}1.000±0.000/0.046±0.003\mathbf{1.000}{\scriptstyle\pm 0.000}/0.046{\scriptstyle\pm 0.003}1.000±0.000/0.048±0.003\mathbf{1.000}{\scriptstyle\pm 0.000}/0.048{\scriptstyle\pm 0.003}
III0.903±0.013/0.048±0.0030.903{\scriptstyle\pm 0.013}/0.048{\scriptstyle\pm 0.003}0.927±0.012/0.046±0.003\mathbf{0.927}{\scriptstyle\pm 0.012}/0.046{\scriptstyle\pm 0.003}0.914±0.013/0.048±0.0030.914{\scriptstyle\pm 0.013}/0.048{\scriptstyle\pm 0.003}
Appendix
Table 9: Power and Type-I error for scalar outcomes at α=0.05\alpha=0.05 , with δ=0.8\delta=0.8 and M=5M=5 . Each cell reports H1/H0H_{1}/H_{0} , with means ±\pm one standard error across 20 independent outer repetitions.
 Power/Type-I error ( H1/H0H_{1}/H_{0} )
ScenarioDR-KTE-DeepIPW-CrossKPT
Scenario II0.961±0.007/0.048±0.003\mathbf{0.961}{\scriptstyle\pm 0.007}/0.048{\scriptstyle\pm 0.003}0.920±0.012/0.085±0.0090.920{\scriptstyle\pm 0.012}/0.085{\scriptstyle\pm 0.009}0.638±0.022/0.070±0.0130.638{\scriptstyle\pm 0.022}/0.070{\scriptstyle\pm 0.013}
Scenario III0.739±0.011/0.048±0.003\mathbf{0.739}{\scriptstyle\pm 0.011}/0.048{\scriptstyle\pm 0.003}0.574±0.008/0.085±0.0090.574{\scriptstyle\pm 0.008}/0.085{\scriptstyle\pm 0.009}0.291±0.017/0.070±0.0130.291{\scriptstyle\pm 0.017}/0.070{\scriptstyle\pm 0.013}
Scenario IV0.945±0.009/0.048±0.003\mathbf{0.945}{\scriptstyle\pm 0.009}/0.048{\scriptstyle\pm 0.003}0.903±0.013/0.085±0.0090.903{\scriptstyle\pm 0.013}/0.085{\scriptstyle\pm 0.009}0.607±0.027/0.070±0.0130.607{\scriptstyle\pm 0.027}/0.070{\scriptstyle\pm 0.013}
Appendix
Table 10: Power and Type-I error at α=0.05\alpha=0.05 , N=1600N=1600 and M=1M=1 for different choices of estimator. Each cell reports H1/H0H_{1}/H_{0} , with means ±\pm one standard error across 20 independent outer repetitions.
 Power/Type-I error ( H1/H0H_{1}/H_{0} )
EnvironmentFitted μ^θ\hat{\mu}_{\theta}μ^θ≡0\hat{\mu}_{\theta}\equiv 0
Causal-Blob0.318±0.024/0.047±0.0060.318{\scriptstyle\pm 0.024}/0.047{\scriptstyle\pm 0.006}0.247±0.017/0.047±0.0070.247{\scriptstyle\pm 0.017}/0.047{\scriptstyle\pm 0.007}
Scenario II0.999±0.001/0.056±0.0040.999{\scriptstyle\pm 0.001}/0.056{\scriptstyle\pm 0.004}0.927±0.012/0.057±0.0030.927{\scriptstyle\pm 0.012}/0.057{\scriptstyle\pm 0.003}
Scenario III0.616±0.015/0.056±0.0040.616{\scriptstyle\pm 0.015}/0.056{\scriptstyle\pm 0.004}0.349±0.013/0.057±0.0030.349{\scriptstyle\pm 0.013}/0.057{\scriptstyle\pm 0.003}
Scenario IV1.000±0.000/0.056±0.0041.000{\scriptstyle\pm 0.000}/0.056{\scriptstyle\pm 0.004}0.999±0.001/0.057±0.0030.999{\scriptstyle\pm 0.001}/0.057{\scriptstyle\pm 0.003}
Appendix
Table 11: Power and Type-I error at α=0.05\alpha=0.05 , N=1500N=1500 and M=1M=1 . Each cell reports H1/H0H_{1}/H_{0} , with means ±\pm one standard error across 20 independent outer repetitions.
Scenario Random WeightsPixel AutoencoderDR-KTE-Deep
OCT-DrusenH1H_{1}0.057±0.0040.057\pm 0.0040.052±0.0040.052\pm 0.0040.371±0.080\mathbf{0.371\pm 0.080}
H0H_{0}0.054±0.0040.054\pm 0.0040.050±0.0030.050\pm 0.0030.058±0.0050.058\pm 0.005
Fake-MNISTH1H_{1}0.204±0.0080.204\pm 0.0080.200±0.0120.200\pm 0.0120.380±0.050\mathbf{0.380\pm 0.050}
H0H_{0}0.053±0.0040.053\pm 0.0040.055±0.0040.055\pm 0.0040.050±0.0040.050\pm 0.004
Appendix
Table 12: Power under H1H_{1} and Type-I error under H0H_{0} at N=1500N=1500 . Values are means and one standard error across 20 independent outer repetitions.
 Power/Type-I error ( H1/H0H_{1}/H_{0} )
ScenarioDR-KTEDR-KTE-OptDR-KTE-Deep
Oncology Notes0.828±0.035/0.057±0.010\mathbf{0.828}{\scriptstyle\pm 0.035}/0.057{\scriptstyle\pm 0.010}0.827±0.035/0.058±0.0100.827{\scriptstyle\pm 0.035}/0.058{\scriptstyle\pm 0.010}0.828±0.036/0.058±0.010\mathbf{0.828}{\scriptstyle\pm 0.036}/0.058{\scriptstyle\pm 0.010}
Movie Reviews0.892±0.009/0.029±0.0040.892{\scriptstyle\pm 0.009}/0.029{\scriptstyle\pm 0.004}0.895±0.008/0.029±0.004\mathbf{0.895}{\scriptstyle\pm 0.008}/0.029{\scriptstyle\pm 0.004}0.895±0.008/0.029±0.004\mathbf{0.895}{\scriptstyle\pm 0.008}/0.029{\scriptstyle\pm 0.004}
Clinical Notes0.894±0.027/0.049±0.007\mathbf{0.894}{\scriptstyle\pm 0.027}/0.049{\scriptstyle\pm 0.007}0.894±0.027/0.049±0.007\mathbf{0.894}{\scriptstyle\pm 0.027}/0.049{\scriptstyle\pm 0.007}0.894±0.027/0.050±0.007\mathbf{0.894}{\scriptstyle\pm 0.027}/0.050{\scriptstyle\pm 0.007}
Appendix
Table 13: Power ( H1H_{1} ) and Type-I error ( H0H_{0} ) at N=1500N=1500 and α=0.05\alpha=0.05 for text outcomes represented as text embeddings.
DatasetStatisticM=1M=1M=2M=2M=3M=3M=5M=5
Causal-BlobPower0.350±0.0030.350{\scriptstyle\pm 0.003}0.373±0.0070.373{\scriptstyle\pm 0.007}0.383±0.0070.383{\scriptstyle\pm 0.007}0.394±0.0080.394{\scriptstyle\pm 0.008}
 SD0.150±0.0110.150{\scriptstyle\pm 0.011}0.157±0.0150.157{\scriptstyle\pm 0.015}0.150±0.0140.150{\scriptstyle\pm 0.014}0.151±0.0140.151{\scriptstyle\pm 0.014}
OCT-DrusenPower0.993±0.0000.993{\scriptstyle\pm 0.000}0.995±0.0000.995{\scriptstyle\pm 0.000}0.995±0.0000.995{\scriptstyle\pm 0.000}0.996±0.0010.996{\scriptstyle\pm 0.001}
 SD0.011±0.0010.011{\scriptstyle\pm 0.001}0.009±0.0020.009{\scriptstyle\pm 0.002}0.009±0.0020.009{\scriptstyle\pm 0.002}0.007±0.0010.007{\scriptstyle\pm 0.001}
Appendix
Table 14: Mean and standard deviation (SD) of empirical power at N=1500N=1500 and δ=1.0\delta=1.0 . Each statistic is computed across 20 independent outer repetitions within each batch; values are means ±\pm one standard error across five independent batches.
Figure 13: Average standard deviation of empirical power against ensemble size at N=1500N=1500 and δ=1.0\delta=1.0 . Shaded bands show ±\pm one standard error across five independent batches.
 Power/Type-I error ( H1/H0H_{1}/H_{0} )
ScenarioDR-KTE-WLDR-KTE-WL-Deep
LLM Code Repairs0.743±0.023/0.059±0.0060.743{\scriptstyle\pm 0.023}/0.059{\scriptstyle\pm 0.006}0.964±0.008/0.050±0.004\mathbf{0.964}{\scriptstyle\pm 0.008}/0.050{\scriptstyle\pm 0.004}
Contact Networks0.971±0.005/0.057±0.0030.971{\scriptstyle\pm 0.005}/0.057{\scriptstyle\pm 0.003}0.972±0.005/0.056±0.004\mathbf{0.972}{\scriptstyle\pm 0.005}/0.056{\scriptstyle\pm 0.004}
Appendix
Table 15: Power ( H1H_{1} ) and Type-I error ( H0H_{0} ) for graph outcomes using Weisfeiler-Lehman base kernels at N=1500N=1500 . Each cell reports H1/H0H_{1}/H_{0} , with means ±\pm one standard error across 20 outer repetitions. Gaussian-kernel results are given in Table 3 .
 Power/Type-I error ( H1/H0H_{1}/H_{0} )
ScenarioDR-ATEDR-KTE-Deep
Breast-Tumor0.052±0.003/0.047±0.0040.052{\scriptstyle\pm 0.003}/0.047{\scriptstyle\pm 0.004}0.989±0.002/0.054±0.003\mathbf{0.989}{\scriptstyle\pm 0.002}/0.054{\scriptstyle\pm 0.003}
Chest-Cardiomegaly0.044±0.003/0.043±0.0030.044{\scriptstyle\pm 0.003}/0.043{\scriptstyle\pm 0.003}0.928±0.046/0.051±0.004\mathbf{0.928}{\scriptstyle\pm 0.046}/0.051{\scriptstyle\pm 0.004}
Chest-Effusion0.046±0.005/0.049±0.0030.046{\scriptstyle\pm 0.005}/0.049{\scriptstyle\pm 0.003}0.999±0.000/0.048±0.004\mathbf{0.999}{\scriptstyle\pm 0.000}/0.048{\scriptstyle\pm 0.004}
Chest-Mass0.052±0.003/0.051±0.0040.052{\scriptstyle\pm 0.003}/0.051{\scriptstyle\pm 0.004}1.000±0.000/0.056±0.004\mathbf{1.000}{\scriptstyle\pm 0.000}/0.056{\scriptstyle\pm 0.004}
Derma-Melanoma0.045±0.004/0.048±0.0020.045{\scriptstyle\pm 0.004}/0.048{\scriptstyle\pm 0.002}0.898±0.040/0.053±0.004\mathbf{0.898}{\scriptstyle\pm 0.040}/0.053{\scriptstyle\pm 0.004}
Appendix
Table 16: Comparison with DR-ATE for medical-image outcomes at N=1500N=1500 . Each cell reports H1/H0H_{1}/H_{0} , with means ±\pm one standard error across 20 outer repetitions.
DatasetScenarioDR-ATEDR-KTE-Deep
MNISTI0.045±0.0030.045{\scriptstyle\pm 0.003}0.051±0.0030.051{\scriptstyle\pm 0.003}
II0.677±0.0270.677{\scriptstyle\pm 0.027}0.870±0.024\mathbf{0.870}{\scriptstyle\pm 0.024}
III0.695±0.0230.695{\scriptstyle\pm 0.023}0.803±0.023\mathbf{0.803}{\scriptstyle\pm 0.023}
RetinaMNISTI0.052±0.0030.052{\scriptstyle\pm 0.003}0.051±0.0040.051{\scriptstyle\pm 0.004}
II0.434±0.0160.434{\scriptstyle\pm 0.016}0.647±0.022\mathbf{0.647}{\scriptstyle\pm 0.022}
III0.235±0.0100.235{\scriptstyle\pm 0.010}0.732±0.035\mathbf{0.732}{\scriptstyle\pm 0.035}
Appendix
Table 17: Comparison with DR-ATE for MNIST, RetinaMNIST, BUSI and Fake-MNIST at N=1500N=1500 . Scenario I and Fake-MNIST H0H_{0} report Type-I error; Scenarios II and III and Fake-MNIST H1H_{1} report power. Values are means ±\pm one standard error across 20 outer repetitions.
 Power/Type-I error ( H1/H0H_{1}/H_{0} )
ScenarioDR-ATEDR-KTE-Deep
Oncology Notes0.375±0.015/0.051±0.0030.375{\scriptstyle\pm 0.015}/0.051{\scriptstyle\pm 0.003}0.809±0.023/0.055±0.003\mathbf{0.809}{\scriptstyle\pm 0.023}/0.055{\scriptstyle\pm 0.003}
Movie Reviews0.141±0.009/0.048±0.0030.141{\scriptstyle\pm 0.009}/0.048{\scriptstyle\pm 0.003}0.449±0.026/0.053±0.004\mathbf{0.449}{\scriptstyle\pm 0.026}/0.053{\scriptstyle\pm 0.004}
Clinical Notes0.053±0.004/0.053±0.0030.053{\scriptstyle\pm 0.004}/0.053{\scriptstyle\pm 0.003}0.939±0.013/0.058±0.004\mathbf{0.939}{\scriptstyle\pm 0.013}/0.058{\scriptstyle\pm 0.004}
Appendix
Table 18: Comparison with DR-ATE for text outcomes at N=1500N=1500 . Each cell reports H1/H0H_{1}/H_{0} , with means ±\pm one standard error across 20 outer repetitions.
 Power/Type-I error ( H1/H0H_{1}/H_{0} )
ScenarioDR-ATEDR-KTE-Deep
LLM Code Repairs0.874±0.019/0.056±0.0040.874{\scriptstyle\pm 0.019}/0.056{\scriptstyle\pm 0.004}1.000±0.000/0.051±0.003\mathbf{1.000}{\scriptstyle\pm 0.000}/0.051{\scriptstyle\pm 0.003}
Contact Networks0.923±0.009/0.059±0.0040.923{\scriptstyle\pm 0.009}/0.059{\scriptstyle\pm 0.004}0.962±0.006/0.051±0.002\mathbf{0.962}{\scriptstyle\pm 0.006}/0.051{\scriptstyle\pm 0.002}
Appendix
Table 19: Comparison with DR-ATE for graph outcomes at N=1500N=1500 . Each cell reports H1/H0H_{1}/H_{0} , with means ±\pm one standard error across 20 outer repetitions.
 Power/Type-I error ( H1/H0H_{1}/H_{0} )
ScenarioDR-ATEDR-KTE-Deep
Causal-Blob0.049±0.003/0.043±0.0020.049{\scriptstyle\pm 0.003}/0.043{\scriptstyle\pm 0.002}0.382±0.024/0.050±0.004\mathbf{0.382}{\scriptstyle\pm 0.024}/0.050{\scriptstyle\pm 0.004}
Appendix
Table 20: Comparison with DR-ATE on Causal-Blob at N=1500N=1500 . Each cell reports H1/H0H_{1}/H_{0} , with means ±\pm one standard error across 20 outer repetitions.
 Power/Type-I error ( H1/H0H_{1}/H_{0} )
ScenarioDR-ATEDR-KTE-Deep
Scenario II0.939±0.009/0.097±0.0070.939{\scriptstyle\pm 0.009}/0.097{\scriptstyle\pm 0.007}0.995±0.002/0.055±0.004\mathbf{0.995}{\scriptstyle\pm 0.002}/0.055{\scriptstyle\pm 0.004}
Scenario III0.050±0.004/0.097±0.0070.050{\scriptstyle\pm 0.004}/0.097{\scriptstyle\pm 0.007}0.614±0.028/0.055±0.004\mathbf{0.614}{\scriptstyle\pm 0.028}/0.055{\scriptstyle\pm 0.004}
Scenario IV0.972±0.006/0.097±0.0070.972{\scriptstyle\pm 0.006}/0.097{\scriptstyle\pm 0.007}0.981±0.004/0.055±0.004\mathbf{0.981}{\scriptstyle\pm 0.004}/0.055{\scriptstyle\pm 0.004}
Appendix
Table 21: Comparison with DR-ATE for scalar outcomes at N=1500N=1500 . Each alternative is paired with the shared Scenario I null. Each cell reports H1/H0H_{1}/H_{0} , with means ±\pm one standard error across 20 outer repetitions.
Figure 14: Causal-Blob at N=4000N=4000 when varying the outcome dimension, with δ=0\delta=0 under H0H_{0} and δ=1.0\delta=1.0 under H1H_{1} . Shaded bands show ±\pm one standard error across 20 independent outer repetitions. The dotted grey line marks α=0.05\alpha=0.05 .
 Power/Type-I error ( H1/H0H_{1}/H_{0} )
ScenarioDR-KTEDR-KTE-DeepJoint maximisation
Breast-Tumor0.749±0.018/0.053±0.0030.749{\scriptstyle\pm 0.018}/0.053{\scriptstyle\pm 0.003}0.989±0.002/0.054±0.003\mathbf{0.989}{\scriptstyle\pm 0.002}/0.054{\scriptstyle\pm 0.003}0.989±0.003/0.053±0.003\mathbf{0.989}{\scriptstyle\pm 0.003}/0.053{\scriptstyle\pm 0.003}
Chest-Cardiomegaly0.156±0.007/0.050±0.0030.156{\scriptstyle\pm 0.007}/0.050{\scriptstyle\pm 0.003}0.928±0.046/0.051±0.004\mathbf{0.928}{\scriptstyle\pm 0.046}/0.051{\scriptstyle\pm 0.004}0.763±0.077/0.051±0.0040.763{\scriptstyle\pm 0.077}/0.051{\scriptstyle\pm 0.004}
Chest-Effusion0.231±0.009/0.053±0.0040.231{\scriptstyle\pm 0.009}/0.053{\scriptstyle\pm 0.004}0.999±0.000/0.048±0.004\mathbf{0.999}{\scriptstyle\pm 0.000}/0.048{\scriptstyle\pm 0.004}0.998±0.001/0.056±0.0040.998{\scriptstyle\pm 0.001}/0.056{\scriptstyle\pm 0.004}
Chest-Mass0.436±0.018/0.050±0.0030.436{\scriptstyle\pm 0.018}/0.050{\scriptstyle\pm 0.003}1.000±0.000/0.056±0.004\mathbf{1.000}{\scriptstyle\pm 0.000}/0.056{\scriptstyle\pm 0.004}0.999±0.000/0.050±0.0040.999{\scriptstyle\pm 0.000}/0.050{\scriptstyle\pm 0.004}
Derma-Melanoma0.171±0.009/0.051±0.0020.171{\scriptstyle\pm 0.009}/0.051{\scriptstyle\pm 0.002}0.898±0.040/0.053±0.004\mathbf{0.898}{\scriptstyle\pm 0.040}/0.053{\scriptstyle\pm 0.004}0.795±0.060/0.046±0.0030.795{\scriptstyle\pm 0.060}/0.046{\scriptstyle\pm 0.003}
Appendix
Table 22: Effect of joint maximisation at N=1500N=1500 and α=0.05\alpha=0.05 . Each cell reports H1/H0H_{1}/H_{0} , with means ±\pm one standard error across 20 outer repetitions.
More in this topic

Two-Sample Testing

Two-sample testing develops statistical methods for deciding whether two sets of observations come from the same distribution.

See all 33 papers

Explore similar work

CardsList
May 8, 2026

Semiparametric Efficient Test for Interpretable Distributional Treatment Effects

Distributional treatment effects can be invisible to means: a treatment may preserve average outcomes while changing tails, modes, dispersion, or rare-event probabilities. Kernel tests can detect discrepancies between interventional outcome laws, but global tests do not reveal where the laws differ. We propose DR-ME, to our knowledge the first semiparametrically efficient finite-location test for interpretable distributional treatment effects. DR-ME evaluates an interventional kernel witness at learned outcome locations, returning causal-discrepancy coordinates rather than only a global rejection. From observational data, we derive orthogonal doubly robust kernel features whose centered oracle form is the canonical gradient of this finite witness. For fixed locations, we characterize the local testing limit: DR-ME is chi-square calibrated under the null, has noncentral chi-square local power, and uses the covariance whitening that optimizes local signal-to-noise for discrepancies visible through the selected coordinates. This efficient local-power geometry yields a principled location-learning criterion, with sample splitting preserving post-selection validity. Experiments show near-nominal type-I error, competitive power against global doubly robust kernel tests, and interpretable learned locations that localize distributional effects in a semi-synthetic medical-imaging study.

Houssam Zenati, Arthur Gretton

1Gatsby Computational Neuroscience Unit, University College London

Semiparametric InferenceCausal Effect EstimationKernel MethodsCausal InferenceCausal InterpretabilityDoubly Robust Estimation+1
AskPDF
Mar 17, 2026

Conditional Distributional Treatment Effects: Doubly Robust Estimation and Testing

Beyond conditional average treatment effects, treatments may impact the entire outcome distribution in covariate-dependent ways, for example, by altering the variance or tail risks for specific subpopulations. We propose a novel estimand to capture such conditional distributional treatment effects, and develop a doubly robust estimator that is minimax optimal in the local asymptotic sense. Using this, we develop a test for the global homogeneity of conditional potential outcome distributions that accommodates discrepancies beyond the maximum mean discrepancy (MMD), has provably valid type 1 error, and is consistent against fixed alternatives---the first test, to our knowledge, with such guarantees in this setting. We then provide a test that aggregates evidence across a grid of kernel-bandwidth choices. Furthermore, we derive exact closed-form expressions for two natural discrepancies (including the MMD), and provide a computationally efficient, permutation-free algorithm for our test.

Saksham Jain, Alex Luedtke

Department of Statistics, University of Washington, Seattle, USA · Departments of Health Care Policy and Statistics, Harvard University, USA

Causal Effect EstimationCausal InferenceDoubly Robust EstimationStatistical Hypothesis Testing
AskPDF
Apr 29, 2026

Deep-testing: the case of dependence detection

Deep learning methods have proved highly effective for classification and image recognition problems. In this paper, we ask whether this success can be transferred to hypothesis testing: if a neural network can distinguish, for example, an image of a handwritten digit from another, can it also distinguish an "image of a sample" (such as a scatter plot) generated under a given statistical model from one generated outside that model? Motivated by this idea, we propose a novel procedure called deep-testing, which approaches the classical inferential problem of hypothesis testing through deep learning. More specifically, the test statistic is a classification map learned by a deep neural network from simulated data satisfying the null and alternative hypotheses, leveraging its strong discriminating power to construct a highly powerful test. As a proof of concept, we apply deep-testing to the problem of independence testing, arguably one of the most important problems in statistics. In a large-scale simulation study, deep-testing achieves the highest overall power against nineteen competing methods across a broad range of complex dependence structures, confirming the viability of the proposed approach.

Gery Geenens, Pierre Lafaye de Micheaux, Ivan Muyun Zou

School of Mathematics and statistics, UNSW Sydney, Australia

Statistical Hypothesis Testing
AskPDF