Kernel-based External Validity of Experimental Studies

Federico Matteucci; Gerhard Neumann

stat.MEOct 9, 2026

Kernel-based External Validity of Experimental Studies

Authors: Federico Matteucci, Gerhard Neumann

Organizations: Karlsruhe Institute of Technology

Abstract

The external validity of a study is the extent to which its findings generalize to unseen experimental conditions, such as datasets. We formalize the results of a study together with its research question, and define external validity as the probability that two independent and same-sized replications of the study produce results whose maximum mean discrepancy (MMD) is below a given tolerance. This approach, centered on the choice of a space of results and of an appropriate reproducing kernel for the MMD, is flexible enough to be deployed on multiple different types of results and research questions. Additionally, we establish an asymptotic power-law relation between the size of a study and the quantiles of the MMD, which we use to estimate the desired size of an externally valid study from a small-scale preliminary study. We deploy our framework to rankings with ties and discuss kernels suited to different research questions. Experiments on existing case studies and on a simulated newly designed study illustrate the applicability of the framework. We release our code as part of the genexpy Python module.

Figures & tables

Figure 1: Example 1 as an experimental study. We are interested in the external validity of the results

P

for a fixed configuration

m=1

Kernel	$d(x_{1},x_{2})$	$g(u)$
Borda $k^{a^{*},\nu}_{\mathrm{b}}$	$\|b_{1}-b_{2}\|/(n_{a}-1)$	$e^{-\nu(n_{a}-1)u}$
Jaccard $k^{t}_{\mathrm{j}}$	$1-J_{t}(r_{1},r_{2})$	$1-u$
Mallows $k^{\nu}_{\mathrm{m}}$	$n_{d}/\binom{n_{a}}{2}$	$e^{-\nu\binom{n_{a}}{2}u}$
RBF $k^{\gamma}_{\mathrm{RBF}}$	$\\|x_{1}-x_{2}\\|^{2}/n_{a}$	$e^{-\gamma n_{a}u}$

Table 1: Interpretable form

k=g\circ d

( Assumption 7 ) of the kernels used in this work.

Figure 2: Actual (

\operatorname{E}_{\hat{n}^{*}_{N}}(P,\varepsilon)

) vs. desired

(\alpha)

validity.

Figure 3: Necessary size

n^{*}

to achieve

(\alpha,\varepsilon(\delta),n^{*})

-validity. (Left)

\delta=0.05

; (Right)

\alpha=0.95

. Variability across configuration.

Figure 4: KEV for varying

\delta

-s and

n

-s. The shaded regions correspond to all configurations.

Figure 5: Necessary size

n^{*}

to achieve

(\alpha,\varepsilon(\delta),n^{*})

-validity. (Left)

\delta=0.05

; (Right)

\alpha=0.95

. Variability across configuration.

Figure 6: KEV for varying

\delta

-s and

n

-s. The shaded regions correspond to all configurations.

Figure 7: Proposed execution of an experimental study. (Top) The KEV is estimated from

N

preliminary experiments, for

n\in[\lfloor N/2\rfloor]

. (Bottom) A power law relation is fitted between

n

and the

0.95

-quantile of

\operatorname{MMD}_{n}

;

\hat{n}^{*}_{N}

is the prediction at

\varepsilon(\delta=0.05)=0.312

(

\star

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: KEV at the estimated $\hat{n}^{*}_{N}$ compared to the desired level $\alpha$ . The cautiously pessimistic scenario corresponds to a slightly positive difference. $P={\mathcal{U}}({\mathcal{R}}_{8})$ and we use the Mallows kernel.

disjoint	replace	Collisions within	Collisions between	Scheme
True	True	yes	no	overpessimistic
True	False	no	no	permutation
False	False	no	yes	overoptimistic
False	True	yes	yes	bootstrap

Appendix

Table 2: Resampling schemes and their expected effect on the estimated KEV.

Panel of Figure 9 — Figure 9: KEV compared for incremental and fresh sampling schemes.

Scientific Reproducibility

Scientific reproducibility studies whether research methods, data, and software can be inspected and rerun to support published findings.

See all 108 papers

Kernel Selection is Model Selection: A Unified Complexity-Penalized Approach for MMD Two-Sample Tests
May 7, 2026Yijin Ni, Xiaoming HuoModel Selection Two-Sample Testing
H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta, GA 30332, USA
Validity Threats for Foundation Model Research
Jun 3, 2026Gunnar König, Martin Pawelczyk, Ulrike von Luxburg +1
University of Tübingen, Tübingen AI Center · University of Vienna
Maximum Mean Discrepancy with Unequal Sample Sizes via Generalized U-Statistics
Dec 16, 2025Aaron Wei, Milad Jalali, Danica J. SutherlandTwo-Sample Testing Maximum Mean Discrepancy
University of British Columbia · Independent researcher, Vancouver, Canada · Alberta Machine Intelligence Institute

May 7, 2026J/K move · Enter open · S save

Kernel Selection is Model Selection: A Unified Complexity-Penalized Approach for MMD Two-Sample Tests

Yijin Ni, Xiaoming Huo

H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta, GA 30332, USA

Model Selection Two-Sample Testing Maximum Mean Discrepancy Kernel Methods Statistical Learning Theory

Open paper AskPDF arXiv

The Maximum Mean Discrepancy (MMD) is a cornerstone statistic for nonparametric two-sample testing, but its test power is dictated entirely by the chosen kernel. Because any fixed kernel inherently fails to distinguish certain distributions, the kernel must be dynamically optimized. However, data-driven optimization violates the foundational i.i.d. assumption, forcing a strict trade-off in existing frameworks. Ratio criteria ignore this dependence, inducing overfitting and variance collapse on rich kernel classes. Conversely, aggregation methods bypass the dependence using finite grids, but this strategy cannot scale to continuous search spaces like deep kernels. To break this dichotomy, we establish data-driven kernel selection as a model selection problem. We propose Complexity-Penalized MMD (CP-MMD), a criterion derived by applying the two-sample uniform concentration inequality of preceding works to the post-optimization MMD problem. The resulting penalty bounds the empirical MMD by the complexity of the kernel search space, mathematically absorbing the cost of optimization, so that CP-MMD enables direct, grid-free maximization over continuous parametric classes, including scalar bandwidths, polynomial feature bandwidths, and deep network parameters. By formally accounting for optimization complexity, we prove that CP-MMD maximizes true test power while ensuring unconditional Type-I validity. Consequently, CP-MMD enables grid-free kernel selection across linear, polynomial-feature, and deep regimes, matching or exceeding state-of-the-art test power.

Abstract

Scientific Reproducibility

Explore similar work

Kernel Selection is Model Selection: A Unified Complexity-Penalized Approach for MMD Two-Sample Tests

Validity Threats for Foundation Model Research

Maximum Mean Discrepancy with Unequal Sample Sizes via Generalized U-Statistics