We propose novel kernel-based tests for assessing the equivalence between distributions. Traditional goodness-of-fit testing is inappropriate for concluding the absence of distributional differences, because failure to reject the null hypothesis may simply be a result of lack of test power, also known as the Type-II error. This motivates equivalence testing, which aims to assess the absence of a statistically meaningful effect under controlled error rates. However, existing equivalence tests are either limited to parametric distributions or focus only on specific moments rather than the full distribution. We address these limitations using two kernel-based statistical discrepancies: the kernel Stein discrepancy and the Maximum Mean Discrepancy. The null hypothesis of our proposed tests assumes the candidate distribution differs from the nominal distribution by at least a pre-defined margin, which is measured by these discrepancies. We propose two approaches for computing the critical values of the tests, one using an asymptotic normality approximation, and another based on bootstrapping. Numerical experiments are conducted to assess the performance of these tests.
Figures & tables
Equivalence Test
Discrepancy
Setting
Method
Reference
E-KSD-Normal
KSD
One-sample
CLT
Theorem 2
E-KSD-Boot
KSD
One-sample
Bootstrapping
Theorem 4
E-MMD-Normal
MMD
Two-sample
CLT
Chen et al. (2023) and Theorem 6
E-MMD-Boot
MMD
Two-sample
Bootstrapping
Theorem 8
Table 1: Summary of kernel-based equivalence tests proposed in this paper, and their connections to existing literature.
Figure 1: Comparison of different hypothesis testing paradigms. The shaded area represents the space of distributions of interest. The null sets are shown in orange, and the alternative sets in grey. Left. standard testing with a point null hypothesis. Middle. Robust testing based on a statistical discrepancy D . Right. Equivalence testing.
Figure 2: Gaussian mean-shift experiments with varying sample sizes. The black dotted vertical line is the equivalence margin θ .
Figure 3: Gaussian mean-shift experiments with varying equivalence margins θ , selected to be the population KSD values for different mean shifts.
Figure 4: Gaussian mean-shift experiments with θ selected using a power guarantee with β=0.2 . Top . KSD-based equivalence tests. Bottom . MMD-based equivalence tests.
Figure 5: Gaussian-Bernoulli RBM experiment with θ selected using a Type-II error guarantee with β=0.2 . The same results are plotted against the KSD values (left) and the standard deviations of the noise (right).
Figure 6: Testing equivalence of MNIST samples. The θ is selected using a power guarantee with β=0.2 .
Statistical hypothesis testing develops methods for assessing evidence against a stated null while controlling false-positive and false-negative rates.
Kernel Stein discrepancies (KSDs) provide a versatile tool for comparing distributions. One of their main applications is in quantifying the goodness-of-fit (GoF) between a data-generating distribution and a prescribed target distribution. In this work, we study the related problem of conditional GoF quantification: given only a (possibly non-normalized) conditional target model, without information on the distribution of its covariates, and samples from a joint distribution, the goal is to assess how well the conditional distribution of the samples matches the target. To tackle this setting, we present a framework that allows lifting unconditional KSDs to the conditional setting through an operator-valued kernel on the covariate space, going beyond the known Euclidean case. We establish that our suggested statistic vanishes if and only if the conditional model and the true conditional distribution agree for almost all covariates and deploy it to test conditional GoF on smooth manifolds and on discrete spaces. Our experiments on level, power, and runtime demonstrate the viability of testing on these domains using the proposed statistic.
Models with intractable normalizing constants are widely used in statistics and machine learning. Assessing the adequacy of such models poses significant challenges: obtaining samples from the fitted model often requires sophisticated sampling algorithms. Moreover, model fitting sometimes requires iterative numerical optimization, making bootstrap procedures that require repeated refitting computationally expensive. In this paper, we leverage the kernel-based testing framework to develop a general semiparametric goodness-of-fit test based on the kernelized Stein discrepancy. We establish the consistency and the asymptotic null distribution of the test statistic under general nuisance estimation. To produce a level-α test, we propose a novel influence-adjusted wild bootstrap that requires neither refitting the model nor sampling from it. We prove the consistency of the proposed bootstrap test procedure under the null and the alternative, and characterize its limiting power under contiguous local alternatives. Across simulations ranging from classical normality testing to models with intractable likelihoods, the proposed test delivers competitive or superior power at a computational cost orders of magnitude lower than that of existing approaches. We illustrate the method by assessing the adequacy of a protein signaling network model for reverse-phase protein array data from lung adenocarcinoma tumors. As a complementary insight, we show that the SKSD test can be regarded as a nonparametric score test under exponentially tilted models, connecting score-based and distance-based goodness-of-fit testing.
Zhihan Huang, Ziang Niu
Department of Statistics and Data Science University of Pennsylvania
Kernel Stein discrepancy (KSD) is among the most popular goodness-of-fit (GoF) measures on general domains with a large number of successful deployments. One of the main applications of KSD is in constructing powerful GoF tests. However, tests relying on the classical U-/V-statistic-based KSD estimators have two major drawbacks. (i) Their runtime scales quadratically in the number of samples. (ii) Their asymptotic null distribution is computationally intractable in most cases, typically handled by bootstrapping. While it is known that the Nyström method permits accelerating KSD estimation with no loss of statistical accuracy under mild conditions, to the best of our knowledge, the fundamental question of its impact on bootstrap-based GoF testing is open; resolving this question is the focus of the current paper. In particular, we prove that the key properties of the quadratic-time bootstrapped KSD-based GoF test (asymptotic level and local consistency) are preserved by its Nyström acceleration. We numerically demonstrate the efficiency of the accelerated KSD estimator and bootstrap in the context of GoF testing of spherical and functional data. Our numerical results show that the Nyström-accelerated method performs statistically on-par with the quadratic-time approach, while requiring substantially smaller runtime.
Florian Kalinke, Zoltán Szabó, Bharath K. Sriperumbudur
Chair of Information Systems Karlsruhe Institute of Technology Am Fasanengarten 5, 76131 Karlsruhe, Germany · Department of Statistics London School of Economics Houghton Street, London, WC2A 2AE, UK · Department of Statistics The Pennsylvania State UniversityMay University Park, PA 16802, USA