Measuring Differences between Conditional Distributions using Kernel Embeddings

Peter Moskvichev; Siu Lun Chau; Dino Sejdinovic

stat.MLMay 4, 2026

Measuring Differences between Conditional Distributions using Kernel Embeddings

Authors: Peter Moskvichev, Siu Lun Chau, Dino Sejdinovic

Organizations: School of Mathematical Sciences, Adelaide University · College of Computing and Data Science, Nanyang Technological University

Abstract

Comparing conditional distributions is a fundamental challenge in statistics and machine learning, with applications across a wide range of domains. While proposed methods for measuring discrepancies using kernel embeddings of distributions in a reproducing kernel Hilbert space (RKHS) provide powerful non-parametric techniques, the existing literature remains fragmented and lacks a unified theoretical treatment. This paper addresses this gap by establishing a coherent framework for studying kernel-based methods to measure divergence between conditional distributions through what we refer to as conditional maximum mean discrepancy (CMMD). The CMMD consists of a family of metrics which we call levels, with three special cases each using a different type of RKHS embedding: CMMD $_0$ (conditional mean operators), CMMD $_1$ (conditional mean embeddings), and CMMD $_2$ (joint mean embeddings). We additionally introduce a general level $s$ CMMD, clarifying the required assumptions, and establishing mathematical connections between the levels through the lens of operator-based smoothing. In addition to reviewing previously proposed estimators, we introduce a novel doubly robust estimator for the CMMD that maintains consistency provided at least one of the underlying models is correctly specified. We provide numerical experiments demonstrating that the CMMD effectively captures complex conditional dependencies for statistical testing.

Explore similar work

Mar 17, 2026stat.ML

Conditional Distributional Treatment Effects: Doubly Robust Estimation and Testing

Beyond conditional average treatment effects, treatments may impact the entire outcome distribution in covariate-dependent ways, for example, by altering the variance or tail risks for specific subpopulations. We propose a novel estimand to capture such conditional distributional treatment effects, and develop a doubly robust estimator that is minimax optimal in the local asymptotic sense. Using this, we develop a test for the global homogeneity of conditional potential outcome distributions that accommodates discrepancies beyond the maximum mean discrepancy (MMD), has provably valid type 1 error, and is consistent against fixed alternatives---the first test, to our knowledge, with such guarantees in this setting. We then provide a test that aggregates evidence across a grid of kernel-bandwidth choices. Furthermore, we derive exact closed-form expressions for two natural discrepancies (including the MMD), and provide a computationally efficient, permutation-free algorithm for our test.

Saksham Jain, Alex Luedtke

Heterogeneous Treatment Effects

Ask PDF

Jul 27, 2026stat.ML

Minimax Lower Bounds of Kernel Discrepancy Estimation: MMD, HSIC, KSD

Over the past 20 years, kernel discrepancies have been leveraged as a highly powerful tool for quantifying the disagreement of distributions, with numerous successful applications in two-sample, goodness-of-fit, and independence testing, among others. Their fastest estimators are known to converge at a parametric rate---

n^{-1/2}

---under mild conditions. While this rate is known to be minimax optimal on

\mathbb R^d

under strict assumptions with bounded kernels, little is known about its optimality beyond the finite-dimensional Euclidean setting with unbounded kernels. In this work, we prove that the minimax lower bound of estimation of the most popular kernel discrepancies (maximum mean discrepancy, Hilbert-Schmidt independence criterion and kernel Stein discrepancy; MMD, HSIC, KSD) is

n^{-1/2}

on general topological spaces, and under mild assumptions on the kernel; the same rates are shown (as corollaries) to hold for the estimation of the mean embedding and the centered cross-covariance operator. Our results settle the question of optimal estimation of these kernel discrepancies.

Jose Cribeiro-Ramallo, Florian Kalinke, Zoltán Szabó

Maximum Mean Discrepancy Kernel Hilbert Spaces Concentration Inequalities Estimators Covariance Estimation Lower Bounds

Ask PDF

Jul 22, 2026stat.ML

Directional Kernel Mean Difference: A Fast Signed Statistic for Univariate Distribution Comparison

We introduce the Directional Kernel Mean Difference (DKMD), a signed statistic for univariate distribution comparison that preserves the direction of distributional shifts. Unlike the squared Maximum Mean Discrepancy (MMD), which discards directional information by squaring the RKHS distance, DKMD integrates the difference of kernel mean embeddings against a fixed odd weighting function. This construction yields three structural properties: antisymmetry, immunity to symmetric distributional differences, and directional monotonicity under stochastic dominance. We derive a data-driven Riemann estimator that ensures asymptotic consistency with the continuous formulation, strictly preserving the theoretical guarantees of the signed statistic in empirical evaluations. To overcome the quadratic computational cost of kernel methods, we develop an

O(N \log N)

prefix--suffix scanning algorithm that exploits the total order of the real line while requiring only

O(N)

memory. Experiments on synthetic benchmarks demonstrate that DKMD correctly isolates directional shifts from symmetric perturbations, remains robust to heavy-tailed outliers that can flip the sign of the mean difference, and scales to millions of samples in seconds.

Shijie Zhong, Jiangfeng Fu

Maximum Mean Discrepancy Kernel Method Density Ratio Estimation Distance Estimators Statistical Inference

Ask PDF