cs.LGNov 5, 2026

Non-Parametric Mutual Information Estimation with Vine Copulas

Authors: Sigurd Holmsen, Berit Omli Øksnes, Ingrid Hobæk Haff, Sylvia Richardson, Ali Ramezani-Kebrya

Organizations: Department of Informatics, University of Oslo · Integreat – Norwegian Centre for Knowledge-driven Machine Learning · Department of Mathematics, University of Oslo · TRUST – The Norwegian Centre for Trustworthy AI · 

Abstract

We introduce VINE: a non-parametric mutual information (MI) estimator based on simplified D-vine copulas. Approximating the joint density with a simplified D-vine allows the density ratio to simplify to a product of pair copulas, which we can estimate non-parametrically. This structural simplification yields VINE, a flexible and computationally efficient MI estimator. We compare VINE against SotA neural MI estimators on a range of datasets, both under settings where the simplified D-vine assumption holds and where it is violated, demonstrating competitive accuracy at a lower computational cost for low- to medium-dimensional datasets.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Apr 22, 2026

Amortized Vine Copulas for High-Dimensional Density and Information Estimation

Modeling high-dimensional dependencies while keeping likelihoods tractable re- mains challenging. Classical vine copulas provide an explicit dependence structure but require repeated, costly bivariate fitting. We propose Vine Denoising Copula (VDC)1, an amortized estimator for simplified vines on continuous data. VDC trains a single bivariate denoising network on simulated copulas and reuses it across all vine edges. For each edge, the network maps an empirical histogram to a density grid, and iterative proportional fitting approximately enforces uniform grid marginals. The predicted densities and cached conditional transforms support stan- dard vine recursion, likelihood evaluation, sampling, and information estimation. This retains the classical vine factorization while replacing repeated per-edge fitting with reusable neural inference. Across synthetic and real-data benchmarks, VDC combines accurate bivariate density estimation, competitive information estimates, and faster high-dimensional vine fitting after pretraining. Performance varies across datasets; finite grid resolution and conditional prediction remain limitations. The explicit fitted structure supports edge- and tree-level information decomposition, making repeated dependence analysis more practical.
Jul 3, 2026

Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation

Mutual information estimation is a central problem in machine learning and statistics; however, existing benchmarks typically evaluate estimators on simplified, low-dimensional distributions, leaving their performance on complex, realistic data largely unexplored. We address this gap with a comprehensive benchmarking framework grounded in a unified copula-theoretic perspective that subsumes existing benchmarks as special cases. Within this framework, we propose two complementary families of tests: a copula-first family that systematically varies ground-truth mutual information, dimensionality, and marginal complexity using synthetic and flow-based transformations; and a marginals-first family that couples real-world image data with controlled dependency structures, extending the classic same-class-pairing paradigm. We use this suite to extensively evaluate three classes of estimators: non-parametric, variational, and generative. Contrary to prevailing assumptions, our results indicate that there is no universal winner: each category can systematically outperform all other estimators under specific setups. By analyzing these cases, we identify fundamental estimation barriers and propose new tests that more effectively stress these specific limitations. We share the open source code at https://github.com/VanessB/mutinfo.
Jul 30, 2026

NMINE: Normalized Mutual Information Neural Estimation

Mutual information is a general measure of statistical dependence that captures both linear and nonlinear relationships between random variables. For continuous and multidimensional variables For continuous multidimensional variables, mutual information must be estimated from samples. Because mutual information is unbounded, its values are not directly comparable across datasets, dimensions, or applications. Normalized mutual information addresses this limitation by converting mutual information into a normalized dependency score. Recent work has demonstrated the practical value of normalized mutual information in applications such as molecular dynamics {arXiv:2405.04980} and interpretable machine learning {arXiv:2409.16768}, but existing estimators remain sensitive to dimensionality and numerical stability {arXiv:2410.07642}. In this paper, we propose a fully neural normalized mutual information estimator for continuous variables. The proposed approach combines a MINE-based neural mutual information estimator {arXiv:1801.04062} with MI-NEE-inspired neural marginal entropy estimators {arXiv:1905.12957}. Mutual information is estimated using the Donsker--Varadhan representation, while marginal entropies are estimated by learning the divergence between each marginal distribution and a uniform reference distribution, from which entropy is recovered. The resulting estimator provides a neural alternative to k-nearest-neighbor-based normalized mutual information estimation {arXiv:2405.04980}. Experiments on Gaussian data from one to eight dimensions show that the proposed estimator improves accuracy over a KSG-based normalized mutual information baseline. These results indicate that neural estimation is a promising direction for normalized dependency measurement in continuous multidimensional settings.