Non-Parametric Mutual Information Estimation with Vine Copulas
Organizations: Department of Informatics, University of Oslo · Integreat – Norwegian Centre for Knowledge-driven Machine Learning · Department of Mathematics, University of Oslo · TRUST – The Norwegian Centre for Trustworthy AI ·
Abstract
We introduce VINE: a non-parametric mutual information (MI) estimator based on simplified D-vine copulas. Approximating the joint density with a simplified D-vine allows the density ratio to simplify to a product of pair copulas, which we can estimate non-parametrically. This structural simplification yields VINE, a flexible and computationally efficient MI estimator. We compare VINE against SotA neural MI estimators on a range of datasets, both under settings where the simplified D-vine assumption holds and where it is violated, demonstrating competitive accuracy at a lower computational cost for low- to medium-dimensional datasets.
Figures & tables
| Dataset | Estimator | Avg. Time | |||
|---|---|---|---|---|---|
| Gaussian | MINE | 0.992 ( 0.017) | 2.848 ( 0.028) | 5.802 ( 0.039) | 16.49s |
| InfoNCE | 0.950 ( 0.031) | 2.604 ( 0.014) | 3.823 ( 0.010) | 465.8s | |
| VCE | 0.027 ( 0.051) | 3.140 ( 0.096) | 7.021 ( 0.021) | 124.60s | |
| VINE (Ours) | 1.089 ( 0.055) | 3.540 ( 0.217) | 6.325 ( 0.412) | 8.82s | |
| Rectangles | MINE | 0.874 ( 0.043) | 2.601 ( 0.066) | 5.727 ( 0.163) | 86.16s |
| InfoNCE | 0.860 ( 0.040) | 2.427 ( 0.057) | 3.576 ( 0.122) | 633.21s |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | Dim ( ) | Var ( ) | Var ( ) | Chosen? |
|---|---|---|---|---|
| Gaussian | 4 | 83.04% | 82.95% | No |
| 8 | 92.59% | 93.58% | No | |
| 12 | 97.27% | 97.26% | Yes | |
| 16 | 99.86% | 99.86% | — | |
| Rectangle | 20 | 88.67% | 88.58% | No |
| 30 | 92.75% | 92.74% | No |
Mutual Information Estimation
Mutual information estimation measures statistical dependence between variables from data, supporting analysis, prediction, and information-aware learning.
See all 23 papers