Controlling Dependence in Implicit Generative Models via Spread Mutual Information

Jiahao Yu; Song Liu; José Miguel Hernández-Lobato; RuiKang OuYang

stat.MLOct 7, 2026

Controlling Dependence in Implicit Generative Models via Spread Mutual Information

Authors: Jiahao Yu, Song Liu, José Miguel Hernández-Lobato, RuiKang OuYang

Organizations: University of Oxford · Work completed at the University of Cambridge. · University of Bristol · University of Cambridge

Abstract

Mutual information (MI) provides an objective for suppressing or encouraging statistical dependence in implicit generative models. However, direct MI evaluation is challenging in implicit models due to typically intractable densities. A remedy is estimating the generator gradient from the difference between conditional and marginal scores. This score difference can, in turn, be estimated by differentiating a log density ratio learned through classification. This construction nevertheless faces two difficulties: (i) singular distributions need not admit the required score functions, and (ii) poor overlap can hinder density-ratio estimation. We therefore introduce Spread Mutual Information (SMI), a weighted integral of MI across noise levels obtained by applying a common spreading kernel to the generated variable. Gaussian spreading yields smooth, strictly positive conditional and marginal densities, extending the gradient construction to distributions that may originally be singular. Across a variaty of experiments, SMI consistently achieves effective dependence control among MI-based methods and remains competitive with established task-specific approaches.

Figures & tables

Panel of Figure 1 — Figure 1: Gaussian spreading near a singular limit. Left pair: Joint and product-of-marginals samples at $b=0$ , before (leftmost) and after (middle) spreading with $\sigma=0.5$ . Right: Relative variance of the pathwise estimator of $\partial_{a}I(\widetilde{X};Y)$ using exact scores.

Figure 2: Synthetic experiments. (a) Conditional supports $x_{2}=\pm a$ and optimisation of their separation $2a$ . (b,c) MI minimisation and maximisation results, pooled over Gaussian, cubic, asinh, signed-power and spiral data; $d\in\{20,100,200\}$ ; a shared correlation coefficient or one per latent coordinate pair; and ten seeds.

Estimator	Score difference	Parameter gradient
SMI	$\mathbf{0.408\pm 0.030}$	$\mathbf{0.275\pm 0.031}$
DSM scores	$2.608\pm 0.376$	$6.225\pm 1.834$

Table 1: Gradient accuracy. Relative errors (

\downarrow

), mean

\pm

SD over ten seeds.

MI-based methods
	(a) 30% MCAR		(b) 60% MCAR		(c) 80% MCAR
Method	MMD 2 rank	RMSE rank	MMD 2 rank	RMSE rank	MMD 2 rank	RMSE rank
SMI (Ours)	$\mathbf{5.43\pm 0.31}$	$\mathbf{7.63\pm 0.15}$	$\mathbf{4.10\pm 0.10}$	$7.63\pm 0.06$	$\mathbf{2.87\pm 0.21}$	$\underline{7.33\pm 0.35}$
CLUB	$12.30\pm 0.66$	$9.93\pm 0.21$	$11.97\pm 0.21$	$\mathbf{7.37\pm 0.12}$	$11.00\pm 0.20$	$\mathbf{5.33\pm 0.12}$
NWJ	$11.90\pm 0.20$	$13.63\pm 0.23$	$8.43\pm 0.40$	$11.80\pm 0.20$	$6.47\pm 0.83$	$10.53\pm 0.95$
MINE	$10.33\pm 0.83$	$12.90\pm 0.40$	$7.43\pm 0.32$	$11.27\pm 0.15$	$7.40\pm 0.66$	$10.40\pm 0.53$

Table 2: Missing data imputation. Ranks across all methods (

\downarrow

), averaged over ten datasets at each missingness rate; mean

\pm

SD over three seeds. Failed runs rank last.

Figure 3: Waterbirds. Training examples from the four bird–background groups.

MI-based methods
	Worst-group	Leakage (points, $\downarrow$ )
Method	accuracy (%, $\uparrow$ )	Linear	MLP
SMI (Ours)	$\mathbf{71.68\pm 1.33}$	$\mathbf{18.19\pm 5.35}$	$\mathbf{9.20\pm 5.49}$
MINE	$70.87\pm 1.16$	$26.21\pm 2.42$	$22.23\pm 4.00$
SMILE	$71.43\pm 1.83$	$25.34\pm 3.15$	$17.69\pm 1.76$
InfoNCE	$71.43\pm 1.40$	$25.59\pm 3.14$	$18.12\pm 4.09$

Table 3: Waterbirds. Test results; mean

\pm

SD over five training seeds.

Figure 4: Edges-to-photos translation. Input, paired photo and two SMI outputs (different latents).

MI-based methods
	Latent-code prediction $R^{2}$ ( $\uparrow$ )
Method	Unperturbed	JPEG (75)	Resize
SMI (Ours)	$\mathbf{0.798\pm 0.008}$	$\mathbf{0.623\pm 0.045}$	$\mathbf{0.726\pm 0.032}$
InfoNCE	$0.757\pm 0.004$	$0.588\pm 0.032$	$0.623\pm 0.071$
NWJ	$\underline{0.788\pm 0.008}$	$\underline{0.596\pm 0.047}$	$\underline{0.682\pm 0.033}$
FLO	$0.725\pm 0.001$	$0.503\pm 0.028$	$0.619\pm 0.025$

Table 4: Edges2Shoes. Latent-code prediction

R^{2}

(

\uparrow

); mean

\pm

SD over three training seeds.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

	Minimisation $\downarrow$			Maximisation (target: 50)
Method	$d=20$	$d=100$	$d=200$	$d=20$	$d=100$	$d=200$
SMI (Ours)	$\mathbf{(1.36\pm 0.366)\times 10^{-8}}$	$\mathbf{(7.83\pm 11.6)\times 10^{-8}}$	$\mathbf{(1.00\pm 2.74)\times 10^{-6}}$	$50.007\pm 0.006$	$50.205\pm 0.111$	$50.230\pm 0.197$
CLUB	$\underline{(1.12\pm 0.373)\times 10^{-5}}$	$(1.20\pm 0.362)\times 10^{-4}$	$0.973\pm 0.025$	$50.006\pm 0.003$	$50.268\pm 0.057$	$50.095\pm 0.058$
NWJ	$(2.44\pm 1.56)\times 10^{-5}$	$(6.59\pm 4.85)\times 10^{-5}$	$(4.84\pm 3.08)\times 10^{-4}$	n.f.	n.f.	$50.430\pm 0.316$
MINE	n.f.	$(6.55\pm 7.41)\times 10^{-5}$	$(1.00\pm 0.827)\times 10^{-4}$	n.f.	n.f.	n.f.
InfoNCE	$(1.90\pm 0.991)\times 10^{-5}$	$\underline{(6.26\pm 5.40)\times 10^{-5}}$	$\underline{(6.90\pm 4.34)\times 10^{-5}}$	$21.391\pm 0.246$	$49.526\pm 0.556$	$50.009\pm 0.007$

Appendix

Table 5: Gaussian MI optimisation. Shared correlation (

r=d

); mean

\pm

SD in nats over ten runs. n.f.: at least one numerical failure.

Panel of Figure 5 — Figure 5: Gaussian data. Outcomes by dimension, pooled over one shared correlation parameter or one per coordinate pair (ten runs each).

Panel of Figure 6 — Figure 6: Cubic data. Outcomes by dimension, pooled over one shared correlation parameter or one per coordinate pair (ten runs each).

Panel of Figure 7 — Figure 7: Asinh data. Outcomes by dimension, pooled over one shared correlation parameter or one per coordinate pair (ten runs each).

Panel of Figure 8 — Figure 8: Signed-power data. Outcomes by dimension, pooled over one shared correlation parameter or one per coordinate pair (ten runs each).

Panel of Figure 9 — Figure 9: Spiral data. Outcomes by dimension, pooled over one shared correlation parameter or one per coordinate pair (ten runs each).

Panel of Figure 10 — Figure 10: Information across noise levels. Exact-density evaluations. (a) $\tfrac{\tau}{2}\mathbb{E}[v_{\tau}^{2}]$ (lines) and $-\partial I/\partial\log\tau$ (markers), with $\tau=\sigma^{2}$ . (b) Multiscale SMI for $\sigma\sim\operatorname{LogUniform}(0.01\eta,\eta)$ .

Dataset	Source	Rows	Features	Licence
Blood	https://www.openml.org/d/46913	748	4	CC BY 4.0
Concrete	https://www.openml.org/d/46917	1,030	8	CC BY 4.0
Fish	https://www.openml.org/d/46954	907	6	CC BY 4.0
Hazelnut	https://www.openml.org/d/46930	2,400	30	CC BY-SA
Houses	https://www.openml.org/d/46934	20,640	8	Public
Maternal	https://www.openml.org/d/46941	1,014	6	CC BY 4.0

Appendix

Table 6: TabArena datasets. All rows; numerical feature counts exclude targets. Licences follow the linked OpenML records.

Panel of Figure 11 — Figure 11: Missing data imputation. MMD 2 and RMSE at 30%, 60% and 80% MCAR; individual runs and mean $\pm$ SD for groups with three successful fits.

Panel of Figure 12 — Figure 12: Missing data imputation (continued).

Panel of Figure 13 — Figure 13: Missing data imputation (continued).

Panel of Figure 14 — Figure 14: Missing data imputation (continued).

MI-based methods
	Accuracy (%, $\uparrow$ )		Leakage (points, $\downarrow$ )
Method	Overall	Worst group	Linear	MLP
SMI (Ours)	$89.96\pm 0.32$	$\mathbf{71.68\pm 1.33}$	$\mathbf{18.19\pm 5.35}$	$\mathbf{9.20\pm 5.49}$
MINE	$90.68\pm 0.32$	$70.87\pm 1.16$	$26.21\pm 2.42$	$22.23\pm 4.00$
SMILE	$90.62\pm 0.36$	$71.43\pm 1.83$	$25.34\pm 3.15$	$17.69\pm 1.76$
InfoNCE	$\underline{90.75\pm 0.22}$	$71.43\pm 1.40$	$25.59\pm 3.14$	$18.12\pm 4.09$

Appendix

Table 7: Waterbirds. Test accuracy and background leakage; mean

\pm

SD over five training runs.

MI-based methods
Method	KID ( $\downarrow$ )	Edge F1 ( $\uparrow$ )	ResNet-50 MLP $R^{2}$ ( $\uparrow$ )
SMI (Ours)	$(1.587\pm 0.145)\times 10^{-2}$	$0.679\pm 0.004$	$\mathbf{0.584\pm 0.008}$
InfoNCE	$\underline{(1.293\pm 0.084)\times 10^{-2}}$	$0.685\pm 0.003$	$0.523\pm 0.007$
NWJ	$(1.481\pm 0.064)\times 10^{-2}$	$0.679\pm 0.002$	$\underline{0.573\pm 0.016}$
FLO	$\mathbf{(1.177\pm 0.014)\times 10^{-2}}$	$\mathbf{0.690\pm 0.002}$	$0.470\pm 0.005$
KNIFE	$(1.451\pm 0.154)\times 10^{-2}$	$0.684\pm 0.002$	$0.533\pm 0.011$

Appendix

Table 8: Edges2Shoes additional results. Reporting-set metrics at 20,000 generator updates; mean

\pm

SD over three training runs.

Figure 15: Computational costs. Iteration time and peak allocated GPU memory across dimension, minibatch size and correlated coordinates; median and range over three repetitions.

Method	Objective or gradient	Time $O(\cdot)$	Peak memory $O(\cdot)$
SMI (Ours)	Spread log-ratio gradient	$(N_{\mathrm{aux}}+1)B(dh+N_{\ell}h^{2})$	$dh+N_{\ell}h^{2}+B(d+N_{\ell}h)$
CLUB	Sampled Gaussian contrast	$(N_{\mathrm{aux}}+1)B(dh+N_{\ell}h^{2})$	$dh+N_{\ell}h^{2}+B(d+N_{\ell}h)$
NWJ	Variational KL bound	$(N_{\mathrm{aux}}+1)B^{2}(dh+N_{\ell}h^{2})$	$dh+N_{\ell}h^{2}+B^{2}(d+N_{\ell}h)$
MINE	DV; moving normaliser	$(N_{\mathrm{aux}}+1)B^{2}(dh+N_{\ell}h^{2})$	$dh+N_{\ell}h^{2}+B^{2}(d+N_{\ell}h)$
InfoNCE	Contrastive bound	$(N_{\mathrm{aux}}+1)B^{2}(dh+N_{\ell}h^{2})$	$dh+N_{\ell}h^{2}+B^{2}(d+N_{\ell}h)$
SMILE	JS gradient; DV readout	$(N_{\mathrm{aux}}+1)B^{2}(dh+N_{\ell}h^{2})$	$dh+N_{\ell}h^{2}+B^{2}(d+N_{\ell}h)$

Appendix

Table 9: MI control methods and computational costs. Estimator costs per generator update for the evaluated synthetic implementations.

Explore similar work

InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate
May 29, 2026Zhengyang Hu, Yanzhi Chen, Hanxiang Ren +5Mutual Information Temporal Dependence
The University of Hong Kong · University of Cambridge · Microsoft +2
Optimal Mixture-of-Experts Model Averaging for Conditional Generative Models
Jul 5, 2026Shijin Gong, Baihua He, Xinyu ZhangGenerative Models Mixture-Of-Experts
School of Management, University of Science and Technology of China · Academy of Mathematics and Systems Science, Chinese Academy of Sciences
An Accuracy--Information Tradeoff for Loss-Difference Conditional Mutual Information
Oct 6, 2026Hazar Yueksel

May 29, 2026, cs.LGJ/K move · Enter open · S save

InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate

Zhengyang Hu, Yanzhi Chen, Hanxiang Ren, Qunsong Zeng+4

The University of Hong Kong · University of Cambridge · Microsoft+2

Mutual Information Temporal Dependence Foundation Model Atlas

Open paper AskPDF arXiv

Measuring statistical dependency between high-dimensional random variables is a fundamental task in data science and machine learning. Neural mutual information (MI) estimators offer a promising avenue, but they typically require costly iterative optimization for each new dataset, making them impractical for real-time applications. We present InfoAtlas, a foundation model-like architecture that eliminates this bottleneck by directly inferring MI in a single forward pass. Pretrained on large-scale synthetic data with rich dependence patterns, InfoAtlas learns to identify diverse dependence structures and predict MI directly from the dataset. Comprehensive experiments demonstrate that InfoAtlas matches state-of-the-art neural estimators in accuracy while achieving

100\times

speedup, can flexibly handle varying dimensions and sample sizes through a single unified model, and generalizes effectively to complex, real-world scenarios. By reformulating MI estimation as an inference task, InfoAtlas establishes a foundation for real-time dependency analysis.

Abstract

Explore similar work

InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate

Optimal Mixture-of-Experts Model Averaging for Conditional Generative Models

An Accuracy--Information Tradeoff for Loss-Difference Conditional Mutual Information