A Stable Neural Statistical Dependence Estimator for Autoencoder Feature Analysis

Bo Hu; Jose C Principe

Iters	MSE	Encoder pairs		Decoder pairs		End-to-end
$\times 10^{2}$	$\times 10^{-3}$	$(X,Y^{\prime})$	$(Y,Y^{\prime})$	$(Y^{\prime},\widehat{X^{\prime}})$	$(\widehat{\,X\,},\widehat{X^{\prime}})$	$(X,\widehat{\,X\,})$	$(X,\widehat{X^{\prime}})$
0	32.0	0.997	0.998	0.997	0.997	0.997	0.997
1	1.88	23.10	24.03	9.39	9.49	23.24	8.38
2	1.64	24.20	24.94	13.26	13.44	24.98	11.27
3	1.56	24.16	25.29	13.98	14.24	25.42	11.83
4	1.47	24.68	25.50	14.66	14.97	25.46	12.23

Appendix

Table 5: Two-moon dataset: dependence vs. iterations.

Iters	MSE	Encoder pairs		Decoder pairs		End-to-end
$\times 10^{2}$		$(X,Y^{\prime})$	$(Y,Y^{\prime})$	$(Y^{\prime},\widehat{X^{\prime}})$	$(\widehat{\,X\,},\widehat{X^{\prime}})$	$(X,\widehat{\,X\,})$	$(X,\widehat{X^{\prime}})$
0	0.239	0.998	0.997	0.997	0.998	0.997	0.997
1	0.119	25.00	24.96	28.84	28.96	25.00	17.74
2	0.089	27.12	27.06	70.17	71.06	27.06	24.78
3	0.077	28.29	28.25	97.60	99.55	28.26	26.92
4	0.071	28.46	28.45	110.14	113.46	28.44	27.43

Appendix

Table 6: MNIST: dependence vs. iterations.

$v_{p}$	MSE	Encoder pairs		Decoder pairs		End-to-end
		$(X,Y^{\prime})$	$(Y,Y^{\prime})$	$(Y^{\prime},\widehat{X^{\prime}})$	$(\widehat{\,X\,},\widehat{X^{\prime}})$	$(X,\widehat{\,X\,})$	$(X,\widehat{X^{\prime}})$
$10^{-7}$	0.035	781.71	723.36	868.93	877.61	762.46	625.97
$10^{-6}$	0.036	275.65	272.43	793.28	840.97	278.31	265.03
$10^{-5}$	0.041	88.56	88.33	387.96	422.73	88.86	86.90
$10^{-4}$	0.046	28.87	28.88	200.74	207.90	28.87	28.87
$10^{-3}$	0.051	9.83	9.83	65.84	67.29	9.83	9.73

Appendix

Table 7: MNIST: dependence vs. noise levels.

Noise type	MSE	NMF-like cost		MINE
		$(X,Y^{\prime})$	$(X,\widehat{\,X\,})$	$(X,Y^{\prime})$	$(X,\widehat{\,X\,})$
Concatenated input noise	0.417	1508.45	1793.36	2.83	4.14
No noise	0.417	1507.52	1782.98	4.68	5.45
Additive feature noise $10^{-7}$	0.424	608.29	583.30	4.64	5.20

Appendix

Table 8: Concatenating input noise does not solve the issue that the variable pairs are overly dependent, which essentially makes the dependence unmeasurable. MINE is very unstable and does not produce a meaningful result.

Figure 3: Learning curves of singular values.

Figure 4: Two-moon dataset: left and right singular functions for the pairs

\{X,Y^{\prime}\}

and

\{X,\widehat{\,X\,}\}

. (a), (c), and (d) display 2D singular functions as heatmaps (nine functions shown per panel). (b) displays 1D singular functions as curves (six functions shown).

Figure 5: MNIST: left and right singular functions for the pairs

\{X,Y^{\prime}\}

and

\{X,\widehat{\,X\,}\}

. In (a) we have excluded the trivial constant singular function that always has a singular value

1

.

Figure 6: Top 2D Hermite polynomials obtained by decomposing the standard Gaussian density. The first component coincides with the density itself and is therefore trivial. We include these polynomials to compare with the right singular functions in Fig. 5 a. Although they do not match exactly, the qualitative similarity is clear, consistent with the result in Fig. 4 b for the toy dataset.

Figure 7: Visualizing the estimator output and comparing the induced metrics. Both the estimator activations and the resulting matrices are highly sparse, and the matrices induced by

\{Y,Y^{\prime}\}

and

\{X,\widehat{X}\}

are visually similar.

Figure 8: MNIST feature projections learned by maximizing statistical dependence directly as an objective without a decoder. In this example the Gaussian assumption on the sample data is needed.

Figure 9: Visualization of the singular values and singular functions associated with Fig. 8(b) . A comparison can be made to Fig. 5 a.

Figure 10: The pdf and data samples of the two-moon distribution we use throughout the paper, with a variance of

0.04

.

Figure 11: Regular autoencoders: reconstructions (Fig. 11(a) –Fig. 11(d) ) and feature heatmaps (Fig. 11(e) –Fig. 11(h) ) at iterations 1K, 9K, 20K, and 200K. The features are latently continuous, so the reconstructions appear as 1D curves in 2D, i.e., a manifold. Training also appears to have distinct stages: reconstructions and the implied decision boundary are initially coarse and become progressively more fine-grained.

Figure 12: Producing the end-to-end autoencoder without neural networks, but with discretized Markovian matrices parameterized simply by two optimizable vectors. The results are consistent with autoencoder features. It also shows the importance of the encoder variance

v_{p}

for features. If it is lower than a certain threshold, the model loses generalization.

Figure 13: The threshold of the lowest feature variance

v_{p}

is tied to the grid size of the sample space, which can be lowered to

10^{-6}

if we increase the grid size from

50\times 50

(Fig. 12 ) to

200\times 200

. The produced mean-squared error is around

2\times 10^{-4}

, which, combined with our finding that the error is the decoder variance

v_{q}

, gives us the marginal

q(X)

in (b).

Figure 14: A Nyström-style analysis of the autoencoder.

Figure 15: The Nyström-style analysis after applying the two modifications from Section D.3 . In Fig. 14 , the encoder spectrum (curve C2 ) consistently exhibits larger singular values than the decoder spectrum (curve C3 ), revealing a spectral mismatch between the two operators. After applying the fixes, the two spectra coincide, indicating a symmetry: the encoder and decoder become inverse mappings of each other (in the sense that they share the same singular values and have swapped left/right singular functions).

Figure 16: The Nyström-style analysis of the setting where we remove the decoder and maximize statistical dependence for the encoder alone. Generalized features can still be learned, but only under the assumption of additive Gaussian input noise.

Measurement	Iterations ( $\times 10^{3}$ )
	0	1	2	3	4	5	6
MSE ( $\times 10^{-3}$ )	29.9	1.28	1.51	1.00	0.95	1.08	0.98
Shannon’s MI	1.15	4.08	4.10	3.93	3.82	3.84	3.85
Rényi’s MI	1479.41	1499.53	1521.31	1515.33	1485.18	1481.55	1544.81

Appendix

Table 9: Results on the two-moon dataset in a fully deterministic setting without injected noise. At initialization, the MINE estimate of Shannon mutual information is closer to its lower bound of

0

, whereas the Rényi’s mutual information is far from its lower bound of

1

. However, at later training iterations (e.g., 1000, 2000, and beyond), the Shannon’s mutual information estimates produced by MINE show little clear separation and appear noisy, even though the dependence is expected to increase as the reconstruction MSE decreases.

	0	1	2	3	4	5	20	50
Proposed fix	Iterations ( $\times 10^{3}$ )
(i) Additive feature noise
MSE ( $\times 10^{-3}$ )	30.2	2.16	1.65	1.64	1.70	1.51	1.20	0.99
Rényi’s MI	1.01	27.89	27.84	27.87	27.95	27.88	28.49	28.78
(ii) Concatenated input noise
MSE ( $\times 10^{-3}$ )	37.3	1.75	1.49	1.37	1.27	1.16	0.83	0.54

Appendix

Table 10: Two modifications that correct the high dependence estimate at initialization in the deterministic setting. Both additive feature noise and concatenated input noise make the estimated Rényi’s mutual information start close to its lower bound of 1 before training.

Figure 17: Toy example illustrating the effect of sample discreteness on dependence estimation. Although the two variables are truly independent, finite-sample estimates can be biased when the empirical distribution is too sparse or discrete. As the number of samples increases, the estimated dependence value may first increase and then gradually decrease toward its true value at the minimum.

Figure 18: Experiments on sample sufficiency versus parameters controlling estimator resolution: the kernel variance

v

in KDE and the grid size

S

in the grid-based estimator.

Figure 19: Neural estimation of statistical dependence for the 2D uniformly distributed independent variables in Fig. 17 . Similar Fig. 18 , the estimated value first increases and then decreases as the sample size grows, indicating that the neural estimator also suffers from a resolution-induced bias when the number of samples is insufficient.

Training Setting	Available Samples
	100	1000	10000	100000	200000
All Samples in a Batch	90.95	530.51	558.99	344.61	325.86
Fix the Batch Size at $1000$	93.16	186.26	134.47	131.23	131.26

Appendix

Table 11: Estimated statistical dependence between the two-moon data and the corresponding autoencoder features as the number of available training samples increases. Two settings are considered: full-batch training using all available samples at each iteration, and stochastic optimization with a fixed batch size of

1000

. In both cases, the estimates eventually decrease as more samples become available, consistent with the sample-resolution analysis above. Training was stopped after

5000

iterations for the full-batch case and after

50000

iterations for the fixed-batch case, so some residual bias due to incomplete optimization may remain. No noise is used in this example.

Figure 20: Dependence estimation in the fully static setting, with no presence of noise. The autoencoder is first trained to convergence, after which the dependence estimator is trained between the input data and the learned features. (a) Learning curve of the estimated dependence (Rényi’s mutual information), which converges to a very large value; the number of singular functions is set to

2000

. (b) Learned density-ratio matrix

\frac{p(X,Y)}{p(X)p(Y)}

, visualized using

100

selected samples from the full set of

10000

samples, resulting in a

100\times 100

matrix. (c) Direct outputs of the estimator network

\mathbf{f}

. The activations are extremely sparse, and the learned density-ratio matrix is close to diagonal.

Figure 21: Direct comparison with Fig. 20 , but now

50

dimensions of uniform random noise are concatenated to the inputs. Since the original data are 2D, the encoder input becomes

52

-dimensional. In this setting, the estimated dependence starts from its independence baseline and converges to about

60

, which is much smaller than the value of about

1500

in Fig. 20 . The learned metric, which is represented by the pairwise density-ratio matrix computed from

100

selected exemplars out of the full set of

10000

samples, is much noisier and less diagonal. The outputs of the estimator network

\mathbf{f}

are also much less sparse. This fix does not sacrifice reconstruction performance, as the MSE remains very similar to that of the noise-free case.

Figure 22: Two-moon with concatenated input noise: evolution of the dependence estimate, the singular values, and the singular functions during training. This figure highlights the difference between the noisy-input setting and the fully static setting. In the static case, the estimated dependence remains artificially large (around

1500

) regardless of the training iteration, the singular values collapse to the trivial solution in which they all equal

1

, and the corresponding left and right singular functions are not meaningful.

Figure 23: Heatmaps of the learned dependence values obtained by fixing one sample and computing the density ratio between this selected representative sample and all other samples in the dataset. Samples that have positive density ratio values are concentrated around the selected sample.

Figure 24: Evolution of the estimated density ratio between one fixed sample and all other samples across autoencoder training iterations. For each autoencoder checkpoint, a separate dependence estimator is trained. As training proceeds, the high-dependence region with positive density ratio values progressively shrinks, which is consistent with the stated Gaussian-ball shrinking effect. The visualized quantity is

\sum_{k=1}^{K}f_{k}(X)g_{k}(Y)

, where

X

is fixed and

Y

ranges over all

10000

samples.

Figure 25: MNIST: learning curves of the dependence estimator for the concatenated-input-noise setting and the static-network setting, evaluated at different autoencoder checkpoints. When the autoencoder is untrained (iteration 0), concatenated input noise keeps the estimate near its lower bound of

1

, whereas the static setting converges to a much larger value, close to

2000

, which matches the output dimension of the estimator. At later checkpoints, concatenated input noise still reduces the estimate during the early stage of autoencoder training (e.g., iteration

1000

), but its effect becomes limited by iteration

10000

, where both settings again converge to large values. In these experiments, the feature dimension is

2

.

Figure 26: MNIST results under full-batch optimization: the learning curve, the learned density-ratio matrix, and per-sample responses.

Figure 27: MNIST results under stochastic minibatch optimization. Compared with the full-batch setting in Fig. 26 , the estimated dependence is reduced and the learned metric becomes much less sparse, although positive responses are still concentrated on relatively few samples.

Figure 28: Heatmap visualization of the learned density-ratio metric on MNIST in the 2D latent space. Each panel fixes one sample and shows its response to the remaining samples. The positive-response region is small and localized, reflecting the sparsity of the learned density-ratio metric. These results correspond to the stochastic minibatch setting in Fig. 27(b) .

Figure 29: Heatmaps generated in the same way as in Fig. 28 , but for the setting in Section C.4 where only the encoder, without the decoder, is trained using the NMF-like cost with additive Gaussian noise on the input. In this case, more samples receive positive responses relative to the query point, even though the 2D projected features in panel (a) remain visually similar to those obtained in the autoencoder-based setting.

Figure 30: Further visualizing the case of the NMF-like scalar cost with additive Gaussian noise on the input for optimizing the encoder. Compared with Fig. 26 and Fig. 27 , which correspond to standard autoencoder training, the learned metric here is substantially less sparse, even though the projected features shown in Fig. 29(a) appear visually similar to training an end-to-end autoencoder.

Figure 31: Learning curves of the exact objective values described in Eq. ( 40 ), i.e., the logarithm of the original cost with small constants added inside the

\log

terms for numerical stability, evaluated at multiple checkpoints during autoencoder training. The training of the dependence estimator is stable and fast. As the autoencoder becomes better trained, the estimated dependence value increases, which is consistent with our assumption.

Figure 32: Layerwise statistical dependence in the trained encoder. IN denotes the input, L1 – L7 denote the outputs of the seven ReLU layers, and L8 denotes the output of the final Sigmoid layer. Each matrix entry represents the estimated dependence between a pair of two layer representations, while diagonal entries represent self-dependence. Panel (a) that uses

6000

samples could be biased, whereas panel (b) that uses

1000

samples can be considered unbiased because the sample size

(1000)

does not exceed the estimator output dimension

(2000)

, i.e., the total number of singular values. The concatenated noise in the encoder is important for producing a meaningful pattern, and its dimensionality affects the values in this matrix.

	Encoder pairs		Decoder pairs		End-to-end
Objective	$(X,Y^{\prime})$	$(Y,Y^{\prime})$	$(Y^{\prime},\widehat{X^{\prime}})$	$(\widehat{\,X\,},\widehat{X^{\prime}})$	$(X,\widehat{\,X\,})$	$(X,\widehat{X^{\prime}})$
NMF-DR	28.57	28.68	22.37	22.38	28.64	16.58
LOGDET	28.35	28.60	22.35	22.36	28.34	16.72
TRACE	28.46	28.66	22.40	22.40	28.44	19.59*
MINE	2.84	3.17	2.63	2.81	2.96	2.36
KDE	20.48	20.57	17.94	20.29	23.22	15.78

	Encoder pairs		Decoder pairs		End-to-end
Objective	$(X,Y^{\prime})$	$(Y,Y^{\prime})$	$(Y^{\prime},\widehat{X^{\prime}})$	$(\widehat{\,X\,},\widehat{X^{\prime}})$	$(X,\widehat{\,X\,})$	$(X,\widehat{X^{\prime}})$
NMF-DR	28.81	28.85	200.74	207.90	28.81	28.51
LOGDET	27.98	28.04	199.30*	200.75*	27.98	27.70
TRACE	28.28	28.34	201.10*	206.97*	28.29	28.02
MINE	1.85	2.09	3.31	4.90	2.07	2.07
KDE	2.50	3.18	3.18	4.93	3.00	2.96

$v_{p}$	MSE	Encoder pairs		Decoder pairs		End-to-end
	$(\times 10^{-3})$	$(X,Y^{\prime})$	$(Y,Y^{\prime})$	$(Y^{\prime},\widehat{X^{\prime}})$	$(\widehat{\,X\,},\widehat{X^{\prime}})$	$(X,\widehat{\,X\,})$	$(X,\widehat{X^{\prime}})$
$10^{-7}$	0.540	519.68	509.61	46.55	46.79	506.74	45.84
$10^{-6}$	0.624	213.59	212.88	40.05	40.25	216.28	37.45
$10^{-5}$	0.650	86.87	87.25	38.71	38.84	87.06	31.90
$10^{-4}$	1.10	28.78	28.83	20.49	20.52	28.80	15.37
$10^{-3}$	2.46	9.82	9.82	9.03	9.04	9.82	6.69

Fea. Dim.	MSE	Encoder pairs		Decoder pairs		End-to-end
		$(X,Y^{\prime})$	$(Y,Y^{\prime})$	$(Y^{\prime},\widehat{X^{\prime}})$	$(\widehat{\,X\,},\widehat{X^{\prime}})$	$(X,\widehat{\,X\,})$	$(X,\widehat{X^{\prime}})$
1	0.048	28.81	28.85	200.74	207.90	28.81	28.51
2	0.030	648.40	675.31	1632.26	1661.05	647.28	634.45
3	0.021	1604.27	1679.04	1859.84	1819.99	1576.40	1623.15
4	0.016	1886.03	1885.60	1899.80	1839.93	1846.92	1857.05
5	0.012	1929.32	1875.47	1914.92	1840.91	1892.85	1886.75

A Stable Neural Statistical Dependence Estimator for Autoencoder Feature Analysis

Abstract

Figures & tables

Appendix

Mutual Information Estimation

Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders

Spectral characteristics of autoencoder parameters as a vector representation of data

InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate

Abstract

Mutual Information Estimation

Explore similar work

Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders

Spectral characteristics of autoencoder parameters as a vector representation of data

InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate