A Stable Neural Statistical Dependence Estimator for Autoencoder Feature Analysis

Bo Hu; Jose C Principe

cs.LGMar 12, 2026

A Stable Neural Statistical Dependence Estimator for Autoencoder Feature Analysis

Authors: Bo Hu, Jose C Principe

Organizations: Department of Electrical and Computer Engineering, University of Florida

Abstract

Statistical dependence measures like mutual information is ideal for analyzing autoencoders, but it can be ill-posed for deterministic, static, noise-free networks. We adopt the variational (Gaussian) formulation that makes dependence among inputs, latents, and reconstructions measurable, and we propose a stable neural dependence estimator based on an orthonormal density-ratio decomposition. Unlike MINE, our method avoids input concatenation and product-of-marginals re-pairing, reducing computational cost and improving stability. We introduce an efficient NMF-like scalar cost and demonstrate empirically that assuming Gaussian noise to form an auxiliary variable enables meaningful dependence measurements and supports quantitative feature analysis, with a sequential convergence of singular values.

Figures & tables

	Encoder pairs		Decoder pairs		End-to-end
Objective	$(X,Y^{\prime})$	$(Y,Y^{\prime})$	$(Y^{\prime},\widehat{X^{\prime}})$	$(\widehat{\,X\,},\widehat{X^{\prime}})$	$(X,\widehat{\,X\,})$	$(X,\widehat{X^{\prime}})$
NMF-DR	28.57	28.68	22.37	22.38	28.64	16.58
LOGDET	28.35	28.60	22.35	22.36	28.34	16.72
TRACE	28.46	28.66	22.40	22.40	28.44	19.59*
MINE	2.84	3.17	2.63	2.81	2.96	2.36
KDE	20.48	20.57	17.94	20.29	23.22	15.78

Table 1: Two-moon dataset: measurement comparisons.

	Encoder pairs		Decoder pairs		End-to-end
Objective	$(X,Y^{\prime})$	$(Y,Y^{\prime})$	$(Y^{\prime},\widehat{X^{\prime}})$	$(\widehat{\,X\,},\widehat{X^{\prime}})$	$(X,\widehat{\,X\,})$	$(X,\widehat{X^{\prime}})$
NMF-DR	28.81	28.85	200.74	207.90	28.81	28.51
LOGDET	27.98	28.04	199.30*	200.75*	27.98	27.70
TRACE	28.28	28.34	201.10*	206.97*	28.29	28.02
MINE	1.85	2.09	3.31	4.90	2.07	2.07
KDE	2.50	3.18	3.18	4.93	3.00	2.96

Table 2: MNIST: measurement comparisons.

Panel of Figure 1 — Figure 1: Learning curves for the NMF-like cost. The curves are smooth and stable because no re-pairing is required.

Figure 2: A learning curve of MINE on MNIST. The sudden “dip” in the curve is largely due to the re-pairing step for sampling from the product of marginals. The learning curve would be smoother if we lowered the learning rate, but convergence would take significantly longer.

$v_{p}$	MSE	Encoder pairs		Decoder pairs		End-to-end
	$(\times 10^{-3})$	$(X,Y^{\prime})$	$(Y,Y^{\prime})$	$(Y^{\prime},\widehat{X^{\prime}})$	$(\widehat{\,X\,},\widehat{X^{\prime}})$	$(X,\widehat{\,X\,})$	$(X,\widehat{X^{\prime}})$
$10^{-7}$	0.540	519.68	509.61	46.55	46.79	506.74	45.84
$10^{-6}$	0.624	213.59	212.88	40.05	40.25	216.28	37.45
$10^{-5}$	0.650	86.87	87.25	38.71	38.84	87.06	31.90
$10^{-4}$	1.10	28.78	28.83	20.49	20.52	28.80	15.37
$10^{-3}$	2.46	9.82	9.82	9.03	9.04	9.82	6.69

Table 3: Two-moon: dependence vs. noise levels (NMF-like).

Fea. Dim.	MSE	Encoder pairs		Decoder pairs		End-to-end
		$(X,Y^{\prime})$	$(Y,Y^{\prime})$	$(Y^{\prime},\widehat{X^{\prime}})$	$(\widehat{\,X\,},\widehat{X^{\prime}})$	$(X,\widehat{\,X\,})$	$(X,\widehat{X^{\prime}})$
1	0.048	28.81	28.85	200.74	207.90	28.81	28.51
2	0.030	648.40	675.31	1632.26	1661.05	647.28	634.45
3	0.021	1604.27	1679.04	1859.84	1819.99	1576.40	1623.15
4	0.016	1886.03	1885.60	1899.80	1839.93	1846.92	1857.05
5	0.012	1929.32	1875.47	1914.92	1840.91	1892.85	1886.75

Table 4: MNIST: dependence vs. feature dims (NMF-like).

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Iters	MSE	Encoder pairs		Decoder pairs		End-to-end
$\times 10^{2}$	$\times 10^{-3}$	$(X,Y^{\prime})$	$(Y,Y^{\prime})$	$(Y^{\prime},\widehat{X^{\prime}})$	$(\widehat{\,X\,},\widehat{X^{\prime}})$	$(X,\widehat{\,X\,})$	$(X,\widehat{X^{\prime}})$
0	32.0	0.997	0.998	0.997	0.997	0.997	0.997
1	1.88	23.10	24.03	9.39	9.49	23.24	8.38
2	1.64	24.20	24.94	13.26	13.44	24.98	11.27
3	1.56	24.16	25.29	13.98	14.24	25.42	11.83
4	1.47	24.68	25.50	14.66	14.97	25.46	12.23

Appendix

Table 5: Two-moon dataset: dependence vs. iterations.

Iters	MSE	Encoder pairs		Decoder pairs		End-to-end
$\times 10^{2}$		$(X,Y^{\prime})$	$(Y,Y^{\prime})$	$(Y^{\prime},\widehat{X^{\prime}})$	$(\widehat{\,X\,},\widehat{X^{\prime}})$	$(X,\widehat{\,X\,})$	$(X,\widehat{X^{\prime}})$
0	0.239	0.998	0.997	0.997	0.998	0.997	0.997
1	0.119	25.00	24.96	28.84	28.96	25.00	17.74
2	0.089	27.12	27.06	70.17	71.06	27.06	24.78
3	0.077	28.29	28.25	97.60	99.55	28.26	26.92
4	0.071	28.46	28.45	110.14	113.46	28.44	27.43

Appendix

Table 6: MNIST: dependence vs. iterations.

$v_{p}$	MSE	Encoder pairs		Decoder pairs		End-to-end
		$(X,Y^{\prime})$	$(Y,Y^{\prime})$	$(Y^{\prime},\widehat{X^{\prime}})$	$(\widehat{\,X\,},\widehat{X^{\prime}})$	$(X,\widehat{\,X\,})$	$(X,\widehat{X^{\prime}})$
$10^{-7}$	0.035	781.71	723.36	868.93	877.61	762.46	625.97
$10^{-6}$	0.036	275.65	272.43	793.28	840.97	278.31	265.03
$10^{-5}$	0.041	88.56	88.33	387.96	422.73	88.86	86.90
$10^{-4}$	0.046	28.87	28.88	200.74	207.90	28.87	28.87
$10^{-3}$	0.051	9.83	9.83	65.84	67.29	9.83	9.73

Appendix

Table 7: MNIST: dependence vs. noise levels.

Noise type	MSE	NMF-like cost		MINE
		$(X,Y^{\prime})$	$(X,\widehat{\,X\,})$	$(X,Y^{\prime})$	$(X,\widehat{\,X\,})$
Concatenated input noise	0.417	1508.45	1793.36	2.83	4.14
No noise	0.417	1507.52	1782.98	4.68	5.45
Additive feature noise $10^{-7}$	0.424	608.29	583.30	4.64	5.20

Appendix

Table 8: Concatenating input noise does not solve the issue that the variable pairs are overly dependent, which essentially makes the dependence unmeasurable. MINE is very unstable and does not produce a meaningful result.

Panel of Figure 3 — Figure 3: Learning curves of singular values.

Panel of Figure 4 — Figure 4: Two-moon dataset: left and right singular functions for the pairs $\{X,Y^{\prime}\}$ and $\{X,\widehat{\,X\,}\}$ . (a), (c), and (d) display 2D singular functions as heatmaps (nine functions shown per panel). (b) displays 1D singular functions as curves (six functions shown).

Panel of Figure 5 — Figure 5: MNIST: left and right singular functions for the pairs $\{X,Y^{\prime}\}$ and $\{X,\widehat{\,X\,}\}$ . In (a) we have excluded the trivial constant singular function that always has a singular value $1$ .

Panel of Figure 6 — Figure 6: Top 2D Hermite polynomials obtained by decomposing the standard Gaussian density. The first component coincides with the density itself and is therefore trivial. We include these polynomials to compare with the right singular functions in Fig. 5 a. Although they do not match exactly, the qualitative similarity is clear, consistent with the result in Fig. 4 b for the toy dataset.

Panel of Figure 7 — Figure 7: Visualizing the estimator output and comparing the induced metrics. Both the estimator activations and the resulting matrices are highly sparse, and the matrices induced by $\{Y,Y^{\prime}\}$ and $\{X,\widehat{X}\}$ are visually similar.

Panel of Figure 8 — Figure 8: MNIST feature projections learned by maximizing statistical dependence directly as an objective without a decoder. In this example the Gaussian assumption on the sample data is needed.

Panel of Figure 9 — Figure 9: Visualization of the singular values and singular functions associated with Fig. 8(b) . A comparison can be made to Fig. 5 a.

Panel of Figure 10 — Figure 10: The pdf and data samples of the two-moon distribution we use throughout the paper, with a variance of $0.04$ .

Panel of Figure 11 — Figure 11: Regular autoencoders: reconstructions (Fig. 11(a) –Fig. 11(d) ) and feature heatmaps (Fig. 11(e) –Fig. 11(h) ) at iterations 1K, 9K, 20K, and 200K. The features are latently continuous, so the reconstructions appear as 1D curves in 2D, i.e., a manifold. Training also appears to have distinct stages: reconstructions and the implied decision boundary are initially coarse and become progressively more fine-grained.

Panel of Figure 12 — Figure 12: Producing the end-to-end autoencoder without neural networks, but with discretized Markovian matrices parameterized simply by two optimizable vectors. The results are consistent with autoencoder features. It also shows the importance of the encoder variance $v_{p}$ for features. If it is lower than a certain threshold, the model loses generalization.

Panel of Figure 13 — Figure 13: The threshold of the lowest feature variance $v_{p}$ is tied to the grid size of the sample space, which can be lowered to $10^{-6}$ if we increase the grid size from $50\times 50$ (Fig. 12 ) to $200\times 200$ . The produced mean-squared error is around $2\times 10^{-4}$ , which, combined with our finding that the error is the decoder variance $v_{q}$ , gives us the marginal $q(X)$ in (b).

Panel of Figure 14 — Figure 14: A Nyström-style analysis of the autoencoder.

Panel of Figure 15 — Figure 15: The Nyström-style analysis after applying the two modifications from Section D.3 . In Fig. 14 , the encoder spectrum (curve C2 ) consistently exhibits larger singular values than the decoder spectrum (curve C3 ), revealing a spectral mismatch between the two operators. After applying the fixes, the two spectra coincide, indicating a symmetry: the encoder and decoder become inverse mappings of each other (in the sense that they share the same singular values and have swapped left/right singular functions).

Panel of Figure 16 — Figure 16: The Nyström-style analysis of the setting where we remove the decoder and maximize statistical dependence for the encoder alone. Generalized features can still be learned, but only under the assumption of additive Gaussian input noise.

Measurement	Iterations ( $\times 10^{3}$ )
	0	1	2	3	4	5	6
MSE ( $\times 10^{-3}$ )	29.9	1.28	1.51	1.00	0.95	1.08	0.98
Shannon’s MI	1.15	4.08	4.10	3.93	3.82	3.84	3.85
Rényi’s MI	1479.41	1499.53	1521.31	1515.33	1485.18	1481.55	1544.81

Appendix

Table 9: Results on the two-moon dataset in a fully deterministic setting without injected noise. At initialization, the MINE estimate of Shannon mutual information is closer to its lower bound of

0

, whereas the Rényi’s mutual information is far from its lower bound of

1

. However, at later training iterations (e.g., 1000, 2000, and beyond), the Shannon’s mutual information estimates produced by MINE show little clear separation and appear noisy, even though the dependence is expected to increase as the reconstruction MSE decreases.

	0	1	2	3	4	5	20	50
Proposed fix	Iterations ( $\times 10^{3}$ )
(i) Additive feature noise
MSE ( $\times 10^{-3}$ )	30.2	2.16	1.65	1.64	1.70	1.51	1.20	0.99
Rényi’s MI	1.01	27.89	27.84	27.87	27.95	27.88	28.49	28.78
(ii) Concatenated input noise
MSE ( $\times 10^{-3}$ )	37.3	1.75	1.49	1.37	1.27	1.16	0.83	0.54

Appendix

Table 10: Two modifications that correct the high dependence estimate at initialization in the deterministic setting. Both additive feature noise and concatenated input noise make the estimated Rényi’s mutual information start close to its lower bound of 1 before training.

Panel of Figure 17 — Figure 17: Toy example illustrating the effect of sample discreteness on dependence estimation. Although the two variables are truly independent, finite-sample estimates can be biased when the empirical distribution is too sparse or discrete. As the number of samples increases, the estimated dependence value may first increase and then gradually decrease toward its true value at the minimum.

Panel of Figure 18 — Figure 18: Experiments on sample sufficiency versus parameters controlling estimator resolution: the kernel variance $v$ in KDE and the grid size $S$ in the grid-based estimator.

Figure 19: Neural estimation of statistical dependence for the 2D uniformly distributed independent variables in Fig. 17 . Similar Fig. 18 , the estimated value first increases and then decreases as the sample size grows, indicating that the neural estimator also suffers from a resolution-induced bias when the number of samples is insufficient.

Training Setting	Available Samples
	100	1000	10000	100000	200000
All Samples in a Batch	90.95	530.51	558.99	344.61	325.86
Fix the Batch Size at $1000$	93.16	186.26	134.47	131.23	131.26

Appendix

Table 11: Estimated statistical dependence between the two-moon data and the corresponding autoencoder features as the number of available training samples increases. Two settings are considered: full-batch training using all available samples at each iteration, and stochastic optimization with a fixed batch size of

1000

. In both cases, the estimates eventually decrease as more samples become available, consistent with the sample-resolution analysis above. Training was stopped after

5000

iterations for the full-batch case and after

50000

iterations for the fixed-batch case, so some residual bias due to incomplete optimization may remain. No noise is used in this example.

Panel of Figure 20 — Figure 20: Dependence estimation in the fully static setting, with no presence of noise. The autoencoder is first trained to convergence, after which the dependence estimator is trained between the input data and the learned features. (a) Learning curve of the estimated dependence (Rényi’s mutual information), which converges to a very large value; the number of singular functions is set to $2000$ . (b) Learned density-ratio matrix $\frac{p(X,Y)}{p(X)p(Y)}$ , visualized using $100$ selected samples from the full set of $10000$ samples, resulting in a $100\times 100$ matrix. (c) Direct outputs of the estimator network $\mathbf{f}$ . The activations are extremely sparse, and the learned density-ratio matrix is close to diagonal.

Panel of Figure 21 — Figure 21: Direct comparison with Fig. 20 , but now $50$ dimensions of uniform random noise are concatenated to the inputs. Since the original data are 2D, the encoder input becomes $52$ -dimensional. In this setting, the estimated dependence starts from its independence baseline and converges to about $60$ , which is much smaller than the value of about $1500$ in Fig. 20 . The learned metric, which is represented by the pairwise density-ratio matrix computed from $100$ selected exemplars out of the full set of $10000$ samples, is much noisier and less diagonal. The outputs of the estimator network $\mathbf{f}$ are also much less sparse. This fix does not sacrifice reconstruction performance, as the MSE remains very similar to that of the noise-free case.

Panel of Figure 22 — Figure 22: Two-moon with concatenated input noise: evolution of the dependence estimate, the singular values, and the singular functions during training. This figure highlights the difference between the noisy-input setting and the fully static setting. In the static case, the estimated dependence remains artificially large (around $1500$ ) regardless of the training iteration, the singular values collapse to the trivial solution in which they all equal $1$ , and the corresponding left and right singular functions are not meaningful.

Panel of Figure 23 — Figure 23: Heatmaps of the learned dependence values obtained by fixing one sample and computing the density ratio between this selected representative sample and all other samples in the dataset. Samples that have positive density ratio values are concentrated around the selected sample.

Panel of Figure 24 — Figure 24: Evolution of the estimated density ratio between one fixed sample and all other samples across autoencoder training iterations. For each autoencoder checkpoint, a separate dependence estimator is trained. As training proceeds, the high-dependence region with positive density ratio values progressively shrinks, which is consistent with the stated Gaussian-ball shrinking effect. The visualized quantity is $\sum_{k=1}^{K}f_{k}(X)g_{k}(Y)$ , where $X$ is fixed and $Y$ ranges over all $10000$ samples.

Panel of Figure 25 — Figure 25: MNIST: learning curves of the dependence estimator for the concatenated-input-noise setting and the static-network setting, evaluated at different autoencoder checkpoints. When the autoencoder is untrained (iteration 0), concatenated input noise keeps the estimate near its lower bound of $1$ , whereas the static setting converges to a much larger value, close to $2000$ , which matches the output dimension of the estimator. At later checkpoints, concatenated input noise still reduces the estimate during the early stage of autoencoder training (e.g., iteration $1000$ ), but its effect becomes limited by iteration $10000$ , where both settings again converge to large values. In these experiments, the feature dimension is $2$ .

Panel of Figure 26 — Figure 26: MNIST results under full-batch optimization: the learning curve, the learned density-ratio matrix, and per-sample responses.

Panel of Figure 27 — Figure 27: MNIST results under stochastic minibatch optimization. Compared with the full-batch setting in Fig. 26 , the estimated dependence is reduced and the learned metric becomes much less sparse, although positive responses are still concentrated on relatively few samples.

Panel of Figure 28 — Figure 28: Heatmap visualization of the learned density-ratio metric on MNIST in the 2D latent space. Each panel fixes one sample and shows its response to the remaining samples. The positive-response region is small and localized, reflecting the sparsity of the learned density-ratio metric. These results correspond to the stochastic minibatch setting in Fig. 27(b) .

Panel of Figure 29 — Figure 29: Heatmaps generated in the same way as in Fig. 28 , but for the setting in Section C.4 where only the encoder, without the decoder, is trained using the NMF-like cost with additive Gaussian noise on the input. In this case, more samples receive positive responses relative to the query point, even though the 2D projected features in panel (a) remain visually similar to those obtained in the autoencoder-based setting.

Panel of Figure 30 — Figure 30: Further visualizing the case of the NMF-like scalar cost with additive Gaussian noise on the input for optimizing the encoder. Compared with Fig. 26 and Fig. 27 , which correspond to standard autoencoder training, the learned metric here is substantially less sparse, even though the projected features shown in Fig. 29(a) appear visually similar to training an end-to-end autoencoder.

Panel of Figure 31 — Figure 31: Learning curves of the exact objective values described in Eq. ( 40 ), i.e., the logarithm of the original cost with small constants added inside the $\log$ terms for numerical stability, evaluated at multiple checkpoints during autoencoder training. The training of the dependence estimator is stable and fast. As the autoencoder becomes better trained, the estimated dependence value increases, which is consistent with our assumption.

Panel of Figure 32 — Figure 32: Layerwise statistical dependence in the trained encoder. IN denotes the input, L1 – L7 denote the outputs of the seven ReLU layers, and L8 denotes the output of the final Sigmoid layer. Each matrix entry represents the estimated dependence between a pair of two layer representations, while diagonal entries represent self-dependence. Panel (a) that uses $6000$ samples could be biased, whereas panel (b) that uses $1000$ samples can be considered unbiased because the sample size $(1000)$ does not exceed the estimator output dimension $(2000)$ , i.e., the total number of singular values. The concatenated noise in the encoder is important for producing a meaningful pattern, and its dimensionality affects the values in this matrix.

Mutual Information Estimation

Mutual information estimation measures statistical dependence between variables from data, supporting analysis, prediction, and information-aware learning.

See all 21 papers

Jun 10, 2026

Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders

Sparse autoencoders (SAEs) are widely used to interpret neural network representations, but their utility depends on whether the learned features are reproducible across training runs. We study this question through feature stability: for each SAE feature, we estimate the probability that a similar feature reappears in an independently trained SAE. This yields a scalable per-feature signal that separates stable from unstable features. In a large-scale study across seeds, models, layers, dictionary sizes, and SAE variants, we find a pronounced functional asymmetry: stable features carry most of the reconstruction- and prediction-relevant signal, while unstable features have weak marginal impact and are dominated by low-frequency surface-form triggers in both activation statistics and automatic explanations. Geometrically, unstable features are individually non-reproducible but concentrate in reproducible lower-rank subspaces, suggesting that seed dependence often reflects basis ambiguity within a shared region of activation space rather than pure noise. A controlled synthetic model makes this mechanism explicit, showing that low-rank ground-truth features can be recovered at the subspace level while remaining non-identifiable as individual SAE latents across seeds. Finally, by pooling unique cross-seed features, we construct more stable SAEs while preserving explained variance in this setting. Together, these results show that unstable features are not merely failed or noisy latents: they have weak individual functional impact, but reflect reproducible low-dimensional structure that standard SAEs resolve differently across seeds.

Gleb Gerasimov, Timofei Rusalev, Nikita Balagansky +3

T-Tech

Representation Identifiability Sparse Autoencoders Mechanistic Interpretability

Ask PDF

Sep 3, 2026

Spectral characteristics of autoencoder parameters as a vector representation of data

This paper examines the relationship between the parameters of autoencoder models and the statistical properties of the data on which they are trained. Autoencoders are defined as models with an encoder-decoder architecture, trained to reconstruct input data through a compressed latent representation. It is proposed that the model parameters can be viewed as a dense vector representation of the corresponding sample. To test this hypothesis, a theoretical and experimental study is conducted in which a vector representation is formed based on the spectral characteristics of the autoencoder parameter matrices. Theoretical analysis shows that the singular values of the model parameter matrices are related to the eigenvalues of the covariance matrix of the training data, ensuring the transfer of information between the data space and the parameter space. Experimental results on the CIFAR-10 and FashionMNIST datasets confirm that the resulting vector representations allow for a high degree of accuracy in distinguishing between models trained on different data subsets, without resorting to complex vector generation algorithms or using the original samples. These results suggest that the parameters of trained autoencoders can be viewed as sample representations.

Maria Nikitina, Anton Bishuk, Oleg Bakhteev

Representation Learning Autoencoders Weight-Space Analysis

Ask PDF

May 29, 2026

InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate

Measuring statistical dependency between high-dimensional random variables is a fundamental task in data science and machine learning. Neural mutual information (MI) estimators offer a promising avenue, but they typically require costly iterative optimization for each new dataset, making them impractical for real-time applications. We present InfoAtlas, a foundation model-like architecture that eliminates this bottleneck by directly inferring MI in a single forward pass. Pretrained on large-scale synthetic data with rich dependence patterns, InfoAtlas learns to identify diverse dependence structures and predict MI directly from the dataset. Comprehensive experiments demonstrate that InfoAtlas matches state-of-the-art neural estimators in accuracy while achieving

100\times

speedup, can flexibly handle varying dimensions and sample sizes through a single unified model, and generalizes effectively to complex, real-world scenarios. By reformulating MI estimation as an inference task, InfoAtlas establishes a foundation for real-time dependency analysis.

Zhengyang Hu, Yanzhi Chen, Hanxiang Ren +5

The University of Hong Kong · University of Cambridge · Microsoft +2

Amortized Inference Mutual Information Estimation

Ask PDF

Abstract

Mutual Information Estimation

Explore similar work

Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders

Spectral characteristics of autoencoder parameters as a vector representation of data

InfoAtlas: A Foundation Model for Zero-Shot Statistical Dependence Estimate