Understanding when neural networks learn interpretable representations is central to mechanistic interpretability. We study this through identifiability: when can supervised learning recover the true latent variables underlying the data? We introduce two data generating processes distinguished by their causal direction where either labels cause latents (generative) or latents cause labels (discriminative). When the representation has at least as many dimensions as the latent space, we prove that the optimal encoders are linear in the latents. Conversely, when dimensions are insufficient, identifiability generically fails and encoders become nonlinear. We then show that additional constraints yield recovery of individual latent variables, not just linear mixtures. In the generative setting, non-negative activations and energy efficiency force each neuron to encode exactly one latent, providing a precise account of selective grandmother cell-like responses. In the discriminative setting, when latents are sparse and the true measurement matrix satisfies the spark condition, the learned encoder inherits compressed sensing structure, enabling sparse dictionary learning to perfectly recover individual latents. This constitutes, to our knowledge, the first identifiability result for undercomplete nonlinear independent component analysis. Our framework unifies nonlinear ICA, compressed sensing, and neural network interpretability, delineating exactly when representations transition from nonlinear distortions to linear representations and, under additional constraints or steps, to clean, axis-aligned codes. Theoretical predictions are validated on synthetic benchmarks and on activations drawn from vision models, large language models, and macaque inferotemporal cortex.
Figures & tables
Figure 1: Linear and axis-aligned latent recovery at exact capacity. A) Hyperspherical DGP [ 19 , 20 ] , training a neural network with linear representation (i.e., no final activation function, blue) and a non-negative representation (ReLU, orange); Disentanglement between true latent variables ( z ) and learned neural representations ( z^=h(z) ), measured up to arbitrary linear transformations ( W ). B) Our proposed sparse, non-negative DGP, same architectures as on the left (with/without activation function), linear disentanglement reported. C) Same as A, but disentanglement is measured only up to permutations ( P ), measuring axis-aligned latent variable recovery. D) Same as B but individual, axis-aligned disentanglement measured. E) Sample visualizations of five optimization runs, learned linear representation h(z)=Az showing a clear permutation only for the non-negative case. Shaded areas in all plots indicate standard error of the mean over 10 random seeds.
Figure 2: Phase transition at the capacity threshold. Final KL (p∗∥q) is reported as a function of encoder dimension m , for linear (blue) and MLP (red) encoders across three latent distributions. The dashed vertical line marks m=n′ , the capacity threshold.
Figure 3: Compressed Sensing Recovery. A) Training dynamics for a linear encoder h(z)=Az trained under different sparsity levels. Left: training KL (p∗∥q) . Right: row-space match cos(θmax) between the learned A and the true A∗ . B) R2 of supervised latent recovery by Lasso and OLS. C) MCCs of supervised latent recovery and unsupervised sparse dictionary learning. Shaded areas in A) and error bars in B) and C) indicate standard error of the mean over 10 random seeds.
Figure 4: Disentanglement predicts cross-system preservation. A,B) Per-neuron interpretability (MIS, Zimmermann et al. 40 ) versus the Maximum Matching Unit (MMU) score, for ImageNet-trained vision models (A) and LLMs (B; MIS logits reported as accuracy is at ceiling). C,D) MIS–MMU correlations for all model pairs in A and B. E) Directional MIS–MMU associations in macaque IT cortex [ 41 ] . Responses to 3,200 images were recorded from 58 IT neurons in "Chabo" and 110 IT neurons in "Tito", two macaques.
Figure 5: Empirical Results. Same as in main text but with exponential instead of ReLU nonlinearity, which is easier to optimize in this setting. Top left) DGP with latents on a hypersphere [ 19 , 20 ] , training a neural network with a linear representation (i.e., no final activation function, blue) and a non-negative representation (i.e., scale normalization and exponential nonlinearity as final activation function, orange); Disentanglement is measured up to arbitrary linear transformations. Bottom left) same as top left, but disentanglement is measured only up to permutations, measuring axis-aligned latent variable recovery. Top right) Our proposed sparse DGP with exponential distributions over non-negative latent variables, same architectures as on the left (with/without activation function), linear disentanglement reported. Bottom right) same as top right but individual, axis-aligned disentanglement measured. Shaded areas in all plots indicate standard error of the mean over 10 random seeds.
Figure 6: Optimization Dynamics. A) MCC curves over 10 random seeds. B) Same as A, but with batch normalisation and exponential nonlinearity. C) ReLU models with better final performance have a flatter power spectrum. D) Adding weight decay 0.03 improves convergence for ReLU model.
Figure 7: Seed-level optimization dynamics and finite-budget reliability. A–D) Exact-capacity experiments under a hyperspherical DGP (A,C) and the class-conditional sparse exponential DGP (B,D), using a linear output (blue) or a non-negative ReLU output (orange). A,B show alignment up to an unrestricted linear map, whereas C,D show permutation-and-scaling axis alignment measured by MCC. Thin lines denote individual initialization seeds, thick lines denote their mean, and shaded bands denote ± s.e.m. over ten seeds. All A–D runs use zero weight decay. E) Paired weight-decay sweep for the sparse-ReLU condition. The upper panel shows final MCC; orange points denote individual runs, gray lines connect the same initialization across weight-decay values, and the counts report the number of runs reaching MCC ≥0.90 . The lower panel shows the first update at which this threshold was reached. Black bars denote the median and interquartile range. Red crosses denote runs that did not reach the threshold within the 106 -update budget.
Figure 8: Sensitivity to the assumptions of the axis-alignment result. Each point shows mean ± s.e.m. over ten paired initialization seeds after 106 updates. Blue denotes held-out alignment up to an unrestricted linear map, and orange denotes permutation-and-scaling MCC. A) Class-dependent conditional entropy, with both bias-free and affine-readout controls. B) Conditional dependence introduced through a Gaussian copula while preserving the exponential marginals. C) Alternative class-conditional marginal families matched in conditional mean. D) Signed latent support introduced by independently flipping latent signs. E) Relaxations of output non-negativity using LeakyReLU and a linear output. F) AdamW weight decay. All other data-generation, training, and evaluation randomness is shared across paired conditions.
Figure 9: Training Dynamics under undercomplete regime. CE loss as a function of gradient update step for linear (solid blue) and MLP (dashed orange) encoders, across three latent distributions with encoder dimensions m∈{2,4,6,8,10} (lighter shades indicate smaller m ). Shaded areas indicate standard error of the mean over 10 random seeds.
Representation identifiability studies when learned internal codes can be uniquely recovered from observations or behavior, and which ambiguities remain.