Representation learning models exhibit a surprising stability in their internal representations. Whereas most prior work treats this stability as a single property, we formalize it as two distinct concepts: statistical identifiability (consistency of representations across runs) and structural identifiability (alignment of representations with some unobserved ground truth). Recognizing that perfect pointwise identifiability is generally unrealistic for modern representation learning models, we propose new model-agnostic definitions of statistical and structural near-identifiability of representations up to some error tolerance ε. Leveraging these definitions, we prove a statistical ε-near-identifiability result for the representations of models with nonlinear decoders, generalizing existing identifiability theory beyond last-layer representations in e.g. generative pre-trained transformers (GPTs) to near-identifiability of the intermediate representations of a broad class of models including (masked) autoencoders (MAEs) and supervised learners. Although these weaker assumptions confer weaker identifiability, we show that independent components analysis (ICA) can resolve much of the remaining linear ambiguity for this class of models, and validate and measure our near-identifiability claims empirically. With additional assumptions on the data-generating process, statistical identifiability extends to structural identifiability, yielding a simple and practical recipe for disentanglement: ICA post-processing of latent representations. On synthetic benchmarks, this approach achieves state-of-the-art disentanglement using a vanilla autoencoder. With a foundation model-scale MAE for cell microscopy, it disentangles biological variation from technical batch effects, substantially improving downstream generalization.
Figures & tables
Figure 1: A simple isometric data-generating process.
Figure 2: Controlling the bi-Lipschitz constant L leads to improved identifiability (reduced ℓ2 error).
Model Pair
Permutation
Supervised
ICA (% eff.)
Rigid
Linear
Pythia-160M-0 → Pythia-160M-1
0.219
0.150
0.131
0.202 (25%)
MAE- timm → MAE-original
0.197
0.109
0.036
0.145 (59%)
CheXpert-small → CheXpert-base
0.218
0.104
0.048
0.175 (38%)
ResNet-18-fc-1 → ResNet-18-fc-2
0.382
0.206
0.175
0.312 (40%)
Table 1: Supervised and unsupervised alignment scores between pairs of models which measure empirical identifiability. The optimal transforms (permutation, rigid, linear, or ICA) are estimated between the two models, and average ℓ2 errors normalized by latent diameter are reported. For ICA, we also report its efficiency as the reduction of ℓ2 error from the permutation to the rigid transform.
model
aggregated
Shapes3D
MPI3D
Falcor3D
Isaac3D
( InfoM InfoE InfoC ) ↑
AE
(0.39 0.76 0.25 )
(0.34 0.99 0.16 )
(0.42 0.40 0.31 )
(0.37 0.83 0.20 )
(0.41 0.80 0.34 )
β -VAE*
(0.59 0.81 0.55 )
(0.59 0.99 0.49 )
(0.45 0.71 0.51 )
( 0.71 0.73 0.70 )
(0.60 0.80 0.51 )
β -TCVAE*
(0.58 0.72 0.59 )
(0.61 0.82 0.62 )
( 0.51 0.60 0.57 )
(0.66 0.74 0.71 )
(0.54 0.70 0.46 )
BioAE*
(0.54 0.75 0.36 )
(0.56 0.98 0.44 )
(0.45 0.66 0.36 )
(0.54 0.73 0.31 )
(0.63 0.65 0.33 )
AE + ICA (ours)
( 0.65 0.83 0.40 )
( 0.79 0.99 0.52 )
(0.44 0.66 0.31 )
( 0.71 0.83 0.33 )
( 0.64 0.82 0.43 )
Table 2: Disentanglement metrics ( InfoM , InfoE , InfoC ), of which InfoM and InfoE are the most important. AE + ICA performs comparably to some of the best disentanglement-specific neural networks, with almost no tuning. Results marked with (*) are quoted without reproduction from Hsu et al. (2023) . Bolded metrics have the highest point estimates. Full details in Appendix A.7.3 .
Mean AUROC ( ↑ )
Sparsity ( ↑ more sparse)
Gene
Base
PCA
PCA + ICA
PCA + Rand
Base
PCA
PCA + ICA
PCA + Rand
CYP11B1 (1)
0.663
0.692
0.709
0.678
0.184
0.204
0.237
0.188
EIF3H (1)
0.682
0.724
0.749
0.725
0.192
0.224
0.268
0.214
HCK (1)
0.670
0.693
0.711
0.668
0.156
0.208
0.241
0.166
MTOR (6)
0.663
0.690
0.705
0.679
0.166
0.201
0.233
0.186
PLK1 (6)
0.803
0.811
0.815
0.792
0.251
0.307
0.305
0.262
Table 3: Results from a downstream perturbation classification task. Each row represents a gene consisting of (#) separate experiments with different CRISPR guides. All experiments use the same set of 22,062 controls. Base = untransformed embeddings, PCA = whitened embeddings, PCA + ICA = whitened embeddings with ICA rotation applied, PCA + Rand = whitened embeddings with a random rotation applied.
Model
Concentration ( ↑ )
Base
0.163
PCA
0.332
PCA + ICA
0.386
PCA + Rand
0.287
Table 4: Concentration of biological variation in the top 25% of features.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.3: The distribution of sample-level bi-Lipschitz constant estimates B(z) tightens around 1 as α→0 .
Figure A.4: Controlling the bi-Lipschitz constant L leads to improved identifiability (reduced ℓ2 error). The proportionality does not appear to differ whether the max or mean bi-Lipschitz constant is estimated.
Figure A.5: Reconstruction error improves as the bi-Lipschitz constant grows (leak α→1 ). Notably, poor reconstruction does not inhibit identifiability.
Figure A.6: In digit-specific models, controlling the bi-Lipschitz constant L leads to improved identifiability (reduced ℓ2 error) with pattern similar to the full-dataset models.
Shapes3D
model
InfoM ↑
InfoE ↑
InfoC ↑
AE
0.41±0.03
0.98±0.01
0.28±0.01
β -VAE
0.59±0.02
0.99±0.02
0.49±0.03
β -TCVAE
0.61±0.03
0.82±0.02
0.62±0.02
BioAE
0.56±0.02
0.98±0.01
0.44±0.02
AE (reproduction)
0.36±0.05
1.00±0.00
0.18±0.06
Appendix
Table 5: Full results across datasets.
Hyperparameter
Value
is_unbalance
True
learning_rate
0.05
num_leaves
31
feature_fraction
0.6
reg_alpha
5.0
reg_lambda
1.0
Appendix
Table 6: LightGBM hyperparameters used in all experiments.
Representation identifiability studies when learned internal codes can be uniquely recovered from observations or behavior, and which ambiguities remain.