cs.LGFeb 6, 2026
SaveFrom Core to Detail: Unsupervised Disentanglement with Entropy-Ordered Flows
Organizations: Heidelberg University
Abstract
The unsupervised discovery of features that are both semantically meaningful and stable across runs remains a central challenge in representation learning. We introduce entropy-ordered flows (EOFlows), a normalizing flow (NF) framework that augments standard maximum likelihood training with an orthogonality regularizer on the decoder Jacobian. The regularizer is rooted in Independent Mechanism Analysis and encourages geometric disentanglement, and a stochastic estimator makes it tractable at image scale (CelebA at and ). Learned features form near-orthogonal curvilinear coordinates and can be ordered by their after training, analogous to the ranking by explained variance in PCA, which turns EOFlows into a non-linear generalization of PCA. EOFlows identify an order of magnitude more stable features than existing methods, and these features emerge in distinguishable categories (global, local, and generic) and support tentative semantic interpretations. The local features have strikingly sparse support in pixel space, although our method never enforces this. Retaining only the most important, i.e. highest entropy, features turns the bijective flow into an autoencoder with adjustable bottleneck, rivaling the rate-distortion performance of dedicated autoencoders.
Figures & tables
Figure 1: Archetypes of the features learned by EOFlows on CelebA, sorted by decreasing entropy. Global features involve the entire image, e.g. head pose. Local features affect only a small number of pixels, e.g. eye gaze. Generic features resemble windowed Fourier transforms when a subregion (e.g. the background) lacks semantic structure. Finally, features with low entropy contain only noise.
Figure 2: Induced coordinates. A standard normal latent (left) is pushed forward by linear (center) and non-linear (right) decoders, which induce affine and curvilinear coordinates in the data space. Both decoders of each pair fit the same distribution, but only the ones on the right, PCA and EOFlows, have orthogonal coordinate lines and that align with the data geometry.
Figure 3: Denoising of held-out images by one Tweedie step and through the top- bottleneck of an EOFlow ( ).
Figure 4: Entropy spectra, compression and denoising rate-distortion for representative models. (a) Manifold entropy spectra define the ordering of latent dimensions. EOFlows and NDFlows concentrate their entropy in a few hundred dimensions; EOFlows flatten towards the noise entropy , separating high-entropy core from low-entropy detail dims. The plain NF spreads it over all dims, and FactorVAE (best VAE) uses at most active latents. (b) Denoising error of held-out images reconstructed through the top- entropy-ordered bottleneck; the plain NF is omitted, as its error is far above the plotted range. Additionally to models in (a): denoising autoencoders trained separately for each (App. F.6 ), and Tweedie denoising of the flows without a bottleneck (dotted).
Figure 5: Stable features across seeds. (a, b) Sketch: coordinate lines of a reference model (gray) against one that reproduces them (a) and one that does not (b), quantified by the alignment of their tangent vectors . (c) Left: matched stability (Sec. 4 ) of every coordinate, sorted in decreasing order (median over seed pairs). EOFlows keep gaining stable coordinates up to epochs, whereas NDFlows and FactorVAE, trained just as long, do not. The plain NF has no stable coordinate, and noisy PCA (dashed) has hundreds of them, by construction. Budgets and seeds in Table F.1 . Right: archetypes of representative EOFlow features in three random seeds (all six: App. G.7 ).
Figure 6: Equivariance of EOFlows.
Figure 7: Sweeping the orthogonality penalty from NF to PCA. Each NF/EOFlow trained with – seeds for epochs, noisy PCA fits. (a) Negative log-likelihood on training and held-out data; the gap between them is overfitting. (b) Hadamard gap . (c) Coordinates reproduced per seed pair (matched stability ). (d) Sparsity via mean and minimum ENZ. (e) Non-linearity of coordinates (one seed, App. F.7.4 ). (f) gFID of prior samples after Tweedie denoising.
| CelebA (fully connected) | dSprites (conv.) | Shapes3D (conv.) | ||||||
| model | stability | orthog- onality | sparsity min/mean | TAD | MIG | DCI | MIG | DCI |
| EOFlow | 121 ±5 | 0.041 ±.001 | 0.03 / 0.20 | 0.72 ±.01 | 0.14 ±.10 | 0.07 ±.04 | 0.15 ±.03 | 0.52 ±.03 |
| EOFlow | 25.0 ±3.6 | 0.17 ±<.01 | 0.09/0.32 | 1.00 ±.01 | 0.26 ±.01 | 0.19 ±<.01 | 0.17 ±.07 | 0.41 ±.09 |
| NDFlow | 8.5 ±1.8 | 0.29 | 0.30/0.40 | 0.24 ±.01 | – | – | – | – |
| NDFlow | 1.7 ±1.2 | 0.46 ±<.01 | 0.32/0.41 | 0.58 ±.07 | – | – | – | – |
| NF | 0 | 1.6 ±<.1 | 0.37/0.41 | 0.00 ±.00 | – | – | – | – |
Table 1: Reproducible, sparse and disentangled. Mean over seeds. CelebA at : epochs (AnnealedVAE: ), fully-connected architectures; stability and sparsity as in Sec. 4 ; TAD = attribute disentanglement score ( Yeats et al., 2022 ) (App. G.6 ). Orthogonality = Hadamard gap of the core coordinates (App. F.7.1 ). dSprites and Shapes3D: , epochs, convolutional architectures (App. G.3 ). PCA fits orthogonal and reproducible components by construction (in parentheses). On the disentanglement benchmarks we compare EOFlows with the VAEs only.
Appendix figures & tables42 assets
Supplementary material from the paper’s appendix.
Appendix
Figure B.1: Variance of the stochastic Hadamard estimate along the sweep (last epochs of training; lines are means over – seeds per , shaded bands one standard deviation, mostly narrower than the line). (a) Per-step variance of the estimate of in the logged training loss (blue) and as predicted by the variance formula at the same weights, from full Jacobians at noise-inflated training images (dashed). (b) The same variance relative to the squared mean, i.e. to the penalty itself. (c) Per-step variance of the two terms of the training loss.
| model | epochs | seeds | notes |
|---|---|---|---|
| EOFlow | 103, 105, 106, 108–110 | ||
| EOFlow | 103, 105, 106 | ||
| EOFlow | 103, 105, 106 | ||
| EOFlow | 103, 105, 106 | snapshots after – epochs | |
| EOFlow | 103, 105, 106 | ||
| EOFlow | 103, 106, 108, 109 |
Table F.1: Trained models behind the reported numbers (CelebA , ; flows at batch size and learning rate unless noted). Cells at and epochs of the same or are the same trainings, evaluated at both lengths.
Figure F.1: Twelve random CelebA training images, clean (top) and inflated with Gaussian noise of as seen by the flows (bottom; one noise draw, redrawn at every training step).
| small AE ( M), denoising (Fig. 4 b) | ||||||
| small AE ( M), pure reconstruction | ||||||
| large AE ( M), denoising | ||||||
| large AE ( M), pure reconstruction | ||||||
| EOFlow |
Table F.2: Held-out denoising error (MSE, ; noisy input, clean target) of the autoencoders, one model per and seed , and of EOFlows ( , mean over three seeds) through their top- bottleneck.
| model | |||
|---|---|---|---|
| EOFlow | 0.035 ±<.001 | 0.041 ±.001 | 0.014 ±<.001 |
| EOFlow | 0.13 ±<.01 | 0.17 ±<.01 | 0.058 ±.002 |
| NDFlow | 0.20 | 0.29 | 0.20 ±<.01 |
| NDFlow | 0.38 ±.01 | 0.46 ±<.01 | 0.44 ±<.01 |
| NF | 1.2 ±<.1 | 1.6 ±<.1 | 1.6 ±<.1 |
| FactorVAE | 0.38 ±.07 | – | – |
Table F.3: Orthogonality of all models: Hadamard gap per coordinate (Eq. F.2 ) over the core coordinates, mean over seeds (NDFlow at : one seed). Table 1 combines for the flows and PCA with , i.e. all latents, for the VAEs; for EOFlows and the NF is Fig. 7 b.
Figure F.2: Stability of coordinate grids (a, the protocol of Fig. 5 c) and correlation of latent codes (b, squared Pearson correlation on held-out images, Hungarian-matched), sorted per seed pair (median, min–max band); same models and seed pairs as Fig. 5 .
Figure F.3: Geometry of the coordinates against the entropy rank for all coordinates of the EOFlow ( ) and of the plain NF (same seed, epochs each) at training images: (a) non-linearity across images (zero iff the lines through all images are parallel), (b) mean norm of the Jacobian column . Dots are single coordinates, lines running medians over rank bins.
| arm | NLL | FID | stable features per seed pair | |
|---|---|---|---|---|
| ref (12 learnable mixings) | / / | |||
| last1 (final mixing learnable) | / / | |||
| frozen (no learnable mixing) | / / | |||
| nomix (permutations only) | 1827 / 1828 / 1824 | |||
| noperm (couplings only) | 1834 / 1838 / 1843 |
Table G.1: Architecture ablation on CelebA, 100 epochs, three seeds per arm (means; two ref seeds reach NLL and FID , the third and ; all other arms agree across seeds to the last digit). NLL: held-out negative log-likelihood per dimension of noise-inflated images; : Hadamard gap; FID at for prior samples; stable: latent dimensions with matched squared cosine similarity , given for the three seed pairs. The stable counts of the two mixing-free arms are set in italics: they count the pixel basis, not learned features (see text).
Figure G.1: Stability of the latent dimensions across seeds for the five arms of the architecture ablation, three seed pairs per arm. (a) Matched squared cosine similarity of every latent dimension, sorted; the crossing with the dotted line at gives the number of stable features. (b) The same quantity along the entropy order of the first seed (mean over ten neighboring dimensions); the two mixing-free arms (dashed) are stable only beyond rank , in the pixel-basis tail.
Figure G.2: Archetypes of the five arms of the architecture ablation: the highest-entropy archetypes of one seed of each arm (difference images as in Appendix G.10 , each normalized to , entropy order row-major).
Figure G.3: Entropy spectra of the five arms of the architecture ablation, mean standard deviation over three seeds; the two mixing-free arms are dashed, and the dotted line is the noise entropy . The spectra coincide over the leading ranks, so the entropy ordering is comparable across the arms.
Figure G.4: Convolutional EOFlow on CelebA ( , , epochs). (a) Sorted manifold entropy spectra of the EOFlow ( ), the same flow trained by maximum likelihood alone (NF, ) and PCA of the noisy data (dashed), with the noise entropy (dotted). (b) Archetypes of the highest-entropy EOFlow coordinates in entropy order (difference images from the latent origin, each normalized). (c) EOFlow prior samples decoded through the leading coordinates and (d) the same latent draws decoded at full after one Tweedie step.
| PCA | |||
|---|---|---|---|
| NLL (nats/dim) | |||
| entropy range, all latents (nats) | |||
| entropy range, coarsest (nats) | – | ||
| MSE, | – | ||
| MSE, | – |
Table G.2: Convolutional EOFlow on CelebA ( , , epochs, one seed each) and PCA of the noisy data. NLL: held-out negative log-likelihood per dimension; entropy range: spread of the manifold entropies over all latents and over the coarsest ones; MSE: denoising error of held-out images through the leading coordinates (noisy input: ).
Figure G.5: Entropy spectra on ground-truth benchmarks (four seeds each, , , epochs). Sorted manifold entropy spectra of EOFlows on (a) dSprites and (b) Shapes3D, the spectrum of PCA of the noisy data (dashed) and the noise entropy (dotted). The vertical line marks the number of ground-truth factors. EOFlows concentrate the entropy in a few coordinates and drop to the noise level at a sharp knee, which PCA lacks.
| dSprites (conv.) | Shapes3D (conv.) | |||||
| model | MIG | DCI | FactorVAE | MIG | DCI | FactorVAE |
| EOFlow | 0.14 ±.10 | 0.07 ±.04 | 0.65 ±.13 | 0.15 ±.03 | 0.52 ±.03 | 0.80 ±.01 |
| EOFlow | 0.26 ±.01 | 0.19 ±<.01 | 0.81 ±.02 | 0.17 ±.07 | 0.41 ±.09 | 0.70 ±.02 |
| FactorVAE | 0.15 ±.07 | 0.14 ±.03 | 0.76 ±.09 | 0.13 ±.03 | 0.14 ±.05 | 0.73 ±.04 |
| -TCVAE | 0.26 ±.09 | 0.25 ±.03 | 0.78 ±.07 | 0.31 ±.12 | 0.42 ±.06 | 0.71 ±.09 |
| -VAE | 0.06 ±.02 | 0.06 ±.01 | 0.64 ±.06 | 0.22 ±.07 | 0.25 ±.08 | 0.83 ±.15 |
Table G.3: Disentanglement on dSprites and Shapes3D ( , epochs). Mean over seeds. MIG ( Chen et al., 2018 ) , DCI disentanglement ( Eastwood & Williams, 2018 ) and the FactorVAE score ( Kim & Mnih, 2018 ) on latents per model (VAEs: all ; EOFlows and noisy PCA: the highest-entropy coordinates), from clean held-out images. VAEs at literature settings ( -VAE , AnnealedVAE and , FactorVAE , -TCVAE ); EOFlows with . Seeds (dSprites / Shapes3D): EOFlow 3 / 3, EOFlow 4 / 4, FactorVAE 3 / 3, -TCVAE 6 / 4, -VAE 4 / 3, AnnealedVAE 3 / 3.
Figure G.6: Archetypes: EOFlow trained on shuffled pixels ( , ep, seed ). Top: the highest-entropy latents after un-shuffling with , in the format of App. G.10 (number above each triplet is the entropy rank). Bottom: the five leading latents in the basis the model was trained in, each image normalized to .
Figure G.7: FID of prior samples through the top- bottleneck (no Tweedie step; mean sd over seeds) for EOFlows at epochs and PCA, and of full-prior samples after one Tweedie step (dotted lines, including the plain NF).
| model | epochs | TAD |
|---|---|---|
| plain NF ( ) | 300 | 0.00 0.00 (3) |
| EOFlow | 300 | 0.48 0.21 (3) |
| EOFlow | 300 | 1.00 0.01 (3) |
| EOFlow | 300 | 0.72 0.01 (3) |
| EOFlow | 300 | 0.60 0.05 (4) |
| EOFlow | 300 | 0.45 0.03 (4) |
Table G.4: Attribute-based disentanglement (TAD) ( Yeats et al., 2022 ) on the CelebA validation split (higher is better; mean sd over seeds). Flows: same architecture and recipe; VAEs: best tuned cell of each family (App. F.4 ); number of seeds in parentheses. The weakly regularized scores highest; ties the best VAE with far less seed scatter.
Figure G.8: The most stable features of EOFlow after epochs in all six seeds. The latents of the reference seed are ranked by their mean matched stability over the five other seeds, column by column; each row shows the difference images of the matched latents in all six seeds (reference first, sign flips resolved). Left of each row: the entropy rank of the feature in the reference seed (#) and the mean sd of its matched over the five seed pairs.
Figure G.9: Prior samples of every model from the same prior codes: decoded from the full prior, after one Tweedie step on these samples, and through bottlenecks that keep the or highest-entropy latents ( epochs, seed ). The plain NF has no meaningful latent order, so its bottlenecks keep arbitrary latents. Continued on the next page.
Figure G.10: Prior samples (continued): PCA (Gaussian samples with the covariance of the noisy data, their posterior mean, and samples of the top components), NDFlows ( epochs, seed unless noted) and the VAEs of App. F.4 ( epochs, seed unless noted), which decode the first entries of each code and give full-prior samples only.
Figure G.11: Reconstructions through the bottleneck, plain NF ( ).
Figure G.12: Reconstructions through the bottleneck, EOFlow .
Figure G.13: Reconstructions through the bottleneck, EOFlow .
Figure G.14: Reconstructions through the bottleneck, EOFlow .
Figure G.15: Reconstructions through the bottleneck, EOFlow .
Figure G.16: Reconstructions through the bottleneck, EOFlow .
Figure G.17: Reconstructions through the bottleneck, PCA (projection onto the leading components; Tweedie = posterior mean of the Gaussian model).
Figure G.18: Archetypes: plain NF, (300 ep, seed 103). For every latent (number above each triplet, ordered by decreasing manifold entropy): decoded image at (top), at (middle) and their difference image, normalized per tile (bottom).
Figure G.19: Archetypes: EOFlow (300 ep, seed 103). For every latent (number above each triplet, ordered by decreasing manifold entropy): decoded image at (top), at (middle) and their difference image, normalized per tile (bottom).
Figure G.20: Archetypes: EOFlow (300 ep, seed 103). For every latent (number above each triplet, ordered by decreasing manifold entropy): decoded image at (top), at (middle) and their difference image, normalized per tile (bottom).
Figure G.21: Archetypes: EOFlow (300 ep, seed 103). For every latent (number above each triplet, ordered by decreasing manifold entropy): decoded image at (top), at (middle) and their difference image, normalized per tile (bottom).
Figure G.22: Archetypes: EOFlow (300 ep, seed 103). For every latent (number above each triplet, ordered by decreasing manifold entropy): decoded image at (top), at (middle) and their difference image, normalized per tile (bottom).
Figure G.23: Archetypes: EOFlow (300 ep, seed 103). For every latent (number above each triplet, ordered by decreasing manifold entropy): decoded image at (top), at (middle) and their difference image, normalized per tile (bottom).
Figure G.24: Archetypes: PCA. For every latent (number above each triplet, ordered by decreasing manifold entropy): decoded image at (top), at (middle) and their difference image, normalized per tile (bottom).
Figure G.25: Archetypes: NDFlow (300 ep, seed 103). For every latent (number above each triplet, ordered by decreasing manifold entropy): decoded image at (top), at (middle) and their difference image, normalized per tile (bottom).
Figure G.26: Archetypes: NDFlow (300 ep, seed 103). For every latent (number above each triplet, ordered by decreasing manifold entropy): decoded image at (top), at (middle) and their difference image, normalized per tile (bottom).
Figure G.27: Archetypes: NDFlow (300 ep, seed 106). For every latent (number above each triplet, ordered by decreasing manifold entropy): decoded image at (top), at (middle) and their difference image, normalized per tile (bottom).
Figure G.28: Archetypes: FactorVAE (300 ep, seed 101). For every latent (number above each triplet, ordered by decreasing manifold entropy): decoded image at (top), at (middle) and their difference image, normalized per tile (bottom).
Figure G.29: Archetypes: -TCVAE (300 ep, seed 101). For every latent (number above each triplet, ordered by decreasing manifold entropy): decoded image at (top), at (middle) and their difference image, normalized per tile (bottom).
Figure G.30: Archetypes: -VAE (300 ep, seed 101). For every latent (number above each triplet, ordered by decreasing manifold entropy): decoded image at (top), at (middle) and their difference image, normalized per tile (bottom).
Figure G.31: Archetypes: AnnealedVAE (100 ep, seed 101). For every latent (number above each triplet, ordered by decreasing manifold entropy): decoded image at (top), at (middle) and their difference image, normalized per tile (bottom).