Organizations: College of Computer Science and Technology, Jilin University, Changchun, China · College of Computing and Data Science, Nanyang Technological University, Singapore · City St George’s, University of London, London, United Kingdom · Systems Research Institute, Polish Academy of Sciences, Warsaw 01-447, Poland
Disentangled representation learning aims to capture the underlying explanatory factors of observed data, enabling a principled understanding of the data-generating process. Recent advances in generative modeling have introduced new paradigms for learning such representations. However, existing diffusion-based methods encourage factor independence via inductive biases, yet frequently lack strong semantic alignment. In this work, we propose a flow matching-based framework for disentangled representation learning, which casts disentanglement as learning factor-conditioned flows in a compact latent space. To enforce explicit semantic alignment, we introduce a non-overlap (orthogonality) regularizer that suppresses cross-factor interference and reduces information leakage between factors. Extensive experiments across multiple datasets demonstrate consistent improvements over representative baselines, yielding higher disentanglement scores as well as improved controllability and sample fidelity.
Figures & tables
Figure 1: Illustration of the MPI3D-toy dataset ( Gondal et al., 2019 ) . The seven rectangles represent the underlying factors of variation in the scene, including object color, shape, size, camera height, background color, vertical axis and horizontal axis.
Figure 2: Training pipeline of the proposed framework. A frozen VQ-GAN encoder E maps I to z1 , which is linearly interpolated with Gaussian noise z0 to obtain zt . Factor tokens Sγ(I)={si}i=1N from Tγ condition the U-Net through cross-attention (CA). Output attention computes soft routing masks Attni from h and the tokens (Eqs. (17)–(18)). These masks decompose the predicted velocity into factor-specific components vθ(i) that sum to vθ (Eqs. (14) and (19)). LFM supervises the aggregate velocity, while Lorth discourages overlap between components. Masks and velocity fields are schematic; dots indicate small velocity magnitudes.
Method
Cars3D
Shapes3D
MPI3D-toy
FactorVAE ↑
DCI-D ↑
FactorVAE ↑
DCI-D ↑
FactorVAE ↑
DCI-D ↑
VAE
FactorVAE
0.906±0.052
0.161±0.019
0.840±0.066
0.611±0.101
0.152±0.025
0.240±0.051
β -TCVAE
0.855±0.082
0.140±0.019
0.873±0.074
0.613±0.114
0.179±0.017
0.237±0.056
DAVA
0.940±0.010
0.230±0.040
0.820±0.030
0.780±0.030
0.410±0.040
0.300±0.030
GAN
ClosedForm
0.873±0.036
0.243±0.048
0.951±0.021
0.525±0.078
0.523±0.056
0.318±0.014
GANSpace
0.932±0.018
0.209±0.031
0.788±0.091
0.284±0.034
0.465±0.036
0.229±0.042
Table 1: Quantitative comparison on Cars3D, Shapes3D, and MPI3D-toy datasets using FactorVAE score and DCI-D (mean ± std).
Metric
FactorVAE
DisDiff
FDAE
EncDiff
DyGA
Ours
TAD ↑
0.137±0.043
0.305±0.010
0.275±0.104
0.638±0.008
0.905±0.021
0.913±0.012
FID ↓
96.3±1.6
18.2±2.1
18.8±1.5
14.8±2.3
12.1±0.9
7.8±0.1
Table 2: Attribute alignment (TAD) and generation quality (FID) on CelebA (mean ± std).
Figure 3: Factor swapping on Shapes3D (left) and MPI3D-toy (right). The top two rows show source and target images. Subsequent rows show images generated after replacing one source factor token with the corresponding target token; each row corresponds to a different exchanged factor.
Variant
Cars3D
Shapes3D
MPI3D-toy
FactorVAE ↑
DCI-D ↑
FactorVAE ↑
DCI-D ↑
FactorVAE ↑
DCI-D ↑
AdaGN
0.954±0.028
0.308±0.084
0.865±0.024
0.684±0.040
0.695±0.039
0.440±0.040
w/o Lorth
0.914±0.002
0.448±0.046
0.999±0.001
0.946±0.021
0.804±0.073
0.637±0.041
Full model
0.980±0.011
0.487±0.034
1.000±0.000
0.977±0.032
0.931±0.017
0.710±0.022
Table 3: Ablation of the conditioning mechanism and orthogonality regularizer (mean ± std).
Cars3D
Shapes3D
MPI3D-toy
N
FactorVAE ↑
DCI-D ↑
FactorVAE ↑
DCI-D ↑
FactorVAE ↑
DCI-D ↑
5
0.909±0.059
0.195±0.086
0.795±0.045
0.710±0.074
0.443±0.040
0.282±0.069
10
0.964±0.013
0.431±0.034
1.000±0.000
0.973±0.020
0.907±0.017
0.649±0.024
15
0.934±0.027
0.521±0.045
1.000±0.000
0.962±0.032
0.926±0.013
0.687±0.004
20
0.980±0.011
0.487±0.034
1.000±0.000
0.977±0.032
0.931±0.017
0.710±0.022
30
0.963±0.036
0.582±0.040
1.000±0.000
0.946±0.037
0.943±0.035
0.757±0.045
Table 4: Sensitivity analysis with respect to the number of latent factor tokens N on the Cars3D, Shapes3D, and MPI3D-toy datasets (mean ± std).
Method
Shapes3D
MPI3D-toy
Acc1000/Acc
Acc100/Acc
Acc1000/Acc
Acc100/Acc
DisDiff
0.928±0.001
0.732±0.002
0.862±0.001
0.700±0.002
FDAE
0.983±0.004
0.746±0.004
0.847±0.011
0.745±0.012
EncDiff
0.975±0.000
0.772±0.002
0.853±0.005
0.701±0.001
DyGA
0.990±0.000
0.863±0.001
0.888±0.002
0.757±0.001
Ours
0.994±0.001
0.890±0.003
0.899±0.002
0.759±0.012
Table 5: Relative downstream accuracy with limited training labels on Shapes3D and MPI3D-toy. Acc denotes accuracy with 10,000 training examples. Higher ratios indicate better retention of this accuracy.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Layer
Configuration
Conv 4×4 , 3→128 , stride 2
BN, ReLU
Conv 4×4 , 128→128 , stride 2
BN, ReLU
Conv 4×4 , 128→128 , stride 2
BN, ReLU
Conv 4×4 , 128→128 , stride 2
BN, ReLU
2 × residual block, 128 ch
Conv 3×3 + Conv 1×1 , BN, ReLU
Flatten
128×4×4=2048
Appendix
Table 6: Architecture of the factor encoder Tγ (input 64×64×3 ).
Parameter
Shapes3D / Cars3D / MPI3D-toy
Input latent (VQ-GAN, f=4 , 2048 codes)
16×16×3
Model channels
64
Channel multipliers
[1, 2, 4, 4]
Res. blocks per level
2
Attention resolutions
16,8,4
Attention heads
8
Appendix
Table 7: U-Net architecture and training settings for velocity prediction on the three synthetic datasets.
Dataset
N=10
N=20
N=30
DCI-I ↑
DCI-C ↑
DCI-I ↑
DCI-C ↑
DCI-I ↑
DCI-C ↑
Cars3D
0.698±0.136
0.251±0.027
0.813±0.101
0.279±0.012
0.814±0.110
0.289±0.014
Shapes3D
0.999±0.000
0.791±0.017
0.999±0.000
0.699±0.038
0.999±0.000
0.693±0.043
MPI3D-toy
0.815±0.010
0.609±0.023
0.829±0.036
0.584±0.019
0.859±0.004
0.573±0.064
Appendix
Table 8: DCI-I and DCI-C under over-specification of N (mean ± std).
Figure 4: Training loss curves. Left: Shapes3D. Right: MPI3D-toy.
Figure 5: Factor swapping on Shapes3D. Each column shows a source–target pair. The first two rows contain the source and target images. The remaining rows show generations after replacing the source token associated with the indicated factor with its target counterpart. Results cover floor hue, wall hue, object hue, object size, object shape, and orientation.
This paper explores unsupervised disentangled representation learning from a functional perspective. We define latent concepts as factors that influence observations through locally orthogonal directions, formalized as an orthogonality constraint on the Jacobian of the generative mapping. We prove that this condition yields identifiability of general nonlinear generative models, without requiring statistical independence or causal assumptions, provided the latent domain admits all combinations of factor values. Experiments with orthogonality-regularized normalizing flows empirically confirm the theory, demonstrate reliable recovery of ground-truth factors, and shed light on the success of VAEs. These findings challenge the prevailing impossibility claims for unsupervised disentanglement and provide a principled alternative foundation.
Mathieu Cyrille Simon, Pascal Frossard, Christophe De Vleeschouwer
Disentangled representation learning seeks latent representations whose indicidual dimensions each align with a distinct covariate. Unsupervised approaches typically target latent dimension independence, yet this gives no guarantee that the resulting dimensions align with semantically meaningful covariates. Supervised approaches structure the latent space using observed covariates, but under correlated covariates they cannot simultaneously control one-to-one latent-covariate alignment and latent independence. We introduce a unified, supervised framework that couples latent dimension-covariate dependence with constraints on the latent structure. Within this framework, we show an inherent trade-off, where enforcing latent independence or exclusive one-to-one latent-covariate dependence comes at a provable cost in latent-covariate alignment. We prove that the resulting disentanglement regimes are ordered by the strength of that alignment. Each regime admits a closed-form transformation of the latent space. We apply these transformations post-hoc to realign the representations of pretrained models such as CLIP, DINOv2, and ViT, and we fold them into the inference of informed factor analysis (iFA), a probabilistic model with covariate-informed factors. On simulated and real multi-omics data, we show that both post-hoc alignment and iFA enable controllability of structured latent representations.
Małgorzata Łazęcka, Ewa Szczurek
Faculty of Mathematics, Informatics, and Mechanics, University of Warsaw Warsaw, Poland · Institute of AI for Health, Helmholtz Munich Munich, Germany
The unsupervised discovery of features that are both semantically meaningful and stable across runs remains a central challenge in representation learning. We introduce entropy-ordered flows (EOFlows), a normalizing flow (NF) framework that augments standard maximum likelihood training with an orthogonality regularizer on the decoder Jacobian. The regularizer is rooted in Independent Mechanism Analysis and encourages geometric disentanglement, and a stochastic estimator makes it tractable at image scale (CelebA at D=2352 and 12288). Learned features form near-orthogonal curvilinear coordinates and can be ordered by their explained (manifold) entropy after training, analogous to the ranking by explained variance in PCA, which turns EOFlows into a non-linear generalization of PCA. EOFlows identify an order of magnitude more stable features than existing methods, and these features emerge in distinguishable categories (global, local, and generic) and support tentative semantic interpretations. The local features have strikingly sparse support in pixel space, although our method never enforces this. Retaining only the most important, i.e. highest entropy, features turns the bijective flow into an autoencoder with adjustable bottleneck, rivaling the rate-distortion performance of dedicated autoencoders.