Organizations: College of Computer Science and Technology, Jilin University, Changchun, China · College of Computing and Data Science, Nanyang Technological University, Singapore · City St George’s, University of London, London, United Kingdom · Systems Research Institute, Polish Academy of Sciences, Warsaw 01-447, Poland
Disentangled representation learning aims to capture the underlying explanatory factors of observed data, enabling a principled understanding of the data-generating process. Recent advances in generative modeling have introduced new paradigms for learning such representations. However, existing diffusion-based methods encourage factor independence via inductive biases, yet frequently lack strong semantic alignment. In this work, we propose a flow matching-based framework for disentangled representation learning, which casts disentanglement as learning factor-conditioned flows in a compact latent space. To enforce explicit semantic alignment, we introduce a non-overlap (orthogonality) regularizer that suppresses cross-factor interference and reduces information leakage between factors. Extensive experiments across multiple datasets demonstrate consistent improvements over representative baselines, yielding higher disentanglement scores as well as improved controllability and sample fidelity.
Figures & tables
Figure 1: Illustration of the MPI3D-toy dataset ( Gondal et al., 2019 ) . The seven rectangles represent the underlying factors of variation in the scene, including object color, shape, size, camera height, background color, vertical axis and horizontal axis.
Figure 2: Training pipeline of the proposed framework. A frozen VQ-GAN encoder E maps I to z1 , which is linearly interpolated with Gaussian noise z0 to obtain zt . Factor tokens Sγ(I)={si}i=1N from Tγ condition the U-Net through cross-attention (CA). Output attention computes soft routing masks Attni from h and the tokens (Eqs. (17)–(18)). These masks decompose the predicted velocity into factor-specific components vθ(i) that sum to vθ (Eqs. (14) and (19)). LFM supervises the aggregate velocity, while Lorth discourages overlap between components. Masks and velocity fields are schematic; dots indicate small velocity magnitudes.
Method
Cars3D
Shapes3D
MPI3D-toy
FactorVAE ↑
DCI-D ↑
FactorVAE ↑
DCI-D ↑
FactorVAE ↑
DCI-D ↑
VAE
FactorVAE
0.906±0.052
0.161±0.019
0.840±0.066
0.611±0.101
0.152±0.025
0.240±0.051
β -TCVAE
0.855±0.082
0.140±0.019
0.873±0.074
0.613±0.114
0.179±0.017
0.237±0.056
DAVA
0.940±0.010
0.230±0.040
0.820±0.030
0.780±0.030
0.410±0.040
0.300±0.030
GAN
ClosedForm
0.873±0.036
0.243±0.048
0.951±0.021
0.525±0.078
0.523±0.056
0.318±0.014
GANSpace
0.932±0.018
0.209±0.031
0.788±0.091
0.284±0.034
0.465±0.036
0.229±0.042
Table 1: Quantitative comparison on Cars3D, Shapes3D, and MPI3D-toy datasets using FactorVAE score and DCI-D (mean ± std).
Metric
FactorVAE
DisDiff
FDAE
EncDiff
DyGA
Ours
TAD ↑
0.137±0.043
0.305±0.010
0.275±0.104
0.638±0.008
0.905±0.021
0.913±0.012
FID ↓
96.3±1.6
18.2±2.1
18.8±1.5
14.8±2.3
12.1±0.9
7.8±0.1
Table 2: Attribute alignment (TAD) and generation quality (FID) on CelebA (mean ± std).
Figure 3: Factor swapping on Shapes3D (left) and MPI3D-toy (right). The top two rows show source and target images. Subsequent rows show images generated after replacing one source factor token with the corresponding target token; each row corresponds to a different exchanged factor.
Variant
Cars3D
Shapes3D
MPI3D-toy
FactorVAE ↑
DCI-D ↑
FactorVAE ↑
DCI-D ↑
FactorVAE ↑
DCI-D ↑
AdaGN
0.954±0.028
0.308±0.084
0.865±0.024
0.684±0.040
0.695±0.039
0.440±0.040
w/o Lorth
0.914±0.002
0.448±0.046
0.999±0.001
0.946±0.021
0.804±0.073
0.637±0.041
Full model
0.980±0.011
0.487±0.034
1.000±0.000
0.977±0.032
0.931±0.017
0.710±0.022
Table 3: Ablation of the conditioning mechanism and orthogonality regularizer (mean ± std).
Cars3D
Shapes3D
MPI3D-toy
N
FactorVAE ↑
DCI-D ↑
FactorVAE ↑
DCI-D ↑
FactorVAE ↑
DCI-D ↑
5
0.909±0.059
0.195±0.086
0.795±0.045
0.710±0.074
0.443±0.040
0.282±0.069
10
0.964±0.013
0.431±0.034
1.000±0.000
0.973±0.020
0.907±0.017
0.649±0.024
15
0.934±0.027
0.521±0.045
1.000±0.000
0.962±0.032
0.926±0.013
0.687±0.004
20
0.980±0.011
0.487±0.034
1.000±0.000
0.977±0.032
0.931±0.017
0.710±0.022
30
0.963±0.036
0.582±0.040
1.000±0.000
0.946±0.037
0.943±0.035
0.757±0.045
Table 4: Sensitivity analysis with respect to the number of latent factor tokens N on the Cars3D, Shapes3D, and MPI3D-toy datasets (mean ± std).
Method
Shapes3D
MPI3D-toy
Acc1000/Acc
Acc100/Acc
Acc1000/Acc
Acc100/Acc
DisDiff
0.928±0.001
0.732±0.002
0.862±0.001
0.700±0.002
FDAE
0.983±0.004
0.746±0.004
0.847±0.011
0.745±0.012
EncDiff
0.975±0.000
0.772±0.002
0.853±0.005
0.701±0.001
DyGA
0.990±0.000
0.863±0.001
0.888±0.002
0.757±0.001
Ours
0.994±0.001
0.890±0.003
0.899±0.002
0.759±0.012
Table 5: Relative downstream accuracy with limited training labels on Shapes3D and MPI3D-toy. Acc denotes accuracy with 10,000 training examples. Higher ratios indicate better retention of this accuracy.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Layer
Configuration
Conv 4×4 , 3→128 , stride 2
BN, ReLU
Conv 4×4 , 128→128 , stride 2
BN, ReLU
Conv 4×4 , 128→128 , stride 2
BN, ReLU
Conv 4×4 , 128→128 , stride 2
BN, ReLU
2 × residual block, 128 ch
Conv 3×3 + Conv 1×1 , BN, ReLU
Flatten
128×4×4=2048
Appendix
Table 6: Architecture of the factor encoder Tγ (input 64×64×3 ).
Parameter
Shapes3D / Cars3D / MPI3D-toy
Input latent (VQ-GAN, f=4 , 2048 codes)
16×16×3
Model channels
64
Channel multipliers
[1, 2, 4, 4]
Res. blocks per level
2
Attention resolutions
16,8,4
Attention heads
8
Appendix
Table 7: U-Net architecture and training settings for velocity prediction on the three synthetic datasets.
Dataset
N=10
N=20
N=30
DCI-I ↑
DCI-C ↑
DCI-I ↑
DCI-C ↑
DCI-I ↑
DCI-C ↑
Cars3D
0.698±0.136
0.251±0.027
0.813±0.101
0.279±0.012
0.814±0.110
0.289±0.014
Shapes3D
0.999±0.000
0.791±0.017
0.999±0.000
0.699±0.038
0.999±0.000
0.693±0.043
MPI3D-toy
0.815±0.010
0.609±0.023
0.829±0.036
0.584±0.019
0.859±0.004
0.573±0.064
Appendix
Table 8: DCI-I and DCI-C under over-specification of N (mean ± std).
Figure 4: Training loss curves. Left: Shapes3D. Right: MPI3D-toy.
Figure 5: Factor swapping on Shapes3D. Each column shows a source–target pair. The first two rows contain the source and target images. The remaining rows show generations after replacing the source token associated with the indicated factor with its target counterpart. Results cover floor hue, wall hue, object hue, object size, object shape, and orientation.