A Unified Latent Space Disentanglement VAE Framework with Robust Disentanglement Effectiveness Evaluation

Xiaoan Lang; Md Mostafizer Rahman; Fang Liu

stat.MLMar 11, 2026

A Unified Latent Space Disentanglement VAE Framework with Robust Disentanglement Effectiveness Evaluation

Authors: Xiaoan Lang, Md Mostafizer Rahman, Fang Liu

Organizations: Department of Applied and Computational Mathematics and Statistics University of Notre Dame, Notre Dame, IN, USA · Lucy Family Institute for Data & Society University of Notre Dame, Notre Dame, IN, USA

Abstract

Evaluating and interpreting latent representations, such as variational autoencoders (VAEs), remains a significant challenge for diverse data types, especially when ground-truth generative factors are unknown. To address this, we unify several state-of-the-art disentangled VAE approaches for latent space disentanglement into one framework -- bfVAE. To assess the effectiveness of a disentangled VAE model and enhance latent space interpretability, we propose Feature Variance Heterogeneity via Latent Traversal (FVH-LT) and Dirty Block Sparse Regression in Latent Space (DBSR-LS). To ensure robust interpretability of learned latent space, we develop a greedy alignment strategy (GAS) that mitigates label switching and aligns latent dimensions across runs to set the foundation of result aggregation. We also introduce a convenient scalar latent space separation index (LSSI) based on the GAS-aligned outputs of FVH-LT and DBSR-LS to summarize the overall latent structural separation without knowledge of the ground-truth generative factors. We compare bfVAE to five VAE models and validate the effectiveness FVH-LT, DBSR-LS, and LSSI in on seven tabular and image datasets. Under our examined experimental settings, bfVAE provides a more flexible disentanglement framework achieves more favorable overall trade-off between disentanglement and reconstruction than the benchmark VAE models; FVH-LT and DBSR-LS reliably uncover semantically meaningful and domain-relevant latent structures and generally yield consistent results; and LSSI makes an effective quantitative summary of latent structural separation.

Figures & tables

Figure 1: The FVH-LT Workflow. $\mathbf{x}_{i}$ contains input features for the $i$ -th observation; $z_{k}$ is the $k$ -th LD and $\hat{\mathbf{x}}_{ikl}$ for $k\!=\!1,\ldots,K$ and $l\!=\!1,\ldots,L$ is reconstructed output for $\mathbf{x}_{i}$ at $l$ -th fixed value from the LT of $z_{k}$ ; $s_{kj}$ is the sample variance of the restructured data for the $j$ -th feature from the LT on LD $k$ .

Figure 2: The GAS procedure.

d_{rk}

denotes the prior-posterior KL divergence of

k

-th LD;

C_{ij}

represents the correlation between the

i

-th LD in reference run

r^{*}

and the

j

-th LD in run

r

in the assocation matrix output from FVH-LT (variance matrix

S

) or DBSR-LS (regression coefficient matrix

|D|

Panel of Figure 3 — Figure 3: Illustration of two degenerate cases resulting in LSSI $=0$ . Left: All entries of matrix $A$ are $0$ , indicating the complete absence of informative LDs. Right: A single LD dominates all other LDs across all input features in $A$ , resulting in no disentanglement.

Dataset	Training $n$ ; test $n$	No. of Features $p$	Known Ground Truth?
FA15 (tabular) †	800/200	15	Yes ( $K_{0}=4$ factors)
FA100 (tabular) †	40,000/10,000	100	Yes ( $K_{0}=6$ factors)
White wine (tabular) $\#$	3,918/980	12	No
FIFA 2018 (tabular) $\#$	102/26 (51/13 matches)	16	No
RNA-seq (tabular) $\#$	640/161	20,264	No
MNIST (image) $\#$	50,000/10,000	784 ( $28\times 28$ )	No

Table 1: Datasets used in the experiments

Panel of Figure 4 — Figure 4: Ground-truth factor-feature association and discovered LD-feature association via bfVAE assessed by FVH-LT in FA15 and FA100. Darker cells indicate stronger associations between LDs and input features. Values in the LD-feature FVH-LT matrix are not shown due to readability reasons. DBSR-LS results for bfVAE and FVH-LT and DBSR-LS in the VAE benchmarks are in Apps. D .1- D .4.

Method	Data	Metric	bfVAE	factor-VAE	$\beta$ -VAE	Vanilla VAE	DIP-VAE-I	DIP-VAE-II
FVH-LT	FA15	$\uparrow$ LSSI	0.891	0.807	0.366	0.544	0.633	0.498
	FA15	$(\to 0)\,K_{\mathrm{max}}\!-\!K_{0}$ †	0	-1	0	0	0	0
	FA100	$\uparrow$ LSSI	0.913	0.940	0.252	0.866	0.874	0.929
	FA100	$(\to 0)\,K_{\mathrm{max}}\!-\!K_{0}$ †	0	0	3	0	0	0
DBSR-LS	FA15	$\uparrow$ LSSI	0.979	0.987	0.906	0.597	0.962	0.841
	FA15	$(\to 0)\,K_{\mathrm{max}}\!-\!K_{0}$ †	0	-1	0	0	0	-1

Table 2: LSSI, reconstruction error, and uncovered informative LD counts in synthetic FA15 and FA100.

Data	Metric	bfVAE	factor-VAE	$\beta$ -VAE	Vanilla VAE	DIP-VAE-I	DIP-VAE-II
FA15	LSSI 0 FVH-LT	0.963	0.811	0.405	0.703	0.661	0.780
	LSSI 0 DBSR-LS	1.000	0.999	0.972	0.851	0.995	0.578
	Higgins et al., (2017) ‡	$\mathbf{1.00\pm 0.00}$	$0.99\pm 0.01$	$0.90\pm 0.07$	$0.96\pm 0.06$	$0.84\pm 0.11$	$0.94\pm 0.07$
	Kim and Mnih, (2018) ‡	$\mathbf{0.99\pm 0.03}$	$0.90\pm 0.06$	$0.89\pm 0.09$	$0.84\pm 0.07$	$0.84\pm 0.13$	$0.81\pm 0.06$
FA100	LSSI 0 FVH-LT	0.990	0.986	0.417	0.860	0.932	0.962
	LSSI 0 DBSR-LS	0.766	0.743	0.351	0.702	0.681	0.738

Table 3: LSSI 0 and baseline disentanglement scores in synthetic FA15 and FA100

Panel of Figure 5 — Figure 5: Disentanglement effects of bfVAE assessed by FVH-LT in datasets without knowledge of ground-truth generative factors. Darker cells indicate stronger associations between LD and input features.DBSR-LS results for bfVAE and FVH-LT and DBSR-LS results for the benchmark VAE models are in Apps. D .5- D .9.

Method	Data	Metric	bfVAE	factor-VAE	$\beta$ -VAE	Vanilla VAE	DIP-VAE-I	DIP-VAE-II
FVH-LT	RNA-seq	$\uparrow$ LSSI	0.081	0.057	0.056	0.052	0.072	0.058
	RNA-seq	$K_{\max}$	11	14	12	12	13	14
	White wine	$\uparrow$ LSSI	0.390	0.308	0.175	0.220	0.090	0.280
	White wine	$K_{\max}$	5	4	5	5	4	4
	FIFA 2018	$\uparrow$ LSSI	0.132	0.020	0.197	0.031	0.104	0
	FIFA 2018	$K_{\max}$	5	2	5	2	5	3

Table 4: LSSI, reconstruction error, and uncovered informative LD counts in real tabular data.

fixed acidity	volatile acidity (VC)	citric acid	residual sugar	chlorides (C)	free SO 2
0.08	0.17	0.08	0.06	0.14	0.05
total SO 2	density (D)	pH	sulphates	alcohol (A)	quality
0.09	0.25	0.07	0.05	0.54	2.9

Table 5: Variance of reconstructed features with LT on wine quality score via FVH-LT.

Panel of Figure 6 — Figure 6: Disentanglement effects of bfVAE on CelebA assessed by FVH-LT. FVH-LT shows which facial features correspond to what informative LD: LT of non-informative LD4 produces no visible change in the reconstruction and a completely dark heatmap in (a) and (b); in (b), LD1, 2, 12, 16, and 20 primary encode sex, hair style and color, skin tone and illumination, and azimuth (horizontal head orientation), background information, respectively. FVH-LT results for the benchmark VAE models are in Apps. D .11.

Panel of Figure 7 — Figure 7: Disentanglement effects of bfVAE in MNIST assessed by FVH-LT. (a) and (b) show that LD1, LD10, LD11, and LD13 capture variation between digits 3 and 2, 1 and 3, 7 and 3, and 0 and 1 respectively. Key structural elements that differentiate the digits are evident such as the loops and center vertical line in LD13 and the curved regions near top-left and bottom-right in LD10. The 7 informative LDs produce obvious changes in reconstructed images under LT (first four rows in (b)) with interpretable semantic patterns. For example, LT of LD1 generates digits 2, 3, 8 and the morphing process is reflected in the last column of (b) that suggests a blend of 2,3,8. LD10 also captures directional variations in handwriting. Along the LT, the generated digit changes from a slanted 1 into a 3 which gradually changes its slant and curvature. FVH-LT results for the benchmark VAE models are in Apps. D .10.

Data	Metric	bfVAE	factor-VAE	$\beta$ -VAE	Vanilla VAE	DIP-VAE-I	DIP-VAE-II
CelebA	$\uparrow$ LSSI	0.116	0.052	0	0.029	0.031	0.029
	$\downarrow$ recon. $\ell_{2}$ ‡	$11.07\pm 0.23$	$10.36\pm 0.03$	$19.58\pm 0.10$	$10.16\pm 0.03$	$10.21\pm 0.03$	$10.21\pm 0.05$
	$K_{\max}$	14	8	1	10	10	10
MNIST	$\uparrow$ LSSI	0.184	0.089	0.053	0.078	0.088	0.095
	$\downarrow$ recon. $\ell_{2}$ ‡	$3.10\pm 0.11$	$3.46\pm 0.02$	$5.11\pm 0.00$	$3.28\pm 0.01$	$3.30\pm 0.02$	$3.33\pm 0.02$
	$K_{\max}$	7	9	14	10	8	9

Table 6: LSSI, reconstruction error, and uncovered informative LD counts in CelebA and MNIST.

	FA15	FIFA 2018	White wine	CVAE White wine	FA100	RNA-seq	MNIST	CelebA
$n$ ‡	800	102	3,918	3,918	40,000	640	50,000 †	35,000
Epochs	150	350	150	75	150	100	35	50
model training time (min)	3 (M2)	1 (M2)	10 (M2)	20 (M2)	4 (G4)	1 (G4)	2 (A100)	4 (A100)
FVH-LT † time (min)	2 (M2)	$<$ 1 (M2)	15 (M2)	16 (M2)	17 (G4)	10 (G4)	14 (A100)	60 (A100)
DBSR-LS time (min)	1 (M2)	$<$ 1 (M2)	1 (M2)	–	2 (G4)	–	40 (A100)	–

Table 7: Approximate single-run time for VAE model training, FVH-LT, and DBSR-LS.

Panel of Figure 8 — Figure 8: An example of posterior statistics across LDs on MNIST (15 LDs $\times$ 1000 MNIST images): (a) posterior distributions of informative LDs, (b) posterior distributions of non-informative LDs, (c) histogram of posterior means, (d) histogram of posterior variances (among the 15,000 LDs, exactly 7,000 display near-zero variances in their distribution and correspondingly exhibit large KL divergence).

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Conceptual illustration of DBSR-LS. Posterior means

\bm{\mu}_{Z}

in the latent space are multi-task regression responses;

\mathbf{x}

are predictors; sparse regression coefficient matrix

\hat{D}

summarizes latent-feature association.

Data	Method	Model	$K_{\max}$	$\tau$	$c_{1}$	$c_{1,\mathrm{info}}$	$c_{2}$	$c_{2,\mathrm{info}}$	LSSI	LSSI info
FA15	FVH-LT	bfVAE	4	0.142	1.000	1.000	0.957	0.957	0.891	0.922
		Factor VAE	3	0.213	0.867	0.867	0.998	0.986	0.807	0.805
		$\beta$ -VAE	4	0.149	0.867	0.867	1.000	0.969	0.366	0.393
		Vanilla VAE	4	1.244	0.867	0.867	0.750	0.750	0.544	0.527
		DIP-VAE-I	4	2.995	1.000	1.000	0.935	0.935	0.633	0.618
		DIP-VAE-II	4	1.048	0.867	0.867	0.635	0.635	0.498	0.496

Appendix

Table 28

Data	Method	Model	$K_{\max}$	$\tau$	$c_{1}$	$c_{1,\mathrm{info}}$	$c_{2}$	$c_{2,\mathrm{info}}$	LSSI	LSSI info
FA100	FVH-LT	bfVAE	6	0.124	1.000	1.000	0.924	0.924	0.913	0.915
		Factor VAE	6	1.171	1.000	1.000	0.949	0.949	0.940	0.935
		$\beta$ -VAE	9	0.040	0.900	0.900	0.591	0.591	0.252	0.218
		Vanilla VAE	6	1.087	1.000	1.000	0.941	0.941	0.866	0.809
		DIP-VAE-I	6	1.177	1.000	1.000	0.952	0.952	0.874	0.887
		DIP-VAE-II	6	1.183	1.000	1.000	0.951	0.951	0.929	0.915

Appendix

Table 29

Data	Method	Model	$K_{\max}$	$\tau$	$c_{1}$	$c_{1,\mathrm{info}}$	$c_{2}$	$c_{2,\mathrm{info}}$	LSSI	LSSI info
RNA-seq	FVH-LT	bfVAE	11	0.145	0.637	0.637	0.285	0.285	0.081	0.054
		Factor VAE	14	0.024	0.542	0.542	0.455	0.454	0.057	0.054
		$\beta$ -VAE	12	0.040	0.498	0.498	0.574	0.566	0.056	0.053
		Vanilla VAE	12	0.018	0.493	0.493	0.559	0.541	0.052	0.045
		DIP-VAE-I	13	0.343	0.502	0.502	0.590	0.589	0.072	0.071
		DIP-VAE-II	14	0.278	0.477	0.477	0.560	0.560	0.058	0.057

Appendix

Table 30

Data	Method	Model	$K_{\max}$	$\tau$	$c_{1}$	$c_{1,\mathrm{info}}$	$c_{2}$	$c_{2,\mathrm{info}}$	LSSI	LSSI info
White wine	FVH-LT	bfVAE	5	1.176	0.909	0.909	0.586	0.586	0.390	0.355
		Factor VAE	4	1.132	0.909	0.909	0.407	0.407	0.308	0.270
		$\beta$ -VAE	5	1.206	0.727	0.727	0.534	0.534	0.175	0.176
		Vanilla VAE	5	1.140	0.909	0.909	0.296	0.296	0.220	0.208
		DIP-VAE-I	4	0.684	0.818	0.727	0.295	0.112	0.090	0.033
		DIP-VAE-II	4	1.250	0.818	0.818	0.434	0.434	0.280	0.236

Appendix

Table 31

Data	Method	Model	$K_{\max}$	$\tau$	$c_{1}$	$c_{1,\mathrm{info}}$	$c_{2}$	$c_{2,\mathrm{info}}$	LSSI	LSSI info
2018 FIFA	FVH-LT	bfVAE	5	0.248	0.875	0.875	0.252	0.252	0.132	0.120
		Factor VAE	2	0.052	0.938	0.938	0.022	0.017	0.020	0.015
		$\beta$ -VAE	5	0.249	0.750	0.750	0.903	0.903	0.197	0.168
		Vanilla VAE	2	0.116	0.812	0.812	0.040	0.040	0.031	0.029
		DIP-VAE-I	5	0.392	0.562	0.562	0.695	0.695	0.104	0.098
		DIP-VAE-II	3	0.159	0.000	0.000	0.000	0.000	0.000	0.000

Appendix

Table 32

Data	Method	Model	$K_{\max}$	$\tau$	$c_{1}$	$c_{1,\mathrm{info}}$	$c_{2}$	$c_{2,\mathrm{info}}$	LSSI	LSSI info
MNIST	FVH-LT	bfVAE	7	0.097	0.453	0.453	0.555	0.555	0.184	0.096
		Factor VAE	9	0.024	0.406	0.406	0.292	0.292	0.089	0.068
		$\beta$ -VAE	14	0.000	0.258	0.258	0.724	0.685	0.053	0.050
		Vanilla VAE	10	0.024	0.411	0.411	0.260	0.260	0.078	0.062
		DIP-VAE-I	8	0.007	0.365	0.364	0.982	0.802	0.088	0.074
		DIP-VAE-II	9	0.024	0.415	0.415	0.302	0.302	0.095	0.072

Appendix

Table 33

Variational Autoencoders

Variational autoencoders are probabilistic encoder–decoder models for learning latent representations, generating data, and estimating uncertainty.

See all 208 papers

Sculpting the Latent Space: Disentanglement with WAE-MMD and Programmable Priors
Oct 13, 2025Quentin Fruytier, Akshay Malhotra, Shahab Hamidi-Rad +3Disentangled Representation Learning Semantic Representation Learning
The University of Texas at Austin · InterDigital Communications, AI Lab
XFACTORS: Disentangled Information Bottleneck via Contrastive Supervision
Jan 29, 2026Alexandre Myara, Nicolas Bourriez, Thomas Boyer +3Variational Autoencoders Disentangled Representation Learning
ENS Paris
Soft-Constrained Optimization of Latent Space in Variational Autoencoders
Jul 26, 2026Ye ShiVariational Autoencoders Disentangled Representation Learning
Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, 47906 USA

Oct 13, 2025J/K move · Enter open · S save

Sculpting the Latent Space: Disentanglement with WAE-MMD and Programmable Priors

Quentin Fruytier, Akshay Malhotra, Shahab Hamidi-Rad, Aditya Sant+2

The University of Texas at Austin · InterDigital Communications, AI Lab

Disentangled Representation Learning Semantic Representation Learning Autoencoders Latent-Space Regularization

Open paper AskPDF arXiv

Learning disentangled representations, where semantic features are independently captured in a latent space, is dominated by Variational Auto Encoders (VAE) and Adversarial Training. In this paper, we provide direct quantitative and visual evidence that these methods consistently fail to enforce the target factorized representation on the aggregate posterior, leading to entangled representations. We quantify this failure using the Latent Predictability Score (LPS), an unsupervised metric of mutual predictability that empirically lower bounds the level of entanglement in a latent space. To address these limitations, we leverage the non-parametric Maximum Mean Discrepancy (MMD) to directly sculpt the aggregate posterior enabling programmable priors. We demonstrate (1) state-of-the-art unsupervised statistical independence, with a 70% reduction in detected entanglement compared to leading baselines; (2) the flexibility to optionally inject semi-supervised information for explicit semantic alignment; and (3) precise shaping of the aggregate posterior distribution (validated through quantization-aware training) without sacrificing reconstruction quality.

Abstract

Variational Autoencoders

Explore similar work

Sculpting the Latent Space: Disentanglement with WAE-MMD and Programmable Priors

XFACTORS: Disentangled Information Bottleneck via Contrastive Supervision

Soft-Constrained Optimization of Latent Space in Variational Autoencoders