Organizations: VEDECOM Institute · DAVID Lab, UVSQ, Paris-Saclay University · TSM Unit UVSQ, Paris-Saclay University · DAVID Lab UVSQ, Paris-Saclay University · LIPN, UMR CNRS 7030 Sorbonne Paris Nord University
We introduce LBM-VAE, a novel framework that integrates Latent Block Models (LBMs) into Variational Autoencoders (VAEs) to jointly cluster observations and latent dimensions. Beyond its clustering capabilities, LBM-VAE provides a block-structured latent representation that naturally supports disentangled representation learning. In contrast to conventional approaches that associate each factor of variation with an individual latent dimension, our framework allows factors of variation to be encoded across groups of latent dimensions. Experimental results demonstrate the promising clustering performance of LBM-VAE, while achieving competitive results in disentangled representation learning. LBM-VAE highlights the potential of combining co-clustering and variational inference to learn structured and interpretable latent representations.
Figures & tables
Figure 1: t-SNE projections of the MNIST test-set latent means at epochs 5, 30, 55 and 90, coloured by digit class (LBM prior activated at epoch 5).
Figure 2: Latent traversals of the LBM-VAE on Shapes3D. Each column varies one of the 15 latent coordinates and each row one traversal value. Columns are grouped by learned cluster.
DCI
Model
MIG
SAP
D
C
I
β -VAE
0.268±0.079
0.077±0.037
0.603±0.129
0.522±0.118
0.926±0.045
β -TCVAE
0.312±0.171
0.092±0.030
0.741±0.085
0.637±0.084
0.977±0.015
β -VAE + HFS
0.616±0.181
0.085±0.055
0.765±0.179
0.788±0.109
0.920±0.093
β -TCVAE + HFS
0.534±0.204
0.078±0.029
0.848±0.079
0.779±0.088
0.991±0.007
LBM-VAE
0.452±0.124
0.129±0.027
0.899±0.107
0.780±0.095
0.990±0.018
Table 1: Disentanglement scores on Shapes3D (best mean in bold, second best underlined).
Evaluating and interpreting latent representations, such as variational autoencoders (VAEs), remains a significant challenge for diverse data types, especially when ground-truth generative factors are unknown. To address this, we unify several state-of-the-art disentangled VAE approaches for latent space disentanglement into one framework -- bfVAE. To assess the effectiveness of a disentangled VAE model and enhance latent space interpretability, we propose Feature Variance Heterogeneity via Latent Traversal (FVH-LT) and Dirty Block Sparse Regression in Latent Space (DBSR-LS). To ensure robust interpretability of learned latent space, we develop a greedy alignment strategy (GAS) that mitigates label switching and aligns latent dimensions across runs to set the foundation of result aggregation. We also introduce a convenient scalar latent space separation index (LSSI) based on the GAS-aligned outputs of FVH-LT and DBSR-LS to summarize the overall latent structural separation without knowledge of the ground-truth generative factors. We compare bfVAE to five VAE models and validate the effectiveness FVH-LT, DBSR-LS, and LSSI in on seven tabular and image datasets. Under our examined experimental settings, bfVAE provides a more flexible disentanglement framework achieves more favorable overall trade-off between disentanglement and reconstruction than the benchmark VAE models; FVH-LT and DBSR-LS reliably uncover semantically meaningful and domain-relevant latent structures and generally yield consistent results; and LSSI makes an effective quantitative summary of latent structural separation.
Xiaoan Lang, Md Mostafizer Rahman, Fang Liu
Department of Applied and Computational Mathematics and Statistics University of Notre Dame, Notre Dame, IN, USA · Lucy Family Institute for Data & Society University of Notre Dame, Notre Dame, IN, USA
Learning disentangled representations, where semantic features are independently captured in a latent space, is dominated by Variational Auto Encoders (VAE) and Adversarial Training. In this paper, we provide direct quantitative and visual evidence that these methods consistently fail to enforce the target factorized representation on the aggregate posterior, leading to entangled representations. We quantify this failure using the Latent Predictability Score (LPS), an unsupervised metric of mutual predictability that empirically lower bounds the level of entanglement in a latent space. To address these limitations, we leverage the non-parametric Maximum Mean Discrepancy (MMD) to directly sculpt the aggregate posterior enabling programmable priors. We demonstrate (1) state-of-the-art unsupervised statistical independence, with a 70% reduction in detected entanglement compared to leading baselines; (2) the flexibility to optionally inject semi-supervised information for explicit semantic alignment; and (3) precise shaping of the aggregate posterior distribution (validated through quantization-aware training) without sacrificing reconstruction quality.
Disentangled representation learning aims to map independent factors of variation to independent representation components. On one hand, purely unsupervised approaches have proven successful on fully disentangled synthetic data, but fail to recover semantic factors from real data without strong inductive biases. On the other hand, supervised approaches are unstable and hard to scale to large attribute sets because they rely on adversarial objectives or auxiliary classifiers. We introduce XFactors, a weakly-supervised VAE framework that disentangles and provides explicit control over a chosen set of factors. Building on the Disentangled Information Bottleneck perspective, we decompose the representation into a residual subspace S and factor-specific subspaces T1,…,TK and a residual subspace S. Each target factor is encoded in its assigned Ti through contrastive supervision: an InfoNCE loss pulls together latents sharing the same factor value and pushes apart mismatched pairs. In parallel, KL regularization imposes a Gaussian structure on both S and the aggregated factor subspaces, organizing the geometry without additional supervision for non-targeted factors and avoiding adversarial training and classifiers. Across multiple datasets, with constant hyperparameters, XFactors achieves state-of-the-art disentanglement scores and yields consistent qualitative factor alignment in the corresponding subspaces, enabling controlled factor swapping via latent replacement. We further demonstrate that our method scales correctly with increasing latent capacity and evaluate it on the real-world dataset CelebA. Our code is available at github.com/ICML26-anon/XFactors.
Alexandre Myara, Nicolas Bourriez, Thomas Boyer +3