Variational decomposition autoencoding improves disentanglement of latent representations

Ioannis N. Ziogas; Aamna Al Shehhi; Ahsan H. Khandoker; Leontios J. Hadjileontiadis

cs.LGJan 11, 2026

Variational decomposition autoencoding improves disentanglement of latent representations

Authors: Ioannis N. Ziogas, Aamna Al Shehhi, Ahsan H. Khandoker, Leontios J. Hadjileontiadis

Organizations: Dept. of Biomedical Engineering, Khalifa University of Science and Technology, Abu Dhabi, UAE · School of Electrical and Computer Engineering, Aristotle University of Thessaloniki, Thessaloniki, Greece

Abstract

Understanding the structure of complex, nonstationary, high-dimensional time-evolving signals is a central challenge in scientific data analysis. In many domains, such as speech and biomedical signal processing, the ability to learn disentangled and interpretable representations is critical for uncovering latent generative mechanisms. Traditional approaches to unsupervised representation learning, including variational autoencoders (VAEs), often struggle to capture the temporal and spectral diversity inherent in such data. Here we introduce variational decomposition autoencoding (VDA), a framework that incorporates a strong structural bias toward signal decomposition into variational representation learning. VDA is instantiated through variational decomposition autoencoders (DecVAEs), i.e., encoder-only neural networks that perform latent autoencoding at the level of distributions to learn multiple latent subspaces aligned with time-frequency characteristics. We evaluate DecVAEs on simulated data and three publicly available scientific datasets, spanning speech recognition, dysarthric speech evaluation, and emotional speech classification. DecVAEs improve on VAE-based methods in disentanglement quality and robustness, and their latent subspaces become factor-specific when the generative factors have distinct spectral signatures. These findings position decomposition-aware architectures as potential tools for extracting structured representations from speech, and potentially from other dynamic signals with factors exhibiting distinct spectral signatures, with applications in clinical diagnostics, human-computer interaction, and adaptive neurotechnologies.

Figures & tables

Fig. 1 : $|$ Variational decomposition autoencoding (VDA) uses a neural network encoder to approximate latent distributions assuming a decomposition generative process and novel disentanglement mechanics. a) , variational autoencoding: directed factor graph over the $F$ frames $x^{(f)}$ of a sequence, and illustration of inference ( $\varphi$ ) and generation ( $\theta$ ). b) , (i) directed factor graph for the proposed VDA; (ii) VDA assumes that each observation is a superposition of component signals in mutually orthogonal subspaces of the signal space, each generated by its own latent subspace $z_{i}$ with a Gaussian prior $p(z_{i})$ , so that $p(z)=\prod_{i}p(z_{i})$ . Under this assumption the posterior factorizes over components (Proposition 1, Methods): a single recognition model $q_{\varphi}$ , shared across components, maps each decomposed component $x_{i}$ to its own latent subspace, $q_{\varphi}(z_{i}\,|\,x_{i})$ , and the original signal $x$ to $z_{0}$ , without conditioning between subspaces; an aggregation function $a(\cdot)$ combines the subspaces. c) , besides the Gaussian prior regularization, VDA imposes two structural biases that come from the decomposition, reconstruction and orthogonality, and together they shape a latent space that disentangles. DecVAE is trained on the VDA objective, which combines a self-supervised loss embodying these biases with Gaussian prior regularization of the component subspaces (detailed in Methods). d) , the variational decomposition autoencoder (DecVAE) can be applied across several domains $\theta_{1},\dots,\theta_{\omega}$ (speech, emotion, dysarthria severity). It can optionally learn latent variables at two time scales: the decomposition model $\mathcal{D}^{C}_{w}$ is applied to the $N$ input sequences ( $T$ ) and to the $F$ windowed frames of each sequence ( $X$ ); e.g., in speech production ( $\theta_{1}$ ) the long-term speaker identity variable $S$ influences the short-term phonetic content $Z$ . In this panel, superscripts index the frame $f$ and the sequence $n$ , subscripts index the components, and primes denote decomposed signals. The decomposed sequences $\{t^{\prime(n)},t^{\prime(n)}_{1},\dots,t^{\prime(n)}_{C}\}$ (domain $T^{\prime}$ ) and frames $\{x^{\prime(f,n)},x^{\prime(f,n)}_{1},\dots,x^{\prime(f,n)}_{C}\}$ (domain $X^{\prime}$ ) are analysed by the sequence and frame branches of DecVAE, with parameters $\varphi_{S}$ and $\varphi_{Z}$ , which learn the latent subspaces $\{\tilde{s}^{(n)}_{0},\dots,\tilde{s}^{(n)}_{C}\}$ and $\{\tilde{z}^{(f,n)}_{0},\dots,\tilde{z}^{(f,n)}_{C}\}$ . Each branch aggregates its subspaces through $a(\cdot)$ (concatenation in our experiments) into sequence-level and frame-level representations $\tilde{S}$ and $\tilde{Z}$ (detailed in Methods).

Fig. 2 : $|$ Simulated speech data (SimVowels) . a) , TSNE L. and G. (2008) of development set frame-level vowel-colored inputs, (left): Mel filterbank features of original signal, (right): decomposed Mel filterbank features of the original and components signals, after aggregation. b) , TSNE of training set sequence-level speaker-colored inputs, (left): Mel filterbank features of original signal, (right): decomposed Mel filterbank features of the third component, after aggregation. c) , TSNE of latent spaces using a as inputs (left) DecAE with empirical wavelet transform (EWT) J. (2013) frequency-resonant frame embedding, (middle) DecVAE with EWT J. (2013) vowel-colored embedding, (right) VAE D. P. and M. (2014) vowel-colored embedding. d) ,TSNE of latent spaces using a,b as inputs, (left) DecVAE with EWT frequency-resonant sequence embedding, (mid-left) DecVAE with EWT J. (2013) speaker-colored sequence embedding, (mid-right) DecVAE with EWT J. (2013) speaker-colored frame embedding, (right) VAE D. P. and M. (2014) speaker-colored frame embedding. e) , TSNE projection of DecVAE with EWT J. (2013) vowel-colored latent space (two mini-batches) along with a multivariate Gaussian distribution (sphere). f) , TSNE projection of VAE D. P. and M. (2014) vowel-colored latent space (two mini-batches) along with a multivariate Gaussian distribution (sphere). g) , (left) disentanglement (x-axis) versus informativeness (y-axis) from DCI C. and C.K.I. (2018) and robustness (circle size), (right) modularity (x-axis) versus explicitness (y-axis) K. and M.C. (2018) and robustness R. et al. (2019) (circle size). h) , speaker identification (x-axis) versus phoneme (vowel) recognition (y-axis) and disentanglement C. and C.K.I. (2018) (circle size) performance of $\beta$ -DecVAE variants ( $\beta=0.1$ ) and VAE D. P. and M. (2014) , $\beta$ -VAE ( $\beta=0.1$ ) I. et al. (2017) , ICA Hyvärinen et al. (2001) , PCA M. et al. (2022) . In the frequency-resonant embeddings ( c , d , left), each point is the latent embedding of a frame ( c ) or an utterance ( d ) of the original signal or of one of its components, colored by its source (original signal, component 1,2, or 3); the separated regions for the different sources show that each component occupies its own region of the latent space. In the rest of the TSNE panels, points are colored by generative factor named in the panel title, and compact well-separated clusters indicate that the factor is captured by the learned representation. The sphere in e , f is a standard multivariate Gaussian, shown as a reference for the prior. In g , h , models towards the upper right corner perform better on both axes, and the circle size (the bigger the better) encodes a third metric.

Fig. 3 : $|$ Real speech data TIMIT J.S. et al. (1993) . a) , TSNE L. and G. (2008) of a subset (13 phonemes from 6 mini-batches) of the development set, frame-level phoneme-colored inputs, (left): Mel filterbank features of original signal, (right): decomposed Mel filterbank features of the original and components signals, after aggregation. b) , TSNE of a subset (10 speakers from 6 mini-batches) of the development set, (left): Mel filterbank features of original signal, (right): decomposed Mel filterbank features of aggregated components. c) , TSNE of latent spaces using a as inputs (6 mini-batches) (left) DecVAE with EWT J. (2013) frequency-resonant frame embedding ( $C=4$ ), (middle) DecVAE with EWT J. (2013) phoneme-colored embedding, (right) VAE D. P. and M. (2014) phoneme-colored embedding. d) , TSNE of latent spaces using a,b as inputs (6 mini-batches), (left) DecVAE with EWT J. (2013) speaker-colored frame embedding, (right) VAE D. P. and M. (2014) speaker-colored frame embedding. e) , TSNE projection of DecVAE with EWT J. (2013) phoneme-colored (left) and speaker-colored (right) latent spaces (6 mini-batches) along with a multivariate Gaussian distribution (sphere). f) , TSNE projection of VAE D. P. and M. (2014) phoneme-colored (left) and speaker-colored (right) latent spaces (6 mini-batches) along with a multivariate Gaussian distribution (sphere). g) , (left) disentanglement (x-axis) versus informativeness (y-axis) from DCI C. and C.K.I. (2018) and robustness R. et al. (2019) (circle size), (right) modularity (x-axis) versus explicitness (y-axis) K. and M.C. (2018) and robustness R. et al. (2019) (circle size). h) , disentanglement metrics scores of $\beta$ -DecVAE variants, and VAE-based and eigenanalysis methods, VAE D. P. and M. (2014) , $\beta$ -VAE I. et al. (2017) , ICA Hyvärinen et al. (2001) , PCA M. et al. (2022) . Results for a subsampled set of IEMOCAP (see ‘Speech emotion recognition fine-tuning evaluation’), for DecVAEs, VAE-based, time-series and self-supervised baselines are presented in Supplementary Tables 16-19. Panels in this figure are organized and read as in Fig. 2 .

Fig. 4 : $|$ Zero-shot transfer to dysarthric speech data VOC-ALS R. et al. (2024) . a) , TSNE L. and G. (2008) of a subset (4 mini-batches) of the dataset, frame-level King’s clinical stage J.C. et al. (2012) -colored inputs, (left): Mel filterbank features of original signal, (right): decomposed Mel filterbank features of the original and components signals, after aggregation. b) , TSNE of a subset (4 mini-batches) of the dataset, frame-level phoneme-colored inputs (left): Mel filterbank features of original signal, (right): decomposed Mel filterbank features of the aggregated components. c) , TSNE of latent spaces using a,b as inputs and a pre-trained $\beta$ -DecVAE on SimVowels ( $\beta=0.1$ ) for zero-shot evaluation (4 mini-batches) (left) $\beta$ -DecVAE with FD, frequency-resonant frame embedding ( $C=4$ ), (middle-left) $\beta$ -DecVAE with FD, King’s clinical stage J.C. et al. (2012) -colored embedding, (middle-right) $\beta$ -DecVAE phoneme-colored embedding, (right) $\beta$ -DecVAE speaker-colored embedding (for 10 speakers). d) , TSNE of latent spaces using a,b as inputs and a pre-trained on SimVowels $\beta$ -VAE ( $\beta=0.1$ ) for zero-shot evaluation (4 mini-batches), (left) $\beta$ -VAE I. et al. (2017) King’s clinical stage J.C. et al. (2012) -colored embedding, (middle) $\beta$ -VAE I. et al. (2017) phoneme-colored embedding, (right) $\beta$ -VAE I. et al. (2017) speaker-colored embedding (for 10 speakers). e , TSNE projection of $\beta$ -DecVAE with FD, King’s clinical stage J.C. et al. (2012) -colored latent space (4 mini-batches) along with a multivariate Gaussian distribution (sphere). f) , TSNE projection of $\beta$ -VAE I. et al. (2017) King’s clinical stage J.C. et al. (2012) -colored latent space (4 mini-batches) along with a multivariate Gaussian distribution (sphere). g) , (left) King’s clinical stage J.C. et al. (2012) detection (x-axis) versus speaker identification (y-axis) and phoneme recognition (circle size), (right) disentanglement (x-axis) versus informativeness (y-axis) from DCI C. and C.K.I. (2018) and robustness R. et al. (2019) (circle size). h) , total classification performance (accuracies and F1-macro with $95\%$ confidence intervals error bars) of $\beta$ -DecVAE variants, and VAE-based and eigenanalysis methods VAE D. P. and M. (2014) , $\beta$ -VAE I. et al. (2017) , ICA Hyvärinen et al. (2001) , PCA M. et al. (2022) , in predicting King’s clinical stage, disease duration, phoneme recognition, speaker identification. King’s stage and disease duration classification are evaluated with participants shared between folds, as is done for speaker and phoneme recognition. If participants are held out, all classifiers are near chance (see Supplementary Fig. 27, Supplementary Table 21). PCA and ICA are fitted on VOC-ALS without labels and serve as non-zero-shot references. Panels in this figure are organized and read as in Fig. 2 .

Fig. 5 : $|$ Fine-tuning disentanglement on emotional speech data IEMOCAP C. et al. (2008) . a) , TSNE L. and G. (2008) of a subset (6 mini-batches) of the dataset, frame-level categorical emotion-colored inputs, (left): Mel filterbank features of original signal, (right): decomposed Mel filterbank features of the original and components signals, after aggregation. b) , TSNE of a subset (6 mini-batches) of the dataset, frame-level speaker-colored inputs (left): Mel filterbank features of original signal, (right): decomposed Mel filterbank features of the aggregated components. c) , TSNE of latent spaces using a as inputs and a pre-trained DecVAE on SimVowels for fine-tuning evaluation (6 mini-batches) (left) DecVAE with FD, frequency-resonant frame embedding ( $C=4$ ), (middle-left) DecVAE with FD, emotion-colored embedding, (middle-right) DecVAE speaker-colored embedding, (right) DecVAE phoneme-colored embedding (for 11 phonemes). d) , TSNE of latent spaces using b as inputs and a pre-trained on SimVowels $\beta$ -VAE ( $\beta=0.1$ ) for fine-tuning evaluation (6 mini-batches), (left) $\beta$ -VAE I. et al. (2017) emotion-colored embedding, (middle) $\beta$ -VAE I. et al. (2017) speaker-colored embedding, (right) $\beta$ -VAE I. et al. (2017) phoneme-colored embedding (for 11 phonemes). e) , TSNE projection of DecVAE with FD, emotion-colored latent space (6 mini-batches) along with a multivariate Gaussian distribution (sphere). f) , TSNE projection of $\beta$ -VAE I. et al. (2017) emotion-colored latent space (6 mini-batches) along with a multivariate Gaussian distribution (sphere). g) , (left) phoneme recognition (x-axis) versus speaker identification (y-axis) and emotion recognition (circle size) performances, (right) disentanglement (x-axis) versus informativeness (y-axis) from DCI C. and C.K.I. (2018) and robustness R. et al. (2019) (circle size), for the full IEMOCAP protocol (Supplementary Tables 14,15). h) , total classification performance (weighted and unweighted accuracy, weighted F1 with $95\%$ confidence intervals error bars) of $\beta$ -DecVAE variants, and VAE-based and eigenanalysis methods VAE D. P. and M. (2014) , $\beta$ -VAE I. et al. (2017) , ICA Hyvärinen et al. (2001) , PCA M. et al. (2022) , in predicting phoneme, speaker and categorical emotion classes, for the full IEMOCAP protocol (Supplementary Tables 14,15). Panels in this figure are organized and read as in Fig. 2 .

Panel of Figure 6 — Fig. 6 : $|$ Latent response interpretability analysis and overall performance . a) , latent responses on TIMIT J.S. et al. (1993) for the DecVAE + EWT model; rows are the latent subspaces of the original signal (X) and of the components (OC1 to OC4), columns are the four dimensions of each subspace with the largest response variance and two with the smallest (‘unused’), and each line is one phoneme, drawn as the speaker varies along the horizontal axis. b) , latent responses on VOC-ALS R. et al. (2024) for the $\beta$ -DecVAE ( $\beta=0.1$ ) + EWT model, laid out as in a ; each line is one King’s clinical stage, traced as the phoneme varies. King’s stage is constant within each participant and therefore confounded with speaker identity (Supplementary Section 11.4. c) , latent responses on IEMOCAP C. et al. (2008) for the DecVAE + EWT model (first five columns, one per subspace) and the $\beta$ -VAE ( $\beta=0.1$ ) (last column), both transferred from TIMIT J.S. et al. (1993) ; for each emotion (rows), the surface shows the response of the most informative latent dimension over the phoneme-speaker plane. DecVAE spreads the responses over its subspaces, each covering a different range of the factors’ values, whereas the $\beta$ -VAE concentrates them in a single latent space. d) , overall disentanglement performance of (i) $\beta$ -DecVAE and (ii) $\beta$ -VAE models, across all utilized datasets; DecVAEs disentangle better than $\beta$ -VAEs on most metrics in all four datasets.

Fig. 7 : $|$ How the VDA objective shapes the latent subspaces on SimVowels. a) , Separation of the components (mean JS divergence between the normalized embeddings of pairs of components, warm colours) and their divergence from the original (cold colours) against $\beta$ , FD and EWT, sequence branch, test set; dotted line, maximum separation $\ln 2$ ; dashed line, divergence from the original at the optimum, $j_{3}=0.318$ (Proposition 2). Below: speaker (circles) and vowel (triangles) accuracy of the same checkpoints. (b-d) Subspace-wise tests on the frame-level latents of DecVAE + EWT ( $\beta=0.1$ ); X, subspace of the original; OC1 to OC3, component subspaces in increasing frequency. b) , Prediction from a single subspace (chance-normalized balanced accuracy of a linear probe), with the full latent as bottom row. c) , Ablation: drop in the full-latent score when the subspace is removed. d) , Swap: transfer of the factor when the subspace is taken from a frame that differs only in it, minus a matched control ( $\Delta T$ ). Mean ± s.d. over three seeds. e) , Selectivity of the DecVAE subspaces (coloured, one marker per seed) and of 20 random partitions (grey), for prediction and ablation, across decompositions and $\beta$ . f) , Linear CKA S. et al. (2019) between the subspaces of DecVAE + EWT models trained with different seeds (mean ± s.d. over seed pairs).

Fig. 8 : $|$ Sensitivity of DecVAE to the decomposition parameters . a) , Components FD keeps per frame ( $K$ ) on 100 test utterances per dataset, for the default peak detection intervals (bars) and, on SimVowels, six alternative layouts (lines); dashed line, generator $C=3$ . b) , Share of frames whose peaks are grouped ( $K>C$ ), match $C$ , are completed with empty components ( $0<K<C$ ) or have none ( $K=0$ ), per $C$ . c-e) , $\beta$ -DecVAE (FD, $\beta=0.1$ ) with $C=2$ to $6$ on SimVowels (blue; dashed line, generator $C$ ) and SimCoupled (red): c) , DCI disentanglement; d) , effective number of component subspaces carrying each factor (dotted line, $n_{\mathrm{eff}}=C$ ); e) , selectivity of the component subspaces minus the mean of 20 random partitions, for prediction, ablation and swap (see Methods). On SimCoupled ablation and swap fall below 0 because the undecomposed signal subspace holds the information (Fig. 9d). f) , DCI disentanglement on SimVowels ( $C=3$ ) for five interval layouts; dots, single runs. Error bars, s.d. over three seeds or three random layouts.

Fig. 9 : $|$ DecVAE on SimCoupled, whose factors are not band-limited. a) , Mean log-power spectrum per lag class (left; gain removed) and per gain class (right), independent split. b) , Share of each factor’s effect on band log-energy per Mel band, independent split; dotted line, uniform share. c) , Lag and gain accuracy of DecVAE (red) and baselines on the correlated test set (filled) and the independent split (open), classifiers trained on correlated data. d) , Prediction score of each subspace of $\beta$ -DecVAE + FD ( $\beta=0.1$ , three random seeds; mean ± s.d.), with the full latent as bottom row.

Disentangled Representation Learning

Disentangled representation learning separates distinct factors of variation so models can interpret, control, and generalize their predictions.

See all 240 papers

Mar 11, 2026

A Unified Latent Space Disentanglement VAE Framework with Robust Disentanglement Effectiveness Evaluation

Evaluating and interpreting latent representations, such as variational autoencoders (VAEs), remains a significant challenge for diverse data types, especially when ground-truth generative factors are unknown. To address this, we unify several state-of-the-art disentangled VAE approaches for latent space disentanglement into one framework -- bfVAE. To assess the effectiveness of a disentangled VAE model and enhance latent space interpretability, we propose Feature Variance Heterogeneity via Latent Traversal (FVH-LT) and Dirty Block Sparse Regression in Latent Space (DBSR-LS). To ensure robust interpretability of learned latent space, we develop a greedy alignment strategy (GAS) that mitigates label switching and aligns latent dimensions across runs to set the foundation of result aggregation. We also introduce a convenient scalar latent space separation index (LSSI) based on the GAS-aligned outputs of FVH-LT and DBSR-LS to summarize the overall latent structural separation without knowledge of the ground-truth generative factors. We compare bfVAE to five VAE models and validate the effectiveness FVH-LT, DBSR-LS, and LSSI in on seven tabular and image datasets. Under our examined experimental settings, bfVAE provides a more flexible disentanglement framework achieves more favorable overall trade-off between disentanglement and reconstruction than the benchmark VAE models; FVH-LT and DBSR-LS reliably uncover semantically meaningful and domain-relevant latent structures and generally yield consistent results; and LSSI makes an effective quantitative summary of latent structural separation.

Xiaoan Lang, Md Mostafizer Rahman, Fang Liu

Department of Applied and Computational Mathematics and Statistics University of Notre Dame, Notre Dame, IN, USA · Lucy Family Institute for Data & Society University of Notre Dame, Notre Dame, IN, USA

Variational Autoencoders Disentangled Representation Learning Representation Disentanglement Neural Network Interpretability

Ask PDF

Aug 31, 2026

XVAE-WMT: Explainable Wavelet-Temporal Variational Autoencoder for Blind Source Separation of Heart and Lung Sounds

The separation of cardiovascular sounds is a critical task in biomedical signal processing. In this paper, we introduce XVAE-WMT1, an unsupervised explainable generative AI algorithm combining a variational autoencoder (VAE) with explainable AI (XAI), wavelet-based inputs, a post-hoc output mask, and temporal consistency (TC) loss. Unlike existing supervised and VAE-based methods that rely on Short-Time Fourier Transform (STFT) and ignore latent interpretability, XVAE-WMT requires no paired clean recordings and integrates a Continuous Wavelet Transform (CWT) front-end for superior time-frequency localization. We assessed the latent space interpretability via different metrics, with SHAP (SHapley Additive exPlanations) enabling dimensionality reduction to the top 75% of latent features while preserving separation quality. Evaluated across two datasets using Signal-to-Distortion Ratio (SDR), Signal-to-Interference Ratio (SIR), and Signal-to-Artifacts Ratio (SAR), XVAE-WMT attains 26.8 dB SDR, 32.8 dB SIR, and 28.6 dB SAR.

Yasaman Torabi, Shahram Shirani, James P. Reilly

Department of Electrical and Computer Engineering, McMaster University, Hamilton, ON L8S 4K1, Canada. · L.R. Wilson/Bell Canada Chair in Data Communications, Hamilton, ON L8S 4L7, Canada.

Variational Autoencoders Heart Sound Classification Audio Source Separation

Ask PDF

Apr 14, 2026

On the Distillation Loss Functions of Speech VAE for Unified Reconstruction, Understanding, and Generation

Continuous speech representations based on Variational Autoencoders (VAEs) have emerged as a promising alternative to traditional spectrogram or discrete token based features for speech generation and reconstruction. Recent research has tried to enrich the structural information in VAE latent representations by aligning with self-supervised learning (SSL) features, aiming for better generation performance. However, it remains unclear whether the widely-used alignment approach based on time-axis distillation is optimal when considering more tasks. To address this problem, this paper systematically explores different alignment approaches and analyzes their impact on the performances over three axes: reconstruction, understanding, and generation. We investigate various design choices in the distillation loss. Extensive experiments show that the joint-marginal alignment approach with adaptive weighting can achieve the best overall performance while allowing for a controllable balance.

Changhao Cheng, Wei Wang, Wangyou Zhang +4

School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China · Auditory Cognition and Computational Acoustics Lab, MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · ByteDance Seed, China

Variational Autoencoders Audio Representation Learning Knowledge Distillation Self-Supervised Speech Representation Learning

Ask PDF

Abstract

Disentangled Representation Learning

Explore similar work

A Unified Latent Space Disentanglement VAE Framework with Robust Disentanglement Effectiveness Evaluation

XVAE-WMT: Explainable Wavelet-Temporal Variational Autoencoder for Blind Source Separation of Heart and Lung Sounds

On the Distillation Loss Functions of Speech VAE for Unified Reconstruction, Understanding, and Generation