cs.LGMar 2, 2026
SaveCoVAE: correlated multimodal generative modeling
Organizations: Scuola Internazionale Superiore di Studi Avanzati, Trieste, Italy
Abstract
Multimodal Variational Autoencoders have emerged as a popular tool to extract effective representations from rich multimodal data. However, such models rely on fusion strategies in latent space that destroy the joint statistical structure of the multimodal data, with profound implications for generation and uncertainty quantification. In this work, we introduce Correlated Variational Autoencoders (CoVAE), a new generative architecture that captures the correlations between modalities. We test CoVAE on a number of real and synthetic data sets demonstrating both accurate cross-modal reconstruction and effective quantification of the associated uncertainties.
Figures & tables
Figure 1: When only one modality is available, common methods, such as Product-of-Experts Wu and Goodman [2018] (left) erroneusly assign the same uncertainty to both modalities. On the contrary, CoVAE (right) correctly assigns a wider posterior to the missing modality. In this case, the two modalities have a linear correlation coefficient
Figure 2: Graphical representation of CoVAE: in the case of two modalities, during training
Figure 3: Schematic process of the data generation, assuming classes coming from MNIST
Figure 4: Examples of synthetic datasets in which both modalities are from the MNIST dataset and latent dimensions . From top to bottom,
Figure 5: Linear correlations measured at the level of the input to the decoders, both in the case of joint and conditional reconstructions. The intervals around the measured values represent
Figure 6: Average standard deviation per dimension for the reported models in a conditional generation setup. On the left, the standard deviation of the distribution of the conditioning modality; on the right, the standard deviation of the distribution of the missing modality
Figure 7: Conditional reconstruction of input modalities for MVAE (first row), DMVAE (second) and CoVAE (third) at (top) and (bottom). For each model, the first three images represent the generated input modality, followed by samples of the missing modality
Figure 8: Negative log-likelihoods for different models at varying levels of correlation.
| Model | Precision | NMI | ARI | Joint LL |
|---|---|---|---|---|
| MVAE | 0.901 ± 0.011 (3) | 0.899 ± 0.011 (3) | 0.844 ± 0.018 (3) | 3839.297 ± 2.571 (3) |
| MVTCAE | 0.905 ± 0.010 (2) | 0.903 ± 0.011 (2) | 0.850 ± 0.017 (2) | 3837.561 ± 2.893 (2) |
| MMVAE | 0.865 ± 0.012 (7) | 0.857 ± 0.013 (7) | 0.780 ± 0.020 (7) | 3859.866 ± 4.449 (6) |
| MoPoE | 0.918 ± 0.009 (1) | 0.918 ± 0.009 (1) | 0.875 ± 0.015 (1) | 3865.613 ± 3.137 (7) |
| JMVAE | 0.892 ± 0.010 (5) | 0.891 ± 0.010 (5) | 0.831 ± 0.016 (5) | 3846.290 ± 2.001 (4) |
| DMVAE | 0.832 ± 0.019 (8) | 0.838 ± 0.018 (8) | 0.738 ± 0.029 (8) | — |
Table 1: Comparison of multimodal VAE models on the pan-cancer mRNA/miRNA dataset on joint tasks.
| Model | mRNA→miRNA MAE | miRNA→mRNA MAE | mRNA→miRNA Cls | miRNA→mRNA Cls | LL mRNA | LL miRNA |
|---|---|---|---|---|---|---|
| MVAE | 0.639 ± 0.033 (8) | 0.563 ± 0.047 (5) | 0.832 ± 0.029 (6) | 0.784 ± 0.068 (7) | 448.392 ± 1.849 (7) | 3466.259 ± 10.359 (7) |
| MVTCAE | 0.623 ± 0.011 (7) | 0.582 ± 0.019 (6) | 0.690 ± 0.026 (8) | 0.739 ± 0.027 (8) | 444.148 ± 0.550 (4) | 3462.294 ± 2.720 (6) |
| MMVAE | 0.465 ± 0.003 (2) | 0.618 ± 0.081 (7) | 0.786 ± 0.056 (7) | 0.815 ± 0.045 (6) | 442.770 ± 1.103 (3) | 3445.171 ± 2.906 (3) |
| MoPoE | 0.460 ± 0.002 (1) | 0.406 ± 0.002 (1) | 0.926 ± 0.008 (2) | 0.927 ± 0.007 (2) | 448.273 ± 0.583 (6) | 3456.986 ± 3.752 (5) |
| JMVAE | 0.479 ± 0.001 (5) | 0.441 ± 0.001 (3) | 0.935 ± 0.007 (1) | 0.938 ± 0.007 (1) | 433.423 ± 0.284 (2) | 3433.295 ± 2.204 (1) |
| DMVAE | 0.575 ± 0.014 (6) | 0.751 ± 0.017 (8) | 0.891 ± 0.011 (5) | 0.904 ± 0.010 (4) | — | — |
Table 2: Comparison of multimodal VAE models on the pan-cancer mRNA/miRNA dataset on conditional tasks.
Figure 9: Violin plots for the distributions of the Spearman coefficients of the joint (top) and conditional (bottom) reconstructions for mRNA (blue) and miRNA (orange)
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Top left: joint correlation. Top right: conditional correlation. Bottom left: joint NLL. Bottom right: conditional NLL
Figure 11: Top left: joint correlation. Top right: conditional correlation. Bottom left: joint NLL. Bottom right: conditional NLL
Figure 12: PCA and UMAP visualization of the dataset according to the two modalities
Figure 13: Labels distribution in the MLOmics pancancer dataset
| mRNA→miRNA | miRNA→mRNA | Joint mRNA | Joint miRNA | |
|---|---|---|---|---|
| Model | ||||
| MVAE | 0.432 +/- 0.055 | 0.578 +/- 0.061 | 0.788 +/- 0.002 | 0.742 +/- 0.002 |
| MVTCAE | 0.465 +/- 0.014 | 0.547 +/- 0.028 | 0.787 +/- 0.001 | 0.749 +/- 0.002 |
| MMVAE | 0.685 +/- 0.004 | 0.705 +/- 0.012 | 0.704 +/- 0.009 | 0.695 +/- 0.004 |
| MoPoE | 0.691 +/- 0.002 | 0.758 +/- 0.002 | 0.781 +/- 0.001 | 0.716 +/- 0.001 |
| JMVAE | 0.668 +/- 0.002 | 0.723 +/- 0.001 | 0.782 +/- 0.001 | 0.735 +/- 0.001 |
Table 3: Table of Spearman coefficients represented in Fig. 9
Figure 14: Decomposition of the terms in Eq. 10 for JMVAE (blue), MVAE (green), CoVAE with the Cholesky covariance parameterization(yellow) and cross-diagonal (red). For each model, we sample 5000 times from the posteriors of 500 points from the test samples. Dataset correlation is (left) and (right)
Figure 15: Comparison of CoVAE models’ posterior density with the one of JMVAE and MVAE with two different latent dimensions
Figure 16: Conditional reconstructions for all models tested at different correlation levels
Figure 17: Joint reconstructions for all models tested at different correlation levels
More in this topic
Multimodal Generation
Multimodal generation creates coordinated text, images, audio, video, 3D, and other outputs from mixed inputs while preserving meaning and consistency.
See all 243 papers