VAE-MS: An Asymmetric Variational Autoencoder for Mutational Signature Extraction

Ida Egendal; Rasmus Froberg Brøndum; Dan J Woodcock; Christopher Yau; Martin Bøgsted

stat.APFeb 24, 2026

VAE-MS: An Asymmetric Variational Autoencoder for Mutational Signature Extraction

Authors: Ida Egendal, Rasmus Froberg Brøndum, Dan J Woodcock, Christopher Yau, Martin Bøgsted

Organizations: Center for Clinical Data Science, Aalborg University and Aalborg University Hospital, Aalborg, 9260, Denmark. · Clinical Cancer Research Center, Aalborg University Hospital, Aalborg, 9000, Denmark. · Nuffield Department of Surgical Sciences, University of Oxford, Oxford, OX3 9DU, England. · Nuffield Department of Women’s and Reproductive Health, University of Oxford, Oxford, OX3 9DU, England.

Abstract

Mutational signature analysis has emerged as a powerful method for uncovering the underlying biological processes driving cancer development. However, the signature extraction process, typically performed using non-negative matrix factorization (NMF), often lacks reliability and clinical applicability. To address these limitations, several solutions have been introduced, including the use of neural networks to achieve more accurate estimates and probabilistic methods to better capture natural variation in the data. In this work, we introduce a Variational Autoencoder for Mutational Signatures (VAE-MS), a novel model that leverages both an asymmetric architecture and probabilistic methods for the extraction of mutational signatures. VAE-MS is compared to with three state-of-the-art models for mutational signature extraction: SigProfilerExtractor, the NMF-based gold standard; MUSE-XAE, an autoencoder that employs an asymmetric design without probabilistic components; and SigneR, a Bayesian NMF model, to illustrate the strength in combining a nonlinear extraction with a probabilistic model. In the ability to reconstruct input data and generalize to unseen data, models with probabilistic components (VAE-MS, SigneR) dramatically outperformed models without (SigProfilerExtractor, MUSE-XAE). The NMF-baed models (SigneR, SigProfilerExtractor) had the most accurate reconstructions in simulated data, while VAE-MS reconstructed more accurately on real cancer data. Upon evaluating the ability to extract signatures consistently, no model exhibited a clear advantage over the others. Software for VAE-MS is available at https://github.com/CLINDA-AAU/VAE-MS.

Figures & tables

Figure 1: Schematic representation of VAE-MS. BN: batch normalization.

Figure 2: Beeswarm plots of the number of signatures suggested by each model for extraction in S8 (A), S14 (B), and PCAWG (C) over the 10 splits by all compared models. The true number of signatures in the simulated datasets is highlighted with a tan line.

		$\mathcal{D}_{\text{KL}}$		$\mathcal{D}_{\text{MSE}}$		Signature			Exposures in CI
Data	Model	Train	Test	Train	Test	ACS	PACS	SS	Train	Test
S8	VAE-MS	$13.1$	$5.9$	$407$	$421$	$0.96$	$\bm{0.99}$	$0.98$	$8.4\%$	$7.3\%$
	SigneR	$7.9$	$\bm{3.1}$	$2$	$\bm{5}$	$\bm{0.99}$	$\bm{0.99}$	$\bm{0.99}$	$\bm{36\%}$	$\bm{24\%}$
	SigProEx	$\bm{3.1}$	$3.2$	$\bm{0.12}$	$12$	$\bm{0.99}$	$\bm{0.99}$	$\bm{0.99}$	-	-
	MUSE-XAE	$4.2$	$4.2$	$458$	$465$	$\bm{0.99}$	$\bm{0.99}$	$\bm{0.99}$	-	-
S14	VAE-MS	$9.1$	$5.2$	$2231$	$2347$	$0.80$	$\bm{0.99}$	$0.48$	$\bm{24.4\%}$	$\bm{24.6\%}$

Table 1: The training and test performance of all models on the simulated data sets S8 and S14 with the number of signatures fixed at the true number of signatures.

\mathcal{D}_{\text{KL}}

: Kullback-Leibler divergence,

\mathcal{D}_{\text{MSE}}

: Mean-square error, ACS: Average Cosine Similarity to true signature set, PACS: Pairwise Average Cosine Similarity, SS: Silhouette Score, SigProEx: SigProfilerExtractor. Lowest error (

\mathcal{D}_{\text{KL}},\mathcal{D}_{\text{MSE}}

), highest similarity measures (ACS, PACS, and SS) and highest fraction of exposures contained in CI are highlighted with bold text .

Figure 3: 2D t-SNE embeddings of the estimated signatures across the 10 splits of the simulated data sets, with the number of signatures fixed to the true value. Embeddings of the true signatures are indicated with black ’x’ markers.

Figure 4: Density plots of the true and estimated exposures for the corresponding signature for the second split of the S8 data.

		$\mathcal{D}_{\text{KL}}$		$\mathcal{D}_{\text{MSE}}$
	Model	Train	Test	Train	Test	PACS
S8	VAE-MS	$11.7$	$4.9$	$255$	$255$	$\bm{0.99}$
	SigneR	$7.9$	$3.1$	$2$	$5$	$\bm{0.99}$
	SigProfilerExtractor	$\bm{3.1}$	$3.2$	$\bm{0.12}$	$12$	$\bm{0.99}$
	MUSE-XAE	$\bm{3.1}$	$\bm{3.1}$	$3.6$	$\bm{1.6}$	$\bm{0.99}$
S14	VAE-MS	$4.7$	$4.5$	$1817$	$1861$	$0.96$

Table 2: Training and test performance of all models on the simulated and PCAWG data with the number of signatures selected by the models.

\mathcal{D}_{\text{KL}}

: Kullback-Leibler divergence,

\mathcal{D}_{\text{MSE}}

: Mean-squared error, PACS: Pairwise Average Cosine Similarity. Lowest error (

\mathcal{D}_{\text{KL}},\mathcal{D}_{\text{MSE}}

), highest PACS are highlighted with bold text .

Cancer Genomics

Cancer genomics uses molecular data to understand tumor biology, identify biomarkers, and guide diagnosis, prognosis, and treatment.

See all 34 papers

Evolutionary Curriculum Learning Improves Biological Sequence Modeling
Aug 1, 2026Richard Zhu, Kento NishiVariational Autoencoders Curriculum Learning
Department of Computer Science, Harvard University, Cambridge, MA, USA · Department of Statistics, Harvard University, Cambridge, MA, USA.
Multimodal Empirical Bayes Variational Autoencoders for Joint Longitudinal and Time-to-Event Modeling
Jul 15, 2026Anders Sjöberg, Nils Olsson, Marcus Baaz +1Empirical Bayes Survival Analysis
Fraunhofer-Chalmers Centre, Gothenburg, SE-412 88, Sweden · Department of Electrical Engineering, Chalmers University of Technology, Gothenburg, SE-412 96, Sweden
SYNPRED: A Synergistic Approach to Multimodal Learning for Clinical Prediction
Oct 13, 2026Ivana Janíčková, Yen Y. Tan, Thomas H. Helbich +6Clinical Prediction Multimodal Disentangled Representation Learning
Computational Imaging Research Lab, Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Austria · Comprehensive Center for Artificial Intelligence in Medicine, Medical University of Vienna, Austria · Christian Doppler Laboratory for Machine Learning Driven Precision Imaging, Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Austria +3

Aug 1, 2026J/K move · Enter open · S save

Evolutionary Curriculum Learning Improves Biological Sequence Modeling

Richard Zhu, Kento Nishi

Department of Computer Science, Harvard University, Cambridge, MA, USA · Department of Statistics, Harvard University, Cambridge, MA, USA.

Variational Autoencoders Curriculum Learning Molecular Generation Sequence Modeling Bioinformatics

Open paper AskPDF arXiv

Variational autoencoders (VAEs) trained on multiple sequence alignments (MSAs) have emerged as powerful generative models for biological sequences, with applications ranging from disease variant prediction to functional RNA design. However, standard biological VAE training treats all sequences as exchangeable, ignoring the rich evolutionary structure that organizes homologous sequences from evolutionarily close to highly divergent. We propose Evolutionary Curriculum Learning (ECL), a training strategy that exploits this structure by progressively exposing the model to sequences of increasing evolutionary distance from sampled anchors, following a power-law expansion schedule. Applied to two architecturally distinct VAE models and two biological domains--protein variant effect prediction with EVE and RNA family sequence generation with RfamGen--ECL improves downstream task performance across five random seeds per configuration. Mean ClinVar classification AUROC rises from 0.981 to 0.989 for p53; for PTEN, ECL attains 1.000 in every seed whereas the baseline is unstable (mean 0.905, falling as low as 0.54). For RNA, ECL raises mean covariance-model bit scores on all three families tested and exceeds its seed-matched baseline in 12 of 15 training runs, though with only three families the effect cannot be established as significant at the family level. Ablation experiments show that progressively expanding the sampled sequences by evolutionary distance outperforms fixed-size neighborhood sampling in addition to uniform random sampling. Evolutionary distance is therefore a useful inductive bias for ordering the training curriculum in biological sequence modeling.

Abstract

Cancer Genomics

Explore similar work

Evolutionary Curriculum Learning Improves Biological Sequence Modeling

Multimodal Empirical Bayes Variational Autoencoders for Joint Longitudinal and Time-to-Event Modeling

SYNPRED: A Synergistic Approach to Multimodal Learning for Clinical Prediction