Mitigating the reconstruction-detection trade-off in VAE-based unsupervised anomaly detection

Agathe Senellart; Maëlys Solal; Stéphanie Allassonnière; Ninon Burgos

cs.LGApr 9, 2026

Mitigating the reconstruction-detection trade-off in VAE-based unsupervised anomaly detection

Authors: Agathe Senellart, Maëlys Solal, Stéphanie Allassonnière, Ninon Burgos

Organizations: UPCité, INSERM, HeKA | U1346 · Université Paris Cité, Inria, Inserm, HeKA, F-75015 Paris, France · ARAMIS, ICM · Sorbonne Université, Institut du Cerveau - Paris Brain Institute - ICM, CNRS, Inria, Inserm, AP-HP, Hôpital de la Pitié Salpêtrière, F-75013, Paris, France

Abstract

Variational autoencoders are widely used for unsupervised anomaly detection. Model selection however remains an open-question: to remain fully unsupervised, hyperparameters are often chosen to minimize the reconstruction error on normal samples. In this paper, we reveal a trade-off between reconstruction quality and anomaly detection among $β$ -VAE models. Models with constrained latent space reach higher detection metrics but lower reconstruction quality. We also assess the performance variability across random seeds and show it is linked to the distance between normal and abnormal latent distributions. From this analysis, we justify and investigate two methods to mitigate the reconstructiondetection tradeoff: beta-scheduling and the Sparse VAE. The latter especially shows an improvement in detection while maintaining high reconstruction quality.

Figures & tables

Figure 1: Trade-off between reconstruction quality (hhMSE) and anomaly detection performance (rMSE and AP) for

\beta

-VAEs with varying

\beta

and latent dimension

d

. We report mean and standard deviation on five seeds for each model.

Figure 2: Scatterplot of the reconstruction healthiness (rMSE) on test-AD-30 against the normalized 2-Wasserstein distance. Each point represents one of 30 seeds.

	test-CN		test-AD-30				test-AD-50
	hhMSE $10^{-3}$ $\downarrow$	SSIM $\uparrow$	ahMSE $10^{-3}$ $\downarrow$	rMSE $\downarrow$	AP $\uparrow$	$\lceil\text{Dice}\rceil$ $\uparrow$	ahMSE $10^{-3}$ $\downarrow$	rMSE $\downarrow$	AP $\uparrow$	$\lceil\text{Dice}\rceil$ $\uparrow$
$\beta=1$	$\mathbf{1.14\pm 0.01}$	$\mathbf{94.29\pm 0.06}$	$1.66\pm 0.09$	$1.48\pm 0.07$	$48\pm 3$	$49\pm 2$	$2.00\pm 0.15$	$1.78\pm 0.13$	$79\pm 2$	$72\pm 2$
$\beta=10$	$1.18\pm 0.02$	$94.20\pm 0.05$	$1.67\pm 0.08$	$1.42\pm 0.09$	$51\pm 5$	$51\pm 3$	$1.97\pm 0.19$	$1.68\pm 0.17$	$81\pm 3$	$73\pm 3$
$\beta=30$	$1.24\pm 0.03$	$94.11\pm 0.12$	$1.70\pm 0.12$	$1.38\pm 0.07$	$53\pm 2$	$52\pm 2$	$2.04\pm 0.31$	$1.67\pm 0.22$	$80\pm 4$	$73\pm 4$
$\beta=50$	$1.26\pm 0.03$	$94.07\pm 0.07$	$1.72\pm 0.13$	$1.37\pm 0.07$	$56\pm 3$	$54\pm 2$	$2.01\pm 0.25$	$1.60\pm 0.16$	$82\pm 3$	$74\pm 2$
$\beta=30$ (cycle)	$1.20\pm 0.02$	$94.15\pm 0.18$	$1.61\pm 0.07$	$1.36\pm 0.06$	$56\pm 3$	$54\pm 2$	$1.89\pm 0.16$	$1.60\pm 0.15$	$83\pm 2$	$75\pm 2$

Table 1: Reconstruction and detection results for healthy dataset (test-CN) and simulated AD datasets (test-AD-30 and test-AD-50). For each metric, we report the mean and standard deviation over 5 splits and seeds. Best metrics in bold. Erratum: the standard deviations in the ISBI 2026 proceedings were computed over the subjects and not splits.

Explore similar work

Unsupervised Anomaly Detection in NSL-KDD Using $β$ -VAE: A Latent Space and Reconstruction Error Approach
Feb 23, 2026Dylan Baptiste, Ramla Saddem, Alexandre Philippot +1Unsupervised Anomaly Detection Network Intrusion Detection
CRESTIC · Université de Reims Champagne-Ardenne, CRESTIC, Reims, France · Seckiot, Paris, France
MLE-UVAD: Minimal Latent Entropy Autoencoder for Fully Unsupervised Video Anomaly Detection
Mar 25, 2026Yuang Geng, Junkai Zhou, Kang Yang +5Video Anomaly Detection Reconstruction-Based Anomaly Detection
Disjoint or Overlapping? Inference Windowing for Reconstruction-Based Time Series Anomaly Detection
Jun 2, 2026Guillaume Coulaud, Reza Akbarinia, Florent MassegliaReconstruction-Based Anomaly Detection Time-Series Anomaly Detection
UM, IROKO · University of Montpellier, Inria, CNRS, LIRMM, Montpellier, France · IROKO +1

Feb 23, 2026J/K move · Enter open · S save

Unsupervised Anomaly Detection in NSL-KDD Using $β$ -VAE: A Latent Space and Reconstruction Error Approach

Dylan Baptiste, Ramla Saddem, Alexandre Philippot, François Foyer

CRESTIC · Université de Reims Champagne-Ardenne, CRESTIC, Reims, France · Seckiot, Paris, France

Unsupervised Anomaly Detection Network Intrusion Detection Reconstruction-Based Anomaly Detection Variational Autoencoders

Open paper AskPDF arXiv

As Operational Technology increasingly integrates with Information Technology, the need for Intrusion Detection Systems becomes more important. This paper explores an unsupervised approach to anomaly detection in network traffic using

β

-Variational Autoencoders on the NSL-KDD dataset. We investigate two methods: leveraging the latent space structure by measuring distances from test samples to the training data projections, and using the reconstruction error as a conventional anomaly detection metric. By comparing these approaches, we provide insights into their respective advantages and limitations in an unsupervised setting. Experimental results highlight the effectiveness of latent space exploitation for classification tasks.

Abstract

Explore similar work

Unsupervised Anomaly Detection in NSL-KDD Using βββ-VAE: A Latent Space and Reconstruction Error Approach

MLE-UVAD: Minimal Latent Entropy Autoencoder for Fully Unsupervised Video Anomaly Detection

Disjoint or Overlapping? Inference Windowing for Reconstruction-Based Time Series Anomaly Detection

Unsupervised Anomaly Detection in NSL-KDD Using $β$ -VAE: A Latent Space and Reconstruction Error Approach