Unsupervised Anomaly Detection in NSL-KDD Using $β$-VAE: A Latent Space and Reconstruction Error Approach

Dylan Baptiste; Ramla Saddem; Alexandre Philippot; François Foyer

cs.LGFeb 23, 2026

Unsupervised Anomaly Detection in NSL-KDD Using $β$ -VAE: A Latent Space and Reconstruction Error Approach

Authors: Dylan Baptiste, Ramla Saddem, Alexandre Philippot, François Foyer

Organizations: CRESTIC · Université de Reims Champagne-Ardenne, CRESTIC, Reims, France · Seckiot, Paris, France

Abstract

As Operational Technology increasingly integrates with Information Technology, the need for Intrusion Detection Systems becomes more important. This paper explores an unsupervised approach to anomaly detection in network traffic using $β$ -Variational Autoencoders on the NSL-KDD dataset. We investigate two methods: leveraging the latent space structure by measuring distances from test samples to the training data projections, and using the reconstruction error as a conventional anomaly detection metric. By comparing these approaches, we provide insights into their respective advantages and limitations in an unsupervised setting. Experimental results highlight the effectiveness of latent space exploitation for classification tasks.

Figures & tables

DoS	Probe	U2R	R2L
neptune, smurf, back, teardrop, pod, land, apache2, mailbomb, processtable, udpstorm, worm	ipsweep, nmap, portsweep, satan, mscan, saint	buffer_overflow, loadmodule, perl, rootkit, httptunnel, ps, sqlattack, xterm	ftp_write, guess_passwd, imap, multihop, phf, spy, warezclient, warezmaster, snmpgetattack, snmpguess, xlock, xsnoop

TABLE I: Categories of attacks in NSL-KDD

Fig. 1: Mean AUROC of

\mathcal{L}_{rec}

-classification and

\mathcal{Z}_{k}

-classification with variables

\beta

and

k

$\beta$	AUROC (%)
$\beta$	$\mathcal{Z}_{1}$	$\mathcal{Z}_{100}$	$\mathcal{Z}_{150}$	$\mathcal{Z}_{200}$	$\mathcal{Z}_{250}$	$\mathcal{Z}_{300}$	$\mathcal{Z}_{400}$	$\mathcal{Z}_{500}$	$\mathcal{Z}_{1000}$	$\mathcal{Z}_{2000}$	$\mathcal{Z}_{3000}$	$\mathcal{Z}_{4000}$	$\mathcal{Z}_{5000}$	$\mathcal{L}_{rec}$
0	94.11	96.43	96.63	96.76	96.89	96.99	97.09	97.14	97.40	97.48	97.56	97.66	97.70	96.78
0.00001	94.49	96.79	97.03	97.16	97.25	97.32	97.46	97.52	97.68	97.75	97.81	97.87	97.90	96.23
0.0001	94.28	96.60	96.76	96.96	97.12	97.19	97.26	97.29	97.52	97.65	97.70	97.73	97.73	96.52
0.001	93.51	95.81	96.20	96.47	96.66	96.71	96.66	96.58	96.69	96.80	96.82	96.85	96.86	96.61
0.01	93.47	96.16	96.37	96.48	96.57	96.64	96.71	96.76	96.96	96.85	96.86	96.85	96.82	96.44

TABLE II: Mean AUROC of

\mathcal{L}_{rec}

-classification and

\mathcal{Z}_{k}

-classification with variables

\beta

and

k

Fig. 2: ROC curves for the binary classification task with

\mathcal{L}_{rec}

-classification and

\mathcal{Z}_{k}

-classification

Fig. 3: ROC curves on

X_{test}

and

X_{attack}

with

\mathcal{L}_{rec}

-classification and

\mathcal{Z}_{5000}

-classification, per attack class

Fig. 4: Distribution of $\mathcal{Z}_{5000}$ -classification and $\mathcal{L}_{rec}$ -classification on $X_{test}$ and $X_{attack}$ . Blue points are classified as normal, purple as Probe , orange as DoS , green as U2R , and red as R2L . The distribution of each category is represented as a density on the opposing axes.

Explore similar work

Mitigating the reconstruction-detection trade-off in VAE-based unsupervised anomaly detection
Apr 9, 2026Agathe Senellart, Maëlys Solal, Stéphanie Allassonnière +1Reconstruction-Based Anomaly Detection Unsupervised Anomaly Detection
UPCité, INSERM, HeKA | U1346 · Université Paris Cité, Inria, Inserm, HeKA, F-75015 Paris, France · ARAMIS, ICM +1
Robust Unsupervised Network Intrusion Detection via Federated Learning with Selective Aggregation under Anomalous Sample Contamination
Jul 28, 2026Shohei Kamiguchi, Takayuki NishioNetwork Intrusion Detection Unsupervised Anomaly Detection
School of Engineering, Institute of Science Tokyo, Tokyo 152-8550, Japan
When Does Unsupervised Learning Succeed or Fail? A PoS Perspective on Reconstruction-Based Anomaly Detection
Sep 23, 2026Mehmet Yamaç, Yagmur Mustu, Muhammad Numan Yousaf +2Unsupervised Representation Learning Reconstruction-Based Anomaly Detection
Tampere University, Tampere, Finland · Radboud University, Nijmegen, The Netherlands

Apr 9, 2026J/K move · Enter open · S save

Mitigating the reconstruction-detection trade-off in VAE-based unsupervised anomaly detection

Agathe Senellart, Maëlys Solal, Stéphanie Allassonnière, Ninon Burgos

UPCité, INSERM, HeKA | U1346 · Université Paris Cité, Inria, Inserm, HeKA, F-75015 Paris, France · ARAMIS, ICM+1

Reconstruction-Based Anomaly Detection Unsupervised Anomaly Detection Variational Autoencoders

Open paper AskPDF arXiv

Variational autoencoders are widely used for unsupervised anomaly detection. Model selection however remains an open-question: to remain fully unsupervised, hyperparameters are often chosen to minimize the reconstruction error on normal samples. In this paper, we reveal a trade-off between reconstruction quality and anomaly detection among

β

-VAE models. Models with constrained latent space reach higher detection metrics but lower reconstruction quality. We also assess the performance variability across random seeds and show it is linked to the distance between normal and abnormal latent distributions. From this analysis, we justify and investigate two methods to mitigate the reconstructiondetection tradeoff: beta-scheduling and the Sparse VAE. The latter especially shows an improvement in detection while maintaining high reconstruction quality.

Abstract

Explore similar work

Mitigating the reconstruction-detection trade-off in VAE-based unsupervised anomaly detection

Robust Unsupervised Network Intrusion Detection via Federated Learning with Selective Aggregation under Anomalous Sample Contamination

When Does Unsupervised Learning Succeed or Fail? A PoS Perspective on Reconstruction-Based Anomaly Detection