Adversarial Robustness Is Not Zero-Day Generalization: A Controlled Evaluation of Flow-Based Intrusion Detectors on Corrected CIC-IDS2017

Abdulhamid Zaro

cs.CROct 31, 2026

Adversarial Robustness Is Not Zero-Day Generalization: A Controlled Evaluation of Flow-Based Intrusion Detectors on Corrected CIC-IDS2017

Authors: Abdulhamid Zaro

Organizations: College of Professions and Applied Sciences, Hebron University, Hebron, Palestine

Abstract

Synthetic variants of known attacks, including feature-space adversarial perturbations, are sometimes used as stand-ins for unseen ("zero-day") attacks when evaluating machine-learning network intrusion detection systems (NIDS). We test whether this proxy holds. On the corrected CIC-IDS2017 release we train Random Forest, XGBoost, a 1D-CNN and an FT-Transformer and compare, at a fixed 1% false-positive rate over five seeds, detection under constrained perturbations of known attacks (gradient-based and black-box attacks on 42 attacker-controllable features) with leave-one-attack-family-out (LOAO) detection, where an entire family is absent from training. Detection under the strongest perturbation we tested, with budgets of at most 0.2, stays between 90.9% and 99.7%, yet mean detection of held-out families falls to 76.2% (RF), 40.2% (XGBoost), 25.8% (1D-CNN) and 22.8% (FT-Transformer). Robustness exceeds LOAO detection for every model (gaps of 24 to 72 percentage points, 23 to 66 when robustness is also averaged over families; Holm-adjusted p $\le$ 0.00096), and the two measures agree at most on a coarse ranking of the models. An exact mixed-integer evasion search confirms the tree ensembles' scores. Benign-only anomaly detectors and a simple multi-class open-set baseline do not close the gap. A temporal split on 5 seeds (3 of the 4 models) leaves the gap in place. On CSE-CIC-IDS2018 the gap recurs for every model (16 to 68 points). On these benchmarks and under this threat model, a robustness score should not be read as a zero-day score, and the two should be reported separately. Code and per-seed results are publicly available.

Figures & tables

Family	Unique flows	Train	Test
Benign	1,432,904	80,000 (+20,000 val.)	40,000
DoS	171,634	15,000	5,000
DDoS	95,144	15,000	5,000
PortScan	7,695	5,386	2,309
BruteForce	6,933	4,853	2,080
Botnet	736	515	221

TABLE I: Flows per family after cleaning, and per-seed train/test sizes used. Infiltration and Heartbleed are excluded from all experiments.

Model	Detection	FPR	F1	AUROC
RF	1.0000 $\pm$ 0.0000	0.0098 $\pm$ 0.0013	0.987 $\pm$ 0.002	1.0000
XGBoost	0.9999 $\pm$ 0.0001	0.0103 $\pm$ 0.0013	0.986 $\pm$ 0.002	1.0000
1D-CNN	0.9995 $\pm$ 0.0003	0.0104 $\pm$ 0.0011	0.986 $\pm$ 0.001	0.9999
FT-Transformer	0.9991 $\pm$ 0.0005	0.0097 $\pm$ 0.0007	0.986 $\pm$ 0.001	0.9999

TABLE II: Known attacks under the random flow-level split: detection rate, measured false-positive rate on benign test flows, F1 and AUROC at the threshold set for 1% FPR on benign validation flows (mean

\pm

95% CI over five seeds). These are ceilings, not field estimates (Section V-A).

Model	Split	Known DR	Test FPR	Robustness	LOAO	Gap
RF	random	1.000	0.010	0.998	0.762	23.6
	temporal	0.995	0.016	0.985	0.696	28.9
XGBoost	random	1.000	0.010	0.998	0.402	59.5
	temporal	0.998	0.014	0.982	0.372	61.0
1D-CNN	random	0.999	0.010	0.909	0.258	65.0
	temporal	0.962	0.016	0.803	0.182	62.1

TABLE III: Temporal split (earliest 70% of each family for training, latest 30% for testing; seed 0, 1, 2, 3, 4; FT-Transformer not run) against the random split on the same seed: known-attack detection and test FPR, worst-case robustness (

\epsilon\leq 0.2

; FGSM, PGD, SimBA), mean LOAO detection and the gap (points).

Procedure	$\epsilon$	$n$	RF	XGBoost	1D-CNN	FT-Transformer
Random sign	0.05	14,642	1.000 $\pm$ 0.000	0.999 $\pm$ 0.000	0.999 $\pm$ 0.000	0.999 $\pm$ 0.000
Random sign	0.1	14,642	1.000 $\pm$ 0.000	0.999 $\pm$ 0.000	0.999 $\pm$ 0.000	0.999 $\pm$ 0.001
Random sign	0.2	14,642	1.000 $\pm$ 0.000	0.999 $\pm$ 0.000	0.999 $\pm$ 0.001	0.997 $\pm$ 0.003
Random sign	0.5	14,642	1.000 $\pm$ 0.000	0.999 $\pm$ 0.000	0.967 $\pm$ 0.020	0.988 $\pm$ 0.013
FGSM (transfer)	0.05	14,642	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	0.999 $\pm$ 0.000	0.999 $\pm$ 0.001
FGSM (transfer)	0.1	14,642	1.000 $\pm$ 0.000	0.999 $\pm$ 0.000	0.999 $\pm$ 0.001	0.998 $\pm$ 0.001

TABLE IV: Detection rate at 1% FPR under every perturbation procedure (mean

\pm

95% CI over five seeds).

n

: attack flows per seed (all six families). Gradient attacks do not apply to tree ensembles (–); the tree-aware search applies only to them. Mixup: cross-family hybrids, no

\epsilon

Held-out family	$n$	RF	XGBoost	1D-CNN	FT-Transformer
DoS	5,000	0.988 $\pm$ 0.007	0.979 $\pm$ 0.010	0.043 $\pm$ 0.061	0.038 $\pm$ 0.023
DDoS	5,000	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000	1.000 $\pm$ 0.000
PortScan	2,309	0.481 $\pm$ 0.396	0.097 $\pm$ 0.009	0.058 $\pm$ 0.027	0.069 $\pm$ 0.015
BruteForce	2,080	0.573 $\pm$ 0.022	0.042 $\pm$ 0.104	0.000 $\pm$ 0.000	0.023 $\pm$ 0.060
Botnet	221	0.940 $\pm$ 0.030	0.206 $\pm$ 0.402	0.285 $\pm$ 0.482	0.176 $\pm$ 0.490
WebAttack	32	0.588 $\pm$ 0.428	0.087 $\pm$ 0.043	0.163 $\pm$ 0.154	0.062 $\pm$ 0.027

TABLE V: Detection rate at 1% FPR on held-out (unseen) attack families, mean

\pm

95% CI over seeds;

n

is the number of test flows of the held-out family per seed. The Botnet and WebAttack rows rest on few flows and vary strongly between seeds; they are reported for completeness only.

		Isolation Forest		Autoencoder
Family	$n$	DR	AUROC	DR	AUROC
DoS	5,000	0.003	0.884	0.047	0.855
DDoS	5,000	0.000	0.883	0.006	0.959
PortScan	2,309	0.610	0.973	0.179	0.974
BruteForce	2,080	0.000	0.867	0.000	0.886
Botnet	221	0.017	0.747	0.045	0.780

TABLE VI: Benign-only anomaly baselines on attack families they never saw: detection rate at 1% FPR and AUROC, mean over five seeds.

	RF		XGBoost
Held-out	Binary	MSP	Binary	MSP
DoS	0.988	0.983	0.979	0.958
DDoS	1.000	1.000	1.000	1.000
PortScan	0.481	0.271	0.097	0.131
BruteForce	0.573	0.573	0.042	0.006
Botnet	0.940	0.910	0.206	0.899

TABLE VII: Open-set baseline: multi-class RF and XGBoost with maximum-softmax-probability rejection (MSP), detection of the held-out family at 1% FPR (mean over 5 seeds), next to the binary detectors of Table V . The score

1-P(\text{benign})

gives identical detection rates at this threshold.

		All six families		Four large families		Flow-weighted
Model	Robustness	LOAO	Gap	LOAO	Gap	LOAO	Gap
RF	0.997	0.762	23.5	0.760	23.6	0.851	14.5
XGBoost	0.997	0.402	59.5	0.530	46.7	0.700	29.6
1D-CNN	0.909	0.258	65.0	0.275	63.3	0.370	53.8
FT-Transformer	0.946	0.228	71.8	0.282	66.3	0.371	57.5

TABLE VIII: Worst-case robustness (

\epsilon\leq 0.2

, all procedures) against LOAO detection averaged three ways, and the resulting gap in percentage points (mean over five seeds; every model-seed gap is positive in all three cases).

Model	No repair	Repair	No repair	Repair	Gap (repair)
	Worst case, $\epsilon\leq 0.2$		SimBA, $\epsilon=0.5$
RF	0.998	0.998	0.998	0.997 ( $p$ =0.59)	0.236 $\pm$ 0.078
XGBoost	0.998	0.998	0.993	0.994 ( $p$ =0.19)	0.595 $\pm$ 0.067
1D-CNN	0.909	0.943	0.187	0.207 ( $p$ =0.23)	0.685 $\pm$ 0.119
FT-Transformer	0.946	0.932	0.667	0.656 ( $p$ =0.56)	0.704 $\pm$ 0.080

TABLE IX: Detection with and without enforcing the derived-feature identities (repair), mean

\pm

95% CI over seeds. Worst case: minimum over FGSM, PGD (transfer, white-box) and SimBA at

\epsilon\leq 0.2

. Gap: repaired worst case minus mean LOAO detection.

p

: paired t-tests.

Model	Known	Robust.	DoS	DDoS	PortScan	BruteForce	Botnet	WebAttack	LOAO	Gap
RF	1.000	0.998	0.999	1.000	0.445	1.000	1.000	0.557	0.833	16.5
XGBoost	1.000	0.991	0.993	0.943	0.179	1.000	0.999	0.145	0.710	28.1
1D-CNN	0.999	0.963	0.060	0.406	0.112	0.073	0.670	0.388	0.285	67.9

TABLE X: CSE-CIC-IDS2018 (uniform per-family sample; 3 seeds): known-attack detection, worst-case robustness (

\epsilon\leq 0.2

; FGSM, PGD, SimBA), LOAO detection per held-out family and on average, and the gap in points.

Network Intrusion Detection

Network intrusion detection identifies malicious or abnormal network activity to help protect computers, connected devices, and critical infrastructure.

See all 174 papers

Categorical Robustness Assessment for Machine Learning based Network Intrusion Detection Systems
Jun 10, 2026Mayank Raj, Nathaniel D. Bastian, Lance Fiondella +1IoT Intrusion Detection Adversarial Attacks
University of Massachusetts Dartmouth, Dartmouth, MA 02747 USA · United States Military Academy, West Point, NY 10996 USA
ThreatFormer-IDS: Robust Transformer Intrusion Detection with Zero-Day Generalization and Explainable Attribution
Feb 26, 2026Srikumar NayakExplainable Intrusion Detection IoT Intrusion Detection
Incedo Inc, USA
A Three-Axis Stress Test of LLM vs Classical ML for Network Intrusion Detection under Distribution Shift and Adversarial Evasion
Sep 11, 2026Muhammad Ebad Atif, Muhammad Haider AliNetwork Intrusion Detection LLMs for Cybersecurity
Dhanani School of Science and Engineering Habib University Karachi, Pakistan · Faculty of Computer Sciences and Engineering Ghulam Ishaq Khan Institute of Engineering Sciences Swabi, Pakistan

Jun 10, 2026J/K move · Enter open · S save

Categorical Robustness Assessment for Machine Learning based Network Intrusion Detection Systems

Mayank Raj, Nathaniel D. Bastian, Lance Fiondella, Gokhan Kul

University of Massachusetts Dartmouth, Dartmouth, MA 02747 USA · United States Military Academy, West Point, NY 10996 USA

IoT Intrusion Detection Adversarial Attacks Network Intrusion Detection Adversarial Robustness Convolutional Neural Networks

Open paper AskPDF arXiv

Network Intrusion Detection Systems (NIDS) heavily utlize Machine Learning (ML) but ML models can be manipulated via adversarial attacks. These attacks add carefully crafted perturbations to network traffic data that leads to misclassifications. While prior work has demonstrated adversarial vulnerabilities in isolated settings, systematic cross-architecture as well as class and category of attack based comparisons under controlled attack conditions remain limited, leaving practitioners without clear guidance on which models to deploy in adversarial environments. This paper asks a simple question: what type of classifier architectures actually hold up when attackers try to manipulate the systems? We put three popular architectures through their paces: a 1D Convolutional Neural Network, a Long Short-Term Memory (LSTM) network, and a Random Forest (RF) ensemble. Using the ACI-IoT-2023 dataset (over 1.2 million samples spanning 12 attack types), we subject each model with FGSM and PGD adversarial attacks, which apply gradient-based perturbations in normalized feature space consistent with established adversarial ML evaluation protocols, at perturbation budgets ranging from

ε=0.01

ε=0.1

. Surprisingly, Random Forest achieved near-perfect baseline accuracy (99.98%), yet collapsed catastrophically under attack, dropping 73 percentage points at the smallest perturbation we tested. CNN, on the other hand, retained 95.5% accuracy at

ε=0.01

and degraded gracefully as perturbations increased. LSTM fell somewhere in between. These findings flip the conventional wisdom where high baseline accuracy means nothing if a model shatters at the first sign of adversarial pressure. For practitioners deploying intrusion detection in adversarial environments, we recommend CNN-based architectures and provide scenario-specific deployment guidance.

Abstract

Network Intrusion Detection

Explore similar work

Categorical Robustness Assessment for Machine Learning based Network Intrusion Detection Systems

ThreatFormer-IDS: Robust Transformer Intrusion Detection with Zero-Day Generalization and Explainable Attribution

A Three-Axis Stress Test of LLM vs Classical ML for Network Intrusion Detection under Distribution Shift and Adversarial Evasion