A Three-Axis Stress Test of LLM vs Classical ML for Network Intrusion Detection under Distribution Shift and Adversarial Evasion

Muhammad Ebad Atif; Muhammad Haider Ali

cs.LGSep 11, 2026

A Three-Axis Stress Test of LLM vs Classical ML for Network Intrusion Detection under Distribution Shift and Adversarial Evasion

Authors: Muhammad Ebad Atif, Muhammad Haider Ali

Organizations: Dhanani School of Science and Engineering Habib University Karachi, Pakistan · Faculty of Computer Sciences and Engineering Ghulam Ishaq Khan Institute of Engineering Sciences Swabi, Pakistan

Abstract

Large language models are increasingly benchmarked against classical machine learning for network intrusion detection (NIDS), almost always using same-dataset evaluation, and that protocol turns out to be incomplete. Evaluating XGBoost and RoBERTa-LoRA on two independently collected NetFlow v2 networks across three axes (same-dataset performance, cross-dataset transfer, and adversarial evasion) reveals no universal winner. The two models are statistically tied same-dataset. XGBoost wins decisively under cross-dataset distribution shift, by 15 points of F1 and 25 points of balanced accuracy; on the target network RoBERTa-LoRA's false positive rate reaches 0.78, leaving it barely above chance despite a superficially moderate F1. RoBERTa-LoRA wins decisively under adversarial evasion, by roughly 17 points of F1 at a representative mid-range perturbation strength, while both models hold false positive rates below 0.01 throughout. The model an evaluator would recommend therefore depends entirely on which axis is tested, not on same-dataset accuracy alone. A staged feature-leakage ablation improves cross-dataset transfer non-monotonically, indicating the leakage signal is distributed across the feature representation rather than confined to a few columns, and cross-dataset transfer between our two networks is strongly directional. These results argue for evaluating NIDS models along multiple independent robustness axes, and with more than one metric per axis.

Figures & tables

Dataset	XGBoost	RoBERTa-LoRA	Gap
UNSW-NB15	0.9966	0.9938	+0.0028
CIC-IDS2018	0.9776	0.9745	+0.0031

TABLE I: Same-dataset F1 (XGBoost: mean over 4 seeds, RoBERTa: single seed)

Fig. 1: Near-identical same-dataset F1 at vastly different inference throughput (log scale). XGBoost processes roughly four orders of magnitude more flows per second than RoBERTa-LoRA.

Feature set	#Feat	SD F1	CD F1	CD FNR
A: drop TTL only	36	0.9972	0.0488	0.9656
B: drop TTL + pkt. len.	32	0.9972	0.0282	0.9851
C: aggressive drop	29	0.9966	0.8116	0.1293

TABLE II: Feature ablation: same-dataset vs. cross-dataset F1 (XGBoost, UNSW

\to

CIC)

Fig. 2: Same-dataset F1 (grey) is flat across all three ablation sets while cross-dataset F1 (blue) swings non-monotonically, recovering only at Set C. Same-dataset accuracy cannot reveal which features leak.

Direction	XGBoost	Random Forest	RoBERTa-LoRA
UNSW $\to$ CIC	0.8116	0.5662–0.7587	0.6582
CIC $\to$ UNSW	0.0730	0.0201–0.0378	0.0599

TABLE III: Cross-dataset F1 under Set C (29 features)

Fig. 3: Cross-dataset transfer across both directions, for both models. On-diagonal (same-dataset) cells are near-perfect; the off-diagonal CIC $\to$ UNSW cell collapses for both models, while UNSW $\to$ CIC remains workable for XGBoost. The asymmetry is architecture-independent.

Metric	XGBoost	RoBERTa-LoRA	Random Forest
F1	0.8116	0.6582	0.5662–0.7587
Precision	0.7601	0.5290	0.5546–0.6932
Recall	0.8707	0.8709	0.5783–0.8380
FPR	0.2748	0.7756	0.3710–0.4852
Balanced accuracy	0.7979	0.5477	0.5569–0.7335

TABLE IV: Cross-dataset UNSW

\to

CIC, decomposed. F1 conceals a near-chance classifier.

Model	F1 range	Std
XGBoost	0.8116 – 0.8116	0.0000
Random Forest	0.5662 – 0.7587	0.0830

TABLE V: Seed variance, cross-dataset UNSW

\to

CIC (seeds 42–45)

	XGBoost		RoBERTa-LoRA
$\varepsilon$	F1	FPR	F1	FPR
0.00	0.9966	0.0057	0.9938	0.0081
0.05	0.9718	0.0057	0.9703	0.0081
0.10	0.9702	0.0057	0.9797	0.0081
0.15	0.6917	0.0057	0.9552	0.0081
0.20	0.6191	0.0057	0.9675	0.0081

TABLE VI: Adversarial evasion by

\varepsilon

: F1 and FPR on a balanced eval set (15k perturbed attacks + 15k untouched benign)

Fig. 4: F1 vs. perturbation strength $\varepsilon$ on the balanced evaluation set (15,000 perturbed attack flows plus 15,000 untouched benign flows). The degenerate $\varepsilon{=}1.0$ endpoint is excluded.

Condition	XGBoost	RoBERTa-LoRA	Gap
Same-dataset (UNSW)	0.9966	0.9938	+0.0028
Same-dataset (CIC)	0.9776	0.9745	+0.0031
Cross-dataset (UNSW $\to$ CIC)	0.8116	0.6582	+0.1534
Cross-dataset (CIC $\to$ UNSW)	0.0730	0.0599	+0.0131
Adversarial ( $\varepsilon=0.2$ )	0.6191	0.9675	$-$ 0.3484
Adversarial ( $\varepsilon=0.5$ )	0.7720	0.9448	$-$ 0.1728

TABLE VII: Master results: F1 across all evaluation axes

Fig. 5: The same two models produce three different verdicts. Same-dataset (both networks): statistically tied. Cross-dataset (UNSW $\to$ CIC): XGBoost wins. Adversarial evasion ( $\varepsilon=0.5$ ): RoBERTa-LoRA wins. No single axis predicts the others.

Network Intrusion Detection

Network intrusion detection identifies malicious or abnormal network activity to help protect computers, connected devices, and critical infrastructure.

See all 168 papers

Categorical Robustness Assessment for Machine Learning based Network Intrusion Detection Systems
Jun 10, 2026Mayank Raj, Nathaniel D. Bastian, Lance Fiondella +1IoT Intrusion Detection Adversarial Attacks
University of Massachusetts Dartmouth, Dartmouth, MA 02747 USA · United States Military Academy, West Point, NY 10996 USA
Jev-IDS: System One Models for Network Intrusion Detection
Oct 1, 2026Paulo Severo, Silvio E. Quincozes, Amanda DiasNetwork Intrusion Detection LLMs for Cybersecurity
Graduate Program in Software Engineering, Federal University of Pampa (UNIPAMPA), Alegrete, Rio Grande do Sul, Brazil
A No-Defense Defense Against Gradient-Based Adversarial Attacks on ML-NIDS: Is Less More?
May 18, 2026Mohamed elShehaby, Ashraf MatrawyAdversarial Attacks Network Intrusion Detection
Carleton University, Ottawa, Canada

Jun 10, 2026J/K move · Enter open · S save

Categorical Robustness Assessment for Machine Learning based Network Intrusion Detection Systems

Mayank Raj, Nathaniel D. Bastian, Lance Fiondella, Gokhan Kul

University of Massachusetts Dartmouth, Dartmouth, MA 02747 USA · United States Military Academy, West Point, NY 10996 USA

IoT Intrusion Detection Adversarial Attacks Network Intrusion Detection Adversarial Robustness Convolutional Neural Networks

Open paper AskPDF arXiv

Network Intrusion Detection Systems (NIDS) heavily utlize Machine Learning (ML) but ML models can be manipulated via adversarial attacks. These attacks add carefully crafted perturbations to network traffic data that leads to misclassifications. While prior work has demonstrated adversarial vulnerabilities in isolated settings, systematic cross-architecture as well as class and category of attack based comparisons under controlled attack conditions remain limited, leaving practitioners without clear guidance on which models to deploy in adversarial environments. This paper asks a simple question: what type of classifier architectures actually hold up when attackers try to manipulate the systems? We put three popular architectures through their paces: a 1D Convolutional Neural Network, a Long Short-Term Memory (LSTM) network, and a Random Forest (RF) ensemble. Using the ACI-IoT-2023 dataset (over 1.2 million samples spanning 12 attack types), we subject each model with FGSM and PGD adversarial attacks, which apply gradient-based perturbations in normalized feature space consistent with established adversarial ML evaluation protocols, at perturbation budgets ranging from

ε=0.01

ε=0.1

. Surprisingly, Random Forest achieved near-perfect baseline accuracy (99.98%), yet collapsed catastrophically under attack, dropping 73 percentage points at the smallest perturbation we tested. CNN, on the other hand, retained 95.5% accuracy at

ε=0.01

and degraded gracefully as perturbations increased. LSTM fell somewhere in between. These findings flip the conventional wisdom where high baseline accuracy means nothing if a model shatters at the first sign of adversarial pressure. For practitioners deploying intrusion detection in adversarial environments, we recommend CNN-based architectures and provide scenario-specific deployment guidance.

Abstract

Network Intrusion Detection

Explore similar work

Categorical Robustness Assessment for Machine Learning based Network Intrusion Detection Systems

Jev-IDS: System One Models for Network Intrusion Detection

A No-Defense Defense Against Gradient-Based Adversarial Attacks on ML-NIDS: Is Less More?