Unmasking Shortcut Learning in IoT Intrusion Detection: A Forensic, Multi-Paradigm Evaluation of Feature Dependence and Data Leakage

Uday Shankar Roy; Mahbuba Jahan Minu

cs.CRSep 23, 2026

Unmasking Shortcut Learning in IoT Intrusion Detection: A Forensic, Multi-Paradigm Evaluation of Feature Dependence and Data Leakage

Authors: Uday Shankar Roy, Mahbuba Jahan Minu

Organizations: Computer Science and Engineering Khulna University of Engineering & Technology Khulna, Bangladesh

Abstract

Machine learning-based Network Intrusion Detection Systems often report near-perfect performance on IoT benchmarks. However, whether these models learn generalizable attack behavior or exploit spurious dataset shortcuts- such as static testbed IP/MAC addresses and chronological recording artifacts-remains an important question. We evaluate the CyberFlowIoT-GICAP benchmark, containing 3,617,388 flow records across 126 PCAP sessions with 849,395 benign flows. Four learning paradigms are evaluated across four feature configurations using PCAP-disjoint splits; LightGBM is additionally evaluated using conventional random-flow splitting. When only statistical flow behavior is used (Fbehav), LightGBM (92.58% +/- 8.18%), Random Forest (92.59% +/- 8.18%), and Deep MLP (92.55% +/- 8.18%) achieve nearly identical Macro-F1, indicating that performance is constrained by feature representation rather than model complexity. With raw timestamps (Ftstamp), tree-based models reach 99.28% Macro-F1, while the linear model remains at 90.62%, showing that nonlinear models can exploit dataset-specific temporal structure. Attack detectability is highly asymmetric: high-rate and active attacks maintain >99.8% recall from flow behavior alone in nonlinear models, whereas the DNS Beaconing drops from 27.78% to 0.00% recall when contextual features are removed. Conventional random-flow splitting increases attack recall by up to 14.00%, highlighting the effect of placing flows from the same sessions in both training and test sets. We conclude with a 4-point protocol checklist for realistic IoT NIDS evaluation.

Figures & tables

Category	PCAPs ∗	Total Flows	Attacks	Atk %
BENIGN (Dedicated)	—	849,395	0	0.00%
Denial of Service (DoS)	43 (35/8)	960,657	593,576	61.79%
ARP Spoofing (MITM)	8 (8/0)	70,915	61,042	86.08%
Nmap Reconnaissance	30 (30/0)	130,229	51,155	39.28%
SQL Injection (SQLi)	6 (6/0)	105,107	33,814	32.17%
DNS Beaconing	22 (13/9)	1,072,347	1,611	0.15%

TABLE I: Complete Forensic Inventory across 126 PCAP Captures

Feature Set	Count	Included Feature Groups
$F_{\text{all}}$	75	55 Behavioral, 12 App/Proto, 2 Ports, 6 Host IDs
$F_{-\text{ident}}$	69	55 Behavioral, 12 App/Proto, 2 Ports
$F_{\text{behav}}$	55	55 Behavioral Features (Volumes, PIATs, TCP Flags, etc.)
$F_{\text{tstamp}}$	61	55 Behavioral, 6 Absolute Epoch Timestamps

TABLE II: Landmark Feature Taxonomy

Model	Paradigm	Core Hyperparameters
LightGBM	GBDT	$N=100$ , $\text{lr}=0.05$ , $\text{num\_leaves}=31$ , $\text{jobs}=10$
Random Forest	Bagging	$N=100$ , $\text{max\_samples}=0.25$ , $\text{depth}=20$
Deep MLP	Deep NN	$128\text{-}64\text{-}32\text{-}2$ , BatchNorm, Dropout 0.2, AdamW
Linear SGD	Linear	log_loss , L2, $\alpha=10^{-4}$ , $\text{max\_iter}=1000$

TABLE III: Evaluated Model Architectures & Hyperparameters

Model	$\mathbf{F_{\text{all}}}$	$\mathbf{F_{-\text{ident}}}$	$\mathbf{F_{\text{behav}}}$	$\mathbf{F_{\text{tstamp}}}$
LightGBM	$88.36{\pm}6.42$	$88.07{\pm}6.50$	$92.58{\pm}8.18$	$99.28{\pm}0.42$
RF	$89.11{\pm}6.49$	$89.09{\pm}6.48$	$92.59{\pm}8.18$	$97.37{\pm}1.17$
Deep MLP	$88.66{\pm}6.42$	$88.65{\pm}6.43$	$92.55{\pm}8.18$	$84.07{\pm}13.03$
Linear SGD	$89.86{\pm}6.42$	$89.33{\pm}6.45$	$90.62{\pm}7.00$	$90.62{\pm}7.00$

TABLE IV: Master Multi-Paradigm Macro-F1 Benchmark Across Feature Sets (Mean

\pm

SD % Across 3 Seeds)

Attack	LightGBM	RF	MLP	SGD
DoS	$99.94{\pm}0.05$	$99.96{\pm}0.02$	$99.89{\pm}0.04$	$94.01{\pm}10.05$
Nmap Recon	$99.92{\pm}0.09$	$99.96{\pm}0.06$	$99.83{\pm}0.07$	$91.91{\pm}6.98$
SQLi	$99.97{\pm}0.01$	$99.99{\pm}0.01$	$99.97{\pm}0.01$	$99.77{\pm}0.14$
FuerzaBruta	$92.75{\pm}8.76$	$98.41{\pm}2.75$	$66.03{\pm}13.64$	$28.82{\pm}24.66$
MQTT	$88.56{\pm}7.95$	$88.84{\pm}7.64$	$86.47{\pm}8.30$	$45.90{\pm}4.33$
ARP Spoofing	$35.66{\pm}0.09$	$35.66{\pm}0.08$	$35.52{\pm}0.14$	$35.48{\pm}0.11$

TABLE V: Cross-Model Per-Attack Recall on Pure Behavior (

F_{\text{behav}}

, Mean

\pm

SD % Across 3 Seeds)

Model	DoS ( $F_{\text{tstamp}}$ )	ARP Spoof ( $F_{\text{tstamp}}$ )	DNS Beacon ( $F_{\text{all}}$ )
LightGBM	$99.96{\pm}0.02$	$99.50{\pm}0.37$	$27.78{\pm}4.81$
RF	$99.96{\pm}0.02$	$47.59{\pm}12.67$	$27.78{\pm}4.81$
Deep MLP	$72.52{\pm}47.57$	$65.72{\pm}11.67$	$27.78{\pm}4.81$
Linear SGD	$94.01{\pm}10.06$	$35.48{\pm}0.11$	$0.00{\pm}0.00$

TABLE VI: Cross-Model Shortcut Sensitivity on Critical Attacks (Mean

\pm

SD %)

Feature Set	PCAP	Rand	$\mathbf{\Delta}$	PCAP	Rand	$\mathbf{\Delta}$
	Macro-F1 (%)			Attack Recall (%)
$F_{\text{all}}$	88.36	87.99	$-0.37$	80.60	93.49	$\mathbf{+12.89}$
$F_{-\text{ident}}$	88.07	88.00	$-0.07$	79.65	93.65	$\mathbf{+14.00}$
$F_{\text{behav}}$	92.58	88.65	$-3.93$	89.25	94.45	$+5.19$
$F_{\text{tstamp}}$	99.28	99.57	$+0.29$	99.86	99.42	$-0.45$

TABLE VII: Evaluation Protocol Benchmark on LightGBM: PCAP-Disjoint vs. Random Flow Splitting (% Across 3 Seeds)

Explore similar work

Cross-Domain Generalization Failure in Lightweight Intrusion Detection Models for IIoT Networks
Jul 1, 2026MD Azizul Hakim, Md Shihab Uddin, Talha Ibne AnisInternet Of Thing Intrusion Detection
Department of Computer Science and Technology, Bangladesh Sweden Polytechnic Institute Kaptai, Rangamati, Chittagong 4530, Bangladesh · Department of Electrical Engineering, Bangladesh Sweden Polytechnic Institute Kaptai, Rangamati, Chittagong 4530, Bangladesh
Evaluating Tabular Representation Learning for Network Intrusion Detection
May 4, 2026Muhammad Usman Butt, Andreas Hotho, Daniel SchlörIntrusion Detection Tabular Learning
Julius-Maximilians-Universität Würzburg Würzburg, Germany · Data Science Chair Center for Artificial Intelligence and Data Science Julius-Maximilians-Universität Würzburg Würzburg, Germany
Improving IoT Intrusion Detection Through SMOTE-Based Oversampling and Extended Multi-Model Evaluation on Side-Channel Power Data
May 29, 2026Muhammad Khuram Shahzad, Haseeb Khan, Muhammad Masood Khan +1Intrusion Detection Oversampling
aSchool of Electrical Engineering and Computer Science (SEECS), NUST, H-12 Campus, Islamabad, Pakistan44000

Jul 1, 2026, cs.CRJ/K move · Enter open · S save

Cross-Domain Generalization Failure in Lightweight Intrusion Detection Models for IIoT Networks

MD Azizul Hakim, Md Shihab Uddin, Talha Ibne Anis

Department of Computer Science and Technology, Bangladesh Sweden Polytechnic Institute Kaptai, Rangamati, Chittagong 4530, Bangladesh · Department of Electrical Engineering, Bangladesh Sweden Polytechnic Institute Kaptai, Rangamati, Chittagong 4530, Bangladesh

Internet Of Thing Intrusion Detection Robust Cross-Domain Generalization Lightweight

Open paper AskPDF arXiv

Lightweight machine learning models are increasingly proposed for intrusion detection in Industrial Internet of Things (IIoT) networks due to their suitability for resource-constrained edge deployment. Most reported results evaluate these models only within their training network, leaving behavior on unseen networks unverified. This study trains four lightweight architectures on one IIoT dataset and evaluates them, without retraining, on two structurally distinct IIoT datasets using a feature representation restricted to attributes available across all three sources. Explainability analysis across two top-performing models shows both rely overwhelmingly on coarse port-category features; the most influential category occurs in source-domain attack traffic at 96 to 435 times the rate in the two target domains, indicating that coarsening port resolution relocates rather than removes a documented shortcut. Evaluation under naturally imbalanced class distributions reveals a further effect: the evaluation protocol used can reverse which target network appears to pose the greater generalization challenge. Adversarial robustness and recovery through limited target-domain exposure are also assessed; robustness to adversarial perturbation is unrelated to cross-network generalization, and recovery through adaptation varies considerably by architecture. These findings suggest deployment readiness should be assessed using cross-network evaluation under realistic class distributions, rather than within-domain accuracy alone.

Abstract

Explore similar work

Cross-Domain Generalization Failure in Lightweight Intrusion Detection Models for IIoT Networks

Evaluating Tabular Representation Learning for Network Intrusion Detection

Improving IoT Intrusion Detection Through SMOTE-Based Oversampling and Extended Multi-Model Evaluation on Side-Channel Power Data