Candidate Comparability Before Promotion: Conditional Validation in Adaptive Network Intrusion Detection

Roberto Fernández-Barrios; Iker Pastor-López; Amaia Pikatza-Huerga; Pablo García Bringas

cs.CRSep 3, 2026

Candidate Comparability Before Promotion: Conditional Validation in Adaptive Network Intrusion Detection

Authors: Roberto Fernández-Barrios, Iker Pastor-López, Amaia Pikatza-Huerga, Pablo García Bringas

Organizations: Faculty of Engineering, University of Deusto, Avda. de las Universidades, 24, 48007 Bilbao, Spain

Abstract

Adaptive network intrusion detection systems retrain classifiers after drift alarms, but an alarm detects change; it does not establish that a challenger should replace the deployed incumbent. Promotion is security-relevant because it changes the model responsible for subsequent attack detection, and evaluating it has a methodological problem: promotion conclusions may depend on how the challenger was constructed and on how much evidence supports it. We test that dependence on CICIDS2017, UNSW-NB15 and ToN-IoT with self-contained challenger pipelines, nested candidate-size controls, a common-harness comparison of nine update policies, and a final sensitivity confining every exact feature vector to one evaluation, training or probe role. Incumbent-owned frozen preprocessing amplified apparent promotion harm; with self-contained challenger pipelines the mean full-drift harm did not persist. Raising nominal candidate evidence from 512 to 2,000 samples per class improved promotion under pool-constructed progressive drift by +0.53, +1.67 and +0.38 balanced-accuracy points: positive and statistically resolved in all three benchmarks, but materially benchmark-dependent rather than homogeneous, and driven mainly by fewer false positives. Policy conclusions were partially robust: policy ordering changed with candidate comparability, no policy globally dominated, and earlier compatibility statements for a label-free estimator and a calibrated ensemble narrowed. Validation helped evidence-disadvantaged challengers but added no average benefit at parity. Thirteen replays on real, time-ordered traffic showed no net harm from always deploying. Challenger construction and evidence should be controlled, reported and interpreted explicitly when promotion is evaluated.

Figures & tables

Approach	Objective	Decision unit	Information required	Output	Guarantees / scope
CARA [ 57 ]	Cost-aware stream retraining	Retraining opportunity	Performance and retraining/staleness costs	Retrain or keep	Cost–performance analysis under its assumptions; no concrete-challenger audit
Regol [ 69 ]	Forecast when retraining pays	Retraining opportunity	Sparse performance history and cost ratio	Retrain or keep	Uncertainty-aware forecast within its model; no paired proposal guarantee
Automated adaptation strategies [ 11 ]	Select among stream adaptation mechanisms	Batch / adaptation opportunity	Recent-batch (cross-validation) evidence	Selected adaptive mechanism, including no update	Adaptive-strategy selection within its framework; no explicit incumbent–challenger comparability audit
IGPC-MSOS [ 80 ]	Select NIDS update mode	Drift/update episode	Dual triggers and online system feedback	Update mode	System-specific decision architecture; no paired concrete-challenger guarantee
CARAVAN [ 87 ]	Label, monitor and retrain in-network models	Labeling window	Generated labels, predictions and accuracy proxy	Retrain/update weights	End-to-end adaptive system and public artifact; no held-out challenger comparison
ADAWU-IDS [ 86 ]	Calibrate drift-response policy	Validation stream / traffic chunk	Delayed chunk labels, drift severity and learner performance	Reweight; optional stronger response	Chronological held-out calibration; selected configuration is dataset-specific

Table 1: Conceptual positioning of related decision formulations. “Guarantee” records the type and scope stated by each formulation, not a common empirical ranking.

Figure 1: Candidate promotion as a decision pipeline. A drift alarm proposes, but does not justify, promotion; candidate construction and evidence comparability precede candidate validation, and the validation stage is optional — conditional on construction, evidence and incumbent health — rather than universal.

Object	Source	Reflects window	Latency knob
Trigger decision	detector score on $W_{t}$	$t$	—
Probe (pool arms)	pool sample at $\mathrm{sev}(t-L_{p})$	$t-L_{p}$	--probe-latency
Probe (observed arm)	observed window $t-9$ , row-disjoint	$t-9$	fixed
Candidate batch (pool)	pool sample at $\mathrm{sev}(t-L_{c})$	$t-L_{c}$	--candidate-latency
Candidate batch (sliding)	8 observed windows ending $t-L_{c}$	$[t-L_{c}-7,\,t-L_{c}]$	--candidate-latency
Recalibration (observed)	last 8 observed windows	$\leq t$	—

Table 2: Temporal (causal-information) availability of every object the update loop touches at a trigger at window

t

. “Latency knob” names the runner flag; 0 = contemporaneous.

Configuration	512/class challenger	2,000/class challenger
Frozen incumbent-owned preprocessing (historical), full drift	$\Delta$ vs never: $+6.31$ / $+1.39$ / $-4.95$ (seeds 3001–3030; descriptive) validation: point $-$ naive $+0.33$ / $-0.00$ / $+5.56$ (same block; descriptive)	not evaluated under a registered protocol
Frozen incumbent-owned preprocessing (historical), zero drift	$\Delta$ vs never: $-4.09$ / $-0.68$ / $-5.87$ (seeds 3001–3030; descriptive) validation: point $-$ naive $+2.93$ / $+0.21$ / $+5.47$ (same block; descriptive)	$\Delta$ vs never: $-4.81$ / $-0.13$ / $-5.76$ (seeds 104–133; descriptive) validation: point $-$ naive $+4.46$ / $+0.03$ / $+5.35$ (same block; descriptive)
Self-contained challenger, zero drift	$\Delta$ vs never: $-1.70$ / $-0.65$ / $-0.24$ (seeds 4001–4030; descriptive) validation: 6/6 gate contrasts Holm-significant gains (seeds 3001–3030, Table 4 )	$\Delta$ vs never: $+0.19$ / $-0.02$ / $-0.01$ (seeds 4001–4030; registered F1; CI90 within $\pm 0.5$ in 3/3) validation: 0/6 Holm-significant gains (same block; registered F3)
Self-contained challenger, full drift	$\Delta$ vs never: $+8.90$ / $+2.13$ / $+1.44$ (seeds 6001–6030; registered G1) validation: 1/6 Holm-significant gain, 1 resolved cost (seeds 3001–3030, Table 4 )	$\Delta$ vs never: $+9.72$ / $+3.79$ / $+2.44$ (seeds 6001–6030; registered G1) validation: 0/6 Holm-significant gains, 1 resolved cost (strict, UNSW $-0.34$ ; same block; registered G3)

Table 3: Central evidence matrix: how the apparent value of promotion depends on challenger construction and evidence. Each cell reports always-deploy minus never-adapt in balanced-accuracy points (PortScan / UNSW-Recon / ToN-IoT) and whether point/strict validation adds Holm-significant average value over always-deploy in that configuration; every value names the 30-seed block it was estimated in, and where a cell’s validation summary comes from a different registered block than its

\Delta

, that block is named in the cell. Rows 1–2 are the historical frozen incumbent-owned preprocessing configuration (descriptive; the frozen arm of the symmetric replication, seeds 3001–3030, and, for the 2,000/class zero-drift cell, the registered frozen size-matched control on seeds 104–133). Rows 3–4 are self-contained challengers: zero-drift

\Delta

s from the registered size-matched control (seeds 4001–4030; 512 column descriptive, 2,000 column registered F1), full-drift

\Delta

s from the registered size-matched control under drift (seeds 6001–6030; registered G1). Absolute frozen-configuration magnitudes differ between seed blocks (§ 5.1 ); only within-block contrasts are paired. “not evaluated” cells were never run under a registered protocol.

Regime	Transf.	Never BA	Naive $-$ never	Point $-$ naive	Strict $-$ naive
PortScan full	frozen	87.44	$+6.31$ [4.11, 8.53]	$+0.33$ [-0.05, 1.00]	$+0.36$ [-0.12, 1.07]
	own	87.44	$+7.21$ [4.89, 9.52] †	$+0.19$ [-0.06, 0.41]	$+0.05$ [-0.33, 0.41]
UNSW-Recon full	frozen	83.03	$+1.39$ [1.17, 1.62]	$-0.00$ [-0.11, 0.10]	$-0.03$ [-0.24, 0.16]
	own	83.03	$+2.55$ [2.34, 2.75] †	$-0.21$ [-0.36, -0.06] †	$-0.15$ [-0.35, 0.05]
ToN-IoT full	frozen	92.40	$-4.95$ [-7.15, -2.86]	$+5.56$ [3.50, 7.75]	$+5.38$ [3.34, 7.56]
	own	92.40	$+1.03$ [0.55, 1.53] †	$+0.64$ [0.26, 1.06] †	$+0.18$ [-0.31, 0.67]

Table 4: The symmetric-pipeline dynamic replication (registered; seeds 3001–3030). Shared raw stream per seed (hash-verified across all seven arms); frozen = historical frozen-initial-transformer policy, own = self-contained pipelines (each challenger fits its own scaler/PCA on its own raw batch, bit-identical across policies; commits deploy the complete bundle). BA points, paired within seed, 30 seeds, deterministic paired-bootstrap CI95;

\dagger

= Holm-significant within its registered family (own rows); frozen rows are descriptive (uncorrected). Zero drift: random trigger

p{=}0.05

, no drift; the detector representation stays the initial transformer under both policies.

Regime	Never BA	Naive ${}_{512}-$ never	Naive ${}_{2000}-$ never	compat.	Naive ${}_{2000}-$ naive 512	Point ${}_{2000}-$ naive 2000	Strict ${}_{2000}-$ naive 2000
PortScan zero	95.66	$-1.70$ [-2.05, -1.36]	$+0.19$ [-0.15, 0.55]	yes	$+1.89$ [1.54, 2.23] †	$+0.05$ [-0.00, 0.12]	$+0.13$ [-0.07, 0.34]
UNSW-Recon zero	89.08	$-0.65$ [-0.77, -0.55]	$-0.02$ [-0.10, 0.06]	yes	$+0.63$ [0.53, 0.73] †	$+0.01$ [-0.02, 0.04]	$+0.06$ [-0.01, 0.12]
ToN-IoT zero	92.56	$-0.24$ [-0.39, -0.10]	$-0.01$ [-0.05, 0.04]	yes	$+0.23$ [0.10, 0.38] †	$+0.02$ [0.00, 0.04]	$+0.03$ [-0.01, 0.08]

Table 5: The size-matched self-contained challenger control (registered; seeds 4001–4030). Zero drift, random proposal trigger, own-transformer only; the sole new variable is the challenger’s per-class training size (512 vs. the incumbent’s 2,000), with nested candidate batches (§ 4.2.2 ). BA points, paired within seed, 30 fresh seeds, paired-bootstrap CI95;

\dagger

= Holm-significant within its registered family; the 512-vs-never column is descriptive (uncorrected). compat. : CI90 fully inside the preregistered

\pm 0.5

-point margin; PortScan is boundary-close (upper bound

0.494

0.006

inside) and does not demonstrate absence of an effect.

Regime (full drift)	Never BA	Naive 512	Naive 2000	Size effect [CI95]	G2 class	Point 2000 [CI95]	Strict 2000 [CI95]
PortScan	85.58	$+8.90$ †	$+9.72$ †	$+0.82$ [0.67, 0.98] †	size benefit	$+0.15$ [-0.03, 0.39]	$-0.00$ [-0.17, 0.23]
UNSW-Recon	83.36	$+2.13$ †	$+3.79$ †	$+1.66$ [1.51, 1.81] †	size benefit	$-0.06$ [-0.16, 0.03]	$-0.34$ [-0.53, -0.15] †
ToN-IoT	91.79	$+1.44$ †	$+2.44$ †	$+1.00$ [0.60, 1.39] †	size benefit	$+0.01$ [-0.06, 0.09]	$-0.14$ [-0.56, 0.28]

Table 6: Candidate evidence under pool-constructed progressive drift: the registered source-row-disjoint size-matched control (seeds 6001–6030, 21 arms). Self-contained challengers, nested candidate batches drawn at the proposal-time mixture (the 512 batch is the first 512 rows per class of the 2,000 batch). BA points; paired within seed, 30 seeds; CI95 from the deterministic centered paired bootstrap;

\dagger

= Holm-significant within its registered family (G1: naive vs never at both sizes; G2: the primary size effect; G3: gate value at 2,000). Historical registered classification of G2 per the frozen protocol; block outcome homogeneous–size benefit . Columns 3–4: always-deploy minus never-adapt at each size; column 5: the size effect; columns 7–8: gate minus always-deploy at 2,000/class. Full matrices in Online Resource 1, §S9.

Dataset	Policy	Cand.	Probe	BA vs never [CI95]	Gate gain vs naive [CI95]	Commits/seed
CICIDS2017-PortScan	naive 512	1,024	0	$-1.70$ [-2.05, -1.36]	—	3.70
	point 512 + 32-label probe	1,024	32	$-0.73$ [-1.10, -0.38]	$+0.97$ [+0.67, +1.31]	1.87
	strict 512 + 32-label probe	1,024	32	$-0.03$ [-0.24, +0.16]	$+1.67$ [+1.36, +2.00]	0.23
	naive 2,000	4,000	0	$+0.19$ [-0.15, +0.55]	—	3.70
UNSW-NB15	naive 512	1,024	0	$-0.65$ [-0.77, -0.55]	—	3.70
	point 512 + 32-label probe	1,024	32	$-0.47$ [-0.61, -0.33]	$+0.19$ [+0.10, +0.30]	2.27

Table 7: Nominal evidence–validation trade-off under the preregistered zero-drift size-matched control (own-transformer, random proposals, balanced pools, SVC-RBF, seeds 4001–4030). “Cand.”/“Probe” are nominal adjudicated labels per proposal; moving from 512 to 2,000 per class adds

2\times 1{,}488=2{,}976

candidate-training labels per proposal versus the 32-label point/strict probe — counts, not costs: inspected-flow acquisition is not modeled and no economic dominance is claimed. Paired per-seed BA contrasts (CI95, deterministic centered paired bootstrap): naive-2,000 vs never is registered F1, gate gains at 2,000 are registered F3 (none Holm-significant), 512-side and gate-vs-never cells are descriptive, uncorrected. The supplementary point/strict-2,000 policies differ from naive-2,000 by

<0.14

pp (non-significant, CI90 compatible within

\pm

0.5 pp). Guardrails: all matched-size gate cells pass recall/FPR non-inferiority; at 512 the UNSW strict cell fails recall non-inferiority (Online Resource 1, §S8).

			Full drift			Zero drift
Policy	Origin	Labels at decision	PortScan	UNSW	ToN-IoT	PortScan	UNSW	ToN-IoT
Never-adapt	anchor	0	86.0	83.1	91.9	95.1	89.0	92.5
Always-deploy (naive)	anchor	0	$+9.44$	$+4.06$	$+2.50$	$+0.25$	$+0.07$	$+0.02$
Point gate, $b{=}32$	authors’ policy	32	$+9.53$	$+4.05$	$+2.70$	$+0.49$	$+0.09$	$+0.03$
Strict gate (reject ties), $b{=}32$	authors’ policy	32	$+9.28$	$+3.83$	$+2.28$	$+0.64$	$+0.06$	$-0.01$
ATC [ 31 ]	published generic	0 ( $+$ 512-row val.)	$+8.80$ ?	$+3.84$ ≈	$+2.60$ ≈	$+0.52$ ≈	$+0.03$ ≈	$+0.02$ ≈

Table 8: Registered common-harness comparison with published and reference baselines (primary condition: self-contained 2,000/class challengers; seeds 5001–5030). Balanced-accuracy points over never-adapting (from the sealed per-arm means; the never-adapt row gives its absolute mean BA; all arms share bit-identical raw streams per seed). Markers record the registered per-cell classification of the policy

-

naive contrast (families PF1 zero drift / PF2 full drift; Holm within family):

\dagger

material gain,

\ddagger

material cost,

\approx

compatible (CI90 within

\pm 0.5

?

unresolved. Anchor rows (naive, point, strict) are descriptive here by amendment. Labels = target labels per decision; ATC/DoC additionally use a 512-row labeled validation sample at each model’s training time; DDM/ADWIN consume 8 monitoring labels per window. Origin: published generic method, reference implementation ( river 0.25.0), standard baseline, or authors’ policy — none is an adaptive-NIDS system reproduced end to end. Full families, the 512/class sensitivity block and statements in Online Resource 1, §S10.

Benchmark	Historical	Exact-feature-disjoint [CI95]	$p_{\mathrm{Holm}}$	Class	Change [CI95]	Change class	Recall 512 $\to$ 2,000	FPR 512 $\to$ 2,000
PortScan	$+0.82$	$+0.53$ [0.27, 0.79]	0.00018	material benefit	$-0.29$ [-0.60, 0.01]	unresolved	89.82 $\to$ 90.13	5.44 $\to$ 4.70
UNSW-Recon	$+1.66$	$+1.67$ [1.47, 1.89]	0.00003	material benefit	$+0.01$ [-0.24, 0.28]	compatible	97.86 $\to$ 97.51	31.20 $\to$ 27.51
ToN-IoT	$+1.00$	$+0.38$ [0.21, 0.57]	0.00018	resolved sub-material	$-0.62$ [-1.04, -0.19]	material attenuation	97.16 $\to$ 97.34	28.39 $\to$ 27.81

Table 9: Exact-feature-disjoint sensitivity of the full-drift size effect. Always-deploy with 2,000/class minus always-deploy with 512/class, balanced-accuracy points, 30 fresh seeds (7001–7030). Exact cleaned raw feature groups are assigned wholly to window, candidate-training or probe roles; multiplicity and original labels are retained. CI95 and Holm-adjusted

p

are from the registered deterministic paired bootstrap. The historical source-row-disjoint estimate is shown for context; “change” is sensitivity minus historical, estimated between independent seed blocks and is not a causal duplicate- leakage effect. The last two columns give always-deploy attack recall and false-positive rate (%) at 512

\to

2,000 per class from the same sealed cells: the balanced-accuracy advantage is driven mainly by the lower false-positive rate, with recall approximately stable and slightly lower on UNSW-Recon.

Extension	Question	Main result	Scope / limitation
Observed-data (§S2.7)	Survives without simulator privileges?	Harm-regime rescue reproduces leakage-free ( $+$ 4.05/ $+$ 4.83 above always-deploy)	Frozen, 512/class; UNSW below no-adapt at 64-flow windows
Classifier / generator (§S1.6, S2.12)	Is the harm SVC-specific?	Sign general at 512; size-matching clears robust learners	Magnitude pipeline-dependent; frozen transformer
Mild drift (§S2.12)	Is harm dataset-specific?	Healthy incumbents make always-deploy harmful in all three	Frozen, 512/class
Quantum monitor (§S1.2)	Does a more expressive detector change conclusions?	No	${\sim}114\times$ simulated cost; invariance check only
VBC-SG frontier (§S2.12)	What do formal guarantees cost?	Non-vacuous from $b{=}256$ ; abstention/labels priced	Frozen; not necessary under the size-matched controls
Chronological replays (§S2.6)	External time-ordered boundary?	No net harm observed in any of 13 replays (7 in Table 11 , 6 earlier); gates pay a premium on collapsed incumbents	No ToN-IoT analogue; no prevalence estimate

Table 10: Supporting evidence blocks. Each is detailed in Online Resource 1 at the section shown; scopes are stated in §7.

Figure 3: Per-trigger test on the hardened harness (seeds 104–133; 250 KS-max triggers): incumbent balanced accuracy over the five windows before the trigger (left) and detector score at the trigger (right), each against the candidate-minus-incumbent balanced accuracy over the five following windows, for the KS-max (top) and QK-ZZ (bottom) detectors; colours and marker shapes mark the three regimes. Panel annotations give the correlation pooled over the three regimes (

r=-0.57

and

-0.59

for degradation,

+0.02

and

+0.01

for the score); the regime-specific correlations quoted in § 5.8 (

-0.67

-0.70

) are computed within each regime and are a different statistic. Predictor and outcome share no algebraic term (disjoint past/future windows); clustered CIs and the hierarchical model are in § 5.8 .

Replay	no-adapt BA	naive	point	strict	VBC-SG
Tue $\to$ Wed+Thu+Fri	59.0	$+14.78$	$+11.52$	$+11.50$	$+0.16$
Wed $\to$ Fri	51.8	$+35.29$	$+27.64$	$+28.81$	$+0.91$
Thu $\to$ Fri	49.0	$+36.07$	$+27.71$	$+26.51$	$+1.56$
Wed intra-day	84.7	$+6.47$	$+6.02$	$+5.77$	$+0.56$
Thu intra-day	48.3	$+0.88$	$+0.88$	$+0.42$	$+0.00$
UNSW (train 20%)	82.9	$+8.35$	$+9.90$	$+11.13$	$+10.76$

Table 11: The registered chronological matrix. Seven pre-enumerated time-ordered replays (CICIDS2017 train

\to

future day pairs, two intra-day splits, UNSW-NB15 at two training fractions), 200 windows of 256 flows in capture order, seeds 601–630. Columns: balanced-accuracy points over never-adapting; no-adapt is the frozen incumbent’s absolute BA (the health indicator). No stream shows net harm. Where the incumbent collapses (CICIDS), the gates preserve part of always-deploying’s recovery but can impose substantial costs; where the incumbent stays healthy, point estimates for point and strict gates are above always-deploying on both UNSW timelines (VBC-SG on the 20% split but not the 40%), with the healthy Wednesday intra-day replay the unresolved counterexample. These are descriptive comparisons, not direct inferential superiority claims.

Network Intrusion Detection

Network intrusion detection identifies malicious or abnormal network activity to help protect computers, connected devices, and critical infrastructure.

See all 168 papers

Jul 2, 2026

RES-DARE: Failure-Aware Expert Adaptation and Rollback-Safe Self-Repair for Intrusion Detection

Intrusion detection systems are often trained under static benchmark conditions, although deployed network environments are affected by traffic drift, sensor noise, changing workloads, and evolving attack behaviour. Under such distribution shifts, static detectors may produce confident but incorrect predictions, leading to silent and unsafe failure modes. In this paper, RES-DARE (Recursive Evolving Specialists-Digital Adaptive Reasoning Engine) is proposed as a failure-aware continual intrusion detection framework with rollback-safe self-repair. Difficult, uncertain, and misclassified samples are treated as failure signals for expert specialisation rather than being discarded as noise. A supervised contrastive encoder, two-pass expert router, failure-buffer mechanism, HDBSCAN-based failure-region discovery, and trust-risk monitor are integrated to support adaptive IDS behaviour. AEHM-v2 is introduced as a rollback-safe repair mechanism, where candidate adaptations are provisionally activated and committed only when macro-F1 is preserved or improved while trust risk remains stable. Otherwise, the system is rolled back to its last validated state. RES-DARE is evaluated on CICIDS2017, UNSW-NB15, and TON_IoT, achieving macro-F1 scores of 0.9850, 0.9736, and 0.9691, respectively. Under Gaussian feature corruption at strength 0.10, RES-DARE retains an Attack-F1 of 0.7920 on CICIDS2017 and achieves near-zero catastrophic forgetting with F = 0.0015. The results show that RES-DARE improves robustness, warning capability, and deployment safety under degraded conditions.

Rahil Aftab, Anyash Prasad, Soumya Mazumdar +2

Engineering Science Homi Bhabha National Institute Anushaktinagar, Mumbai 400094, Maharashtra, India Computer and Informatics Group Variable Energy Cyclotron Centre 1/AF, Bidhannagar, Kolkata 700064, West Bengal, India

Adaptive Model Routing Continual Learning Network Intrusion Detection

Ask PDF

Aug 2, 2026

How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection

Provenance-based intrusion detection systems (PIDS) frequently report strong performance, but the conclusions drawn from these results can be highly sensitive to benchmarking choices and evaluation protocols. We investigate this dependency by re-evaluating representative PIDS on public datasets that meet our audit, labeling, and calibration requirements. Focusing primarily on the audited DARPA TC E3 datasets, we apply a unified protocol with temporally separated test periods and validation-only checkpoint selection and threshold calibration, and ask which architectural claims are empirically supported. We find that alerting success and investigation utility can diverge sharply, as several systems surface attacks without providing enough process-level context to support forensic investigation. Across the four primary datasets, a simple allowlist built from executable names and paths observed during training matches or exceeds the selected learned baselines on key operating-point metrics, showing that comparable performance on these metrics is achievable using lexical novelty alone. Quantifying semantic signal quality through feature completeness and field entropy helps explain why several audited E3 datasets support alerting performance without reliably separating model architectures. Theia, however, provides the richest semantic signal and is the only primary dataset on which our reference model, Theseus, clearly improves both ranking and node-level recovery over the other learned systems. Overall, these findings reinforce the importance of interpreting architectural claims in PIDS together with the benchmark properties and evaluation protocol that produced them.

Lorenzo Guerra, Thomas Chapuis, Guillaume Duc +2

LTCI, T´el´ecom Paris, Institut Polytechnique de Paris · Ampere Software Technology

Data Provenance Benchmark Design Network Intrusion Detection

Ask PDF

Sep 28, 2026

Multi-Class, Multi-Tier Network Intrusion Detection: A Comprehensive and Reproducible Benchmark

Machine learning (ML) and deep learning (DL) have dominated Intrusion Detection System (IDS) research in recent years. Unfortunately, many existing studies have produced inflated results and unreliable benchmarks due to critical oversights and mistakes in the ML and DL pipeline, from data collection and labeling to feature engineering and model training and evaluation. CIC-IDS2017 is a standard benchmark for network intrusion detection. Still, many published results on this dataset are difficult to compare due to labeling errors, inconsistent flow extraction, potential leakage, and performance evaluation metrics dominated by benign traffic. In this paper, we present a comprehensive benchmark with corrected PCAP-level labeling and a complete evaluation pipeline with diverse ML models. We evaluate eleven tabular classifiers at three nested levels: binary attack detection, nine-class attack-family attribution, and fifteen-class fine-grained classification. A soft-voting ensemble of Random Forest, XGBoost, and LightGBM obtains the best fine-tier macro-F1 of 0.955, with coarse and binary macro-F1 scores of 0.980 and 0.999, respectively. We further conducted a feature selection study based on an analysis of feature importance. This comprehensive benchmark pipeline is configurable and open-source, enabling new feature extraction and model plugins for new datasets. Future work should use this pipeline as a reference point for richer features, rare-class analysis, and model generalization towards new datasets and attack classes.

Yufeng Xin, Bryant Goseland, Mohamed Rahouti

RENCI, University of North Carolina at Chapel Hill Chapel Hill, NC, USA · Dept. of Computer and Information Science Fordham University New York, NY, USA

Network Intrusion Detection Multiclass Classification

Ask PDF

Abstract

Network Intrusion Detection

Explore similar work

RES-DARE: Failure-Aware Expert Adaptation and Rollback-Safe Self-Repair for Intrusion Detection

How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection

Multi-Class, Multi-Tier Network Intrusion Detection: A Comprehensive and Reproducible Benchmark