Recording-Aware Evaluation of Multimodal OT Intrusion Detection

Mustafa Umut Ozbek; Khalil El-Khatib; Li Yang

cs.CROct 16, 2026

Recording-Aware Evaluation of Multimodal OT Intrusion Detection

Authors: Mustafa Umut Ozbek, Khalil El-Khatib, Li Yang

Organizations: Ontario Tech University, Oshawa, Ontario, Canada

Abstract

Machine-learning-based intrusion detection systems (IDSs) for operational technology (OT) are commonly evaluated on benchmark datasets in which normal and attack data are collected in separate recordings. In such settings, high performance may reflect recording-associated differences as well as attack behavior. We introduce a recording-aware evaluation procedure that compares attack windows with background from the same recording, uses prior data for model fitting and calibration, tests temporal alignment against randomized event schedules, and reports event coverage together with alarm burden. On the ProvICS multimodal dataset, cyber modalities strongly distinguish the benign and attack recordings before the first labeled attack. Same-recording evaluation reverses the apparent modality ranking across all studied window sizes: the physical view becomes the strongest individual modality, while cyber-only fusion shows the largest difference between cross-recording and same-recording AUC, reaching 0.54. Operational analysis further shows that permissive event-overlap metrics can overstate detection when a small number of long alarm episodes cover multiple actions. An external analysis on HAI shows smaller and reference-dependent recording sensitivity. These findings show that cross-recording benchmark performance should be interpreted alongside same-recording, forward, temporal, and operational evaluations.

Figures & tables

Fig. 1: Multimodal OT evidence and recording-aware evaluation. Host, controller, network, and physical-process evidence are aligned as observations for an IDS. When recording context co-varies with labels, cross-recording scores can reflect either attack evidence or recording-associated variation. The workflow makes that alternative testable through recording-aware, forward, temporal, and operational checks.

Check	What is compared	What it tells us
Pre-attack recording check	benign recording vs. attack-recording windows before the first labeled attack	whether recordings are distinguishable before attack activity
Same-recording evaluation	attack windows vs. buffered non-attack windows from the same recording	attack-vs.-background discrimination without changing recordings
Cross-to-within gap $\Delta_{\mathrm{CW}}$	cross-recording AUC minus same-recording AUC	sensitivity of apparent performance to recording structure

TABLE I: Recording checks and same-recording evaluation.

Control	Expected behavior	What it verifies
Add recording variation while holding attack evidence fixed	Cross-recording AUC rises; same-recording AUC is stable	$\Delta_{\mathrm{CW}}$ responds to recording-associated information
Change score timing relative to events	Randomization separates action-aligned from unrelated score timing	Schedule test measures alignment with labeled periods

TABLE II: Synthetic validation of the evaluation checks.

Panel A: synchronized views
View	Evidence source	Source format	Availability (%)
M1	supervisory-host provenance	event/provenance graph	94.9%
M2	controller/PLC provenance	event/provenance graph	77.6%
M3	Modbus protocol semantics	protocol-event sequence ∗	94.9%
M4	physical process	22-variable time series, 1 Hz	98.7%
Panel B: attack campaigns

TABLE III: ProvICS modalities and attack campaigns [ 23 , 29 ] .

Fig. 2: Window-size sensitivity of the per-view comparison. (a) Cross-recording (solid) and same-recording (dashed) AUC. (b) The cross-to-within gap

\Delta_{\mathrm{CW}}

. The four window sizes are correlated re-aggregations of one recording pair, not independent replications.

Fig. 3: Fusion and operational performance at 60 s. (a) Cross-recording (open) and same-recording (filled) AUC for selected score families; connecting segments encode the cross-to-within gap. (b) One-to-one event recall versus the fraction of non-action windows in alarm at the nominal 1.4% calibration budget.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Target	Reference	Cross AUC	Same AUC	$\Delta_{\mathrm{CW}}$
test2	train1	0.815	0.879	-0.064
test3	train1	0.979	0.957	- 0.022
test4	train1	0.945	0.926	- 0.019
test5	train1	0.963	0.921	- 0.042
test2	train2	0.765	0.797	-0.032
test3	train2	0.964	0.898	- 0.066

Appendix

TABLE IV: HAI external check.

\Delta_{\mathrm{CW}}=

cross-recording AUC minus same-recording AUC. Rows sharing a target or reference are not independent replications.

Window (s)	View	$\Delta_{\mathrm{CW}}$ , 300 s	$\Delta_{\mathrm{CW}}$ , 600 s	$\Delta_{\mathrm{CW}}$ , 900 s	$n_{\mathrm{within}}$ (300/900 s)
5	M1	0.439	0.440	0.441	12,067/11,107
5	M2	0.538	0.544	0.546	9,895/9,214
5	M3	0.463	0.467	0.468	12,093/11,133
5	M4	0.174	0.174	0.174	12,629/11,676
10	M1	0.450	0.450	0.450	6,035/5,555
10	M2	0.535	0.541	0.544	4,949/4,607

Appendix

TABLE V: Same-recording buffer sensitivity. Entries are seed-median

\Delta_{\mathrm{CW}}

values.

n_{\mathrm{within}}

gives the corresponding seed-median number of retained same-recording background windows at 300/900 seconds. Detector, features, attack positives, and cross-recording negatives are fixed throughout.

Network Intrusion Detection

Network intrusion detection identifies malicious or abnormal network activity to help protect computers, connected devices, and critical infrastructure.

See all 168 papers

How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection
Aug 2, 2026Lorenzo Guerra, Thomas Chapuis, Guillaume Duc +2Data Provenance Benchmark Design
LTCI, T´el´ecom Paris, Institut Polytechnique de Paris · Ampere Software Technology
Multi-Class, Multi-Tier Network Intrusion Detection: A Comprehensive and Reproducible Benchmark
Sep 28, 2026Yufeng Xin, Bryant Goseland, Mohamed RahoutiNetwork Intrusion Detection Multiclass Classification
RENCI, University of North Carolina at Chapel Hill Chapel Hill, NC, USA · Dept. of Computer and Information Science Fordham University New York, NY, USA
Do Transformers Actually Help Intrusion Detection? A Temporal Sequence Evaluation on CIC-IDS2017
Jun 9, 2026Zach Moczkodan, Hany RagabData Leakage Network Intrusion Detection
Royal Military College of Canada, Kingston, Canada · Department of Electrical and Computer Engineering, Faculty of Engineering, Royal Military College of Canada (RMC), Kingston, Ontario, K7K 7B4, Canada.

Aug 2, 2026J/K move · Enter open · S save

How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection

Lorenzo Guerra, Thomas Chapuis, Guillaume Duc, Pavlo Mozharovskyi+1

LTCI, T´el´ecom Paris, Institut Polytechnique de Paris · Ampere Software Technology

Data Provenance Benchmark Design Network Intrusion Detection

Open paper AskPDF arXiv

Provenance-based intrusion detection systems (PIDS) frequently report strong performance, but the conclusions drawn from these results can be highly sensitive to benchmarking choices and evaluation protocols. We investigate this dependency by re-evaluating representative PIDS on public datasets that meet our audit, labeling, and calibration requirements. Focusing primarily on the audited DARPA TC E3 datasets, we apply a unified protocol with temporally separated test periods and validation-only checkpoint selection and threshold calibration, and ask which architectural claims are empirically supported. We find that alerting success and investigation utility can diverge sharply, as several systems surface attacks without providing enough process-level context to support forensic investigation. Across the four primary datasets, a simple allowlist built from executable names and paths observed during training matches or exceeds the selected learned baselines on key operating-point metrics, showing that comparable performance on these metrics is achievable using lexical novelty alone. Quantifying semantic signal quality through feature completeness and field entropy helps explain why several audited E3 datasets support alerting performance without reliably separating model architectures. Theia, however, provides the richest semantic signal and is the only primary dataset on which our reference model, Theseus, clearly improves both ranking and node-level recovery over the other learned systems. Overall, these findings reinforce the importance of interpreting architectural claims in PIDS together with the benchmark properties and evaluation protocol that produced them.

Abstract

Network Intrusion Detection

Explore similar work

How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection

Multi-Class, Multi-Tier Network Intrusion Detection: A Comprehensive and Reproducible Benchmark

Do Transformers Actually Help Intrusion Detection? A Temporal Sequence Evaluation on CIC-IDS2017