Protocol-Sensitive Evaluation of Log Anomaly Detection: Component Costs and Target-Access Sensitivity on HDFS and BGL

Hang Xiao; Janet Sung; Zhaoyi Li; Gangzhen Qian; Chuhong Xu

cs.DBOct 5, 2026

Protocol-Sensitive Evaluation of Log Anomaly Detection: Component Costs and Target-Access Sensitivity on HDFS and BGL

Authors: Hang Xiao, Janet Sung, Zhaoyi Li, Gangzhen Qian, Chuhong Xu

Organizations: Fortinet, Inc. Sunnyvale, USA · Google LLC Mountain View, USA · Independent Researcher Mountain View, USA · Sony Corporate of America San Jose, USA

Abstract

Protocol choices can change the conclusions drawn from log anomaly detection benchmarks even when detector settings are fixed. We present a joint empirical study of split construction, representation visibility, and component costs using six fixed count, sequence, and semantic configurations on Hadoop Distributed File System (HDFS) and Blue Gene/L (BGL) logs. Random splits place several configurations near the average-precision ceiling, whereas group-disjoint HDFS and chronological BGL evaluation produce lower scores and different observed orderings. At a fixed BGL cutoff, parser choice spans 0.124 in semantic XGBoost mean average precision while preserving its lead over count XGBoost; the earliest rolling period reverses that ordering. A two-factor cross-system ablation contrasts source-only representations with offline transductive access to unlabeled target templates through the representation corpus and inverse document frequency: HDFS-to-BGL mean average precision moves from 0.191 with source-only access to 0.325 with union-corpus, target-IDF access, and the intermediate conditions reveal direction-dependent interactions in average precision and retrieval at fixed review budgets. Component-level profiling separates parsing and representation costs from classifier training, prediction, and storage. Together, these findings connect detector comparisons to the test population, preprocessing state, visible information, and measured pipeline stages, and identify the protocol fields needed alongside a score to support interpretable comparisons of log anomaly detection accuracy and resource use.

Figures & tables

Characteristic	HDFS	BGL
Unit	block session	100-line window
Sessions/windows	575,061	47,479
Anomalies	16,838 (2.93%)	4,802 (10.11%)
Event vocabulary	29	2,746
Mean length	19.4	100
Distinct count vectors	589	6,280

TABLE I: Dataset characteristics. HDFS and BGL differ in scale, vocabulary, prevalence, repetition, and session structure.

Model	Configuration
LR	inverse regularization; balanced weights; 1,000 iterations; standardized input
RF	200 trees; balanced-subsample class weights
XGBoost	300 trees; depth 6; rate 0.1; histogram method; class-ratio positive weight
Word2vec	skip-gram; 100 dimensions; window 5; minimum count 1; 30 epochs
LSTM	32-d embedding; 64 hidden; Adam $10^{-3}$ ; batch 256; at most five epochs

TABLE III: Fixed hyperparameters used across protocols and seeds.

	Random split		Group-disjoint
Configuration	AP	$F_{1}$	AP	$F_{1}$
Count/LR	.991 $\pm$ .004	.989 $\pm$ .006	.867 $\pm$ .103	.934 $\pm$ .046
Count/RF	1.000 $\pm$ .000	.998 $\pm$ .000	.743 $\pm$ .227	.808 $\pm$ .259
Count/XGB	.999 $\pm$ .001	.998 $\pm$ .000	.692 $\pm$ .175	.582 $\pm$ .337
Sequence/LSTM	.999 $\pm$ .001	.993 $\pm$ .002	.627 $\pm$ .210	.502 $\pm$ .324
Semantic/LR	.982 $\pm$ .003	.981 $\pm$ .003	.725 $\pm$ .174	.792 $\pm$ .275

TABLE IV: HDFS results across five seeds (mean

\pm

SD); random/group access is transductive.

	Random split		Chronological
Configuration	AP	$F_{1}$	AP	$F_{1}$
Count/LR	.987 $\pm$ .003	.976 $\pm$ .004	.277 $\pm$ .000	.423 $\pm$ .000
Count/RF	.998 $\pm$ .001	.984 $\pm$ .002	.388 $\pm$ .005	.400 $\pm$ .012
Count/XGB	.998 $\pm$ .001	.989 $\pm$ .004	.448 $\pm$ .000	.396 $\pm$ .000
Sequence/LSTM	.946 $\pm$ .012	.916 $\pm$ .017	.236 $\pm$ .151	.282 $\pm$ .091
Semantic/LR	.916 $\pm$ .006	.870 $\pm$ .007	.205 $\pm$ .095	.267 $\pm$ .043

TABLE V: BGL results across five seeds (mean

\pm

SD); random access is transductive, chronological access is inductive.

Fig. 1: Mean AP (

\pm

standard deviation over five seeds) of the six configurations under random splitting and each dataset’s alternative protocol, from Tables IV and V . Group-disjoint HDFS and chronological BGL yield lower mean scores and different leading configurations.

Panel	Setting	rate	AP	AP	AP
	Test anomaly		Count/XGB	LSTM	Semantic/XGB
Parser	Compact $s=.3$	10.1%	.460 $\pm$ .000	–	.741 $\pm$ .054
Parser	Compact $s=.4$	10.1%	.448 $\pm$ .000	–	.676 $\pm$ .086
Parser	Compact $s=.5$	10.1%	.448 $\pm$ .000	–	.631 $\pm$ .109
Parser	Drain3 $s=.4$	10.1%	.432 $\pm$ .000	–	.617 $\pm$ .177
Rolling	50 $\rightarrow$ 60%	1.3%	.902 $\pm$ .000	.475 $\pm$ .147	.838 $\pm$ .047

TABLE VI: BGL chronological sensitivity over five seeds. Parser: 70/30 split; rolling: next 10% after each cutoff.

System	Configuration	Fit (s)	Infer. (k/s)	Size (MB)
HDFS	Count/LR	2.6 $\pm$ 1.1	6771 $\pm$ 2151	.002
HDFS	Count/RF	17.8 $\pm$ 2.1	312 $\pm$ 91	2.234
HDFS	Count/XGB	3.8 $\pm$ 1.8	660 $\pm$ 94	.423
HDFS	Sequence/LSTM	309.3 $\pm$ 13.2	37 $\pm$ 4	.116
HDFS	Semantic/LR	31.4 $\pm$ 21.3	1257 $\pm$ 113	.004
HDFS	Semantic/XGB	9.4 $\pm$ .8	696 $\pm$ 189	.416

TABLE VII: Component-level cost over five seeds. Throughput covers classifier-only inference; size covers the serialized classifier.

HDFS $\rightarrow$ BGL
Word2vec	IDF	AP	P@1%	R@1%	P@5%	R@5%
Source	Source	.191 $\pm$ .107	.242 $\pm$ .160	.024 $\pm$ .016	.220 $\pm$ .174	.109 $\pm$ .086
Source	Target	.187 $\pm$ .084	.259 $\pm$ .163	.026 $\pm$ .016	.253 $\pm$ .163	.125 $\pm$ .081
Union	Source	.297 $\pm$ .136	.443 $\pm$ .266	.044 $\pm$ .026	.293 $\pm$ .021	.145 $\pm$ .010
Union	Target	.325 $\pm$ .159	.446 $\pm$ .312	.044 $\pm$ .031	.318 $\pm$ .122	.157 $\pm$ .060
BGL $\rightarrow$ HDFS

TABLE VIII: Cross-system XGBoost target-catalog visibility over five seeds. Union corpus and target IDF are offline transductive; budgets use tie-aware allocation.

Explore similar work

NLLog: Lightweight, Explainable SOC Anomaly Detection via Log-to-Language Rewriting
Jun 3, 2026Samuel Ndichu, Tao Ban, Seiichi Ozawa +2Interpretable Anomaly Detection Threat Detection
National Institute of Information and Communications Technology, Tokyo, Japan · Kobe University, Kobe, Japan
LogNEO: A GPT-Neo Reinforcement Learning Framework for Accurate Real-Time Log Anomaly Detection
Jun 6, 2026David Eje, Tanmay Sharma, Khush Patel +2Interpretable Anomaly Detection Openai
Department of Computer Science, Innopolis University, Innopolis, Russia
FAME: Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection
May 21, 2026Huanchi Wang, Zihang Huang, Yifang Tian +3Interpretable Anomaly Detection Early Failure Prediction
Department of Electrical and Computer Engineering University of Toronto, Toronto, Ontario

Jun 3, 2026, cs.CRJ/K move · Enter open · S save

NLLog: Lightweight, Explainable SOC Anomaly Detection via Log-to-Language Rewriting

Samuel Ndichu, Tao Ban, Seiichi Ozawa, Takeshi Takahashi+1

National Institute of Information and Communications Technology, Tokyo, Japan · Kobe University, Kobe, Japan

Interpretable Anomaly Detection Threat Detection Rewrite Lightweight

Open paper AskPDF arXiv

System-generated logs underpin security monitoring, yet their rigid template-based format hinders both automated analysis and human comprehension. We present NLLog (Natural-Language Log), a lightweight pipeline that deterministically rewrites parsed templates into WHO-WHAT-SEVERITY sentences, pools them with term-frequency-inverse-document-frequency weighting, classifies sessions with tree ensembles, and back-projects evidence with TreeSHAP for analyst review. On Hadoop Distributed File System (HDFS) and Blue Gene/L (BGL) corpora, NLLog exceeds two reproduced matched-protocol baselines; across HDFS, BGL, and the AIT Alert Data Set, it sustains low false-positive rates with commodity-hardware latency suitable for security operations center triage. Coverage, sparse-versus-dense, faithfulness, and adversarial ablations show that fallback sufficiency is corpus-dependent, that an enrollment-time coverage check can surface refinement requirements before deployment, and that an auditable deterministic rewrite combined with lightweight dense encoding provides a measurable representation layer for log-anomaly detection and triage.

Abstract

Explore similar work

NLLog: Lightweight, Explainable SOC Anomaly Detection via Log-to-Language Rewriting

LogNEO: A GPT-Neo Reinforcement Learning Framework for Accurate Real-Time Log Anomaly Detection

FAME: Failure-Aware Mixture-of-Experts for Message-Level Log Anomaly Detection