SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence
Authors: Yifan Wang, Dongliang Xu, Xiaoyong Li, Jing Li, Peter S. Pan, Yue Yao
Organizations: School of Airspace Science and Engineering, Shandong University, Shandong 264209, China. · School of Computer Science, University of Technology Sydney, Ultimo, NSW 2007, Australia. · an independent researcher in Braddon, ACT 2612, Australia.
Large language models (LLMs) have shown strong potential for cybersecurity, leading to growing interest in LLM-based automated cybersecurity log analysis. However, current datasets (1) are limited in scale; (2) rely mainly on single-source evidence; and (3) have only binary or multi-class attack labels, without dense annotations of the reasoning process. To address these limitations, SecLog-Bench has three key features. First, we build an extensible simulation platform that covers 29 attack scenarios, supports repeated execution and the integration of new scenarios, and enables scalable generation of cybersecurity logs with attack-stage labels. Second, we use multi-source logs, including audit logs, application logs, and network traffic, to jointly analyze cyber attacks. Third, we annotate each sample with the essential context for cybersecurity log reasoning and a structured step-by-step evidence chain. We evaluate eight mainstream LLMs on this benchmark for post-detection attack-progress reasoning. Results show that, when provided with scaffolded evidence, current LLMs exhibit modest capability in attack-progress reasoning, with the best model achieving only 44.44% accuracy and 45.07% Macro-F1. Their reasoning remains unreliable; i.e., the highest Stage Reasoning F1 (SRF1) score is only 54.87%. Moreover, even correct predictions may be accompanied by flawed reasoning, and reasoning quality is often misaligned with prediction accuracy.
Figures & tables
Fig. 1: Motivation. Left: Typical log datasets provide only binary labels for anomalous and normal logs. Right: SecLog-Bench instead evaluates the reasoning process for attack-progress inference.
Fig. 2: Overview of the SecLog-Bench workflow. The workflow constructs multi-source evidence windows, builds evaluation-only reference reasoning chains, and evaluates model attack-stage predictions and reasoning reliability.
Fig. 3: Example of multi-source context attachment within the window evidence E(W) . Application records, network flows, and retained audit context are attached to the core audit sequence through auxiliary indices and entity matching. Rounded rectangles, ovals, and diamonds denote processes, files, and sockets or network endpoints, respectively.
Fig. 4: Example of model evaluation input and output in SecLog-Bench. Evidence highlights denote application semantics (pink), network interactions (green), and retained audit context (gray); yellow marks the predicted label. The example is shortened for readability.
Fig. 5: Overview of SecLog-Bench data composition and evidence reasoning: (a) hierarchy of the 29 reproducible attack scenarios; (b) raw-data scale comparison; and (c) comparison between typical single-source analysis focused on final attack labels and the SecLog-Bench pipeline for scaffolded multi-source evidence reasoning with attack-stage labels and four-step reference reasoning chains.
Scenario
Technique and Sub-technique IDs
GeoServer/CVE-2023-25157
T1190, T1518
GitList/CVE-2018-1000533
T1016, T1033, T1059.004, T1083, T1087.001, T1190
Joomla/CVE-2023-23752
T1005, T1087, T1190, T1552
Metabase/CVE-2023-38646
T1059.007, T1190
Solr/CVE-2017-12629-XXE
T1087.001, T1190
Solr/CVE-2019-17558
T1016, T1033, T1059, T1083, T1087.001, T1190
TABLE I: Direct ATT&CK mappings for ten representative source scenarios.
Scenario group
Scenarios
Windows
Preparation
Execution
Result
SU
Java
14
92
20
36
12
24
PHP & C
7
58
14
2
27
15
Python
3
29
8
8
9
4
Node.js
3
22
9
3
4
6
Sandworm
1
8
0
3
0
5
Multi-hop
1
7
3
2
2
0
TABLE II: Scenario-group and stage-label distribution of the final evaluation set. The Scenarios and Windows columns report reproducible construction-pool scenarios and retained evaluation windows, respectively.
Dataset
Multi-source
Evidence organization
Extensibility
Reasoning chains
App. log #lines
Audit log #lines
Network traffic #lines
Total #lines
Size (GiB)
HDFS-v1 [ 6 ]
No
Single-source logs
No
No
–
11,175,629
–
11,175,629
1.470
Spark [ 6 ]
No
Single-source logs
No
No
–
33,236,604
–
33,236,604
2.750
OpenStack [ 6 ]
No
Single-source logs
No
No
–
207,820
–
207,820
0.057
BGL [ 6 ]
No
Single-source logs
No
No
–
4,747,963
–
4,747,963
0.692
Thunderbird [ 6 ]
No
Single-source logs
No
No
–
211,212,192
–
211,212,192
29.600
Linux [ 6 ]
No
Single-source logs
No
No
–
25,567
–
25,567
0.002
TABLE III: Comparison with representative public log and network-traffic datasets. SecLog-Bench provides 216 evidence-window-level reference reasoning chains with 864 annotated reasoning steps.
Model
Accuracy ↑
Macro-F1 ↑
Prep. Recall ↑
Exec. Recall ↑
Res. Recall ↑
SU Recall ↑
Qwen3-8B
23.15%
19.27%
3.70%
38.89%
7.41%
42.59%
Qwen3-32B
26.39%
25.95%
12.96%
50.00%
24.07%
18.52%
Llama-3.1-8B-Instruct
24.07%
20.03%
62.96%
16.67%
9.26%
7.41%
DeepSeek-R1
27.31%
26.77%
12.96%
14.81%
46.30%
35.19%
Qwen3.6-Plus
33.80%
34.98%
37.04%
29.63%
31.48%
37.04%
Gemini 3.1 Pro Preview
37.50%
37.17%
31.48%
44.44%
24.07%
50.00%
TABLE IV: Attack-stage prediction performance and per-label recall. Prep., Exec., Res., and SU denote the preparation, execution, result, and SU labels, respectively. Best and second-best results are highlighted in dark and light green, respectively.
Model
Reasoning Quality
Prediction–Reasoning Consistency
Efficiency
SRP ↑
SCC ↑
SRF1 ↑
TT ↑
TF ↓
FT ⋄
FF ↓
TT/Correct ↑
FT/Wrong ⋄
SRF1/1K Tok. ↑
Avg. Tok.
Qwen3-8B
32.27%
56.05%
40.96%
12.50%
10.65%
19.44%
57.41%
54.00%
25.30%
84.99
481.92
Qwen3-32B
29.26%
52.14%
37.49%
12.50%
13.89%
18.06%
55.56%
47.37%
24.53%
109.33
342.92
Llama-3.1-8B-Instruct
32.65%
58.58%
41.93%
9.26%
14.81%
25.00%
50.93%
38.46%
32.93%
136.11
308.07
DeepSeek-R1
26.73%
51.55%
35.21%
15.74%
11.57%
13.43%
59.26%
57.63%
18.47%
83.15
423.43
Qwen3.6-Plus
28.84%
65.10%
39.97%
17.13%
16.67%
16.20%
50.00%
50.68%
24.48%
88.09
453.76
TABLE V: Reasoning quality, prediction–reasoning consistency, and efficiency across models. Avg. Tok. denotes average reasoning length, and SRF1/1K Tok. denotes SRF1 per 1K output tokens. ⋄ denotes a diagnostic metric with no inherently preferred direction.
Metric
AUROC (95% CI)
AP (95% CI)
ROUGE-L F1
0.620 [0.557, 0.681]
0.612 [0.521, 0.708]
BERTScore F1
0.618 [0.542, 0.688]
0.612 [0.524, 0.700]
SRP
0.901 [0.869, 0.930]
0.908 [0.868, 0.942]
SCC
0.853 [0.813, 0.893]
0.851 [0.789, 0.902]
SRF1
0.918 [0.886, 0.947]
0.933 [0.904, 0.958]
TABLE VI: Comparison of the proposed reasoning metrics with conventional reference-similarity metrics.
Fig. 6: Model-level relationship between attack-stage prediction accuracy and reasoning quality. Each point denotes one model. The orange line shows the least-squares fit, and the yellow band denotes the 95% bootstrap confidence interval of the fitted trend. The Pearson correlation is weak ( r=0.15 , 95% CI [−0.62,0.77] ).
Fig. 7: Label-wise prediction–reasoning consistency. Each subfigure groups TT/TF/FT/FF outcomes by the ground-truth stage label y , following the definitions in Section IV-D .
Fig. 8: Conditional failure patterns across models. The columns report failure rates for non- execution windows predicted as execution (Non-Exec as Exec), result windows predicted as non- result labels (Result as Non-Result), SU windows predicted as concrete stages (Unclosed as Stage), and concrete-stage windows predicted as SU (Stage as Unclosed).
Fig. 9: Ablation results for evidence organization and source coverage. (a) compares audit-only and multi-source settings under original and compressed evidence views, with flat and scaffolded variants for compressed multi-source evidence. (b) compares source combinations under the scaffolded compressed view. Invalid Ref. denotes Invalid Evidence Reference; Ori. and Comp. denote uncompressed and compressed evidence views, respectively; Flat and Scaf. denote flat and scaffolded presentations; L and T denote application logs and network traffic. SRP † and SCC † are source-availability diagnostics computed against the full-view reference checklist.
Temperature
Decoding top- k
Qwen3-8B
Llama-3.1-8B-Instruct
n=1
n=3
n=5
n=1
n=3
n=5
0.0
Off
23.15 / 19.27
—
—
24.07 / 20.03
—
—
0.3
Off
18.06 / 15.53
22.22 / 18.86
21.76 / 18.69
21.30 / 19.92
18.52 / 17.11
18.06 / 16.16
0.7
Off
22.69 / 19.36
21.30 / 17.66
24.54 / 19.87
21.30 / 20.92
21.76 / 20.25
20.37 / 18.54
0.7
20
23.61 / 20.00
22.69 / 18.58
25.00 / 20.50
21.30 / 21.32
21.76 / 20.29
20.37 / 18.56
TABLE VII: Effects of sampling and voting settings on attack-stage prediction. Entries are Accuracy / Macro-F1 (%). Here, n denotes the number of answers used for voting. Off denotes decoding top- k=0 (disabled), and — denotes not evaluated.