Investigating cybersecurity incidents requires collecting and analyzing evidence from multiple log sources, including intrusion detection alerts, network traffic records, and authentication events. This process is labor-intensive: analysts must sift through large volumes of data to identify relevant indicators and piece together what happened. We present a RAG-based system that performs security incident analysis through targeted query-based filtering and LLM semantic reasoning. The system uses a query library with associated MITRE ATT&CK techniques to extract indicators from raw logs, then retrieves relevant context to answer forensic questions and reconstruct attack sequences. We evaluate the system with eight LLM configurations on malware traffic incidents and a multi-stage Active Directory attack. We find that LLMs have different performance and tradeoffs, with Claude Sonnet 4 achieving 94% and DeepSeek V3 achieving 89% average recall across 17 malware scenarios, while DeepSeek costs 15× less than Claude per analysis, and locally-deployed Llama 3.1:70b achieves 81% recall at zero per-query cost. Attack step detection on the Active Directory scenario reaches 100% precision and up to 96% recall with an enumeration prompt. These results demonstrate that combining targeted query-based filtering with RAG-based retrieval -- confirmed essential by ablation studies -- enables accurate, cost-effective security analysis within LLM context limits.
Figures & tables
Figure 1. System architecture. Security Context Extraction queries the SIEM data, filtering the traffic to expose indicators of compromise. RAG-LLM Analysis embeds this data, retrieves relevant context for each question, and prompts an LLM to generate an incident report. System architecture. Security Context Extraction queries the SIEM data, filtering the traffic to expose indicators of compromise. RAG-LLM Analysis embeds this data, retrieves relevant context for each question, and prompts an LLM to generate an incident report.
Scenario
Cl.
Deep-
Ll.
Cisco
GPT
GPT
GPT
GPT
Seek
8B
4
4o
5m
5.2
apr21isc
92
79
79
71
85
85
85
85
burnincandle
100
93
84
77
70
70
70
70
dec21isc
100
100
94
81
75
75
75
75
dirty_rat
89
75
71
43
50
64
50
50
easy_123
100
100
92
100
75
75
75
92
Table 1. Incident Analysis Recall (%) by Scenario and Model
LLM
Recall on set-valued
Multi-chunk
questions, hard (%)
cite rate, soft (%)
Claude Sonnet 4
89
95
DeepSeek V3
78
88
Llama 3.1:70b
56
76
Cisco 8B
46
36
GPT-4
52
38
Table 2. Cross-source integration on set-valued questions.
LLM
k=1
k=3
k=5
k=7
k=14
Claude Sonnet 4
39
72
86
94
94
DeepSeek V3
35
71
80
89
89
Llama 3.1:70b
40
68
78
81
78
Cisco 8B
43
62
72
71
67
GPT-4
46
60
64
67
66
GPT-4o
46
59
65
69
64
Table 3. Context-Size Ablation: Recall (%) by k
LLM
RAG (k=7)
No-RAG
Logs Seen by LLM
Claude Sonnet 4
100
50
158 (4.3%)
DeepSeek V3
100
50
104 (2.8%)
GPT-4
69
38
22 (0.6%)
GPT-4o
69
38
22 (0.6%)
GPT-5-mini
56
38
22 (0.6%)
GPT-5.2
69
38
22 (0.6%)
Table 4. RAG vs No-RAG Baseline: Recall (%) on the fake_auth scenario
LLM
Recall
Avg time/
Avg time/
In
Out
Cost/
(%)
scenario
question
($/M)
($/M)
Anlys.
Claude Sonnet 4
94
100s
13.1s
3.00
15.00
$0.12
DeepSeek V3
89
110s
14.4s
0.28
0.42
$0.008
GPT-4
67
18s
2.4s
3.00 †
12.00 †
$0.11 †
GPT-4o
69
19s
2.6s
2.50
10.00
$0.09
GPT-5-mini
64
18s
2.4s
0.25
2.00
$0.01
Table 5. Timing vs Cost per LLM
Figure 2. Cost vs. Recall Across LLM models. Cost vs. Recall Across LLM models, showing that Claude Sonnet 4 stands in the High Recall High Cost Quadrant, most of the GPT models lie in the Low Recall High Cost and the DeepSeek and LLama 3.1 70B land in the High Recall Low Cost quadrant.
LLMs
15:30–35
15:35–40
15:40–46
Avg Recall
( R=11 )
( R=8 )
( R=7 )
(%)
Claude Sonnet 4
11
8
6
96
DeepSeek V3
11
8
6
96
Llama 3.1:70b
9
6
6
81
Cisco 8B
8
7
6
81
GPT-4o
5
7
6
69
Table 6. AD attack reconstruction - steps correctly identified per window. Per-timestamp recall counts and overall average across LLMs. R = reference step count.
School of Airspace Science and Engineering, Shandong University, Shandong 264209, China. · School of Computer Science, University of Technology Sydney, Ultimo, NSW 2007, Australia. · an independent researcher in Braddon, ACT 2612, Australia.