Retrieval-Augmented LLMs for Security Incident Analysis

Xavier Cadet; Aditya Vikram Singh; Harsh Mamania; Edward Koh; Alex Fitts; Dirk Van Bruggen; Simona Boboila; Peter Chin

cs.CRMar 18, 2026

Retrieval-Augmented LLMs for Security Incident Analysis

Authors: Xavier Cadet, Aditya Vikram Singh, Harsh Mamania, Edward Koh, Alex Fitts, Dirk Van Bruggen, Simona Boboila, Peter Chin,

Organizations: Dartmouth College Hanover, NH, USA · Northeastern University Boston, MA, USA · PUNCH Cyber Analytics Reston, VA, USA

Abstract

Investigating cybersecurity incidents requires collecting and analyzing evidence from multiple log sources, including intrusion detection alerts, network traffic records, and authentication events. This process is labor-intensive: analysts must sift through large volumes of data to identify relevant indicators and piece together what happened. We present a RAG-based system that performs security incident analysis through targeted query-based filtering and LLM semantic reasoning. The system uses a query library with associated MITRE ATT&CK techniques to extract indicators from raw logs, then retrieves relevant context to answer forensic questions and reconstruct attack sequences. We evaluate the system with eight LLM configurations on malware traffic incidents and a multi-stage Active Directory attack. We find that LLMs have different performance and tradeoffs, with Claude Sonnet 4 achieving 94% and DeepSeek V3 achieving 89% average recall across 17 malware scenarios, while DeepSeek costs 15 $\times$ less than Claude per analysis, and locally-deployed Llama 3.1:70b achieves 81% recall at zero per-query cost. Attack step detection on the Active Directory scenario reaches 100% precision and up to 96% recall with an enumeration prompt. These results demonstrate that combining targeted query-based filtering with RAG-based retrieval -- confirmed essential by ablation studies -- enables accurate, cost-effective security analysis within LLM context limits.

Figures & tables

Figure 1. System architecture. Security Context Extraction queries the SIEM data, filtering the traffic to expose indicators of compromise. RAG-LLM Analysis embeds this data, retrieves relevant context for each question, and prompts an LLM to generate an incident report. System architecture. Security Context Extraction queries the SIEM data, filtering the traffic to expose indicators of compromise. RAG-LLM Analysis embeds this data, retrieves relevant context for each question, and prompts an LLM to generate an incident report.

Scenario	Cl.	Deep-	Ll.	Cisco	GPT	GPT	GPT	GPT
		Seek		8B	4	4o	5m	5.2
apr21isc	92	79	79	71	85	85	85	85
burnincandle	100	93	84	77	70	70	70	70
dec21isc	100	100	94	81	75	75	75	75
dirty_rat	89	75	71	43	50	64	50	50
easy_123	100	100	92	100	75	75	75	92

Table 1. Incident Analysis Recall (%) by Scenario and Model

LLM	Recall on set-valued	Multi-chunk
	questions, hard (%)	cite rate, soft (%)
Claude Sonnet 4	89	95
DeepSeek V3	78	88
Llama 3.1:70b	56	76
Cisco 8B	46	36
GPT-4	52	38

Table 2. Cross-source integration on set-valued questions.

LLM	$k{=}1$	$k{=}3$	$k{=}5$	$k{=}7$	$k{=}14$
Claude Sonnet 4	39	72	86	94	94
DeepSeek V3	35	71	80	89	89
Llama 3.1:70b	40	68	78	81	78
Cisco 8B	43	62	72	71	67
GPT-4	46	60	64	67	66
GPT-4o	46	59	65	69	64

Table 3. Context-Size Ablation: Recall (%) by

k

LLM	RAG (k=7)	No-RAG	Logs Seen by LLM
Claude Sonnet 4	100	50	158 (4.3%)
DeepSeek V3	100	50	104 (2.8%)
GPT-4	69	38	22 (0.6%)
GPT-4o	69	38	22 (0.6%)
GPT-5-mini	56	38	22 (0.6%)
GPT-5.2	69	38	22 (0.6%)

Table 4. RAG vs No-RAG Baseline: Recall (%) on the fake_auth scenario

LLM	Recall	Avg time/	Avg time/	In	Out	Cost/
	(%)	scenario	question	($/M)	($/M)	Anlys.
Claude Sonnet 4	94	100s	13.1s	3.00	15.00	$0.12
DeepSeek V3	89	110s	14.4s	0.28	0.42	$0.008
GPT-4	67	18s	2.4s	3.00 †	12.00 †	$0.11 †
GPT-4o	69	19s	2.6s	2.50	10.00	$0.09
GPT-5-mini	64	18s	2.4s	0.25	2.00	$0.01

Table 5. Timing vs Cost per LLM

Figure 2. Cost vs. Recall Across LLM models. Cost vs. Recall Across LLM models, showing that Claude Sonnet 4 stands in the High Recall High Cost Quadrant, most of the GPT models lie in the Low Recall High Cost and the DeepSeek and LLama 3.1 70B land in the High Recall Low Cost quadrant.

LLMs	15:30–35	15:35–40	15:40–46	Avg Recall
	( $R=11$ )	( $R=8$ )	( $R=7$ )	(%)
Claude Sonnet 4	11	8	6	96
DeepSeek V3	11	8	6	96
Llama 3.1:70b	9	6	6	81
Cisco 8B	8	7	6	81
GPT-4o	5	7	6	69

Table 6. AD attack reconstruction - steps correctly identified per window. Per-timestamp recall counts and overall average across LLMs. R = reference step count.

RC=Revoke Cert, RK=Reset KRBTGT, RD=Restart DC,
Window	Blue	LLM	P	R
15:30–35	RC, RK, RD	RC, RK, RD, DU, RP	3/5	3/3
15:35–40	RK, RD, RP, DU	DU, RK, RD	3/3	3/3
15:40–46	RD, RP, DU, RK	RC, DU, RD	2/3	2/3
DU=Disable User, RP=Reset Password. DU covers RP.

Table 7. Defense Recommendations (Claude Sonnet 4).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Source	Size	No. of Logs
jun21isc	ISC	43.1 MB	47,991
oct21isc	ISC	46.7 MB	7,278
apr21isc	ISC	12.9 MB	5,699
dec21isc	ISC	31.4 MB	3,869
may21isc	ISC	17.9 MB	1,544
iced_id	Unit-42	22.3 MB	5,701

Appendix

Table 8. Evaluated Malware Scenario Dataset Statistics

			Claude			DeepSeek			Ollama			Cisco			GPT-4			GPT-4o			GPT-5-mini			GPT-5.2
Dataset	Qs	Ref	N	TP/N	Rec.	N	TP/N	Rec.	N	TP/N	Rec.	N	TP/N	Rec.	N	TP/N	Rec.	N	TP/N	Rec.	N	TP/N	Rec.	N	TP/N	Rec.
apr21isc	2	8	6	4/6	66.5	5	4/5	66.5	6	4/6	66.5	4	2/4	33.5	7	3/7	41.5	7	3/7	41.5	7	3/7	41.5	7	3/7	41.5
burnincandle	3	14	16	14/16	100.0	14	11/14	83.3	12	10/12	62.7	7	5/7	45.7	10	10/10	62.7	10	10/10	62.7	10	10/10	62.7	10	10/10	62.7
dec21isc	3	6	10	6/10	100.0	10	6/10	100.0	9	5/9	83.3	7	5/7	83.3	8	4/8	66.7	8	4/8	66.7	8	4/8	66.7	8	4/8	66.7
dirty_rat	2	6	4	3/4	62.5	5	1/5	12.5	3	0/3	0.0	4	0/4	0.0	11	1/11	25.0	11	1/11	25.0	11	1/11	25.0	11	1/11	25.0
easy_123	1	2	2	2/2	100.0	2	2/2	100.0	1	1/1	50.0	2	2/2	100.0	1	1/1	50.0	1	1/1	50.0	1	1/1	50.0	1	1/1	50.0

Appendix

Table 9. Precision-Recall Analysis: Set-Valued Questions Across All Scenarios (single-answer questions excluded)

Query	MITRE ATT&CK
PowerShell Injection	T1059 – Command and Scripting Interpreter
File Drop	T1204 – User Execution
Certificate Enumeration	T1649 – Steal/Forge Authentication Certificates
Certificate Request	T1649 – Steal/Forge Authentication Certificates
Certificate Issued	T1649 – Steal/Forge Authentication Certificates
Kerberos Authentication	T1558 – Steal/Forge Kerberos Tickets

Appendix

Table 10. Query Library for Active Directory Attack Detection

Cybersecurity

Cybersecurity develops methods to protect digital and cyber-physical systems from attacks, vulnerabilities, data loss, and misuse.

See all 354 papers

Jun 9, 2026

Benchmarking and Exploring the Capabilities of LLMs for Attack Investigations

This paper presents AuditBench, a new benchmark dataset for evaluating the capabilities of LLMs at investigating security-related system audit logs. We design and use this benchmark to explore the performance of LLMs on four log-investigation tasks that incident response teams commonly perform, ranging from triaging alerts generated by detectors to identifying persistence mechanisms on compromised systems. AuditBench consists of system audit logs collected from Linux and Windows machines, and spans over 50 different security investigation scenarios, including both malicious and benign activity. Using our benchmark, we evaluate and analyze the performance of five frontier LLMs at analyzing audit logs for attack investigations. Our analysis illuminates how LLM performance and error profiles vary according to different design choices, such as differences in model size, data representation, prompt construction, and specific investigation tasks. Additionally, we characterize the quality of the explanations produced by LLMs and the types of errors that models make across our benchmark. Collectively, our work provides a foundation for assessing the capabilities of LLMs for investigating security logs, novel insights for practitioners using LLMs in security operations, and important directions for future research.

Aniket Anand, Yiwei Hou, Daniel Fields +4

University of Chicago · USA · University of California, Berkeley +1

LLM Evaluation Cybersecurity Digital Forensics LLMs for Cybersecurity

Ask PDF

Apr 28, 2026

Towards Agentic Investigation of Security Alerts

Security analysts are overwhelmed by the volume of alerts and the low context provided by many detection systems. Early-stage investigations typically require manual correlation across multiple log sources, a task that is usually time-consuming. In this paper, we present an experimental, agentic workflow that leverages large language models (LLMs) augmented with predefined queries and constrained tool access (structured SQL over Suricata logs and grep-based text search) to automate the first stages of alert investigation. The proposed workflow integrates queries to provide an overview of the available data, and LLM components that selects which queries to use based on the overview results, extracts raw evidence from the query results, and delivers a final verdict of the alert. Our results demonstrate that the LLM-powered workflow can investigate log sources, plan an investigation, and produce a final verdict that has a significantly higher accuracy than a verdict produced by the same LLM without the proposed workflow. By recognizing the inherent limitations of directly applying LLMs to high-volume and unstructured data, we propose combining existing investigation practices of real-world analysts with a structured approach to leverage LLMs as virtual security analysts, thereby assisting and reducing the manual workload.

Even Eilertsen, Vasileios Mavroeidis, Gudmund Grov

University of Oslo · Norwegian Defence Research Establishment (FFI) · & University of Oslo

Cybersecurity Agentic Workflows LLMs for Cybersecurity LLM Agents

Ask PDF

Oct 22, 2026

SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence

Large language models (LLMs) have shown strong potential for cybersecurity, leading to growing interest in LLM-based automated cybersecurity log analysis. However, current datasets (1) are limited in scale; (2) rely mainly on single-source evidence; and (3) have only binary or multi-class attack labels, without dense annotations of the reasoning process. To address these limitations, SecLog-Bench has three key features. First, we build an extensible simulation platform that covers 29 attack scenarios, supports repeated execution and the integration of new scenarios, and enables scalable generation of cybersecurity logs with attack-stage labels. Second, we use multi-source logs, including audit logs, application logs, and network traffic, to jointly analyze cyber attacks. Third, we annotate each sample with the essential context for cybersecurity log reasoning and a structured step-by-step evidence chain. We evaluate eight mainstream LLMs on this benchmark for post-detection attack-progress reasoning. Results show that, when provided with scaffolded evidence, current LLMs exhibit modest capability in attack-progress reasoning, with the best model achieving only 44.44% accuracy and 45.07% Macro-F1. Their reasoning remains unreliable; i.e., the highest Stage Reasoning F1 (SRF1) score is only 54.87%. Moreover, even correct predictions may be accompanied by flawed reasoning, and reasoning quality is often misaligned with prediction accuracy.

Yifan Wang, Dongliang Xu, Xiaoyong Li +3

School of Airspace Science and Engineering, Shandong University, Shandong 264209, China. · School of Computer Science, University of Technology Sydney, Ultimo, NSW 2007, Australia. · an independent researcher in Braddon, ACT 2612, Australia.

Cybersecurity LLMs for Cybersecurity Language Model Reasoning Evaluation

Ask PDF

Abstract

Cybersecurity

Explore similar work

Benchmarking and Exploring the Capabilities of LLMs for Attack Investigations

Towards Agentic Investigation of Security Alerts

SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence