SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents

Daniel Begimher; Cristian Leo; Jack Huang; Pat Gaw; Bonan Zheng

cs.CRApr 13, 2026

SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents

Authors: Daniel Begimher, Cristian Leo, Jack Huang, Pat Gaw, Bonan Zheng

Organizations: Amazon Web Services

Abstract

We present SIR-Bench, a benchmark of 794 test cases for evaluating autonomous security incident response agents that distinguishes genuine forensic investigation from alert parroting. Derived from 129 anonymized incident patterns with expert-validated ground truth, SIR-Bench measures not only whether agents reach correct triage decisions, but whether they discover novel evidence through active investigation. To construct SIR-Bench, we develop Once Upon A Threat (OUAT), a framework that replays real incident patterns in controlled cloud environments, producing authentic telemetry with measurable investigation outcomes. Our evaluation methodology introduces three complementary metrics: triage accuracy (M1), novel finding discovery (M2), and tool usage appropriateness (M3), assessed through an adversarial LLM-as-Judge that inverts the burden of proof -- requiring concrete forensic evidence to credit investigations. Evaluating our SIR agent on the benchmark demonstrates 97.1% true positive (TP) detection, 73.4% false positive (FP) rejection, and 5.67 novel key findings per case, establishing a baseline against which future investigation agents can be measured.

Figures & tables

Attack Category	TP Cases	FP Cases	Total	%
Brute Force	135	59	194	24.4%
Unauthorized Access	186	112	298	37.5%
Misconfiguration	100	75	175	22.0%
Malicious File Execution	54	73	127	16.0%
Total	475	319	794	100%

Table 1: SIR-Bench dataset composition

Panel of Figure 1 — Figure 1: OUAT pipeline: Real incident patterns seed attack simulations in controlled environments. Key findings are extracted, validated by security Subject Matter Experts (SMEs), and transformed into measurable ground truth for the SIR-Bench dataset.

Category	Infrastructure Provisioned	Attack Actions Simulated
Brute Force	IAM users, CloudTrail logging, authentication endpoints	Failed auth attempts, credential stuffing, successful compromise
Unauthorized Access	IAM roles, S3 buckets, cross-account trust relationships	Role assumption chains, S3 enumeration, data exfiltration
Misconfiguration	S3 buckets, security groups, IAM policies	Public exposure, overly permissive policies, unintended access
Malicious File Exec.	EC2 instances, SSM agents, security groups	Malware download, reverse shells, persistence mechanisms

Table 2: OUAT attack simulation components by category

	$M1_{\text{TP}}$	$M1_{\text{FP}}$	$M1_{F_{3}}$	Missed TPs	False Alarms
SIR Agent	97.1%	73.4%	0.942	14/475	85/319

Table 3: Triage accuracy (

M_{1}

) results

Attack Category	$M1_{\text{TP}}$	$M1_{\text{FP}}$	$M1_{F_{3}}$	n
Brute Force	99.3%	61.0%	0.958	194
Unauthorized Access	97.3%	76.8%	0.951	298
Misconfiguration	94.0%	82.7%	0.929	175
Malicious File Execution	96.3%	47.8%	0.891	127
Overall	97.1%	73.4%	0.942	794

Table 4: Triage accuracy by attack category

	Avg. Novel KF	Novel Coverage	Hit 5+ KF	Hit 7+ KF
SIR Agent	5.67	41.9%	68.4%	47.4%

Table 5: Investigation depth (

M_{2}

) results

Attack Category	Novel Coverage	Hit 3+ KF	Hit 5+ KF	Hit 7+ KF
Unauthorized Access	41.9%	75.0%	61.2%	47.9%
Brute Force	35.7%	75.6%	42.2%	23.0%
Misconfiguration	25.2%	54.9%	37.8%	27.0%
Malicious File Execution	18.0%	42.6%	7.4%	1.9%
Overall (TP cases)	41.9%	64.6%	68.4%	47.4%

Table 6: Novel finding coverage by attack category

Cybersecurity

Cybersecurity develops methods to protect digital and cyber-physical systems from attacks, vulnerabilities, data loss, and misuse.

See all 354 papers

SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response
Jul 29, 2026Lehan Wang, Boli Chen, Ruixue Ding +7LLM Agent Evaluation Cybersecurity
Tongyi Lab, Alibaba Group · Alibaba Cloud Computing, Alibaba Group · The Hong Kong University of Science and Technology
APTInvestBench: Evaluating Autonomous APT Investigation under Varying Telemetry
Sep 30, 2026Yu Wang, Shuhao Li, Tao Yin +4Distribution Shift Robustness AI Agent Benchmarks
Zhongguancun Laboratory
Incident-Arena: Getting agents to the last nine of reliability
Sep 30, 2026Andre Fu, Malik Drabla, Leon Liu +5Agent Reliability AI Coding Agents
Abundant AI · Adrenaline AI · Carnegie Mellon University +2

Jul 29, 2026J/K move · Enter open · S save

SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response

Lehan Wang, Boli Chen, Ruixue Ding, Pengjun Xie+6

Tongyi Lab, Alibaba Group · Alibaba Cloud Computing, Alibaba Group · The Hong Kong University of Science and Technology

LLM Agent Evaluation Cybersecurity Digital Forensics AI Agent Security Benchmarks

Open paper AskPDF arXiv

Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities. However, existing cybersecurity benchmarks focus on pre-compromise settings where agents are placed in a clean and idealized environment before an attack occurs. This leaves the post-compromise setting underexplored. To address this gap, we introduce SecRespond, the first benchmark for evaluating LLM agents on the post-compromise incident-response workflow. Given a forensic disk snapshot of a compromised host together with the alerts, vulnerability scans, and baseline checks reported by a host security product, agents are required to produce forensic reports on intrusions, baseline risks, and vulnerability risks, together with a remediation plan. We instantiate this task across 10 cyber ranges, each constructed from a distinct compromised cloud host, spanning 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. We evaluate 23 frontier LLMs on the OpenCode agent harness. Experimental results show that although current agents can reliably uncover the problems exposed by alerts, they struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans, with no model achieving complete detection and remediation on any single range. This reveals a fundamental bottleneck in building agents for real-world incident response. The benchmark is publicly available at https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond.

Abstract

Cybersecurity

Explore similar work

SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response

APTInvestBench: Evaluating Autonomous APT Investigation under Varying Telemetry

Incident-Arena: Getting agents to the last nine of reliability