Design Principles for the Construction of a Benchmark Evaluating Security Operation Capabilities of Multi-agent AI Systems

Yicheng Cai; Mitchell John DeStefano; Guodong Dong; Pulkit Handa; Peng Liu; Tejas Singhal; Peiyu Tseng; Winston Jen White

cs.CRMar 30, 2026

Design Principles for the Construction of a Benchmark Evaluating Security Operation Capabilities of Multi-agent AI Systems

Authors: Yicheng Cai, Mitchell John DeStefano, Guodong Dong, Pulkit Handa, Peng Liu, Tejas Singhal, Peiyu Tseng, Winston Jen White

Organizations: Cyber Security Laboratory Pennsylvania State University, University Park

Abstract

As Large Language Models (LLMs) and multi-agent AI systems are demonstrating increasing potential in cybersecurity operations, organizations, policymakers, model providers, and researchers in the AI and cybersecurity communities are interested in quantifying the capabilities of such AI systems to achieve more autonomous SOCs (security operation centers) and reduce manual effort. In particular, the AI and cybersecurity communities have recently developed several benchmarks for evaluating the red team capabilities of multi-agent AI systems. However, because the operations in SOCs are dominated by blue team operations, the capabilities of AI systems & agents to achieve more autonomous SOCs cannot be evaluated without a benchmark focused on blue team operations. To our best knowledge, no systematic benchmark for evaluating coordinated multi-task blue team AI has been proposed in the literature. Existing blue team benchmarks focus on a particular task. The goal of this work is to develop a set of design principles for the construction of a benchmark, which is denoted as SOC-bench, to evaluate the blue team capabilities of AI. Following these design principles, we have developed a conceptual design of SOC-bench, which consists of a family of five blue team tasks in the context of large-scale ransomware attack incident response.

Figures & tables

Panel of Figure 1 — Figure 1: Observed blue team activity dependency for Task Fox.

Ring	Pts	Meaning
Bullseye	3	Correct conclusion + evidence-backed justification
		( $\geq$ 2 concrete evidence objects where applicable).
Inner	2	Correct conclusion but missing one key justification element
		(weak correlation, thin evidence, or vague tie).
Outer	1	Directionally plausible but underspecified
		(e.g., “multi-host activity increasing” without anchors).

Table 1: Ring-based scoring model for Task Fox outcomes.

Panel of Figure 2 — Figure 2: Observed blue team activity dependency for Task Goat.

Field	Value / Interpretation
EventID	7 (Sysmon ImageLoad event).
Provider.Name	Microsoft-Windows-Sysmon .
Provider.Guid	5770385F-C22A-43E0-BF4C-06F5698FFBD9 .
Version	Event schema version ( 3 in this instance).
Level	Severity or verbosity level ( 4 ).
Task	Subcategory associated with this event ( 7 ).

Table 2: Example fields extracted from a Sysmon Event 7 record using Chainsaw.

Outcome/Action	Description	Available Points
Relevant Data Sources		15 points total
Irrelevant Data Source	Identifies a data source that is not part of the large-scale attack.	-2 points each
Inaccurate Relationship	Incorrectly identifies that information correlates between two or more data sources.	-1 points each
Threat Graph		35 points total
Node Accuracy
Incorrect Artifact	Identifies artifact (i.e., IP Address, Process ID) unrelated to large-scale attack.	-1 point each

Table 3: Scoreboard Rubric.

Figure 3: Sample of Ground Truth Threat Graph

{ "stage_timestamp": "01/01/2025, 00:00:00 AM", "containment_actions": [ "Isolate ENG-WKS-12 and ENG-WKS-17 from the corporate LAN", "Block outbound SMB and HTTPS traffic from ENG-SRV-05 to 185.199.110.0/24" ], "action_targets": { "hosts": ["ENG-WKS-12", "ENG-WKS-17", "ENG-SRV-05"], "ips": [], "subnets": ["185.199.110.0/24"], "user_accounts": [], "user_groups": [], "services": [], "other": [] }, "reasoning_evidence": "Isolation and outbound blocking are selected to maximize security impact while minimizing operational disruption: both workstations are non-critical endpoints and blocking ENG-SRV-05 outbound traffic preserves service availability while halting lateral movement. Current data shows ENG-WKS-12 and ENG-WKS-17 exhibit rapid encryption and unauthorized C2 connections. ENG-SRV-05 is initiating anomalous SMB flows to a malicious subnet. If containment is delayed, ransomware will likely propagate across engineering hosts and enable further data exfiltration through ENG-SRV-05." }

Table 7

Level	Points	Description
Fully correct	20	All actions exactly match the ground truth.
Partially correct	10	Actions address part of the ground truth but are incomplete or partially misapplied.
Incorrect or missing	-20	Actions are completely wrong, absent, or mostly incorrect, increasing operational risk and potential consequences.

Table 8

Level	Points	Description
Fully supported	40	Reasoning is clear, logical, and fully supported by evidence from the provided data sources.
Partially supported	20	Reasoning is somewhat clear but incomplete or partially unsupported by evidence.
Unclear or unsupported	-20	Reasoning is unclear, illogical, or unsupported.

Table 9

AI Agent Security Benchmarks

AI agent security benchmarks measure how tool-using and autonomous systems handle attacks, privacy risks, authorization, and harmful actions.

See all 305 papers

ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testing
Sep 8, 2026Yi Ting Shen, Kentaroh Toyoda, Alex LeungLLM Safety Benchmarks Language Model Safety Evaluation
Vulcan Research, AIFT Singapore
Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps
Apr 21, 2026Alankrit Chona, Igor Kozlov, Ambuj KumarAI Agent Security Benchmarks LLMs for Cybersecurity
Simbian AI
CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat
Oct 13, 2026Bangya Liu, Jianhong Tu, Yueyang Pan +19AI Agent Security Benchmarks Cybersecurity
2077AI Open Source Foundation · University of Wisconsin–Madison · UC Santa Cruz +8

Sep 8, 2026J/K move · Enter open · S save

ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testing

Yi Ting Shen, Kentaroh Toyoda, Alex Leung

Vulcan Research, AIFT Singapore

LLM Safety Benchmarks Language Model Safety Evaluation Adversarial Attacks on LLMs LLM Red Teaming

Open paper AskPDF arXiv

Automated red-team attacks and blue-team defenses for large language models (LLMs) are advancing quickly. However, attackers and defenders are built and tested in isolation, and the resulting scores are hard to trust. To tackle this, we present ACEA (Adversarial Co-Evolution Arena), a platform that connects a pluggable red-team adapter and a pluggable blue-team adapter to a shared target LLM and scores their attack and defense rates with an LLM judge. ACEA contributes four components. First, a pluggable, model-agnostic arena. Any red or blue project connects over a minimal HTTP protocol, which we call the ACEA Standard Adapter Protocol (ASAP). It can be written in any language, and a project that exposes nothing but the protocol is a full participant. Second, an evaluation methodology built for adversarial rounds. Seeding the target with canonical secrets gives verifiable ground truth that separates real leakage from hallucination. We also send each attack to the target even when the defense blocks it, which measures the attack's raw potency independently of whether it was stopped. Together these yield a per-round decomposition of attack strength and defense effectiveness. Third, a real-time, game-style visualization with a detailed end-of-battle report that localizes each failure. The evaluation thus becomes an actionable signal for improving a red or blue project. Fourth, an optional in-context improvement loop that turns each round's outcome into advisory hints for the next. An adapter can then adapt across rounds without keeping state, provided it reads the hints. We describe the design of ACEA and the metrics through which red and blue teams are scored head to head.

Abstract

AI Agent Security Benchmarks

Explore similar work

ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testing

Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps

CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat