CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat

Bangya Liu; Jianhong Tu; Yueyang Pan; Hanhong Zhao; Zhe Ye; Hongwei Li; Saastha Vasan; Katherine Lambert

cs.CROct 13, 2026

CyberArena: Benchmarking AI Agents in Live Symmetric Attack-and-Defense Combat

Authors: Bangya Liu, Jianhong Tu, Yueyang Pan, Hanhong Zhao, Zhe Ye, Hongwei Li, Saastha Vasan, Katherine Lambert,

Organizations: 2077AI Open Source Foundation · University of Wisconsin–Madison · UC Santa Cruz · Massachusetts Institute of Technology ·

Abstract

AI agents are demonstrating increasingly strong offensive and defensive cybersecurity capabilities in the real world. Current benchmarks mostly evaluate agents in isolation against static targets. Real cyber operations, however, are dynamic and coupled: security postures constantly change as attackers and defenders adapt to each other in real time. Static benchmarks quickly saturate and miss an agent's ability to attack and defend simultaneously under live pressure. To address this gap, we present CyberArena, a dynamic, symmetric adversarial benchmark built around multi-round attack-and-defense combat. In isolated sandboxes, two agents compete under identical rules, acting concurrently as both attackers and defenders over continuously changing services. This converts static targets into evolving, semi-open challenges, directly testing an agent's speed and precision in discovering, exploiting, and patching vulnerabilities. To sustain the benchmark's longevity, an automated agentic pipeline continuously generates, reviews, and validates new environments. Furthermore, because pairwise contests are notoriously hard to quantify, we introduce a dynamical model that maps match dynamics into unified, multidimensional scores across offensive, defensive, and domain-specific capabilities. Evaluating 12 frontier LLM agents across 40 challenges in 466 matches, CyberArena separates model capabilities where single-agent baselines hit a ceiling. The trajectories read like a fight rather than an exam: defenders patch one hole and miss the next and certify repairs on stale evidence, while capable agents think a move ahead, planting backdoors for later rounds and, in one case, writing their own packet sniffer to watch the opponent. Beyond a benchmark, CyberArena serves as an adaptable, scalable experimental platform for future research on cybersecurity agents.

Figures & tables

Benchmark	Scope	# Instances / Projects	Attack	Defense	Live opp.
NYU CTF Bench ( Shao et al., 2024 )	CTF	200 / -	Exploit	✗	✗
Cybench ( Zhang et al., 2025b )	CTF	40 / -	Exploit	✗	✗
CVE-Bench ( Zhu et al., 2025 )	Real-world	40 / 26	Exploit	✗	✗
BountyBench ( Zhang et al., 2025a )	Real-world	40 / 25	Exploit	Patch	✗
SEC-bench ( Lee et al., 2025 )	Real-world	200 / 29	PoC	Patch	✗
ExploitGym ( Wang et al., 2026a )	Real-world	898 / 163	Exploit	✗	✗

Table 1: Comparing CyberArena with existing cybersecurity benchmarks for AI agents. We evaluate benchmarks across scope, instance scale, supported roles, and operational dynamics. Attack distinguishes exploit execution from fault-triggering proof-of-concept (PoC) tests, and Defense indicates patching support. CyberArena’s counts are the 45 objectives across the 40 challenges of the ranking reference set. Across all existing works, CyberArena is the first to evaluate autonomous agents against live, adaptive opponents with concurrent attack and defense.

	Model	Harness	Score	90% interval	Games	$/game
1	GPT-5.6 Sol	Codex	78.6	[76.8, 80.2]	83	17.7
2	Claude Opus 5	Claude Code	73.8	[72.1, 75.4]	62	30.6
3	GLM-5.3	OpenCode	68.5	[67.2, 69.9]	78	4.7
4	Gemini 3.7 Flash	Gemini CLI	67.1	[65.9, 68.2]	76	7.3
5	Grok 4.6	Grok CLI	65.5	[63.9, 66.9]	86	5.4
6	Claude Opus 4.8	Claude Code	65.0	[63.7, 66.1]	73	13.0

Table 2: CyberArena standings and costs. We list posterior mean expected scores (0–100%) and their model-conditional 90% credible intervals across the 12 evaluated entrants. We also report valid games per entrant and each entrant’s mean model-token cost per valid game in US dollars.

Category	Recorded behavior	Games
Partial patch	A guard leaves another observed route to the capability open.	21%
Unverified fix claim	A claim of repair success exceeds what the recorded tests establish.	23%
Sleeps through rounds	The agent spends active round time in deliberate sleeps.	14%
Over-blocking patch	Authorization repairs reject legitimate application workflows.	8%
Fix left undeployed	The agent defers deploying an identified defensive repair.	9%
Stale round state	The agent reuses an earlier target or proof after objective rotation.	8%

Table 3: Agent-error patterns from the automated trajectory audit. Percentages give the share of the 466 audited games with at least one finding in a category, and categories overlap.

Game	Behavioral evidence
e0dfaa2ff108	On RCEaaS Shell Jail, GPT-5.6 Sol captures at 0.7 minutes, deploys a patch at 1.2 minutes, and attempts repairs on additional routes, while Claude Opus 4.8 deploys at 18.9 minutes.
d5ce44f15e00	MiniMax M3’s flight-ID authorization fix precedes GPT-5.6 Sol’s failed replay.
98d7809fac58	An MLflow attacker switches to the trace search endpoint after direct trace access is blocked.
864473292ff4	Qwen3.8 Max deploys a packet sniffer, identifies authenticated opponent traffic, and later rotates the credential.
dea23a986d06	An Openfire plugin remains executable after an entry-route patch is deployed.
e072ff837c4d	An agent cites health checks from before its patch deployment as verification of the patch.

Table 4: Case studies of selected games. We select a subset of games from the campaign and analyze their trajectories manually. The examples illustrate both counter-strategies (packet sniffing, dynamic endpoint evasion, persistent backdoors) and verification errors (inspecting stale logs from before the patch deployment).

Figure 5

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Challenge	Origin	Vulnerability class	Diff.	Mech.	CVE
8-Bit Oracle	saarCTF 2024	Weak Challenge Generation / Authorization Bypass	hard	C, S	–
aiohttp Static Route Traversal	aiohttp	Directory Traversal	medium	F	CVE-2024-23334
Apache Airflow Nested Variable Redaction	Apache Airflow	Secret Redaction	medium	A, R	CVE-2026-32690
Birthdaygram Photo Feed	FAUST CTF 2025	Broken Access Control / Private Image Disclosure	medium	A, R	–
Btx Bildschirmtext	saarCTF 2024	Broken Access Control / Ownership Bypass	medium	A, S	–
Caddy Nextcloud SSO Boundary	Caddy	Forward-auth Header Confusion	hard	A, R	CVE-2026-30851, CVE-2026-52845

Appendix

Table 5: The 40-challenge ranking reference set. Diff. : curator-assigned difficulty. Mech. : vulnerability mechanisms by decreasing weight (A access control, X input and execution, F files and resources, C cryptography and randomness, R parsing, S protocol and state).

Model	Developer	Harness
GPT-5.6 Sol ( OpenAI, 2026b )	OpenAI	Codex ( OpenAI, 2026a )
Claude Opus 5 ( Anthropic, 2026c )	Anthropic	Claude Code ( Anthropic, 2026a )
GLM-5.3 ( Z.ai, 2026 )	Z.ai	OpenCode ( Anomaly, 2025 )
Gemini 3.7 Flash ( Google DeepMind, 2026b )	Google DeepMind	Gemini CLI ( google-gemini, 2026 )
Grok 4.6 ( SpaceXAI, 2026a )	SpaceXAI	Grok CLI ( SpaceXAI, 2026b )
Claude Opus 4.8 ( Anthropic, 2026b )	Anthropic	Claude Code ( Anthropic, 2026a )

Appendix

Table 6: The twelve entrants, in order of expected score. Each model runs in its developer’s command-line coding agent, or in OpenCode for GLM-5.3, MiniMax M3 and HY4 Preview.

Symbol	Meaning	Range
Games and observations
$\mathcal{E}$ , $N$	the entrants and their number
$\mathcal{C}$ , $C$ , $[C]$	the challenges, their number, and $\{1,\dots,C\}$
$g=(i,j,c)$	a game between entrants $i$ and $j$ on challenge $c$
$t$ , $T$	a round and the number of rounds	$T=3$
$Y_{g,i,t}$	whether $i$ captured at least one objective from $j$ in round $t$	$\{0,1\}$

Appendix

Table 7: Symbols of the observation model and the standing.

		Held out by
Model	Rounds of one direction	batch	pair	challenge
M0	independent	$-195.0$	$-189.2$	$-209.3$
M1	exchangeable	$-184.5$	$-182.2$	$-204.8$
M2	round and shared effects	$-164.6$	$-165.3$	$-189.2$
M3	M2 with carry-over	$\mathbf{-158.0}$	$\mathbf{-158.4}$	$\mathbf{-186.1}$
Uniform over the 64 outcomes		$-220.4$	$-220.4$	$-220.4$

Appendix

Table 8: Held-out log score of the two-sided capture sequences of the 53 calibration games (higher is better). All splits hold out whole games, and pair folds group games by challenge and unordered entrant pair.

Parameters	Mean	Covariance or SD
$(\gamma_{2},\gamma_{3},\kappa,\log\tau)$	$(-2.39,-2.16,1.61,-0.16)$	$\Sigma_{\text{cal}}+\mathrm{diag}(0.5^{2},0.5^{2},0.5^{2},0.35^{2})$
$(\omega_{2},\omega_{3},\log\phi)$	$(1.46,2.12,0.17)$	$\Sigma_{\text{cal}}+\mathrm{diag}(0.5^{2},0.5^{2},0.5^{2})$
$\alpha_{c}$ ; $\delta_{c}$	$0.36$ ; $3.15$	SD $2$
$a_{i}$ , $d_{i}$ , $s_{i}$	$0$ for every entrant	SD $1$
$u_{ik}$ , $v_{ik}$ , centered	$0$	marginal SD $0.5$
$\log\lambda_{c}$ contrasts	$0$	SD $1.6$

Appendix

Table 9: Priors for a new campaign. The two dynamics blocks and the intercept means are calibrated on the 53 games (

\Sigma_{\text{cal}}

is each block’s calibrated covariance), while the diagonal additions and all other prior settings use fixed defaults.

AI Agent Security Benchmarks

AI agent security benchmarks measure how tool-using and autonomous systems handle attacks, privacy risks, authorization, and harmful actions.

See all 301 papers

Jun 3, 2026

CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities

AI has the potential to transform cybersecurity by enabling systems that can autonomously detect, analyze, and remediate software vulnerabilities. However, existing cybersecurity evaluations of AI systems are limited in scale or scope, and fail to capture the end-to-end lifecycle of real-world software vulnerability discovery and remediation. To address this gap, we propose CyberGym-E2E, a large-scale and realistic end-to-end cybersecurity benchmark that comprehensively evaluates AI agents' abilities across the full lifecycle of vulnerability discovery, PoC generation, and patch generation. CyberGym-E2E is comprehensive and scalable, as we build an automated, agent-enhanced pipeline for transforming open-source vulnerability data into realistic evaluation environments. Currently, the benchmark consists of 920 real-world vulnerabilities across 139 different open-source projects.

Tianneng Shi, Robin Rheem, Dongwei Jiang +13

1UC Berkeley · 2Johns Hopkins University · 3UC Santa Cruz +1

Automated Program Repair Software Vulnerability Detection AI Agent Security AI Agent Security Benchmarks

Ask PDF

Jun 12, 2026

AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Ranges

Frontier AI systems are increasingly capable of cybersecurity tasks, including codebase inspection, vulnerability detection, and exploitation. However, evaluating their offensive capabilities remains constrained by limited access to open, reproducible, multi-host cyber ranges. Existing public benchmarks capture isolated skills such as CTF solving, vulnerability reproduction, and exploit generation, but often abstract away realistic intrusion workflows: discovering exposed services, gaining a foothold, collecting internal information, and expanding compromise across hosts. This gap makes it difficult to observe emerging risks early, because frontier AI systems are rarely evaluated under realistic attack conditions. We introduce AgentCyberRange, the first open, multi-range infrastructure for measuring autonomous cyber attack capability in realistic cyber ranges. It combines 110 vulnerabilities across 15 real web applications and 8 enterprise-like cyber ranges with 156 internal hosts, plus Cage, a toolchain for execution, orchestration, result collection, and verification. The benchmark covers two core stages: web exploitation, where agents explore exposed applications and validate vulnerabilities, and post exploitation, where agents turn an initial foothold into broader internal compromise. We evaluate six frontier AI systems under matched prompts and budgets. GPT-5.5 with Codex performs best, solving 16.1% of web exploitation tasks and 31.7% of post-exploitation tasks; with more concrete hints, these rates increase to 33.0% and 46.3%. We also observe out-of-benchmark findings, including unknown vulnerabilities in popular projects, and payload mutation that bypasses host defenses. These results show that open cyber-range evaluation is necessary for observing emerging offensive capabilities under realistic and reproducible conditions.

Fengyu Liu, Jiarun Dai, Yihe Fan +11

Fudan University

Cybersecurity AI Agent Security Benchmarks Automated Penetration Testing LLMs for Cybersecurity

Ask PDF

Oct 12, 2026

CWE-bench: Refocusing Agentic Security from Exploitation to Defense

Cybersecurity evaluations of AI agents have focused largely on offense, yet a working exploit does not show that an agent can make software safe. An attacker needs to exploit only one vulnerability; a defender must find and fix all of them. We introduce CWE-bench, a private benchmark of 120 held-out repository audit-and-repair tasks spanning 73 CWE types, eight languages, and all OWASP Top 10 2025 categories. Agents are given a repository to audit and a description of its threat model, but are not told the weakness, where it occurs, or how many vulnerable sites there are. Each patch must block the exploit while preserving original functionality, as checked by a programmatic verifier. We validated the verifier against a panel of agentic judges, manually reviewing disagreements during development and refining the checks. The best agents reach 68% pass@1 and 81% pass@4. We further conduct analyses showing that (i) agents with the same pass@1 differ widely in retry gains and cost, and (ii) no single agent is best across languages or vulnerability categories, and difficulty varies far more by vulnerability category than by language. As the offensive capabilities of agents improve, CWE-bench provides a broad and rigorous measure of whether their defensive capabilities keep pace.

Gonzalo Gonzalez-Pumariega, Ajay Jagannath, Adit Jain +3

AI Agent Security AI Agent Security Benchmarks Software Vulnerability Detection Automated Program Repair

Ask PDF

Abstract

AI Agent Security Benchmarks

Explore similar work

CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilities

AgentCyberRange: Benchmarking Frontier AI Systems in Realistic Cyber Ranges

CWE-bench: Refocusing Agentic Security from Exploitation to Defense