Cybersecurity evaluations of AI agents have focused largely on offense, yet a working exploit does not show that an agent can make software safe. An attacker needs to exploit only one vulnerability; a defender must find and fix all of them. We introduce CWE-bench, a private benchmark of 120 held-out repository audit-and-repair tasks spanning 73 CWE types, eight languages, and all OWASP Top 10 2025 categories. Agents are given a repository to audit and a description of its threat model, but are not told the weakness, where it occurs, or how many vulnerable sites there are. Each patch must block the exploit while preserving original functionality, as checked by a programmatic verifier. We validated the verifier against a panel of agentic judges, manually reviewing disagreements during development and refining the checks. The best agents reach 68% pass@1 and 81% pass@4. We further conduct analyses showing that (i) agents with the same pass@1 differ widely in retry gains and cost, and (ii) no single agent is best across languages or vulnerability categories, and difficulty varies far more by vulnerability category than by language. As the offensive capabilities of agents improve, CWE-bench provides a broad and rigorous measure of whether their defensive capabilities keep pace.
Figures & tables
Benchmark
Tasks
CWEs
OWASP 2025
Lang.
Codebase
Vuln. source
Information given to agent
PatchEval ( Wei et al., 2025 )
230
39
7/10
3
Real
Public CVEs
Vulnerability description
CyberGym-E2E † ( Shi et al., 2026 )
920
NR
1/10
2
Real
Public OSS-Fuzz bugs
None
LiveCVEBench ( Luo et al., 2026a )
190
74
9/10
14
Real
Public CVEs
Bug report
ZeroDayBench † ( Lau et al., 2026 )
22
15
4/10
6
Real
Injected
Varies (5 levels)
PatchBench ( Shen et al., 2026 )
213
16
1/10
2
Real
Injected
PoC and sanitizer report
Vul4Py ( Bui et al., 2026 )
100
60
7/10
1
Real
Public CVEs
CVE and file/line hints
Table 1: Comparison of recent vulnerability repair benchmarks. OWASP 2025 counts the OWASP Top 10:2025 categories covered under each category’s official CWE mapping. Codebase indicates whether tasks are built on real open-source repositories or LLM-generated ones. Injected denotes vulnerabilities inserted into a codebase, such as real CVEs transplanted into new repositories. C/C++ and JavaScript/TypeScript count as separate languages. PatchEval counts its 230-task sandboxed subset. NR: not reported. † Some values (in italics) are derived by us: from source-CVE records (ZeroDayBench), sanitizer crash types (CyberGym-E2E), or the 20 most frequent CWEs reported (SecProbe, a lower bound).
Figure 1: CWE-bench evaluation. Each task provides an open-source repository with injected vulnerabilities and an audit instruction that does not name them. The agent audits and patches the repository without Internet access. A patch passes only if both the security verifier ( S ) and the functional verifier ( F ) succeed. The verifiers are validated against a panel of agentic judges, with disagreements reviewed during development to refine the checks.
Figure 2: Composition of the 120-task evaluation set. Left: tasks per programming language. Right: tasks and distinct CWEs per OWASP Top 10:2025 category, with abbreviated category names.
Programmatic
Judge panel
Mean cost
System
Harness
Pass@1
Pass@4
Pass@1
Pass@4
(USD)
Grok 4.7
OpenCode
68%
81%
64%
82%
2.75
Gemini 4 Argon
Antigravity
68%
75%
62%
73%
6.63
GPT-6 Astra
Codex
68%
74%
66%
77%
2.85
Claude Opus 5.5
Claude Code
67%
79%
67%
81%
0.79
Claude Fable 5.1
Claude Code
58%
73%
60%
77%
3.47
Table 2: Programmatic and judge-panel pass@1 and pass@4, with mean API cost per rollout. Scores are rounded percentages.
Figure 3: Outcomes across four attempts and task coverage. Left: for each agent, the number of tasks passed on all four attempts, on one to three, and on none; agents tied at 68% pass@1 are in bold. Right: tasks passed at least once by each of the three tied agents, and their overlap.
Figure 4: Programmatic pass@1 against mean API cost per rollout, using the values in Table 2 . Orange points mark the Pareto frontier: agents for which no agent of equal or lower cost scores higher.
Figure 5: Programmatic pass@1 by language (left) and OWASP Top 10:2025 category (right). C/C++ and JS/TS are grouped. Outlined cells mark the best score in each column.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Criterion
Requirement
Security
The repair closes the target vulnerability.
Functionality
Ordinary, legitimate behavior remains available.
Generalization
The repair blocks attack variants beyond the particular verifier probe.
Integrity
The submission repairs the software rather than manipulating the grading mechanism.
Soundness
The patched codebase builds and runs in the task environment.
Appendix
Table 3: Judge-panel criteria. A judge accepts a repair only if all five criteria are satisfied.
Panel pass
Panel fail
Programmatic pass
2,860
200
Programmatic fail
120
2,100
Appendix
Table 4: Attempt-level agreement between the programmatic verifier and the agentic judge panel.
Category
Tasks
CWEs
A01
Broken Access Control
13
22, 285, 441, 497, 601, 639, 862, 863, 918
A02
Security Misconfiguration
11
16, 310, 489, 611, 614, 776, 942, 1004
A03
Software Supply Chain Failures
10
1104, 1329, 1357, 1395
A04
Cryptographic Failures
13
208, 319, 326, 327, 330, 338, 347, 916
A05
Injection
14
20, 79, 89, 93, 94, 116, 470, 1336
A06
Insecure Design
12
73, 269, 362, 434, 444, 522, 693, 770, 913, 1284
Appendix
Table 5: CWEs in the evaluation set by OWASP Top 10:2025 category.
Figure 6: Predicted chance of solving a task as a function of task difficulty, from a Rasch model fit to the programmatic outcomes in Table 2 . Each curve uses the mean ability of one tier: top (Grok 4.7, Gemini 4 Argon, GPT-6 Astra, Claude Opus 5.5), middle (Claude Fable 5.1, Grok 4.6, DeepSeek V4.1 Flash, Muse Spark 1.3, Hy4 Preview, GPT-6 Sol), and lower (Inkling). The vertical line marks the task of median difficulty. Below, each circle is one task placed by its estimated difficulty; hollow circles are tasks that every agent always or never solved. The shaded region contains the 30 tasks the strongest agent is predicted to solve less than half the time.
Artificial intelligence agent security studies attacks and safeguards for tool-using, persistent, and collaborative systems across digital and physical settings.