ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefense

Nancy Lau; Louis Sloot; Jyoutir Raj; Giuseppe Marco Boscardin; Evan Harris; Dylan Bowman; Mario Brajkovski; Jaideep Chawla

cs.CRMar 2, 2026

ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefense

Authors: Nancy Lau, Louis Sloot, Jyoutir Raj, Giuseppe Marco Boscardin, Evan Harris, Dylan Bowman, Mario Brajkovski, Jaideep Chawla,

Organizations: UC Santa Cruz · Carnegie Mellon University · Independent · HUD ·

Abstract

Large language models (LLMs) are increasingly being deployed as software engineering agents that autonomously contribute to repositories. A major benefit these agents present is their ability to find and patch security vulnerabilities in the codebases they oversee. To estimate the capability of agents in this domain, we introduce ZeroDayBench, a benchmark where LLM agents find and patch 22 novel critical vulnerabilities in open-source codebases. We focus our efforts on three popular frontier agentic LLMs: GPT-5.2, Claude Sonnet 4.5, and Grok 4.1. We find that frontier LLMs are not yet capable of autonomously solving our tasks and observe some behavioral patterns that suggest how these models can be improved in the domain of proactive cyberdefense.

Figures & tables

Figure 1: Overview of ZeroDayBench . The LLM agent is provided task file(s) and interacts with the necessary codebase in a dockerized container environment before testing/evaluation.

Software	CVE ID	Type	Vulnerability Description
MLFlow	CVE-2019-7164	SQL Injection	Malicious order_by clauses injected via text() allow arbitrary SQL execution.
MLFlow	CVE-2024-9053	RCE (Deserialization)	Unsafe cloudpickle.loads() in RPC server executes arbitrary code from malicious scorers.
MLFlow	CVE-2021-21300	Command Injection	Unsafe subprocess.run(shell=True) in git cloning allows shell injection via crafted URIs.
MLFlow	CVE-2024-3660	RCE (Deserialization)	Loading models with malicious Keras Lambda layers executes bytecode via marshal.loads() .
MLFlow	CVE-2020-11978	Command Injection	Malicious JSON configs in Airflow DAGs are interpolated into shell commands without sanitization.
Flyte	CVE-2021-37678	RCE (Deserialization)	yaml.unsafe_load() processes configs with Python object instantiation tags.

Table 1: Summary of task examples. Full details can be found in the Appendix.

Figure 2: Pass rates by model and difficulty.

Difficulty	Claude	GPT	Grok
zero-day	12.8%	14.4%	12.1%
cwe	32.9%	32.9%	18.0%
post-exploit	60.7%	43.0%	36.6%
one-day	78.0%	74.6%	44.7%
full-info	95.7%	76.2%	58.8%
Overall	56.0%	48.2%	34.0%

Table 2: Average pass rates by difficulty

Figure 3: Failure types for each model.

	Claude		GPT		Grok
Difficulty	Tool Calls	Cost	Tool Calls	Cost	Tool Calls	Cost
zero-day	45.3	$0.75	54.4	$0.41	39.4	$0.04
cwe	41.6	$0.69	48.1	$0.36	37.9	$0.04
post-exploit	35.2	$0.52	25.5	$0.19	27.9	$0.02
one-day	25.5	$0.41	22.8	$0.17	11.3	$0.01
full-info	24.7	$0.37	20.5	$0.15	11.1	$0.01

Table 3: Average number of tool calls and cost per rollout by difficulty level

Difficulty	Claude	GPT-5.2	Grok
zero-day	0/10 (0%)	4/10 (40%)	6/10 (60%)
cwe	8/10 (80%)	8/10 (80%)	9/10 (90%)
post-exploit	10/10 (100%)	10/10 (100%)	10/10 (100%)
one-day	10/10 (100%)	10/10 (100%)	10/10 (100%)
full-info	10/10 (100%)	10/10 (100%)	10/10 (100%)

Table 4: Pass rates by difficulty for MLFlow CVE-2021-21300.

Difficulty	Claude	GPT	Grok
zero-day	0/10 (0%)	0/10 (0%)	1/10 (10%)*
cwe	5/10 (50%)	0/10 (0%)	0/10 (0%)
post-exploit	6/10 (60%)	0/10 (0%)	2/10 (20%)
one-day	10/10 (100%)	0/10 (0%)	2/10 (20%)
full-info	10/10 (100%)	0/10 (0%)	1/10 (10%)*

Table 5: Pass rates by difficulty for Jenkins CVE-2022-29078.

AI Agent Security Benchmarks

AI agent security benchmarks measure how tool-using and autonomous systems handle attacks, privacy risks, authorization, and harmful actions.

See all 305 papers

SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?
May 26, 2026Hwiwon Lee, Jiawei Liu, Dongjun Kim +3Software Vulnerability Detection Software Security
University of Illinois Urbana-Champaign
CyberForge: Verified Vulnerability Injection at Repository Level for Cybersecurity Agent Training
Aug 6, 2026Amine Lbath, Manan Suri, Aurelien Delaitre +4Automated Program Repair Software Vulnerability Detection
National Institute of Standards and Technology · Université Grenoble Alpes, CNRS · University of Maryland, College Park
Can Agentic Code Review Catch Vulnerabilities Before They Are Merged?
Oct 24, 2026Yihan Yang, Chenyuan Yang, Zijie Zhao +5AI Agent Security Benchmarks Automated Code Review
University of Illinois Urbana-Champaign · Quantum AI & Cyber Security Institute, FPT Corporation; work done during an internship at UIUC.

May 26, 2026J/K move · Enter open · S save

SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?

Hwiwon Lee, Jiawei Liu, Dongjun Kim, Ziqi Zhang+2

University of Illinois Urbana-Champaign

Software Vulnerability Detection Software Security LLM Agent Security AI Agent Security Benchmarks

Open paper AskPDF arXiv

Large language models (LLMs) now support automated software security tasks, including vulnerability discovery and proof-of-concept (PoC) generation. Existing benchmarks do not faithfully evaluate LLMs in real-world bug hunting scenarios because they rely on fuzzing harnesses, target-specific descriptions, or vulnerability-reproduction tasks. We present SEC-bench Pro, a benchmark for measuring agent bug hunting on critical, high-complexity software systems. This work discloses reports with concrete PoC inputs and links fixes into reproducible tasks through a three-phase pipeline for vulnerability collection, environment reconstruction, and oracle-based validation. We instantiate SEC-bench Pro with 183 validated vulnerabilities across V8 and SpiderMonkey, including a V8 subset with more than $1.5 million in cumulative Google Vulnerability Reward Program awards. These instances span memory-safety, sandbox, JIT, and race-condition bugs under browser-grade and runtime-grade execution conditions. Our evaluation shows that coding agents with frontier models remain below 40% success on both evaluated engines. The open-weight Kimi-K2.6 baseline reaches 11.7% on V8, while the strongest frontier configuration reaches 32.0% on V8 and 38.8% on SpiderMonkey. ClaudeCode and Codex solve complementary instance sets, and their two-agent union reaches 37.9% on V8 and 48.8% on SpiderMonkey. SEC-bench Pro provides robust environments for assessing LLM-based security agents and exposes limitations in long-horizon bug hunting tasks.

Abstract

AI Agent Security Benchmarks

Explore similar work

SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?

CyberForge: Verified Vulnerability Injection at Repository Level for Cybersecurity Agent Training

Can Agentic Code Review Catch Vulnerabilities Before They Are Merged?