Organizations: University of Illinois Urbana-Champaign · Quantum AI & Cyber Security Institute, FPT Corporation; work done during an internship at UIUC.
LLM agents have been increasingly adopted for security-focused code review, but existing benchmarks mostly derive the reviewed changes from their later fixes instead of the real-world bug-introducing commits. In this paper, we present SecPRBench, the first comprehensive benchmark for evaluating agentic security review on validated real-world code changes. More specifically, we have built an automated agentic pipeline to autonomously harvest, build, and validate massive real-world bug-introducing commits for popular software systems. To ensure precise evaluation, we also adopt LLM-as-a-judge to credit the findings matching ground-truth vulnerabilities. To date, we have collected 265 tasks from nine critical systems, including Firefox, V8, and OpenSSL with millions of lines of code. Across eight frontier models, recall ranges from 32.5% to 78.1%, with Grok 4.6 and GPT-6 sol ranking at the top. Our study also shows that most misses are not simply oversights: in 68% of them, the agent reports on a file that holds the flaw but argues for a different mechanism. Security-focused harnesses also behave differently: OpenAI's Codex Security lowers recall by 8.3 points, mostly by abstaining, while Anthropic's Defending Code harness raises it by 7.7 points, at a higher cost.
Figures & tables
Figure 1: One SecPRBench case (Wasmtime CVE-2026-34946; diff and message paraphrased, findings illustrative ). The flaw depends on unchanged code outside the diff; the LLM judge credits a finding only if it explains the target’s root cause and that cause is still live in the reviewed code.
Table 1: How SecPRBench differs from related benchmarks. Only SecPRBench uses complete vulnerability-introducing changes (no hint, full repo) and pairs each with its repaired version.
Figure 2: Benchmark construction and validation. From evidence withheld from evaluated agents, a construction agent writes G and recovers ci ; Δf adds the later fix (agent-ported if needed); labels are accepted by mechanical checks, including replay on Sp , Si , Sr , or by human adjudication.
Figure 3: SecPRBench by project: (a) bug classes, (b) fixing dates, and (c) time from introduction to fix, grouped by fixing half-year. Each dot is a case; colors denote fixing half-years in (b) and (c). Duration panels share a log scale.
Table 2: Results on 265 tasks (rates in %); darker is higher within a column. Hit@1: first finding detects the target; unresolved ranks count as none or all first. Rej./Empty: rejected/empty reports; Flagged: positive target verdicts. Δ Findings: change in mean findings per scan from the buggy change to its fixed companion ( ↓ fewer, ↑ more). Effort: mean output tokens, estimated cost, and minutes per scan.
Figure 4: Results on the 265 buggy tasks and fixed companions. (a) Targets found by the three leading agents. (b) Misses: a finding in a root-cause file for another mechanism (%), findings only elsewhere, no finding, or no root-cause file opened. (c) Mean findings per buggy and fixed scan; Nonempty: fixed companions with any finding.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Stage
Count
Raw source records, all rounds
> 7,000
Triaged candidates
768
Selected cases
265
Validated cases: mechanical
217
Validated cases: human adjudication
48
Appendix
Table 3: Collection and validation counts. Raw records include overlapping reports and repeated records of the same issue. Selected cases are accepted through executable checks (mechanical) or additional human adjudication.
Project
Collector input
Security signal
curl
Advisory index and machine-readable records
Project-maintained advisory information
FFmpeg
Trac, advisories, CVEs, and Git history
Advisory or commit evidence of a concrete security failure
Firefox
Bugzilla records and comments
Security-bug flags and critical/high/moderate security labels
FreeType
OSS-Fuzz-linked commits and GitLab issues
Source-based evidence of memory errors or malicious-font crashes
OpenSSL
OpenSSL advisory data and GitHub issues/PRs
Upstream ratings and source-based security screening
Redis
GitHub Security Advisories and OSV
Published advisories with security-fix references
Appendix
Table 4: Sources and security-relevance signals used by each project’s collectors. All projects additionally require a traceable code change suitable for benchmark task construction.
Project
Cases
Mechanical
Human
curl
26
23
3
FFmpeg
61
53
8
Firefox
27
18
9
FreeType
15
12
3
OpenSSL
29
28
1
Redis
16
12
4
Appendix
Table 5: Validation routes for the 265 evaluated cases. Mechanical: cases accepted through recorded executable checks; Human: additional cases accepted by adjudication. Each case contributes two review inputs.
Information
Agent
Constructor/evaluator
Review diff, commit message
Yes
Yes
Task source snapshot
Yes
Yes, with exact revision recorded
Target security record
No
Yes
Later fixing commit/diff
No, except in a fixed companion
Yes
Reference findings and annotated root-cause locations
No
Yes
Buggy/fixed label and case identifier (e.g., CVE or bug ID)
No
Yes
Appendix
Table 6: Information boundary. Reference information is withheld from the reviewing agent.
Audited group
Opus 5.5
Sonnet 4.6
Judge disagreements
21/25
4/25
Agreement controls
25/25
25/25
Appendix
Table 7: Agreement with our manual adjudications.
Class
n
Opus
Sonnet
Grok
GLM
MiniMax
GPT-6s
GPT-5.6
GPT-6l
Codex
Memory safety
125
85.6
68.8
84.0
85.6
40.0
82.4
69.6
68.8
60.8
Logic error
86
61.6
44.2
69.8
62.8
18.6
67.4
43.0
44.2
33.7
Integer overflow
44
68.2
63.6
84.1
70.5
45.5
79.5
45.5
45.5
43.2
Concurrency
10
60.0
50.0
50.0
60.0
0.0
60.0
40.0
30.0
20.0
Appendix
Table 8: Recall (%) by reference bug class; n is the row denominator. Abbreviations: Opus 5.5, Sonnet 5, Grok 4.6, GLM 5.3, MiniMax M3, GPT-6 sol (GPT-6s), GPT-5.6 sol, GPT-6 luna (GPT-6l), and Codex Security.
Agent
Misses
(A) Never opened
(B) No finding
(C) Only elsewhere
(D) Same file, other mechanism
Claude Opus 5.5
69
5.8
7.2
2.9
84.1
Claude Sonnet 5
108
5.6
7.4
11.1
75.9
Grok 4.6
58
0.0
8.6
12.1
79.3
GLM 5.3
67
0.0
1.5
6.0
92.5
MiniMax M3
179
1.1
29.1
8.4
61.5
GPT-6 sol
63
0.0
22.2
15.9
61.9
Appendix
Table 9: Scanned misses by category (% of each row’s misses). Categories A–D are mutually exclusive; totals may differ slightly from 100% due to rounding. The pooled row includes the eight direct-review agents.
Mean findings per scan
Agent
Buggy
Fixed
Fixed nonempty (%)
Claude Opus 5.5
2.6
2.2
85.3
Claude Sonnet 5
2.2
2.1
92.8
Grok 4.6
3.1
3.0
92.5
GLM 5.3
3.6
3.4
97.4
MiniMax M3
2.2
2.4
75.8
Appendix
Table 10: Report volume across 265 completed pairs per agent. Fixed nonempty: companions with at least one finding. No fixed companion receives a positive target verdict.
Arm
Recall (%)
Mean findings
Median $
Median min.
Anthropic harness, Opus 5.5
81.5
8.4
3.65
7.6
Direct Opus 5.5, same tasks
73.8
2.5
0.35
1.1
Appendix
Table 11: Anthropic’s reference harness and direct-review Opus 5.5 on the same 260 completed buggy tasks. Findings, nominal cost, and time are per scan; five assigned pairs are missing.
Figure 5: Direct-review recall by time: (a) all cases, (b) shared project–bug-class groups with fixed weights, and (c) within-model cutoff differences with conditional 95% cluster-bootstrap intervals. Headers give sample sizes.
Large language models (LLMs) are increasingly deployed in automated code-review systems, where their approvals can determine which code is merged into shared repositories. However, it is unclear whether review agents can detect vulnerability-introducing code when an attacker controls both the code change and the persuasive Pull Request (PR) narrative designed to mask it. We introduce SEVRA-BENCH (Social Engineering of Vulnerabilities in Review Agents), a benchmark that measures how often a review agent approves such adversarial PR s. Each PR in SEVRA-BENCH is built from a historical commit that fixed a vulnerability. We automatically reverse that fix to extract the original vulnerable code, and submit the resulting code change as a PR wrapped in one of 15 social-engineering framings. To test review-agent resilience to narrative manipulation, these framings vary dimensions such as supporting evidence, conveyed urgency, signals of prior approval, and appeals to authority. SEVRA-BENCH evaluates a retained challenge split of roughly 1000 adversarial PRs drawn from publicly disclosed vulnerability fixes across the top 10 entries of the MITRE's 2025 most dangerous software weaknesses. Evaluating 8 review agents against this benchmark, we reveal that review agents are susceptible to narrative manipulation, exposing a significant gap in security capabilities.
Rui Melo, Riccardo Fogliato, Sean Zhou +2
1Carnegie Mellon University · 2Microsoft Core AI · 3Independent Researcher +1
Large language models (LLMs) now support automated software security tasks, including vulnerability discovery and proof-of-concept (PoC) generation. Existing benchmarks do not faithfully evaluate LLMs in real-world bug hunting scenarios because they rely on fuzzing harnesses, target-specific descriptions, or vulnerability-reproduction tasks. We present SEC-bench Pro, a benchmark for measuring agent bug hunting on critical, high-complexity software systems. This work discloses reports with concrete PoC inputs and links fixes into reproducible tasks through a three-phase pipeline for vulnerability collection, environment reconstruction, and oracle-based validation. We instantiate SEC-bench Pro with 183 validated vulnerabilities across V8 and SpiderMonkey, including a V8 subset with more than $1.5 million in cumulative Google Vulnerability Reward Program awards. These instances span memory-safety, sandbox, JIT, and race-condition bugs under browser-grade and runtime-grade execution conditions. Our evaluation shows that coding agents with frontier models remain below 40% success on both evaluated engines. The open-weight Kimi-K2.6 baseline reaches 11.7% on V8, while the strongest frontier configuration reaches 32.0% on V8 and 38.8% on SpiderMonkey. ClaudeCode and Codex solve complementary instance sets, and their two-agent union reaches 37.9% on V8 and 48.8% on SpiderMonkey. SEC-bench Pro provides robust environments for assessing LLM-based security agents and exposes limitations in long-horizon bug hunting tasks.
Large language models (LLMs) are increasingly being deployed as software engineering agents that autonomously contribute to repositories. A major benefit these agents present is their ability to find and patch security vulnerabilities in the codebases they oversee. To estimate the capability of agents in this domain, we introduce ZeroDayBench, a benchmark where LLM agents find and patch 22 novel critical vulnerabilities in open-source codebases. We focus our efforts on three popular frontier agentic LLMs: GPT-5.2, Claude Sonnet 4.5, and Grok 4.1. We find that frontier LLMs are not yet capable of autonomously solving our tasks and observe some behavioral patterns that suggest how these models can be improved in the domain of proactive cyberdefense.
Nancy Lau, Louis Sloot, Jyoutir Raj +6
UC Santa Cruz · Carnegie Mellon University · Independent +2