Vulnerability discovery is becoming an important ability of large language model (LLM) agents: agents that silently miss real defects leave critical software exposed. Rigorously measuring this ability is therefore urgent, but existing benchmarks are gameable through data contamination, score recall against an unknowable vulnerability set, often rely on synthetic bugs, and report a single end-to-end verdict that cannot localize where an agent fails. Vulnerability discovery is a composite ability: an agent must comprehend source code, infer input constraints, construct inputs, execute them, and iteratively correct from feedback. We recast its measurement as an input-prediction task with a closed, deterministic ground truth: using coverage-guided fuzzing, we mine deep target branches in real-world C/C++ programs and ask an agent to predict an input that drives execution to a given branch. This decomposes discovery into three task modes over 22 real-world C/C++ programs spanning 15 domains. Open-loop and Feedback-enabled share 60 fixed-target task instances across 16 of these codebases (13 domains), testing input construction without and with a distance oracle to isolate code comprehension from feedback-driven correction. Online Arena instead removes the predefined target and scores path exploration by coverage gain on a separate, partially overlapping pool of 11 programs; agents collectively confirmed 13 distinct bugs across six programs. Evaluating 15 default-effort model-scaffold configurations, the best reaches only 55.0% pass rate in the Feedback-enabled mode, and the mean across seven paired Claude Code configurations is 36.4% with feedback versus 19.3% without. Decomposing failures, we find constraint inference, not navigation, is the dominant bottleneck. We release SWE-Test with a turnkey evaluation environment.
Figures & tables
Benchmark
Real software
Predefined target
Open discovery
Input grading
Controlled feedback
Anti- contam.
Long horizon
CyberGym
✓
✓
–
–
✓
–
–
CVE-Bench
✓
✓
–
–
✓
–
–
CyBench
–
✓
–
–
✓
–
–
BountyBench
✓
–
✓
–
✓
–
–
AgentCyberRange
✓
–
✓
–
✓
–
–
SWE-Test
✓
✓
✓
✓
✓
✓
✓
Table 1: Comparison with selected vulnerability-discovery benchmarks.
Figure 1: Composition of SWE-Test’s 60 fixed-target input-prediction tasks (Open-loop and Feedback-enabled modes), covering 16 real-world C/C++ programs across 13 application domains.
Figure 2: Task example contrasting Open-loop and Feedback-enabled modes on a shared target branch.
Figure 3: Online Arena execution and verification pipeline.
Figure 4: Construction pipeline for Open-loop and Feedback-enabled tasks.
Claude Code
Terminus 2
Codex
Kimi Code
Qwen Coder
Qwen3.8-Max
55.0 / 50.0 %
–
–
–
–
DeepSeek-V4-Pro
35.0/8.3%
20.7/2.0%
–
–
–
GLM-5.1
23.3/1.7%
1.7/1.7%
–
–
–
GLM-5.2
41.7 / 26.7 %
10.0/6.9%
–
–
–
Kimi-K2.6
35.0/8.3%
3.7/3.6%
–
16.7/3.8%
–
Kimi-K3
35.0/25.0%
–
–
–
–
Table 2: Feedback-enabled/Open-loop pass rates by model and scaffold.
Program
Qwen3.8- Max-Preview
Kimi- K2.6
GLM- 5.2
DeepSeek- V4-Pro
Qwen3.7- Max
CPython
0.22
0.21
0.23
0.33
0.19
HarfBuzz
0.22
0.11
0.19
0.20
0.18
libjpeg
0.14
0.14
0.14
0.22
0.14
libpcap
0.60
0.60
0.59
0.58
0.56
libpng
0.27
0.27
0.27
0.26
0.27
libxml2
0.19
0.19
0.19
0.16
0.18
Table 3: Normalized coverage reward in 12-hour Online mode.
Program
Qwen3.8- Max-Preview
Kimi- K2.6
GLM- 5.2
DeepSeek- V4-Pro
Qwen3.7- Max
CPython
2
0
0
0
0
HarfBuzz
1
1
0
1
0
libjpeg
0
0
0
0
0
libpcap
1
0
0
0
0
libpng
0
0
1
0
0
libxml2
0
0
0
0
0
Table 4: Per-model counts of confirmed bugs from 12-hour Online runs.
Model
Nav.
Constr.
Pass
Feedback-enabled
Qwen3.8-Max
10
17
33
DeepSeek-V4-Pro
12
27
21
GLM-5.1
11
35
14
GLM-5.2
10
25
25
Kimi-K2.6
19
20
21
Table 5: Outcome counts under Claude Code on identical targets ( n=60 ).
Figure 5: Navigation and target-constraint satisfaction by mode.
Task family
Nav. fail (%)
Constr. fail (%)
Pass (%)
Ada URL parsing
0
100
0
Bloaty
14
86
0
cURL
14
14
72
FreeType
14
29
57
HarfBuzz
16
77
7
JQ
0
100
0
Table 6: Failure outcomes by task family (paired Claude Code configurations).
Effort
n
Nav.
Constr.
Pass
Rate
Feedback-enabled
low
60
13
32
15
25.0%
medium
60
10
33
17
28.3%
high
60
14
23
23
38.3%
Open-loop
low
60
26
30
4
6.7%
Table 7: Reasoning-effort ablation; n denotes valid trials.
Large language models (LLMs) now support automated software security tasks, including vulnerability discovery and proof-of-concept (PoC) generation. Existing benchmarks do not faithfully evaluate LLMs in real-world bug hunting scenarios because they rely on fuzzing harnesses, target-specific descriptions, or vulnerability-reproduction tasks. We present SEC-bench Pro, a benchmark for measuring agent bug hunting on critical, high-complexity software systems. This work discloses reports with concrete PoC inputs and links fixes into reproducible tasks through a three-phase pipeline for vulnerability collection, environment reconstruction, and oracle-based validation. We instantiate SEC-bench Pro with 183 validated vulnerabilities across V8 and SpiderMonkey, including a V8 subset with more than $1.5 million in cumulative Google Vulnerability Reward Program awards. These instances span memory-safety, sandbox, JIT, and race-condition bugs under browser-grade and runtime-grade execution conditions. Our evaluation shows that coding agents with frontier models remain below 40% success on both evaluated engines. The open-weight Kimi-K2.6 baseline reaches 11.7% on V8, while the strongest frontier configuration reaches 32.0% on V8 and 38.8% on SpiderMonkey. ClaudeCode and Codex solve complementary instance sets, and their two-agent union reaches 37.9% on V8 and 48.8% on SpiderMonkey. SEC-bench Pro provides robust environments for assessing LLM-based security agents and exposes limitations in long-horizon bug hunting tasks.
Evaluating the ability of large language models (LLMs) to discover software bugs is increasingly important. Existing benchmarks typically evaluate this capability by asking the model to generate a proof-of-concept input that triggers a predefined target vulnerability. However, this setup may overlook valid crashes discovered by the model when they do not match the predefined target. As a result, the evaluation may not reflect the model's real capability. We present FuzzingBrain-Bench, a benchmark for assessing AI models' ability to discover bugs in open-source software. Models are given an open-source project and a sanitizer-instrumented harness in a self-contained Docker image. Their goal is to generate inputs that trigger as many distinct crashes as possible through the harness. A model's performance on each challenge is scored based on the number of distinct crash signatures it produces, capped at a predefined maximum and weighted by a difficulty coefficient. FuzzingBrain-Bench V1 consists of 77 challenges drawn from 43 open-source projects, with 36 C, 32 C++, and 9 Java/JVM challenges. We evaluate Claude Haiku 4.5, Claude Sonnet 4.6, and Claude Opus 4.8 on the full benchmark. Claude Opus 4.8 performs best, triggering crashes in 60 of 77 challenges and achieving a score of 196 out of 579. None of the three models triggers a crash in 13 challenges. The FuzzingBrain-Bench corpus and harnesses are publicly available at https://github.com/fuzzingbrain/FuzzingBrain-Bench.
Automated vulnerability detection is a fundamental task in software security, yet existing learning-based methods still struggle to capture the structural dependencies, domain-specific vulnerability knowledge, and complex program semantics required for accurate detection. Recent Large Language Models (LLMs) have shown strong code understanding ability, but directly prompting them with raw source code often leads to missed vulnerabilities or false alarms, especially when vulnerable and benign functions differ only in subtle semantic details. To address this, we propose VulTriage, a triple-path context augmentation framework for LLM-based vulnerability detection. VulTriage enhances the LLM input through three complementary paths: a Control Path that extracts and verbalizes AST, CFG, and DFG information to expose control and data dependencies; a Knowledge Path that retrieves relevant CWE-derived vulnerability patterns and examples through hybrid dense--sparse retrieval; and a Semantic Path that summarizes the functional behavior of the code before the final judgment. These contexts are integrated into a unified instruction to guide the LLM toward more reliable vulnerability reasoning. Experiments on the PrimeVul pair test set show that VulTriage achieves state-of-the-art performance, outperforming existing deep learning and LLM-based baselines on key pair-wise and classification metrics. Further ablation studies verify the effectiveness of each path, and additional experiments on the Kotlin dataset demonstrate the generalization ability of VulTriage under low-resource and class-imbalanced settings. Our code is available at https://github.com/vinsontang1/VulTriage
Wenxin Tang, Xiang Zhang, Junliang Liu +11
Tsinghua University · Henan University · Dalian Maritime University +8