FuzzingBrain-Bench V1: Evaluating Open-Ended Bug Discovery by LLMs

Ze Sheng; Aleksandar Kezic; Zhicheng Chen; Jeff Huang

cs.AIAug 25, 2026

FuzzingBrain-Bench V1: Evaluating Open-Ended Bug Discovery by LLMs

Authors: Ze Sheng, Aleksandar Kezic, Zhicheng Chen, Jeff Huang

Organizations: Texas A&M University · University of Novi Sad

Abstract

Evaluating the ability of large language models (LLMs) to discover software bugs is increasingly important. Existing benchmarks typically evaluate this capability by asking the model to generate a proof-of-concept input that triggers a predefined target vulnerability. However, this setup may overlook valid crashes discovered by the model when they do not match the predefined target. As a result, the evaluation may not reflect the model's real capability. We present FuzzingBrain-Bench, a benchmark for assessing AI models' ability to discover bugs in open-source software. Models are given an open-source project and a sanitizer-instrumented harness in a self-contained Docker image. Their goal is to generate inputs that trigger as many distinct crashes as possible through the harness. A model's performance on each challenge is scored based on the number of distinct crash signatures it produces, capped at a predefined maximum and weighted by a difficulty coefficient. FuzzingBrain-Bench V1 consists of 77 challenges drawn from 43 open-source projects, with 36 C, 32 C++, and 9 Java/JVM challenges. We evaluate Claude Haiku 4.5, Claude Sonnet 4.6, and Claude Opus 4.8 on the full benchmark. Claude Opus 4.8 performs best, triggering crashes in 60 of 77 challenges and achieving a score of 196 out of 579. None of the three models triggers a crash in 13 challenges. The FuzzingBrain-Bench corpus and harnesses are publicly available at https://github.com/fuzzingbrain/FuzzingBrain-Bench.

Figures & tables

Panel of Figure 1 — Figure 1: Four generations of evaluating the bug discovery capability of LLMs. Generation 1 and 2 are static without running any code. Generation 3 and 4 are dynamic with harness execution.

Figure 2: Data selection and challenge construction. A finding reached through a fuzzing harness is extracted to four artifacts and built into an image. The reference proof-of-concept is then run three times inside that image: a candidate that does not fault with the same crash signature in all three runs is discarded rather than repaired. After PoC verification, all information is removed from the image.

Figure 3: The benchmark architecture. The agent loop reaches the model over an API and the challenge container over MCP’s stdio transport. The MCP server in the image exposes three tools: setup hands out the task, exec is the only filesystem access and has no network, and run_poc_on_harness is the only route to the instrumented harness binary.

Figure 4: The left panel lists the conditions one PoC run on the harness is judged against; any one of them makes the run a crash. The four examples show two valid crashes, one run that printed nothing, and one that faulted inside the harness.

CWE	Bug type	Challenges
Memory safety		45
CWE-125	out-of-bounds read	23
CWE-476	NULL pointer dereference	6
CWE-416	use after free	5
CWE-787	out-of-bounds write	5
CWE-121	stack-based buffer overflow	3

Table 1: The bug classes the 77 challenges were built from, grouped into memory safety and denial of service (DoS) and other faults.

Figure 5: An example of a signature generated from a sanitizer output. The signature consists of the fault class and up to three relevant function names, joined by bars.

	difficulty	distinct crash signatures
Challenge	coefficient	Opus 4.8	Sonnet 4.6	Haiku 4.5
arrow-01	5	0	0	0
assimp-01	2	4	1	0
avro-01	2	8	9	0
avro-02	1	7	10	2
avro-03	1	2	2	2

Table 2: Overview of the amount of crash signatures found for each challenge in the benchmark, by each of three Anthropic models. Difficulty coefficient is assigned to each challenge according to criteria presented in Table 3

Tier	Classification criteria	$D$	Challenges	Points available
D1	all three models crashed it	1	33	99
D2	two models crashed it, one with $\geq 3$ distinct crash signatures	2	9	54
D3	anything else	3	11	99
D4	one model crashed it, with at most 2 distinct crash signatures	4	11	132
D5	no model crashed it	5	13	195
	total		77	579

Table 3: The five difficulty tiers for the fixed three-model reference panel.

D

is the difficulty coefficient assigned to each challenge in the tier. Points available equals

3D

times the number of challenges in the tier.

Project	Challenges	Description	Confirmed via
arrow	1	columnar format and toolbox for fast data interchange and in-memory analysis	fix commit
assimp	1	loading 3d-file-formats into unified data structures	OSS-Fuzz issue
avro	3	Apache data serialization framework	GitHub PR
binutils	1	collection of binary development tools	Bugzilla
cups	1	standard-based UNIX/Linux printing system	GitHub issue
dtc	1	device tree compiler and utilities for flattened device trees	GitHub issue

Table 4: All 43 open-source projects in FuzzingBrain-Bench. Some projects contribute more than one challenge, each representing a confirmed vulnerability at its buggy commit.

Figure 6: Total benchmark score of each model over the 77-challenge corpus, placed against the maximum achievable score of 579.

Figure 7: Effect of the per-challenge cap. The solid part of each bar is the score that counts, with at most three distinct crash signatures per challenge; the hatched part is what the cap removes.

Figure 8: Challenges crashed at each difficulty tier D1-D5 (tier size in parentheses), shaded by the share of the challenges in each tier crashed.

Figure 9: Points received by each model per difficulty tier. It is worth pointing out that Sonnet performed slightly better than Opus is lower difficulty tier challenges, whereas Opus closed that gap by scoring better in challenges that belong in tiers D3 and D4.

Figure 10: Amount of challenges per turns used, divided into intervals, for each model in the panel.

Model	D1 (33)	D2 (9)	D3 (11)	D4 (11)	D5 (13)	All (77)	Total cost
	average cost per challenge
Opus 4.8	$1.10	$2.20	$3.09	$4.00	$4.54	$2.51	$193.11
Sonnet 4.6	$3.53	$3.25	$3.25	$3.54	$2.52	$3.29	$253.34
Haiku 4.5	$0.60	$0.58	$0.54	$0.54	$0.50	$0.56	$43.42

Table 5: Average cost per challenge for each model, for each difficulty tier. Total cost of entire 77 challenge corpus run for each model.

		average tokens per challenge
Model	Tokens	D1 (33)	D2 (9)	D3 (11)	D4 (11)	D5 (13)	All (77)
Opus 4.8	input (M)	0.83	1.66	2.33	2.74	3.67	1.89
	output (k)	13.2	22.3	28.0	30.2	35.4	22.5
Sonnet 4.6	input (M)	7.26	6.54	6.86	8.06	5.74	6.99
	output (k)	56.2	56.5	50.6	43.2	28.1	49.0
Haiku 4.5	input (M)	4.04	3.43	3.56	3.68	3.53	3.77

Table 6: Average input and output token count for each model, each difficulty tier.

	average episode duration per challenge (s)
Model	D1 (33)	D2 (9)	D3 (11)	D4 (11)	D5 (13)	All (77)
Opus 4.8	260	427	539	716	889	491
Sonnet 4.6	1192	1213	1043	900	633	1040
Haiku 4.5	264	311	277	233	220	259

Table 7: Average duration of an episode in each of the difficulty tiers for each model.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11: Terminal output while an episode is running.

Figure 12: Terminal output once the episode has completed.

Software Engineering Benchmarks

Software engineering benchmarks evaluate how well AI systems understand, build, repair, test, and maintain software in realistic settings.

See all 156 papers

SWE-Test: Benchmarking LLM Vulnerability Discovery via Input Prediction
Sep 5, 2026Yuanxiang Shi, Jiayi Lin, Xuanyong Lin +8Software Vulnerability Detection AI Agent Security Benchmarks
SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?
May 26, 2026Hwiwon Lee, Jiawei Liu, Dongjun Kim +3Software Vulnerability Detection Software Security
University of Illinois Urbana-Champaign
Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking
Jun 27, 2026Balázs Szalontai, Ábel Szauter, Balázs Márton +3Software Engineering Automated Program Repair
Eötvös Loránd University · Faculty of Informatics

Sep 5, 2026J/K move · Enter open · S save

SWE-Test: Benchmarking LLM Vulnerability Discovery via Input Prediction

Yuanxiang Shi, Jiayi Lin, Xuanyong Lin, Liangcai Su+7

Software Vulnerability Detection AI Agent Security Benchmarks LLM Security

Open paper AskPDF arXiv

Vulnerability discovery is becoming an important ability of large language model (LLM) agents: agents that silently miss real defects leave critical software exposed. Rigorously measuring this ability is therefore urgent, but existing benchmarks are gameable through data contamination, score recall against an unknowable vulnerability set, often rely on synthetic bugs, and report a single end-to-end verdict that cannot localize where an agent fails. Vulnerability discovery is a composite ability: an agent must comprehend source code, infer input constraints, construct inputs, execute them, and iteratively correct from feedback. We recast its measurement as an input-prediction task with a closed, deterministic ground truth: using coverage-guided fuzzing, we mine deep target branches in real-world C/C++ programs and ask an agent to predict an input that drives execution to a given branch. This decomposes discovery into three task modes over 22 real-world C/C++ programs spanning 15 domains. Open-loop and Feedback-enabled share 60 fixed-target task instances across 16 of these codebases (13 domains), testing input construction without and with a distance oracle to isolate code comprehension from feedback-driven correction. Online Arena instead removes the predefined target and scores path exploration by coverage gain on a separate, partially overlapping pool of 11 programs; agents collectively confirmed 13 distinct bugs across six programs. Evaluating 15 default-effort model-scaffold configurations, the best reaches only 55.0% pass rate in the Feedback-enabled mode, and the mean across seven paired Claude Code configurations is 36.4% with feedback versus 19.3% without. Decomposing failures, we find constraint inference, not navigation, is the dominant bottleneck. We release SWE-Test with a turnkey evaluation environment.

Abstract

Software Engineering Benchmarks

Explore similar work

SWE-Test: Benchmarking LLM Vulnerability Discovery via Input Prediction

SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?

Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking