PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix

Marco Paina; Mark Niklas Müller; Niels Mündler-Sasahara; Martin Vechev

cs.SEOct 26, 2026

PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix

Authors: Marco Paina, Mark Niklas Müller, Niels Mündler-Sasahara, Martin Vechev

Organizations: ETH Zurich

Abstract

Proactive bug discovery is an important software-engineering problem. Even though coding agents are already widely deployed in this domain, their capabilities remain understudied. In particular, the challenging task of finding all bugs that human maintainers would repair across an entire repository has not been studied yet. To close this gap, we introduce PROBE, a benchmark consisting of all 2,457 bugs fixed over the course of 18 months across 20 diverse GitHub repositories, together with a method for matching candidate findings to this partial reference set. We evaluate coding agents across frontier models and find they all achieve a recall of below $21\%$ and a precision of less than $9\%$ with respect to the reference set. While stronger models increase recall substantially from $8\%$ to $17\%$ , they do not improve precision. Surprisingly, despite the low precision, LLM and human evaluators still judge $85\%$ of findings valid. Crucially, we find that found bug distributions are similar across agents but differ substantially from those of bugs fixed by maintainers. The practical relevance of bug discovery and the low recall of frontier models make this benchmark a valuable resource for both model users and developers.

Figures & tables

Figure 1: Overview of PROBE. (1) We snapshot a repository at

T_{0}

. (2a) We mine the maintainers’ commits in the reference window

T_{0}\to T_{1}

, extract all bugs they fix, and keep those already present at

T_{0}

as the reference set. (2b) The bug finder we are benchmarking audits the

T_{0}

snapshot. (3) Reported bugs are matched against the reference set to compute recall and precision.

Stage	Count	$\Delta$
Commits	13,254	–
Bug-Fix Commits	4,296	$-67.6\%$
Candidate Bugs	4,763	$+10.9\%$
Anchored Candidates	3,345	$-29.8\%$
Present Candidates	2,501	$-25.2\%$
Deduplicated Bugs	2,457	$-1.8\%$

Table 1: Commits and candidate bugs remaining after each stage of the bug collection pipeline.

\Delta

is the change relative to the preceding stage.

Figure 3: Fix-position density observed (gray bars) and fitted based on the first half of each repository’s fix-commit sequence.

Model	Effort	Recall (%)	Precision (%)	F1 (%)	Findings	Cost
DeepSeek V4.1 Flash	high	$15.2$	8.1	10.5	5,019	$73
GLM 5.3 Flash	high	$15.8$	$7.0$	$9.7$	6,029	$120
GPT-5.4	medium	$11.0$	$6.9$	$8.5$	4,204	$655
GPT-5.6 Terra	medium	$9.1$	$7.6$	$8.3$	3,093	$224
Gemini 3.5 Flash	medium	$9.8$	$6.4$	$7.8$	3,867	$493
Muse Spark 1.3	high	$15.9$	$4.8$	$7.4$	8,372	$164

Table 3: Recall, precision, and F1 score w.r.t. the reference set, sorted by F1, with the effort sweeps of GPT-5.6 Sol and GPT-5.6 Luna last, from highest to lowest reasoning effort. Cells for Muse Spark 1.3 and GPT-5.6 Luna (medium) are means over 5 and 10 runs, respectively.

Figure 4: F1 scores over API cost. GPT-5.6 Luna (blue) and Sol (orange) across reasoning efforts. Reruns are faint markers.

Figure 5: Bug validity assessed by the LLM panel over precision w.r.t. the reference set, with 95% confidence intervals. For GPT-5.6 Luna and Muse Spark 1.3, both axes use their first run.

Figure 6: Jensen–Shannon distance of the bug-type distribution of each agent run to the other runs and of the human reference set to all runs.

	Human		Agents
Category	$n$	%	$\bar{n}$	%	Gap (pp)	Ratio
Wrong arithmetic or comparison expression	243	9.9	916	18.5	$+8.6$	1.86
Unhandled degenerate value	222	9.1	674	13.9	$+4.8$	1.53
Incorrect regular expression pattern	47	1.9	245	4.7	$+2.8$	2.47
Missing state reset or restoration	57	2.3	242	4.9	$+2.6$	2.11
Wrong data source read	57	2.3	224	4.5	$+2.1$	1.91

Table 4: Bug types most over- (top) and under-represented (bottom) among agent findings relative to the human reference set, by gap in percentage points (pp). Agent counts (

\bar{n}

) and shares are means over one run of each of the 14 configurations.

Figure 7: Per-category overlap of repeated GPT-5.6 Luna runs compared to human-fixed bugs.

Figure 8: Share of reference bugs that other models find, conditioned on how many of 9 Luna runs find the bug. The dashed line picks bugs at random within each type; the orange line is a held-out tenth Luna run.

Dataset	Model	Method	Recall (%)	Precision (%)	F1 (%)	Findings	Cost
Full	Claude Sonnet 5	Vanilla	$0.6$	17.4	$1.2$	86	$94
		Passes	$5.3$	$8.0$	$6.4$	1,656	$487
		Full harness (ours)	7.9	5.8	6.7	3,472	$421
	GPT-5.6 Luna	Vanilla	$0.9$	32.3	$1.7$	65	$1
		Passes	$5.8$	$8.3$	$6.8$	1,826	$18
		Hound	$3.2$	$5.0$	$3.9$	1,371	$129

Table 5: Comparison of our bug finding harness against a single prompt (Vanilla), the same pass decomposition but with a minimal prompt (Passes), and prior methods: Hound ( Mueller, 2025 ) and agentic-pbt ( Maaz et al., 2025 ) . Recall, precision and F1 are in %.

Figure 9: Historical precision and recall for findings reported in at least

n

of 10 (Luna) or 5 (Muse Spark) runs, from

n=1

(union) to all runs (intersection). Faint curves use only the first two or five runs.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Repository	Domain	Files	Passes	kLOC	Tests	Commits	Fix com.	Bugs
tobymao/sqlglot	SQL parsing	70	16	49.9	646	2,848	979	512
prefecthq/prefect	Workflow orchestration	420	70	119.5	11,831	3,465	910	429
py-pdf/pypdf	PDF manipulation	52	14	35.7	562	562	190	165
kozea/weasyprint	HTML/CSS rendering	65	19	30.5	3,053	315	196	115
fonttools/fonttools	Font tooling	308	69	122.1	4,372	318	108	113
networkx/networkx	Graph algorithms	283	44	112.5	5,646	481	93	84

Appendix

Table 6: The 20 repositories of PROBE. Files and kLOC count the audited non-test source files ( Appendix B ), and Passes the audit passes they are split into ( Section 3 ). Tests counts the tests that pass at the snapshot in our Docker image (for echarts, its unit tests only). Commits and Fix com. (bug-fixing commits) are counted over the 18-month reference window, and Bugs are the reference bugs they yield.

Field	Description
repo	GitHub repository.
file	Primary bug location file.
line_start/end	Primary bug location line numbers.
title	Short bug title.
description	What is wrong, its trigger and consequence.
severity	Impact, from low to critical .

Appendix

Table 7: Fields of a reference bug record.

Figure 10: Calendar density of mined bug-fixing commits. Top: the share of all 4,265 commits in each month; the dashed line is uniform activity over 18 months. Bottom: monthly shares normalized separately within each repository (each row sums to 100%). Fixing activity varies across repositories and over time.

Family	Parameter	Full window	First half
Exponential	$\beta$	0.00112	0.00190
Weibull	$a$	0.926	0.933
	$\lambda$	1,264	763
Lognormal	$\mu$	15.6	15.5
	$\sigma$	3.77	3.82

Appendix

Table 8: Fitted parameters of the three fix-position families. Full-window estimates use all 2,445 bugs; first-half estimates use 1,396.

	5 years	10 years	$\infty$
Exponential	5,195	7,197	13,030
Weibull	5,326	7,471	15,089
Lognormal	5,884	9,389	747,859

Appendix

Table 9: Estimated number of

T_{0}

bugs fixed within different horizons.

Signal	Inputs	Tokens / model	Score	Threshold
line overlap	file , line_start , line_end	line ranges in the same file	Jaccard	0.02
full-text embedding	title , description , first 1,000 characters of code_snippet	all-MiniLM-L6-v2	cosine	0.65
description embedding	title , description	all-MiniLM-L6-v2	cosine	0.57
snippet tokens	code_snippet	words a of $\geq 2$ characters	IDF-weighted Jaccard c	0.60
identifiers	description , code_snippet	words a of $\geq 3$ characters	IDF-weighted Jaccard c	0.10
API calls	code_snippet	called names b	IDF-weighted Jaccard c	0.22

Appendix

Table 10: The six pairwise similarity signals that pre-filter finding pairs before the LLM duplicate judge.

Model	Effort	Recall (%)	Precision (%)	F1 (%)	Findings	Cost
DeepSeek V4.1 Flash	high	$17.5$	11.7	14.0	1,172	$15
GLM 5.3 Flash	high	$18.7$	$10.4$	$13.3$	1,428	$25
GPT-5.4	medium	$12.5$	$10.3$	$11.3$	942	$138
GPT-5.6 Terra	medium	$10.6$	$10.9$	$10.7$	761	$46
Gemini 3.5 Flash	medium	$10.7$	$9.8$	$10.2$	867	$104
Muse Spark 1.3	high	$20.6$	$8.0$	$11.6$	1,921	$35

Appendix

Table 11: Recall, precision and F1 against the reference set on the five repositories Claude Opus 5 audited (755 reference bugs), for every configuration of Table 3 , in its order. Cells for Muse Spark 1.3 and GPT-5.6 Luna (medium) are means over 5 and 10 runs, respectively, and cost covers the five repositories.

Repository	Bugs	DeepSeek V4.1 Flash (high)	GLM 5.3 Flash (high)	GPT-5.4 (medium)	GPT-5.6 Terra (medium)	Gemini 3.5 Flash (medium)	Muse Spark 1.3 (high, 5 runs)	Claude Sonnet 5 (medium)	GPT-5.6 Sol (high)	GPT-5.6 Sol (medium)	GPT-5.6 Sol (low)	GPT-5.6 Luna (xhigh)	GPT-5.6 Luna (high)	GPT-5.6 Luna (medium, 10 runs)	GPT-5.6 Luna (low)	Claude Opus 5 (medium)	Mean
sqlglot	512	$3.9$	$4.5$	$2.0$	$1.8$	$1.6$	$3.2$	$0.8$	8.2	$3.3$	$2.7$	$5.5$	$3.1$	$1.1$	$0.8$	–	3.0
prefect	429	$12.6$	$13.5$	$10.5$	$6.8$	$11.4$	$15.2$	$8.2$	17.9	$13.5$	$8.9$	$16.3$	$12.4$	$6.4$	$5.1$	–	11.3
pypdf	165	$17.0$	$22.4$	$9.1$	$8.5$	$11.5$	$27.8$	$12.7$	$24.2$	$23.6$	$16.4$	$21.2$	$14.5$	$11.4$	$10.3$	30.9	16.5
weasyprint	115	$10.4$	$10.4$	$7.0$	$6.1$	$6.1$	$9.7$	$4.3$	16.5	$13.9$	$6.1$	$15.7$	$7.0$	$4.3$	$1.7$	$9.6$	8.5
fonttools	113	$28.3$	$27.4$	$19.5$	$20.4$	$17.7$	$29.0$	$15.9$	31.9	$30.1$	$23.9$	$28.3$	$22.1$	$14.6$	$8.8$	–	22.7
networkx	84	$28.6$	$32.1$	$19.0$	$15.5$	$20.2$	$27.6$	$13.1$	35.7	$32.1$	$26.2$	$31.0$	$23.8$	$13.1$	$9.5$	$32.1$	23.4

Appendix

Table 12: Recall (%) against the reference set per repository, for every configuration of Table 3 and Claude Opus 5. The shaded italic rows pool the bugs of one language. For configurations run several times, each cell is the mean over the runs. The shaded Mean column averages the configurations of Table 3 , counting each repeated configuration once and excluding Claude Opus 5. Bold marks each row’s best configuration.

Repository	DeepSeek V4.1 Flash (high)	GLM 5.3 Flash (high)	GPT-5.4 (medium)	GPT-5.6 Terra (medium)	Gemini 3.5 Flash (medium)	Muse Spark 1.3 (high, 5 runs)	Claude Sonnet 5 (medium)	GPT-5.6 Sol (high)	GPT-5.6 Sol (medium)	GPT-5.6 Sol (low)	GPT-5.6 Luna (xhigh)	GPT-5.6 Luna (high)	GPT-5.6 Luna (medium, 10 runs)	GPT-5.6 Luna (low)	Claude Opus 5 (medium)	Mean
sqlglot	15.3	$12.9$	$8.3$	$11.7$	$9.2$	$7.2$	$4.7$	$13.1$	$7.9$	$14.6$	$11.9$	$13.4$	$8.6$	$8.9$	–	10.6
prefect	$10.2$	$8.5$	10.3	$8.2$	$10.1$	$6.4$	$7.7$	$9.4$	$7.8$	$9.1$	$9.2$	$8.9$	$8.2$	$8.7$	–	8.8
pypdf	$25.2$	$23.5$	$18.8$	$20.3$	$22.1$	$21.4$	$24.4$	$18.1$	$22.2$	$26.2$	$19.2$	$18.6$	$25.7$	34.7	$24.8$	22.9
weasyprint	$6.3$	$5.8$	$6.7$	$7.4$	$6.9$	$4.7$	$4.6$	$5.8$	$6.4$	$6.0$	8.7	$6.6$	$5.8$	$3.6$	$4.6$	6.1
fonttools	$7.1$	$5.6$	$6.1$	$7.6$	$5.6$	$3.9$	$4.9$	$5.0$	$6.0$	7.8	$5.4$	$5.6$	$6.2$	$4.5$	–	5.8
networkx	6.9	$6.0$	$4.9$	$4.6$	$6.4$	$3.8$	$5.4$	$4.8$	$4.8$	$6.5$	$5.3$	$5.0$	$4.7$	$4.6$	$5.3$	5.3

Appendix

Table 13: Precision (%) against the reference set per repository, for every configuration of Table 3 and Claude Opus 5. The shaded italic rows pool the findings of one language. For configurations run several times, each cell is the mean over the runs. The shaded Mean column averages the configurations of Table 3 , counting each repeated configuration once and excluding Claude Opus 5. Bold marks each row’s best configuration.

Model	Effort	Confirmed	Rejected	Abstained	Validity (%)	95% CI
GPT-5.4	medium	94	5	1	95	[89, 98]
GPT-5.6 Sol	medium	92	5	3	95	[88, 98]
DeepSeek V4.1 Flash	high	93	6	1	94	[87, 98]
GPT-5.6 Terra	medium	89	7	4	93	[86, 97]
GPT-5.6 Luna	medium	85	10	5	89	[81, 95]
GLM 5.3 Flash	high	84	10	6	89	[81, 95]

Appendix

Table 14: The panel’s verdicts on the 100 sampled findings of one run per configuration. Validity is the confirmed share of the decided verdicts, with its 95% Clopper–Pearson interval. For GPT-5.6 Luna (medium) and Muse Spark 1.3, the run is the first of their repeats. Bold marks the most confirmed and fewest rejected findings and the highest validity.

	Panel
Author	Confirmed	Rejected	Abstained	Total
Confirmed	103	0	0	103
Rejected	9	19	6	34
Unsure	35	5	3	43
Total	147	24	9	180

Appendix

Table 15: The author’s blind verdict (rows) against the panel’s (columns) on the 180 findings of the validity sample graded by both.

Model	Effort	Run	Matched	$D$	$p$	$p_{\mathrm{Holm}}$
DeepSeek V4.1 Flash	high		373	0.076	0.008	0.204
GLM 5.3 Flash	high		388	0.042	0.329	1.000
GPT-5.4	medium		270	0.039	0.692	1.000
GPT-5.6 Terra	medium		223	0.044	0.677	1.000
Gemini 3.5 Flash	medium		240	0.067	0.157	1.000
Muse Spark 1.3	high	1	387	0.044	0.282	1.000

Appendix

Table 16: Fix-rank permutation tests for every run, grouped as in Table 3 ; Run numbers the repeated runs of GPT-5.6 Luna (medium) and Muse Spark 1.3. Column

p

is unadjusted;

p_{\mathrm{Holm}}

adjusts for all 27 tests. Bold denotes

p<0.05

in each column; no result remains significant after Holm correction.

Model	Knowledge cut-off	Source
Gemini 3.5 Flash	January 2025	Google (2026)
GPT-5.4	31 August 2025	OpenAI (2026b)
Claude Sonnet 5	January 2026	Anthropic (2026b)
GPT-5.6 Luna, Terra, Sol	16 February 2026	OpenAI (2026c) ; OpenAI (2026d) ; OpenAI (2026e)
Claude Opus 5	May 2026	Anthropic (2026a)
GLM 5.3 Flash	not published (released 26 August 2026)	Z.ai (2026)

Appendix

Table 17: Knowledge cut-offs of the evaluated models as published by their vendors. For the three models without a published cut-off, we give their release date instead.

Category	Definition
Missing case coverage
Missing branch for a supported input variant	Inline conditional logic handles some supported kinds, syntaxes or shapes of input but omits another equally supported variant, so it is processed by the wrong path.
Missing rejection of invalid input	Malformed, out-of-range, duplicated or otherwise unsupported input is silently accepted and processed instead of being validated and refused at the boundary.
Unhandled degenerate value	Code fails to special case a boundary value that is legal input, such as an empty collection, zero, a single element, or an absent or null field, and then indexes, divides or dereferences it.
Unimplemented advertised feature	A documented option, method, mode or format is exposed but its behaviour is absent or a stub, so the request is silently ignored or handled by an unrelated path.
Missing dispatch or lookup table entry	A registry, mapping, keyword list, transform table or alias set lacks an entry for a supported case, so lookup falls through to a generic or absent handler.

Appendix

Table 18: The 33 bug types and their definitions, grouped by family (continued in Table 19 ).

Category	Definition
State, shared data and lifecycle
Missing state reset or restoration	Flags, counters, cursors, module globals or cached bookkeeping are left over from a previous operation or are not restored when an operation exits, including exceptional exits without a finally.
Colliding key construction	A keyed store is built unsafely, either because the identifier used for caching, indexing, deduplication or grouping lacks a delimiter or a distinguishing component so distinct items collide, or because a plain object or inherited property lookup is used as the container so keys such as proto or constructor corrupt or leak results.
Mutation of shared input data	Code mutates data that others alias, either by modifying an argument, option object, cached entry or otherwise shared structure instead of copying it, or by modifying or re-consuming a live view or single pass iterator while it is being traversed, so unrelated callers see changed data or items are skipped, duplicated or lost.
Missing cache invalidation	A change to underlying inputs is not propagated to dependents, either because a memoised or cached value is never invalidated or because a dependency, subscription or change notification needed by an observer, effect or rendering pipeline is never registered or emitted, so later reads observe stale data.
Missing resource release	Listeners, subscriptions, timers, buffers, sockets, processes or graphics objects are created but never unregistered, cancelled, closed or deleted, so they accumulate or fire after teardown.

Appendix

Table 19: The 33 bug types and their definitions, grouped by family (continued from Table 18 ).

	Human		Agents
Category	$n$	%	$\bar{n}$	%	Range	Gap (pp)	Ratio
Wrong arithmetic or comparison expression	243	9.9	916	18.5	16.9–20.8	$+8.6$	1.86
Unhandled degenerate value	222	9.1	674	13.9	10.7–19.6	$+4.8$	1.53
Incorrect regular expression pattern	47	1.9	245	4.7	3.1–5.7	$+2.8$	2.47
Missing state reset or restoration	57	2.3	242	4.9	3.5–5.9	$+2.6$	2.11
Wrong data source read	57	2.3	224	4.5	3.3–5.8	$+2.1$	1.91

Appendix

Table 20: Bug-type distribution of the human reference set and of coding agent findings, for all categories, sorted by gap. Agent counts (

\bar{n}

) and shares are means over one run of each of the 14 configurations; Range gives the lowest and highest share of a single run. Gap is the agent share minus the human share in percentage points; Ratio is the agent share over the human share.

		Human		Agents
Level	Needed to observe the failure	$n$	%	%	Range
Ordinary use	Any execution of the affected path with common inputs fails or gives a wrong result.	105	4.3	6.1	4.6–7.8
A specific input	A particular value, shape or edge case is needed, such as an empty or absent value, an unusual variant, malformed data or a boundary.	1,815	74.3	83.9	79.2–86.3
A specific environment	A particular platform, runtime or dependency version, locale, toolchain or type checker is needed.	265	10.8	4.7	3.7–6.3
Interaction or inspection	The failure shows only in rendered output, a user interface or a document, or through a user’s sequence of actions, and a person has to look at it.	92	3.8	0.7	0.3–1.0
Timing, concurrency or load	Concurrent execution, a particular ordering of asynchronous events, a long-running process or large inputs are needed.	167	6.8	4.6	3.2–6.3

Appendix

Table 21: Trigger levels of the human reference set and of coding agent findings. Agent shares are means over one run of each of the 14 configurations; Range gives the lowest and highest share of a single run.

Model	$k$	$\chi^{2}$	df	$p$	$\sqrt{\chi^{2}/N}$
Muse Spark 1.3, high: 5 runs, 15,233 distinct findings
Binomial	1	22,197.0	3	$<10^{-99}$	1.207
Beta-binomial	2	318.0	2	$8.9\times 10^{-70}$	0.144
Stable-beta process	2	2.1	2	0.357	0.012
GPT-5.6 Luna, medium: 10 runs, 7,472 distinct findings
Binomial	1	$1.7\times 10^{6}$	8	$<10^{-99}$	14.908

Appendix

Table 22: Goodness of fit to the number of distinct findings reported in exactly

j

r

runs, conditional on

j\geq 1

k

is the number of fitted parameters and

N

the number of distinct findings.

Model	Parameters	Pool size
Muse Spark 1.3, high: 5 runs, 15,233 distinct findings
Binomial	$q=0.493$	15,761
Beta-binomial	$a=0.010^{\dagger},\ b=0.612$	—
Stable-beta process	$\alpha=7{,}772,\ c=0.172,\ \sigma=0.263$	$\infty$
GPT-5.6 Luna, medium: 10 runs, 7,472 distinct findings
Binomial	$q=0.348$	7,577

Appendix

Table 23: Fitted parameters of the models in Table 22 and the pool size each implies. † marks a parameter at its bound; the pool size of that Beta-binomial is set by the bound and omitted. The latent pool of the stable-beta process is infinite.

AI Coding Agents

AI coding agents use language models and software tools to plan, edit, test, and maintain code with varying levels of autonomy.

See all 329 papers

Jun 5, 2026

SWE-Explore: Benchmarking How Coding Agents Explore Repositories

Repository-level coding benchmarks such as SWE-bench have driven a rapid surge in the capabilities of coding agents. Yet they usually treat coding tasks as a holistic, binary prediction problem (e.g., resolved or unresolved), neglecting fine-grained agent capabilities such as repository understanding, context retrieval, code localization, and bug diagnosis. In this paper, we introduce SWE-Explore, a benchmark that isolates the evaluation of repository exploration, a critical capability of coding agents. Given a repository and an issue, SWE-Explore asks an explorer to return a ranked list of relevant code regions under a fixed line budget. SWE-Explore covers 848 issues across 10 programming languages and 203 open-source repositories. For each instance, we derive line-level ground truth from independent agent trajectories that successfully solved the same issue, distilling the specific code regions their solution paths actually consulted. We evaluate exploration along coverage, ranking, and context-efficiency dimensions, showing that these metrics strongly track downstream repair behavior. Across a broad set of retrieval methods, general coding agents, and specialized localizers, we find that agentic explorers form a clear tier above classical retrieval. While file-level localization is already strong for modern methods, line-level coverage and efficient ranking remain the key axes differentiating state-of-the-art explorers.

Shaoqiu Zhang, Yuhang Wang, Jialiang Liang +8

1Shanghai Jiao Tong University · 2Xinjiang University · 4Independent Researcher +2

AI Coding Agents Agentic Search Software Engineering Benchmarks Code Localization Information Retrieval

Ask PDF

Aug 5, 2026

Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports

Coding agents powered by large language models (LLMs) are increasingly adopted in software engineering (SWE) scenarios, capable of fixing a specific bug in large-scale codebase. However, existing SWE benchmarks typically assume that high-quality issue reports with detailed information are always available, which is easily violated in practice due to the complexity of report acquisition and curation. To address this, we introduce Active-SWE, a benchmark for evaluating coding agents on proactively discovering and fixing multiple bugs without report guidance, covering 1,663 tasks across six bug categories and eight languages. Beyond shifting the focus from existing reactive bug fixing to proactive bug fixing, Active-SWE enables a more in-depth evaluation by expanding the scope from fixing a specific recorded bug to multiple-bug fixing and potential bug discovery scenarios. To construct Active-SWE, we propose a novel difficulty-aware task formulation pipeline with a dual-track evaluation framework, facilitating comprehensive evaluation of proactive bug-fixing capability. Extensive experiments reveal that most state-of-the-art coding agents struggle with proactive bug-fixing tasks, demonstrating limited performance in locating and resolving recorded bugs, handling multiple bug fixing scenarios, and discovering valid potential bugs.

Haobin Li, Ping Deng, Weizhong Qian +4

Sichuan University · University of Electronic Science and Technology of China · Independent Researchers

Automated Program Repair AI Coding Agents Software Engineering Benchmarks Automated Software Engineering

Ask PDF

Mar 3, 2026

BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?

Current code-agent benchmarks primarily evaluate localized issue resolution within a single target repository, leaving under-tested many software engineering tasks that require external knowledge or broader repository-level changes. We introduce BeyondSWE, a 500-instance benchmark drawn from 246 real-world GitHub repositories to evaluate code agents beyond single-repository bug fixing. BeyondSWE covers four representative settings: cross-repository issue resolution, domain-specific issue resolution, dependency-driven migration, and document-to-repository generation, spanning both broader knowledge scope and broader resolution scope. Our evaluation shows that BeyondSWE remains far from saturated: the best OpenHands-based agent reaches 46.12 average score, while the strongest Codex harness with GPT-5.4 (xhigh) reaches 56.65 under a search-aware prompt. To study whether external information access closes this gap, we use SearchSWE as a controlled diagnostic baseline for search-augmented coding. Search access improves most models and substantially helps some tasks, but the gains remain limited and uneven, showing that current agents still struggle to convert retrieved information into precise, version-compatible, and locally actionable code changes. These results suggest that deep search for coding remains an open problem: progress requires agents that can reliably combine external evidence with repository-local reasoning and execution-based verification.

Guoxin Chen, Fanzhe Meng, Jiale Zhao +12

Retrieval-Augmented Code Generation Software Engineering Benchmarks LLM Agent Evaluation Code Generation Benchmarks Code Generation

Ask PDF

Abstract

AI Coding Agents

Explore similar work

SWE-Explore: Benchmarking How Coding Agents Explore Repositories

Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports

BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?