Organizations: The State Key Laboratory of Blockchain and Data Security, Zhejiang University, China and Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security, China · Graz University of Technology, Austria
Repository-level issue-resolution benchmarks such as SWE-bench evaluate submitted patches primarily based on whether they pass the correctness tests, leaving the runtime performance of different test-passing patches largely unexplored. Evaluating patch performance is essential but challenging since designing performance workloads for each benchmark instance is costly. However, these benchmarks already provide a natural candidate: the built-in FAIL_TO_PASS (F2P) tests, which exercise the behavior affected by the issue fix. We therefore investigate whether the provided F2P tests can be repurposed to evaluate patch performance. Overall, we choose four frontier models, Claude Opus5, GPT5.6, DeepSeek V4 Pro, and Qwen~3.8 Max, paired with SWE-agent to resolve the pre-filtered measurable corpus from verified splits of SWE-bench and SWE-bench-Live, yielding 262 cases with at least two different correct solutions, including the developer ground-truth patch. We then measure each solution's runtime performance in instruction count and wall-clock time on the native F2P test, and use Welch ANOVA instead of the RSD proposed in prior work as a more robust statistical discrimination criterion. We find that this methodology exhibits notable discrimination capability in runtime performance and identifies performance differences among patches in almost half of the instances. Our empirical study shows that the runtime performance of generated patches is independent of the issue resolution rate of LLMs, with nearly reversed model rankings. The case study also shows that the performance analysis reveals architectural regression in some patches that the correctness evaluation overlooked. These findings suggest that resolution rate does not adequately characterize LLMs' capability for repository-level issue resolution, motivating future evaluations to consider patch performance alongside it.
Figures & tables
Figure 1. The decorator-based probing, simplified from the injected wrapper.
Stage
SWE-bench
SWE-bench-Live
Total
Original dataset
500
500
1,000
w/o a single test
155
213
368
w/o repeated exec.
52
35
87
w/o I/O-free exec.
63
143
206
measurable corpus
230
109
339
Table 1. Corpus construction funnel.
Model
SWE-bench
SWE-bench-Live
Total
Claude
220 (95.65%)
76 (69.72%)
296 (87.32%)
Qwen
207 (90.00%)
75 (68.81%)
282 (83.19%)
GPT
208 (90.43%)
70 (64.22%)
278 (82.01%)
DeepSeek
206 (89.57%)
70 (64.22%)
276 (81.42%)
Table 2. Resolution rates of the four models on the corpus.
Figure 2. Solution identity between the four models and the ground truth (GT), on the measurable corpus (left: SWE-bench, right: SWE-bench-Live).
# distinct solutions
SWE-bench
SWE-bench-Live
Total
1
42 (18.26%)
35 (32.11%)
77 (22.71%)
2
55 (23.91%)
23 (21.10%)
78 (23.01%)
3
45 (19.57%)
23 (21.10%)
68 (20.06%)
4
60 (26.09%)
18 (16.51%)
78 (23.01%)
5
28 (12.17%)
10 (9.17%)
38 (11.21%)
≥ 2 ( as study corpus )
188 (81.74%)
74 (67.89%)
262 (77.29%)
Table 3. The number of distinct solutions after merging identical patches (ground truth included).
Dataset
Instructions
Wall time
All (n=262)
110 (41.98%)
91 (34.73%)
SWE-bench (n=188)
80 (42.55%)
65 (34.57%)
SWE-bench-Live (n=74)
30 (40.54%)
26 (35.14%)
Table 4. Number and share of distinguishable cases ( p<0.05 ) with F2P tests.
Instructions
Wall time
Dataset
median
max
mean
median
max
mean
All (n=262)
0.21%
132.68%
3.13%
0.59%
131.87%
3.49%
SWE-bench (n=188)
0.27%
132.68%
3.71%
0.58%
131.87%
4.10%
SWE-bench-Live (n=74)
0.15%
45.22%
1.68%
0.62%
42.55%
1.97%
Table 5. Per-case between-solution RSD with F2P tests.
Figure 3. Per-case between-solution RSD with F2P tests, sorted ascending. The gray curve shows RSD for all cases; colored segments and dots mark cases with p<0.05 .
Table 6. Per-model comparison with the ground-truth. A win (loss) means the model patch is faster (slower) than the ground-truth patch, and a tie means the two patches are identical. Speedup is the geometric mean of per-case ratios (lower is faster). Scored gives the number of cases the model resolved, on all 262 cases in (a), and on that metric’s distinguishable cases (110 for instructions, 91 for wall time) in (b).
Instructions
Wall time
Model
All
Distinguishable
All
Distinguishable
GPT
98.93%
97.53%
99.01%
97.23%
DeepSeek
99.07%
97.89%
98.97%
97.21%
Claude
99.33%
98.42%
99.33%
98.18%
Qwen
100.71%
101.68%
100.71%
102.00%
Table 7. Model speedup against the ground truth on the commonly solved cases, aggregated as the geometric mean. All means the full common subset (n=211), while Distinguishable intersects it with each metric’s distinguishable cases (n=90 for instructions, n=75 for wall time).
Table 8. Pairwise head-to-head speedup with F2P tests. In each pair, model A is faster on instruction count. Speedup aggregates the per-case A/B ratios. Win (loss) means A is faster (slower), and Tie means identical. Total counts the commonly solved cases in (a) and the distinguishable ones among them in (b).
Figure 4. Champion distribution with F2P tests. Each bar shows the share of cases whose fastest solution comes from each source, with the score split evenly among the sources that produced it.
Figure 5. The F2P test and the three repair strategies for astropy__astropy-7671 , with their instruction counts and wall-clock times. The test patch introduces the highlighted code.
Table 9. Instances excluded by the in-process repeatability rule, by failure class.
Figure 6. Density of the per-case between-solution RSD with F2P tests, grouped by the number of distinct solutions k , estimated with a kernel density smoother on the log scale of RSD. Left: instruction count. Right: wall-clock time.
Instructions
Wall time
Dataset
Median
Mean
Max
Median
Mean
Max
All (n=262)
0.27%
0.49%
36.80%
0.82%
1.10%
36.41%
Dist. on instructions (n=110)
0.25%
0.31%
1.48%
0.84%
0.93%
2.75%
Dist. on wall time (n=91)
0.24%
0.32%
1.72%
0.80%
0.90%
2.75%
Appendix
Table 10. Within-solution RSD, median, mean, and max over cases.
Repository-level issue resolution benchmarks have become a standard testbed for evaluating LLM-based agents, yet success is still predominantly measured by test pass rates. In practice, however, acceptable patches must also comply with project-specific design constraints, such as architectural conventions, error-handling policies, and maintainability requirements, which are rarely encoded in tests and are often documented only implicitly in code review discussions. This paper introduces design-aware issue resolution and presents \bench{}, a benchmark that makes such implicit design constraints explicit and measurable. \bench{} is constructed by mining and validating design constraints from real-world pull requests, linking them to issue instances, and automatically checking patch compliance using an LLM-based verifier, yielding 495 issues and 1,787 validated constraints across six repositories, aligned with SWE-bench-Verified and SWE-bench-Pro. Experiments with state-of-the-art agents show that test-based correctness substantially overestimates patch quality: fewer than half of resolved issues are fully design-satisfying, design violations are widespread, and functional correctness exhibits negligible statistical association with design satisfaction. While providing issue-specific design guidance reduces violations, substantial non-compliance remains, highlighting a fundamental gap in current agent capabilities and motivating design-aware evaluation beyond functional correctness.
Kai Yu, Zhenhao Zhou, Junhao Zeng +8
Fudan University, China · Nanyang Technological University, Singapore
SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired with its linked issue by extracting issue references from the PR description; the issue description is used as the problem statement, and the PR patch serves as the test oracle. However, due to the inherent complexity of developing and maintaining large repositories, such PR-Issue pairings are often misaligned in practice. In this work, we systematically study SWE-bench Verified instances, finding that 13.6% exhibit misalignment across five patterns in eleven fine-grained scenarios. To enable reliable and scalable construction of those benchmarks in the future, we propose PAIChecker, a multi-agent system for checking PR-Issue misalignment in SWE-bench-like benchmarks. Specifically, PAIChecker adopts a three-phase design that combines specific pattern identification, cross-agent label synthesis, and code-level validation, thereby enabling more accurate, generalizable, and progressively verified detection. Experiments on SWE-Gym and SWE-bench Multilingual show that PAIchecker achieves the best performance across all four LLM backbones, reaching up to 92.12% and 91.67% binary accuracy, respectively.
Optimizing the performance of large-scale software repositories demands expertise in code reasoning and software engineering (SWE) to reduce runtime while preserving program correctness. However, most benchmarks emphasize what to fix rather than how to fix code. We introduce SWE-fficiency, a benchmark for evaluating repository-level performance optimization on real workloads. Our suite contains 498 tasks across nine widely used data-science, machine-learning, and HPC repositories (e.g., numpy, pandas, scipy): given a complete codebase and a slow workload, an agent must investigate code semantics, localize bottlenecks and relevant tests, and produce a patch that matches or exceeds expert speedup while passing the same unit tests. To enable this how-to-fix evaluation, our automated pipeline scrapes GitHub pull requests for performance-improving edits, combining keyword filtering, static analysis, coverage tooling, and execution validation to both confirm expert speedup baselines and identify relevant repository unit tests. Empirical evaluation of state-of-the-art agents reveals significant underperformance. On average, agents achieve less than 0.23x the expert speedup: agents struggle in localizing optimization opportunities, reasoning about execution across functions, and maintaining correctness in proposed edits. We release the benchmark and accompanying data pipeline to facilitate research on automated performance engineering and long-horizon software reasoning.
Jeffrey Jian Ma, Milad Hashemi, Amir Yazdanbakhsh +5
Harvard University · Google DeepMind · Princeton University +2