ExpertCrowd-SWE (EC-SWE) is a benchmark of 100 expert-authored, multi-file software engineering tasks for evaluating autonomous coding agents in large, unfamiliar codebases. Each task is a change request written by a practising engineer against a pinned commit of a real repository, paired with a sealed reference solution and a task-specific grader whose weighted, mostly executable criteria award partial credit; must-pass blockers and pass/fail gates encode what a maintainer would refuse to merge. The tasks span 87 open-source projects and 11 languages. The median ticket is 148 words, while the median reference solution changes 471 lines. Every task image is calibrated before release so that the reference solution scores 1.0 and both the untouched repository and an inert change score 0; nearly half of the tasks also ship alternative, lazy and adversarial control patches that the grader must classify correctly. The final 100 tasks were selected for difficulty against current frontier models. We evaluate seven frontier models, five runs each (3,495 scored runs), in a single platform harness that gives each model its provider-native tool interface. Claude Opus 5.5 (67.3%) and GPT-6 Astra (63.7%) lead without a statistically significant gap, followed by Claude Fable 5.1 (46.5%), Gemini 3.8 Flash (40.0%), and a statistically indistinguishable group of Qwen 3.8 Max, DeepSeek V4.1 Flash and Grok 4.6 (32-34%). No task is fully solved by every model, and on 80 tasks no model earns full credit on all five runs. Scores span 35 points. Depending on the model, agents take a median of 33 to 166 steps per run, with little relation to score. We release 24 tasks with their environments, graders and results, and hold out 76 to limit contamination.
Figures & tables
Property
Value
Tasks (public / held-out)
100 (24 / 76)
Open-source projects / languages
87 / 11
Ticket length, words: median (IQR; range)
148 (69–261; 39–768)
Reference solution, changed lines: median (IQR)
471 (270–808)
Reference solution, files: median (IQR)
4.5 (3–7)
Agent steps per run: per-model median (range across models)
33–166
Table 1: Corpus statistics for the 100 released tasks. Prompt length counts the author-written ticket only, excluding the shared environment preamble. Changed lines count added plus removed lines in the reference patch.
Figure 1: Composition of the 100 tasks. (a) Primary language of each task’s reference change. (b) Size of the reference solution in changed lines (log scale). (c) Number of weighted criteria in each task’s grader. Green lines mark medians.
Table 2: Evaluated configurations. All models run in the HUD harness, which calls the provider API through the HUD inference gateway and exposes the provider’s native tool interface. Reasoning effort and output-token limits are the platform defaults.
Model
Score (95% CI)
Run-to-run CI
Full-credit runs
Tasks ≥ 1 of 5 / 5 of 5
Within-task SD
Claude Opus 5.5
67.3±4.7
±1.9
26.6%
47 / 9
13.6
GPT-6 Astra
63.7±5.5
±1.5
23.4%
36 / 15
7.3
Claude Fable 5.1
46.5±3.7
±0.4
8.6%
25 / 1
16.9
Gemini 3.8 Flash a
40.0±4.3
±1.5
3.8%
9 / 1
11.0
Qwen 3.8 Max
33.5±3.3
±1.4
1.0%
5 / 0
10.9
DeepSeek V4.1 Flash
32.5±3.7
±1.4
3.0%
9 / 0
13.0
Table 3: Main results on the 100 released tasks, 3,495 scored runs. Score is the macro-average of task scores, with a 95% interval over tasks; the run-to-run interval covers agent sampling noise only (Sec. 3.9 ). “Full-credit runs” is the share of runs scoring 1.0. “ ≥ 1 of 5” and “5 of 5” count tasks with at least one, or all five, full-credit runs. “Within-task SD” is the mean standard deviation of run scores within a task, in points.
Figure 2: EC-SWE leaderboard. (a) Mean task score on the 100 tasks (solid bars), with 95% intervals over tasks. Dotted outlines show the mean of each model’s best run out of five. The right-hand column gives Holm-adjusted p -values from paired sign-flip permutation tests against Claude Opus 5.5. (b) Number of tasks on which a model earned full credit on at least one of its five runs (solid) and on all five runs (hatched). Every model was run five times per task with a 1,000-step limit.
Comparison
Δ (points)
95% paired CI
W / T / L
Claude Opus 5.5 − GPT-6 Astra
+3.6
[−1.4,+9.7]
53 / 6 / 41
Claude Opus 5.5 − Claude Fable 5.1
+20.8
[+16.9,+25.3]
82 / 5 / 13
GPT-6 Astra − Claude Fable 5.1
+17.2
[+11.8,+22.1]
70 / 2 / 28
Claude Fable 5.1 − Gemini 3.8 Flash
+6.5
[+1.6,+10.5]
63 / 3 / 33
Gemini 3.8 Flash − Qwen 3.8 Max
+6.5
[+3.0,+11.0]
50 / 6 / 43
Qwen 3.8 Max − DeepSeek V4.1 Flash
+1.0
[−2.2,+3.5]
64 / 4 / 32
Table 4: Paired differences between models on shared tasks. Δ is the difference in mean task score in points (first model minus second), with a 95% paired bootstrap interval over tasks (2,000 resamples). W / T / L counts tasks where the first model scores higher, ties, or scores lower.
Figure 3: Task-by-model score matrix for the 100 tasks, ordered left to right by seven-model average. Each cell is the mean of five runs. The empty cell in the Gemini 3.8 Flash row is the one task without a valid Gemini run. Colour swatches identify models consistently across all figures.
Figure 4: Distribution of run scores across all scored runs for each model. Segments are labelled with their percentage where space allows.
Model
Capped runs
Mean loss (capped)
Zeroed runs
Mean loss (zeroed)
Points lost
Claude Opus 5.5
62 (12.4%)
0.22
11 (2.2%)
0.43
3.7
GPT-6 Astra
42 (8.4%)
0.26
12 (2.4%)
0.30
2.9
Claude Fable 5.1
37 (7.4%)
0.27
12 (2.4%)
0.18
2.4
Gemini 3.8 Flash
31 (6.3%)
0.25
8 (1.6%)
0.00
1.6
Qwen 3.8 Max
29 (5.8%)
0.21
10 (2.0%)
0.00
1.2
DeepSeek V4.1 Flash
24 (4.8%)
0.23
9 (1.8%)
0.03
1.1
Table 5: Credit lost to blockers and gates. “Capped” counts runs whose score was reduced by a failed blocker; “zeroed” counts runs set to zero by a failed gate. “Mean loss” is the average difference between the uncapped criteria score and the final score for those runs. “Points lost” is the total effect on the benchmark score.
Figure 5: Score against median agent steps per run; whiskers show the interquartile range of steps across each model’s runs.
Model
Score
Median steps (IQR)
Mean steps
Median container time
Claude Opus 5.5
67.3
160 (128–205)
171
80 min
GPT-6 Astra
63.7
33 (26–44)
37
36 min
Claude Fable 5.1
46.5
127 (94–166)
134
56 min
Gemini 3.8 Flash
40.0
166 (127–217)
182
40 min
Qwen 3.8 Max
33.5
137 (103–178)
147
100 min
DeepSeek V4.1 Flash
32.5
160 (114–199)
162
51 min
Table 6: Effort per scored run. Steps are agent steps (model turns). Container time is the wall-clock lifetime of the task container, including grading.
DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one. DeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream, so their reference solutions stay out of the public record that model training scrapes; and each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that provides it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE's verifier about an order of magnitude less often than with SWE-Bench Pro's inherited tests (1.4% versus 32.4%). Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code, and the benchmark separates frontier agents across a wider score band than the leaderboards on which they otherwise cluster. We release the benchmark, its verifiers, and the full record of evaluation trajectories.
We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refactoring (70 tasks). SWE Atlas differs from prior SWE benchmarks in three key ways: it targets underrepresented but practically important task categories, uses comprehensive category-specific evaluation protocols, and adopts under-specified, agentic task formulations that better reflect real-world usage. Its evaluation framework combines programmatic checks with rubric-based assessment. This goes beyond functional correctness, evaluating software engineering quality, including test and refactor completeness, maintainability, reusable abstractions, and codebase hygiene. We evaluate a range of frontier and open-weight models on SWE Atlas and find that GPT-5.4 and Opus 4.7 achieve the strongest overall performance, while even the best open-weight models score poorly. Our analysis suggests that top models rely on extensive codebase exploration and runtime-driven reasoning. However, even top models consistently struggle with subtle edge cases, complex runtime analysis, and adherence to software engineering best practices. Overall, SWE Atlas provides a complementary evaluation suite for measuring both correctness and engineering quality in coding agents.
Mohit Raghavendra, Soham Dan, Miguel Romero Calvo +12
SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents. As agents improve, they reach task issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate. We present SWE-Bench Pro V2, a revised public split built to keep the benchmark accurate as capabilities advance. Agentic audits, expert adjudication and blind re-implementation repaired or retired 480 of the 731 tasks: 23 contracted engineers spent 404 hours, each working in a language they write professionally, and every revised instruction was verified by a second engineer solving the task from the text alone. A hardened protocol restricts network access to the model endpoint and re-grades every patch in a clean container, so agents cannot retrieve upstream fixes or get credit for changes outside their patch. On the 642 V2 tasks frontier agents resolve up to 99.4% while other systems stay clearly separated, and a 51-task Hard split spreads systems from 52.9% to 98.0%. We release the benchmark and distill lessons for maintaining agent benchmarks over time.