ExpertCrowd-SWE (EC-SWE) is a benchmark of 100 expert-authored, multi-file software engineering tasks for evaluating autonomous coding agents in large, unfamiliar codebases. Each task is a change request written by a practising engineer against a pinned commit of a real repository, paired with a sealed reference solution and a task-specific grader whose weighted, mostly executable criteria award partial credit; must-pass blockers and pass/fail gates encode what a maintainer would refuse to merge. The tasks span 87 open-source projects and 11 languages. The median ticket is 148 words, while the median reference solution changes 471 lines. Every task image is calibrated before release so that the reference solution scores 1.0 and both the untouched repository and an inert change score 0; nearly half of the tasks also ship alternative, lazy and adversarial control patches that the grader must classify correctly. The final 100 tasks were selected for difficulty against current frontier models. We evaluate seven frontier models, five runs each (3,495 scored runs), in a single platform harness that gives each model its provider-native tool interface. Claude Opus 5.5 (67.3%) and GPT-6 Astra (63.7%) lead without a statistically significant gap, followed by Claude Fable 5.1 (46.5%), Gemini 3.8 Flash (40.0%), and a statistically indistinguishable group of Qwen 3.8 Max, DeepSeek V4.1 Flash and Grok 4.6 (32-34%). No task is fully solved by every model, and on 80 tasks no model earns full credit on all five runs. Scores span 35 points. Depending on the model, agents take a median of 33 to 166 steps per run, with little relation to score. We release 24 tasks with their environments, graders and results, and hold out 76 to limit contamination.
Figures & tables
Property
Value
Tasks (public / held-out)
100 (24 / 76)
Open-source projects / languages
87 / 11
Ticket length, words: median (IQR; range)
148 (69–261; 39–768)
Reference solution, changed lines: median (IQR)
471 (270–808)
Reference solution, files: median (IQR)
4.5 (3–7)
Agent steps per run: per-model median (range across models)
33–166
Table 1: Corpus statistics for the 100 released tasks. Prompt length counts the author-written ticket only, excluding the shared environment preamble. Changed lines count added plus removed lines in the reference patch.
Figure 1: Composition of the 100 tasks. (a) Primary language of each task’s reference change. (b) Size of the reference solution in changed lines (log scale). (c) Number of weighted criteria in each task’s grader. Green lines mark medians.
Table 2: Evaluated configurations. All models run in the HUD harness, which calls the provider API through the HUD inference gateway and exposes the provider’s native tool interface. Reasoning effort and output-token limits are the platform defaults.
Model
Score (95% CI)
Run-to-run CI
Full-credit runs
Tasks ≥ 1 of 5 / 5 of 5
Within-task SD
Claude Opus 5.5
67.3±4.7
±1.9
26.6%
47 / 9
13.6
GPT-6 Astra
63.7±5.5
±1.5
23.4%
36 / 15
7.3
Claude Fable 5.1
46.5±3.7
±0.4
8.6%
25 / 1
16.9
Gemini 3.8 Flash a
40.0±4.3
±1.5
3.8%
9 / 1
11.0
Qwen 3.8 Max
33.5±3.3
±1.4
1.0%
5 / 0
10.9
DeepSeek V4.1 Flash
32.5±3.7
±1.4
3.0%
9 / 0
13.0
Table 3: Main results on the 100 released tasks, 3,495 scored runs. Score is the macro-average of task scores, with a 95% interval over tasks; the run-to-run interval covers agent sampling noise only (Sec. 3.9 ). “Full-credit runs” is the share of runs scoring 1.0. “ ≥ 1 of 5” and “5 of 5” count tasks with at least one, or all five, full-credit runs. “Within-task SD” is the mean standard deviation of run scores within a task, in points.
Figure 2: EC-SWE leaderboard. (a) Mean task score on the 100 tasks (solid bars), with 95% intervals over tasks. Dotted outlines show the mean of each model’s best run out of five. The right-hand column gives Holm-adjusted p -values from paired sign-flip permutation tests against Claude Opus 5.5. (b) Number of tasks on which a model earned full credit on at least one of its five runs (solid) and on all five runs (hatched). Every model was run five times per task with a 1,000-step limit.
Comparison
Δ (points)
95% paired CI
W / T / L
Claude Opus 5.5 − GPT-6 Astra
+3.6
[−1.4,+9.7]
53 / 6 / 41
Claude Opus 5.5 − Claude Fable 5.1
+20.8
[+16.9,+25.3]
82 / 5 / 13
GPT-6 Astra − Claude Fable 5.1
+17.2
[+11.8,+22.1]
70 / 2 / 28
Claude Fable 5.1 − Gemini 3.8 Flash
+6.5
[+1.6,+10.5]
63 / 3 / 33
Gemini 3.8 Flash − Qwen 3.8 Max
+6.5
[+3.0,+11.0]
50 / 6 / 43
Qwen 3.8 Max − DeepSeek V4.1 Flash
+1.0
[−2.2,+3.5]
64 / 4 / 32
Table 4: Paired differences between models on shared tasks. Δ is the difference in mean task score in points (first model minus second), with a 95% paired bootstrap interval over tasks (2,000 resamples). W / T / L counts tasks where the first model scores higher, ties, or scores lower.
Figure 3: Task-by-model score matrix for the 100 tasks, ordered left to right by seven-model average. Each cell is the mean of five runs. The empty cell in the Gemini 3.8 Flash row is the one task without a valid Gemini run. Colour swatches identify models consistently across all figures.
Figure 4: Distribution of run scores across all scored runs for each model. Segments are labelled with their percentage where space allows.
Model
Capped runs
Mean loss (capped)
Zeroed runs
Mean loss (zeroed)
Points lost
Claude Opus 5.5
62 (12.4%)
0.22
11 (2.2%)
0.43
3.7
GPT-6 Astra
42 (8.4%)
0.26
12 (2.4%)
0.30
2.9
Claude Fable 5.1
37 (7.4%)
0.27
12 (2.4%)
0.18
2.4
Gemini 3.8 Flash
31 (6.3%)
0.25
8 (1.6%)
0.00
1.6
Qwen 3.8 Max
29 (5.8%)
0.21
10 (2.0%)
0.00
1.2
DeepSeek V4.1 Flash
24 (4.8%)
0.23
9 (1.8%)
0.03
1.1
Table 5: Credit lost to blockers and gates. “Capped” counts runs whose score was reduced by a failed blocker; “zeroed” counts runs set to zero by a failed gate. “Mean loss” is the average difference between the uncapped criteria score and the final score for those runs. “Points lost” is the total effect on the benchmark score.
Figure 5: Score against median agent steps per run; whiskers show the interquartile range of steps across each model’s runs.
Model
Score
Median steps (IQR)
Mean steps
Median container time
Claude Opus 5.5
67.3
160 (128–205)
171
80 min
GPT-6 Astra
63.7
33 (26–44)
37
36 min
Claude Fable 5.1
46.5
127 (94–166)
134
56 min
Gemini 3.8 Flash
40.0
166 (127–217)
182
40 min
Qwen 3.8 Max
33.5
137 (103–178)
147
100 min
DeepSeek V4.1 Flash
32.5
160 (114–199)
162
51 min
Table 6: Effort per scored run. Steps are agent steps (model turns). Container time is the wall-clock lifetime of the task container, including grading.