Software engineering (SWE) benchmarks are the primary way to measure the capabilities of coding agents: an agent is placed in a code repository or machine environment, given a request, and judged by hidden tests. These benchmarks almost always start the agent from a clean state that contains no earlier work on the task. In practice, however, agents often continue work that a developer or another agent has already done and reported as complete, with no indication of what remains incorrect or missing. We introduce SWE-Residual, a benchmark for Starting-State Robustness: the ability to maintain performance on a task when the starting state already contains earlier work on that task. SWE-Residual contains 198 Residual States drawn from 111 tasks in DeepSWE and Terminal-Bench, where the earlier work modifies a code repository or a whole machine environment. Each Residual State is the state a predecessor agent left behind after submitting its work as complete, although the hidden tests still fail. The evaluated agent receives only the original request and this state, without the predecessor's trajectory or any sign that the attempt failed; because the request and hidden tests are unchanged, the starting state is the only difference from the source benchmark. Across five frontier agents, success drops by 20.6--53.5 percentage points on DeepSWE and 20.6--34.9 points on Terminal-Bench relative to a Clean Start, and agent rankings on DeepSWE reverse. Failed continuations show a pattern we call acceptance bias: agents treat the inherited work as the definition of what remains to be done instead of checking it against the request. On DeepSWE, in 58.9--88.2% of failed runs, they submit with exactly the same tests failing, although they pass most of these tests from a Clean Start. SWE-Residual thus exposes a capability gap that clean-start evaluation does not capture.
Figures & tables
Figure 1: Left: Standard benchmarks evaluate an agent from the original repository ( Clean Start ). SWE-Residual evaluates the same agent on the same task from the repository left by a predecessor agent that submitted its work as complete but failed the hidden tests ( Residual Start ), with the same request and hidden tests and no information about the earlier attempt. Right: Success rates of the five evaluated agents under both settings, averaged over DeepSWE and Terminal-Bench.
Benchmark
Environment
Prior work
Submitted as done
Failed
Blind continuation
SWE-bench ( Jimenez et al., 2024 )
Repository
✗
–
–
–
Terminal-Bench ( Merrill et al., 2026 )
Machine
✗
–
–
–
Handoff Debt ( KC and Budathoki, 2026 )
Repository
✓
✗
✗
✗
Handoff Tax ( Ganz et al., 2026 )
Repository
✓
✗
✗
✗
FeedbackEval ( Dai et al., 2025 )
Repository
✓
✗
✓
✗
OmniCode ( Sonwane et al., 2026 )
Repository
✓
✗
✓
✗
Table 1: A comparison of SWE-Residual to existing coding agent benchmarks. Prior work : the starting state contains residual artifacts from another agent’s earlier attempt. Submitted as done : the predecessor submitted the prior work as complete, rather than being interrupted or selected without regard to submission. Failed : the prior work does not fully satisfy the task specification. Blind continuation : the evaluated agent receives only the resulting state and task request, with no explicit feedback, trajectory, or results of the prior attempt. ✓ yes, ✗ no, – not applicable.
DeepSWE
Terminal-Bench
Source benchmark tasks
113
89
Source tasks covered
90
21
Residual States
165
33
from DeepSeek
62
17
from Qwen
72
11
from GLM
31
5
Table 2: Statistics of SWE-Residual. Task counts refer to unique source tasks. Prior-attempt steps and execution time are medians over retained states; steps count recorded agent turns. Test pass rates include all tests used for task acceptance and are summarized with equal weight per state.
Track
Agent
Residual (%)
Ref. Clean (%)
Gap
SWE-Residual (DeepSWE)
Gemini-3.8-Flash
20.0
72.2
52.2
Qwen-3.8-Flash
30.6
51.1
20.6
GLM-5.3-Flash
15.4
68.9
53.5
GPT-6-Sol
33.5
65.6
32.0
Kimi-K2.8-Preview
13.0
57.8
44.8
SWE-Residual (Terminal-Bench)
Gemini-3.8-Flash
14.3
47.6
33.3
Table 3: Performance on the two SWE-Residual tracks. Residual reports success averaged first over Residual States within each task, then equally across tasks. Ref. Clean reports success on the same underlying tasks. Gap is Ref. Clean minus Residual, in percentage points, computed before rounding. Bold indicates the highest residual and clean success rates within each track.
Figure 2: Residual Start outcomes on tasks that the same agent successfully completes under Clean Start. Red indicates failure and green indicates success. Outcomes are averaged over Residual States within each task, then equally across tasks. Each agent is evaluated on its own set of observed Clean Start successes.
Initially failing
Left-out files
Agent
Gap
Same failing
Clean
Residual
Kept
Clean
Residual
Steps
GLM-5.3-Flash
53.5
78.9
87.3
23.4
95.7
40.5
0 6.7
0.57
Gemini-3.8-Flash
52.2
76.8
88.4
27.5
97.2
43.5
0 9.4
0.70
Kimi-K2.8-Preview
44.8
88.2
77.6
14.4
98.6
45.7
8.1
0.56
GPT-6-Sol
32.0
58.9
84.1
47.7
91.1
41.2
15.0
1.02
Qwen-3.8-Flash
20.6
72.0
72.8
39.1
86.4
47.9
14.6
0.84
Table 4: Acceptance bias on DeepSWE, with agents sorted by Residual Gap. Same failing : failed continuations that end with exactly the tests that failed in the Residual State. Initially failing : share of those tests that the agent passes from each start. Kept : lines written by the predecessor agent that remain after failed continuations. Left-out files : runs that change a file changed by the reference solution but not by the predecessor agent, over states where both runs are normally submitted. Steps : median ratio of Residual to Clean steps. Normally submitted runs; task-level averages in % (Gap in points).
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
DeepSWE
Terminal-Bench
Total
Passed human review
170
37
207
Patch application or command replay failed
2
1
3
Environment unusable for rollouts
1
2
3
No valid verifier result, or no longer failing
2
1
3
Included in SWE-Residual
165
33
198
Appendix
Table 5: Reconstruction and validation of the candidates that passed human review.
Item
Clean Start
Residual Start
Starting state
Original repository or machine
Residual State
Request
Original request
Original request
System prompt, harness, and budget
Source benchmark defaults
Source benchmark defaults
Earlier work inside the state
None
Retained, including code, tests, documentation, and notes written by the predecessor
Predecessor trajectory, summaries, or handoff notes
–
Not explicitly supplied; traces left in the state, such as shell history, may be accessible
Indication that an earlier attempt exists or failed
–
Not provided
Appendix
Table 6: Information available to the evaluated agent under Clean Start and Residual Start. Both conditions use the same request, system prompt, harness, and budget.
Table 7: Models used in SWE-Residual. Predecessor agents produce Residual States in Stage I; evaluated agents are run from Clean Starts and Residual Starts.
Evaluated agent
Overall
Predecessor agent
DeepSeek
Qwen
GLM
Clean
Residual
Clean
Residual
Clean
Residual
Clean
Residual
Gemini-3.8-Flash
67.6
18.9
62.0
19.0
68.7
16.9
44.4
5.6
Qwen-3.8-Flash
47.7
27.2
40.5
21.5
51.8
32.5
38.9
8.3
GLM-5.3-Flash
63.1
15.2
57.0
11.4
62.7
15.7
33.3
2.8
GPT-6-Sol
66.7
34.1
63.3
32.9
62.7
34.9
47.2
19.4
Appendix
Table 8: Performance by predecessor agent, pooling DeepSWE and Terminal-Bench. All values are success rates (%). Each group uses its own matched Clean Start reference, with each source task weighted equally. Overall includes all Residual States; the remaining groups include states produced by the indicated predecessor agent. DeepSeek, Qwen, and GLM denote DeepSeek-V4-Flash-0731, Qwen-3.8-27B, and GLM-5.3, respectively.
Repository-level coding benchmarks largely assume that tasks and environments remain unchanged throughout execution. In software development, code, dependencies, runtime state, and requirements can change while an agent is still working. We introduce SWE-Shift, a benchmark for evaluating coding agents under such mid-execution changes. From the same saved history and workspace, a Clean continuation proceeds without intervention, while a matched Shifted continuation receives a task-relevant update. Success requires completing the current task, preserving still-valid work, and satisfying the applicable state and evidence checks. The benchmark contains 120 scenarios across 12 change families, each evaluated at nine scheduled intervention positions. We evaluate 17 coding-agent systems, all of which have lower Shifted than Clean success. GPT-6 Astra drops from 67.35% to 24.79%, with 20.34% paired success. Tool/session changes and stale evidence yield zero paired success across all systems. Late-position paired success is lower for some systems and higher for others. These results show that success on the unchanged task can coexist with failures to preserve work, recover state, or refresh evidence after an update.
Victor Shea-Jay Huang, Rongqing Li, Zhangyue Yin +7
MMLab, The Chinese University of Hong Kong · Somewhere on the Earth · Fudan NLP Lab, Fudan University +2
SWE-Bench Pro has emerged as a standard benchmark for evaluating software engineering agents on challenging repository-level tasks. However, our analysis work show that its evaluation is undermined by two sources of unreliability: reward hacking, enabled by leakage of gold solutions or hidden evaluation information, and task quality issues, including misleading problem statements and improperly scoped tests. These issues can inflate benchmark performance and obscure agents' true coding ability. We present SWE-Bench Pro Verified, a verified version of SWE-Bench Pro that addresses both problems. Our approach combines anti-hacking safeguards that eliminate major leakage channels without disrupting normal agent functionality, with task refinement that minimally corrects inconsistencies within flawed instances. Evaluations on SWE-Bench Pro Verified reveal that some models perform substantially worse than previously evaluated, suggesting that existing results on SWE-Bench Pro may overestimate real software engineering capability. SWE-Bench Pro Verified offers a more trustworthy benchmark for assessing software engineering agents.
Pujun Zheng, Zixin Shang, Shufan Jiang +5
East China Normal University · Shanghai Artificial Intelligence Laboratory · Fudan University
DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one. DeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream, so their reference solutions stay out of the public record that model training scrapes; and each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that provides it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE's verifier about an order of magnitude less often than with SWE-Bench Pro's inherited tests (1.4% versus 32.4%). Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code, and the benchmark separates frontier agents across a wider score band than the leaderboards on which they otherwise cluster. We release the benchmark, its verifiers, and the full record of evaluation trajectories.