SWE-Shift: Evaluating Coding Agents in Evolving Workspaces

Victor Shea-Jay Huang; Rongqing Li; Zhangyue Yin; Kelong Mao; Yaqi Zhao; Xinyuan Wang; Yao Yin; Lirui Luo

cs.AIOct 23, 2026

SWE-Shift: Evaluating Coding Agents in Evolving Workspaces

Authors: Victor Shea-Jay Huang, Rongqing Li, Zhangyue Yin, Kelong Mao, Yaqi Zhao, Xinyuan Wang, Yao Yin, Lirui Luo,

Organizations: MMLab, The Chinese University of Hong Kong · Somewhere on the Earth · Fudan NLP Lab, Fudan University · The Hong Kong University of Science and Technology ·

Abstract

Repository-level coding benchmarks largely assume that tasks and environments remain unchanged throughout execution. In software development, code, dependencies, runtime state, and requirements can change while an agent is still working. We introduce SWE-Shift, a benchmark for evaluating coding agents under such mid-execution changes. From the same saved history and workspace, a Clean continuation proceeds without intervention, while a matched Shifted continuation receives a task-relevant update. Success requires completing the current task, preserving still-valid work, and satisfying the applicable state and evidence checks. The benchmark contains 120 scenarios across 12 change families, each evaluated at nine scheduled intervention positions. We evaluate 17 coding-agent systems, all of which have lower Shifted than Clean success. GPT-6 Astra drops from 67.35% to 24.79%, with 20.34% paired success. Tool/session changes and stale evidence yield zero paired success across all systems. Late-position paired success is lower for some systems and higher for others. These results show that success on the unchanged task can coexist with failures to preserve work, recover state, or refresh evidence after an update.

Figures & tables

Figure 1: Success drops after workspace changes. Clean (red) and Shifted (blue) success rates across 17 coding-agent systems, ordered by Clean success.

Figure 2: Per-system evaluation pipeline of SWE-Shift. From the same saved execution state, the agent resumes in two matched continuations: Clean proceeds without intervention, while Shifted introduces a task-relevant change. Outcomes are evaluated under the original and updated conditions, respectively.

Benchmark	Mid- episode	Same prefix	Cross- plane	Preserve/ harm	Multi- position	Trajectory oracle	Fresh evidence
SpecPath	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$
SyncMind	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$
SWE-INTERACT	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$
ToolSandbox	$\times$	$\times$	$\times$	$\checkmark$	$\times$	$\checkmark$	$\times$
SWE-Touch	$\checkmark$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\times$	$\times$
Gaia2	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$

Table 1: Benchmark design at a glance.

Figure 3: What SWE-Shift evaluates. The 12 change families are organized into three broad groups. World Shift (F01–F07) tests changes to executable state or dependencies. Goal Interaction (F08–F10) tests newly disclosed requirements, corrective feedback, and revised goals. Observation Shift (F11–F12) tests stale or misbound evidence. Each panel illustrates the intervention and the required continuation.

System	Cln. $\uparrow$	Shf. $\uparrow$	PSR $\uparrow$	CRR $\uparrow$
GPT-6 Astra [ 2026b ]	67.35	24.79	20.34	30.20
Claude Fable 5.1 [ 2026a ]	62.01	18.33	16.88	26.02
Claude Opus 5 [ 2026b ]	53.76	16.09	14.78	26.54
GPT-5.6 Sol [ 2026a ]	46.22	12.56	11.30	23.94
Kimi-K3 [ 2026b ]	31.41	12.09	8.20	25.12
DeepSeek V4-Pro [ 2026b ]	29.37	10.29	8.83	29.15

Table 2: Continuation performance (%). Full identities appear in Table 8 .

Figure 4: Paired success across changes and intervention positions. PSR by change family (a) and intervention position (b). Solid coral outlines highlight tool/session changes (F06) and stale evidence (F11). Dashed gold outlines highlight runtime/resource changes (F05) and disclosure (F08).

Panel of Figure 5 — Figure 5: Effects of state handling and post-fork token budget on PSR. (a) Bars show cumulative state-access interventions from the Table 2 baseline to the final PSR. (b) Curves show PSR across post-fork token budgets. $B$ is the baseline of 1,000,000 input plus output tokens per continuation, and the gray region marks $8B$ – $16B$ .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

State or execution	Required result
Initial root	Original task checks fail.
Clean reference repair	All Clean success checks pass.
Update without repair	External state remains valid; the task still fails.
Unadapted continuation	Clean succeeds; Shifted task or preservation checks fail.
Repair based on the pre-update state	Execution is protocol-valid; Shifted strict success fails.
Valid recovery after the update	Current task and preservation checks both pass.

Appendix

Table 3: Scenario admission checks. Each scenario must pass all checks before evaluation.

Base task source	Change scenarios
SWE-bench Verified [ 5 ]	60
Multi-SWE-bench [ 37 ]	35
SWE-bench Multilingual [ 15 ]	25
Total	120

Appendix

Table 4: Base task sources for SWE-Shift. Number of SWE-Shift change scenarios built from tasks in each source benchmark.

Family	What changes	Required continuation
F01 User implementation	Valid code is added in the agent’s edit region.	Preserve the user’s behavior while completing the original task.
F02 User contract	A public API or data contract changes.	Meet the updated contract and retained compatibility checks.
F03 Direct collaborator	A collaborator changes the target code or a direct dependency.	Integrate the changes while preserving the collaborator’s behavior.
F04 Indirect collaborator	A transitive dependency changes while the local target stays untouched.	Account for the dependency change and preserve the collaborator’s behavior.
F05 Runtime and resources	Process, port, lock, cache, or resource state changes.	Restore a valid runtime state while preserving unrelated resources.
F06 Tool and session	Session, authorization, schema, or protocol changes invalidate earlier calls.	Use the current permitted interface and preserve unrelated session state.

Appendix

Table 5: Recovery requirements for the 12 change families. Each row pairs a task-relevant change with the behavior a successful continuation must preserve or restore.

Check	Required evidence
Task	Executable task checks pass in the terminal state.
Preservation	Specified behavior and state invariants remain valid.
Prohibited effects	State and event checks find no forbidden outcome.
Current authority	Goal and verifier versions match the scenario’s current requirements.
Fresh observation	An allowed observation occurs after the update and before the first agent mutation.
Recovery order	Required recovery steps occur in the specified order.

Appendix

Table 6: Evidence required for strict success. Every applicable check must pass in a protocol-valid execution.

Benchmark	Changed object	When	Started?	History control	Preservation / harm checks	Outcome
SWE-bench [ 13 ]	Repository repair	Initial task snapshot	N/A	Fixed issue and base commit	Previously passing tests	Resolved issue
SWE-Touch [ 30 ]	Code + contextual user message	Region access; relative schedules	Yes	Matched task conditions; independent runs	Task regression tests; Co-Edit control	Resolve, retention, steps, tokens
SyncMind [ 10 ]	Caller or callee made out of sync	Before recovery	Recovery starts stale	Rollback along Git history	Test-output consistency	Recovery, localization, assistance, cost
SWE-INTERACT [ 28 ]	Requirements and feedback	During user dialogue	Yes	Single- and multi-turn task comparison	Original task verifier	Resolve, goal progress, interaction cost
SWE-Together [ 33 ]	Requirements and corrective feedback	Reactive user turns	Yes	Reconstructed session intent; repeated runs	Frozen task rubric; separate external-work check not reported	Judge correctness; correction turns
SpecPath [ 32 ]	Specification history	Before implementation	No	Equivalent final contract; fresh identical repository	Fixed final-contract probes	Realization; conditional path violation

Appendix

Table 7: Different controls answer different questions about change. “Started?” refers to evaluated execution before the described update; N/A marks a fixed-snapshot baseline. Each row lists what the cited protocol controls and checks. “Not reported” means that the source leaves a separate check unspecified.

System and model card	Inference path	Decoding	Context	Max output
GPT-6 Astra [ 2026b ]	Codex app	xhigh	–	–
Claude Fable 5.1 [ 2026a ]	Claude Code app	xhigh	–	–
Claude Opus 5 [ 2026b ]	Claude Code app	xhigh	–	–
GPT-5.6 Sol [ 2026a ]	Codex app	xhigh	–	–
Kimi-K3 [ 2026b ]	vLLM 0.27.1	T=0, p=1	131,072	4,096
DeepSeek-V4-Pro [ 2026b ]	vLLM 0.27.1	T=0, p=1	131,072	4,096

Appendix

Table 8: Evaluated coding-agent systems. The year links each official model reference. rc1* denotes vLLM 0.28.1rc1.dev452+g144e79c81 ; dashes mark backend limits not exposed to the runner. Context and Max output are per-call limits in tokens; the post-fork token budget is

B=1{,}000{,}000

cumulative input plus output tokens per continuation.

Coding Agents

Coding agents use language models, tools, and feedback to modify software and complete engineering tasks.

See all 139 papers

Aug 3, 2026

SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

Real-world software development requires coding agents to operate in shared workspaces where users may inspect and modify code during an ongoing task, yet existing repository-level benchmarks typically evaluate agents working alone or restrict user participation to messages. This leads us to ask: how do coding agents understand and respond to code changes in a shared workspace? We introduce SWE-Touch, a framework that stress-tests this setting through validated Counter-Edits: plausible edits to task-relevant code that conflict with task completion. SWE-Touch mines task-critical regions from multiple repair trajectories, uses a separate User Patch Generator to construct the edits, and injects them with contextual user messages when agents reach the relevant code. We evaluate nine coding models on SWE-bench Verified, with additional experiments on longer-horizon tasks from SWE-Bench Pro and DeepSWE. Counter-Edit lowers average resolve rate by 7.7 percentage points on SWE-bench Verified, with degradation also persisting on both longer-horizon benchmarks. Trajectory analysis links these failures to limited awareness of the evolving workspace: agents may retain conflicting code or replace it without sufficiently re-inspecting the repository and validating the revised code with targeted tests. These findings show that strong autonomous performance does not yet ensure the state awareness and adaptive behavior needed for shared-workspace collaboration, and point to detecting workspace changes, reconciling conflicting edits with the task, and verifying the affected behavior as key capabilities for future optimization.

Yuqiao Tan, Jinxiang Meng, Fangyu Lei +4

Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences

Coding Agents Software Engineering Benchmarks

Ask PDF

Oct 16, 2026

SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

SWE-Bench Pro is a widely used benchmark of long-horizon software engineering tasks for coding agents. As agents improve, they reach task issues earlier agents never touched and interact with the evaluation environment in ways the original harness did not anticipate. We present SWE-Bench Pro V2, a revised public split built to keep the benchmark accurate as capabilities advance. Agentic audits, expert adjudication and blind re-implementation repaired or retired 480 of the 731 tasks: 23 contracted engineers spent 404 hours, each working in a language they write professionally, and every revised instruction was verified by a second engineer solving the task from the text alone. A hardened protocol restricts network access to the model endpoint and re-grades every patch in a clean container, so agents cannot retrieve upstream fixes or get credit for changes outside their patch. On the 642 V2 tasks frontier agents resolve up to 99.4% while other systems stay clearly separated, and a 51-task Hard split spreads systems from 52.9% to 98.0%. We release the benchmark and distill lessons for maintaining agent benchmarks over time.

Soham Dan, Jeff Da, Sanyam Bhutani +8

Scale AI · Reflection AI

Coding Agent Benchmarks Software Engineering Benchmarks Code Generation Benchmarks Long-Horizon Agent Evaluation Software Engineering Agents

Ask PDF

Jul 8, 2026

DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks

DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks follow SWE-bench in mining merged fixes from public GitHub repositories, which creates two problems: the fixes and their discussion were likely seen during pretraining, so a high score can reflect recall rather than problem-solving; and each task is graded by the tests that shipped with its merged fix, which were written to confirm one specific fix rather than grade an arbitrary solution, so they can fail a correct alternative or pass an incomplete one. DeepSWE avoids both. Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream, so their reference solutions stay out of the public record that model training scrapes; and each task is graded by a hand-written verifier that checks the requested functionality and accepts any implementation that provides it. When an independent LLM judge re-reviews graded runs, it disagrees with DeepSWE's verifier about an order of magnitude less often than with SWE-Bench Pro's inherited tests (1.4% versus 32.4%). Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code, and the benchmark separates frontier agents across a wider score band than the leaderboards on which they otherwise cluster. We release the benchmark, its verifiers, and the full record of evaluation trajectories.

Wenqi Huang, Charley Lee, Leonard Tng +1

1Datacurve

AI Coding Agents Software Engineering Benchmarks Long-Horizon Agent Evaluation Coding Agent Benchmarks

Ask PDF

Abstract

Coding Agents

Explore similar work

SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks