SWE-Shift: Evaluating Coding Agents in Evolving Workspaces

Victor Shea-Jay Huang; Rongqing Li; Zhangyue Yin; Kelong Mao; Yaqi Zhao; Xinyuan Wang; Yao Yin; Lirui Luo

cs.AIOct 23, 2026

SWE-Shift: Evaluating Coding Agents in Evolving Workspaces

Authors: Victor Shea-Jay Huang, Rongqing Li, Zhangyue Yin, Kelong Mao, Yaqi Zhao, Xinyuan Wang, Yao Yin, Lirui Luo,

Organizations: MMLab, The Chinese University of Hong Kong · Somewhere on the Earth · Fudan NLP Lab, Fudan University · The Hong Kong University of Science and Technology ·

Abstract

Repository-level coding benchmarks largely assume that tasks and environments remain unchanged throughout execution. In software development, code, dependencies, runtime state, and requirements can change while an agent is still working. We introduce SWE-Shift, a benchmark for evaluating coding agents under such mid-execution changes. From the same saved history and workspace, a Clean continuation proceeds without intervention, while a matched Shifted continuation receives a task-relevant update. Success requires completing the current task, preserving still-valid work, and satisfying the applicable state and evidence checks. The benchmark contains 120 scenarios across 12 change families, each evaluated at nine scheduled intervention positions. We evaluate 17 coding-agent systems, all of which have lower Shifted than Clean success. GPT-6 Astra drops from 67.35% to 24.79%, with 20.34% paired success. Tool/session changes and stale evidence yield zero paired success across all systems. Late-position paired success is lower for some systems and higher for others. These results show that success on the unchanged task can coexist with failures to preserve work, recover state, or refresh evidence after an update.

Figures & tables

Figure 1: Success drops after workspace changes. Clean (red) and Shifted (blue) success rates across 17 coding-agent systems, ordered by Clean success.

Figure 2: Per-system evaluation pipeline of SWE-Shift. From the same saved execution state, the agent resumes in two matched continuations: Clean proceeds without intervention, while Shifted introduces a task-relevant change. Outcomes are evaluated under the original and updated conditions, respectively.

Benchmark	Mid- episode	Same prefix	Cross- plane	Preserve/ harm	Multi- position	Trajectory oracle	Fresh evidence
SpecPath	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$
SyncMind	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$
SWE-INTERACT	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$
ToolSandbox	$\times$	$\times$	$\times$	$\checkmark$	$\times$	$\checkmark$	$\times$
SWE-Touch	$\checkmark$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\times$	$\times$
Gaia2	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\times$

Table 1: Benchmark design at a glance.

Figure 3: What SWE-Shift evaluates. The 12 change families are organized into three broad groups. World Shift (F01–F07) tests changes to executable state or dependencies. Goal Interaction (F08–F10) tests newly disclosed requirements, corrective feedback, and revised goals. Observation Shift (F11–F12) tests stale or misbound evidence. Each panel illustrates the intervention and the required continuation.

System	Cln. $\uparrow$	Shf. $\uparrow$	PSR $\uparrow$	CRR $\uparrow$
GPT-6 Astra [ 2026b ]	67.35	24.79	20.34	30.20
Claude Fable 5.1 [ 2026a ]	62.01	18.33	16.88	26.02
Claude Opus 5 [ 2026b ]	53.76	16.09	14.78	26.54
GPT-5.6 Sol [ 2026a ]	46.22	12.56	11.30	23.94
Kimi-K3 [ 2026b ]	31.41	12.09	8.20	25.12
DeepSeek V4-Pro [ 2026b ]	29.37	10.29	8.83	29.15

Table 2: Continuation performance (%). Full identities appear in Table 8 .

Figure 4: Paired success across changes and intervention positions. PSR by change family (a) and intervention position (b). Solid coral outlines highlight tool/session changes (F06) and stale evidence (F11). Dashed gold outlines highlight runtime/resource changes (F05) and disclosure (F08).

Panel of Figure 5 — Figure 5: Effects of state handling and post-fork token budget on PSR. (a) Bars show cumulative state-access interventions from the Table 2 baseline to the final PSR. (b) Curves show PSR across post-fork token budgets. $B$ is the baseline of 1,000,000 input plus output tokens per continuation, and the gray region marks $8B$ – $16B$ .

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

State or execution	Required result
Initial root	Original task checks fail.
Clean reference repair	All Clean success checks pass.
Update without repair	External state remains valid; the task still fails.
Unadapted continuation	Clean succeeds; Shifted task or preservation checks fail.
Repair based on the pre-update state	Execution is protocol-valid; Shifted strict success fails.
Valid recovery after the update	Current task and preservation checks both pass.

Appendix

Table 3: Scenario admission checks. Each scenario must pass all checks before evaluation.

Base task source	Change scenarios
SWE-bench Verified [ 5 ]	60
Multi-SWE-bench [ 37 ]	35
SWE-bench Multilingual [ 15 ]	25
Total	120

Appendix

Table 4: Base task sources for SWE-Shift. Number of SWE-Shift change scenarios built from tasks in each source benchmark.

Family	What changes	Required continuation
F01 User implementation	Valid code is added in the agent’s edit region.	Preserve the user’s behavior while completing the original task.
F02 User contract	A public API or data contract changes.	Meet the updated contract and retained compatibility checks.
F03 Direct collaborator	A collaborator changes the target code or a direct dependency.	Integrate the changes while preserving the collaborator’s behavior.
F04 Indirect collaborator	A transitive dependency changes while the local target stays untouched.	Account for the dependency change and preserve the collaborator’s behavior.
F05 Runtime and resources	Process, port, lock, cache, or resource state changes.	Restore a valid runtime state while preserving unrelated resources.
F06 Tool and session	Session, authorization, schema, or protocol changes invalidate earlier calls.	Use the current permitted interface and preserve unrelated session state.

Appendix

Table 5: Recovery requirements for the 12 change families. Each row pairs a task-relevant change with the behavior a successful continuation must preserve or restore.

Check	Required evidence
Task	Executable task checks pass in the terminal state.
Preservation	Specified behavior and state invariants remain valid.
Prohibited effects	State and event checks find no forbidden outcome.
Current authority	Goal and verifier versions match the scenario’s current requirements.
Fresh observation	An allowed observation occurs after the update and before the first agent mutation.
Recovery order	Required recovery steps occur in the specified order.

Appendix

Table 6: Evidence required for strict success. Every applicable check must pass in a protocol-valid execution.

Benchmark	Changed object	When	Started?	History control	Preservation / harm checks	Outcome
SWE-bench [ 13 ]	Repository repair	Initial task snapshot	N/A	Fixed issue and base commit	Previously passing tests	Resolved issue
SWE-Touch [ 30 ]	Code + contextual user message	Region access; relative schedules	Yes	Matched task conditions; independent runs	Task regression tests; Co-Edit control	Resolve, retention, steps, tokens
SyncMind [ 10 ]	Caller or callee made out of sync	Before recovery	Recovery starts stale	Rollback along Git history	Test-output consistency	Recovery, localization, assistance, cost
SWE-INTERACT [ 28 ]	Requirements and feedback	During user dialogue	Yes	Single- and multi-turn task comparison	Original task verifier	Resolve, goal progress, interaction cost
SWE-Together [ 33 ]	Requirements and corrective feedback	Reactive user turns	Yes	Reconstructed session intent; repeated runs	Frozen task rubric; separate external-work check not reported	Judge correctness; correction turns
SpecPath [ 32 ]	Specification history	Before implementation	No	Equivalent final contract; fresh identical repository	Fixed final-contract probes	Realization; conditional path violation

Appendix

Table 7: Different controls answer different questions about change. “Started?” refers to evaluated execution before the described update; N/A marks a fixed-snapshot baseline. Each row lists what the cited protocol controls and checks. “Not reported” means that the source leaves a separate check unspecified.

System and model card	Inference path	Decoding	Context	Max output
GPT-6 Astra [ 2026b ]	Codex app	xhigh	–	–
Claude Fable 5.1 [ 2026a ]	Claude Code app	xhigh	–	–
Claude Opus 5 [ 2026b ]	Claude Code app	xhigh	–	–
GPT-5.6 Sol [ 2026a ]	Codex app	xhigh	–	–
Kimi-K3 [ 2026b ]	vLLM 0.27.1	T=0, p=1	131,072	4,096
DeepSeek-V4-Pro [ 2026b ]	vLLM 0.27.1	T=0, p=1	131,072	4,096

Appendix

Table 8: Evaluated coding-agent systems. The year links each official model reference. rc1* denotes vLLM 0.28.1rc1.dev452+g144e79c81 ; dashes mark backend limits not exposed to the runner. Context and Max output are per-call limits in tokens; the post-fork token budget is

B=1{,}000{,}000

cumulative input plus output tokens per continuation.

Coding Agents

Coding agents use language models, tools, and feedback to modify software and complete engineering tasks.

See all 139 papers

SWE-Touch: Benchmarking Coding Agents When Users Touch the Code
Aug 3, 2026Yuqiao Tan, Jinxiang Meng, Fangyu Lei +4Coding Agents Software Engineering Benchmarks
Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
Oct 16, 2026Soham Dan, Jeff Da, Sanyam Bhutani +8Coding Agent Benchmarks Software Engineering Benchmarks
Scale AI · Reflection AI
DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks
Jul 8, 2026Wenqi Huang, Charley Lee, Leonard Tng +1AI Coding Agents Software Engineering Benchmarks
1Datacurve

Aug 3, 2026J/K move · Enter open · S save

SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

Yuqiao Tan, Jinxiang Meng, Fangyu Lei, Minzheng Wang+3

Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences

Coding Agents Software Engineering Benchmarks

Open paper AskPDF arXiv

Real-world software development requires coding agents to operate in shared workspaces where users may inspect and modify code during an ongoing task, yet existing repository-level benchmarks typically evaluate agents working alone or restrict user participation to messages. This leads us to ask: how do coding agents understand and respond to code changes in a shared workspace? We introduce SWE-Touch, a framework that stress-tests this setting through validated Counter-Edits: plausible edits to task-relevant code that conflict with task completion. SWE-Touch mines task-critical regions from multiple repair trajectories, uses a separate User Patch Generator to construct the edits, and injects them with contextual user messages when agents reach the relevant code. We evaluate nine coding models on SWE-bench Verified, with additional experiments on longer-horizon tasks from SWE-Bench Pro and DeepSWE. Counter-Edit lowers average resolve rate by 7.7 percentage points on SWE-bench Verified, with degradation also persisting on both longer-horizon benchmarks. Trajectory analysis links these failures to limited awareness of the evolving workspace: agents may retain conflicting code or replace it without sufficiently re-inspecting the repository and validating the revised code with targeted tests. These findings show that strong autonomous performance does not yet ensure the state awareness and adaptive behavior needed for shared-workspace collaboration, and point to detecting workspace changes, reconciling conflicting edits with the task, and verifying the affected behavior as key capabilities for future optimization.

Abstract

Coding Agents

Explore similar work

SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks