Software development sessions often involve several unfinished tasks rather than one issue carried through to completion. As coding agents take on repository-level development, they need to recover an earlier task's requirements and partial progress after intervening work. Existing repository-level benchmarks typically score one issue-to-patch episode and do not directly measure whether an agent can resume a task after advancing other unfinished work in the same repository. We introduce \ourmethod, a benchmark for focus-task resumption in sessions that interleave unfinished tasks drawn from the same source repository, with a persistent checkout for each task. Each focus task is evaluated in matched uninterrupted and switched sessions with identical focus requirements and executable verifiers. The switched session interleaves ordered stages from other unfinished tasks in the same repository, while only the final focus patch is scored. \ourmethod~contains 200 focus tasks from 19 Python repositories, reconstructed from issue, pull-request, and review histories and rendered as 400 matched session specifications. Across 7 coding agents, interleaved unfinished work produces an average focus-task resolution gap of 11.5 percentage points, ranging from 6.0 to 15.0 percentage points. Trajectory analysis identifies cross-task requirement confusion and loss of task state as recurrent resumption behaviors. We introduce a plug-in context-management module that selects task-relevant history and maintains concise task-state summaries. On OpenCode, the module improves focus-task completion under same-repository task switching by 5.5 to 15.5 percentage points across the evaluated models. Code is available at github.com/Miraclecsc/Switch-SWE.
Figures & tables
Figure 1: Task switching increases cost while reducing completion. (a) Moving from uninterrupted to switched execution lowers focus-task resolution and raises cost across all seven configurations. (b) Failures span implementation, submission, and incomplete execution.
Figure 2: Overview of Switch-SWE. (a) Repository-level tasks are represented as ordered, cumulative requests grounded in issue, pull-request, and review histories. (b) Each focus task is evaluated in matched uninterrupted and switched sessions. The switched session interleaves unfinished same-repository tasks as interruption. (c) Only the final focus-task patch is scored using the same executable verifier in both conditions. Interruption-task implementations do not enter primary score.
Model
Harness
Overall
Near
Far
Unint.
Switch ↑
Gap ↓
Unint.
Switch ↑
Gap ↓
Unint.
Switch ↑
Gap ↓
DeepSeek-V4.1-Flash
OpenCode
46.5
31.5
15.0
45.0
34.0
11.0
48.0
29.0
19.0
GLM-5.3
OpenCode
49.0
43.0
6.0
47.0
46.0
1.0
51.0
40.0
11.0
Kimi K3
KimiCode
48.0
34.5
13.5
47.0
35.0
12.0
49.0
34.0
15.0
GPT-5.6 Terra (medium)
Codex
41.0
28.0
13.0
42.0
32.0
10.0
40.0
24.0
16.0
GPT-5.6 Sol (medium)
Codex
52.5
38.0
14.5
53.0
40.0
13.0
52.0
36.0
16.0
Table 1: Focus-task resolution with and without task switching. Overall includes 200 tasks; Near and Far include 100 each. Unint. and Switch report resolution (%); Gap is Unint. minus Switch (% ). Within each group, bold marks the highest Switch and lowest Gap; underlining marks the lowest Switch and highest Gap, including ties.
Model
Harness
Tokens ( 106 )
Model calls
Tool calls
Unint.
Switch All
Switch Focus
Unint.
Switch All
Switch Focus
Unint.
Switch All
Switch Focus
DeepSeek-V4.1-Flash
OpenCode
8.594
101.953
25.375
112.2
402.2
120.3
115.2
405.5
124.7
GLM-5.3
OpenCode
4.904
14.970
12.511
94.5
163.7
94.5
102.8
181.4
103.1
Kimi K3
KimiCode
4.739
24.335
6.677
70.5
191.0
79.7
71.5
193.1
80.8
GPT-5.6 Terra (medium)
Codex
1.511
9.741
2.880
38.4
104.8
35.6
32.9
87.5
30.0
GPT-5.6 Sol (medium)
Codex
2.294
16.921
4.047
48.4
170.1
49.3
48.6
135.6
47.0
Table 2: Execution workload with and without task switching. Values are mean per-run totals; tokens are in millions. Unint. covers the full uninterrupted session; Switch-All covers the full switched session; Switch-Focus covers resumed substantive focus stages, excluding the initial request and finalization. Reasoning settings follow Table 1 .
Model
Harness
Overall
Near
Far
Unint.
Switch
Gap
Unint.
Switch
Gap
Unint.
Switch
Gap
Native
DeepSeek-V4.1-Flash
OpenCode
46.5
31.5
15.0
45.0
34.0
11.0
48.0
29.0
19.0
GLM-5.3
OpenCode
49.0
43.0
6.0
47.0
46.0
1.0
51.0
40.0
11.0
+Router
DeepSeek-V4.1-Flash
OpenCode
45.5
42.5
3.0
47.0
42.0
5.0
44.0
43.0
1.0
Table 3: Context management improves switched completion on OpenCode. Overall/Near/Far contain 200/100/100 tasks. Unint. and Switch report resolution (%); Gap is their difference in %. Bold marks the highest Switch within each model and task group.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset profile
Benchmark capabilities
Benchmark name
Size
Repos / sites
Primary scope
MSI
SCT
IFT
RTR
PPTW
SWE-bench ( Jimenez et al., 2024 )
2,294
12
Issue repair
✗
✗
✗
✗
✗
SWE-bench-Live ( Zhang et al., 2026a )
1,319
93
Fresh issue repair
✗
✗
✗
✗
✗
SWE-ContextBench ( Zhu et al., 2026 )
300+99
12
Context reuse
◐
✓
✗
✗
✗
InterruptBench ( Zou et al., 2026 )
165
5
Web goal changes
✓
✗
◐
◐
✗
SWE-Touch ( Tan et al., 2026 )
500
12
User code edits
✓
✗
◐
◐
✗
Appendix
Table 4: Dataset profiles and capabilities of nearby agent benchmarks . The capability headings denote multi-stage interaction (MSI), separate coding tasks (SCT), interleaved unfinished tasks (IFT), repeated task resumption (RTR), and persistent per-task workspaces (PPTW). Check marks indicate support, circles partial or adjacent coverage, and crosses no support.
Harness
Models and reasoning effort
Version
OpenCode
DeepSeek-V4.1-Flash, GLM-5.3
1.18.2
KimiCode
Kimi K3
0.42.0
Codex
GPT-5.6 Terra, GPT-5.6 Sol
0.154.0
Claude Code
Claude Opus 5
2.1.269
Gemini CLI
Gemini 3.7 Flash
0.60.0
Appendix
Table 5: Evaluated harnesses and models. One row denotes one native coding harness.
Focus tasks
Repos.
Interruption tasks
Requests
Workspace switches
Focus returns
200
19
4
22
15
3
Appendix
Table 6: Benchmark profile. Each focus task yields one matched pair of uninterrupted and switched sessions. Focus-task and repository counts are dataset totals; the interruption count is the number of distinct interruption tasks per switched session. Request, workspace-switch, and focus-return counts are medians over switched sessions; requests include finalization.
Repository
Cumulative objective
F2P / P2P
Packaging
Restore serialized tags across supported state formats while preserving an existing object’s identity and hash when restoration fails.
13 / 217
Click
Reconcile flag activation with literal and callable defaults while preserving Boolean values and callable flag values.
7 / 732
AnyIO
Compose child-session creation with validated, single-pass file-descriptor inheritance.
10 / 18
Moto
Preserve S3–SNS–SQS raw delivery while selecting the subscription-specific payload before message-body filtering.
5 / 8
NetworkX
Construct planar dual multigraphs and contract selected dual edges without losing original face provenance.
25 / 41
Appendix
Table 7: Illustrative Switch-SWE tasks. F2P and P2P are the designated fail-to-pass and pass-to-pass tests used by the final focus verifier.
Signature
Trace evidence
Requirement-state loss
A returned focus edit or plan violates an earlier focus requirement, forgets a prior decision, or fails to preserve an invariant established before interruption.
Cross-task interference
Work associated with another checkout enters the focus task’s reasoning, edits, or final-submission scope.
Appendix
Table 8: Trajectory signatures used in the case studies. The signatures describe observable resumption behavior and are separate from the binary resolution metric.
Figure 3: Resumption Context Manager (RCM). When task A resumes, the Task History Router selects relevant interactions from the shared history, while Task State Memory supplies accumulated requirements, progress, and pending work. These signals are combined with the current request and shared context for the coding model. Completed interactions inform subsequent state updates; the native model–tool loop and task checkout are preserved.
Repository-level coding benchmarks largely assume that tasks and environments remain unchanged throughout execution. In software development, code, dependencies, runtime state, and requirements can change while an agent is still working. We introduce SWE-Shift, a benchmark for evaluating coding agents under such mid-execution changes. From the same saved history and workspace, a Clean continuation proceeds without intervention, while a matched Shifted continuation receives a task-relevant update. Success requires completing the current task, preserving still-valid work, and satisfying the applicable state and evidence checks. The benchmark contains 120 scenarios across 12 change families, each evaluated at nine scheduled intervention positions. We evaluate 17 coding-agent systems, all of which have lower Shifted than Clean success. GPT-6 Astra drops from 67.35% to 24.79%, with 20.34% paired success. Tool/session changes and stale evidence yield zero paired success across all systems. Late-position paired success is lower for some systems and higher for others. These results show that success on the unchanged task can coexist with failures to preserve work, recover state, or refresh evidence after an update.
Victor Shea-Jay Huang, Rongqing Li, Zhangyue Yin +7
MMLab, The Chinese University of Hong Kong · Somewhere on the Earth · Fudan NLP Lab, Fudan University +2
Coding-agent benchmarks evaluate whether a single uninterrupted agent can resolve a repository issue. Real software work is messier: tasks are interrupted, reassigned, reviewed, and resumed from partial states left by another agent or engineer. We study this missing dimension through handoff debt: the rediscovery cost imposed when a predecessor's work is opaque or incomplete. Our takeover protocol interrupts a coding agent at deterministic handoff points, freezes the repository, and evaluates successor agents under four handoff views: repository state only, raw trace, summary notes, and structured notes. Across 75 source tasks, the protocol generates 181 handoff-point tasks and 724 takeover runs per successor model. Across three successor models, context-bearing handoffs reduce median agent events by 20-59% and cumulative prompt tokens by 42-63% relative to repository-only takeover. Solved-rate effects are smaller and model-dependent, but efficiency gains are consistent. These findings suggest that coding-agent evaluation should report not only whether a task is solved, but also how costly that work is for another agent to resume.
Development teams need reproducible tasks from their own repositories and workflows to evaluate coding agents. Conversational developer-agent collaboration ("vibe coding") produces abundant coding sessions and repository checkpoints, but requirements evolve across turns, unrelated conversations co-occur with commits, and engineering efforts can span multiple sessions. We present SessionFarmer, an automated framework that turns these records into self-contained, executable software engineering tasks. It analyzes dialogue and commit history to filter unrelated sessions and group multi-turn, potentially multi-session interactions into work episodes centered on one engineering goal. For each episode, it reconstructs a hermetic execution container, synthesizes a task statement from the full context, and authors repository-grounded engineering-quality rubrics and unit tests. Closed-loop test refinement uses requirement-breaking mutants and a blind solver to reduce false rejections of valid alternatives and false acceptances of incomplete solutions. We construct CropChat, an open benchmark of 242 tasks across 61 public repositories and 14 programming languages, and CropEnterprise from proprietary enterprise logs. In a CropEnterprise audit, the original developers endorse the vast majority of reconstructed episode boundaries, task statements, and quality rubrics. Full-episode context covers substantially more user requirements than single-turn or single-session inputs. Test refinement cuts false-rejection rates and raises balanced accuracy against reference patch labels by 7-9 percentage points across solvers. Across 12 model-harness configurations on CropChat, the strongest setup achieves 39.3% Pass@1, while harness choice shifts Pass@1 by up to 8.4 percentage points for the same model.
Ao Lu, Yuanhao Zeng, Shaocheng Lan +5
ShanghaiTech University · ByteDance, China · Shanghai AI Laboratory