Coding agents increasingly collaborate on long-term projects. When taking over a project, they need to recover currently applicable requirements from the interaction history and update them in response to new requests. However, most coding benchmarks do not include long interaction histories, and some evaluations that consider history still do not cover the full process from requirement reconstruction to project delivery. We therefore provide agents with the interaction history, current request, and pre-request project environment, ask them to make the requested changes, and assess takeover performance through delivery tests. Across six model-harness configurations and 37 targets eligible for executable validation, the mean complete-delivery rate under Full History is only 4.95%. We then use requirement atoms to independently diagnose a larger set of takeover targets, examining relevant information identification, pre-request state reconstruction, and current request processing. We further supply correct pre-request or target states to examine the remaining difficulties in requirement processing and project modification. The diagnostics show that none of the six configurations produces fully correct post-request requirement states, and a case illustrates omission of previously recovered constraints during updating. With correct target states supplied, the mean complete-delivery rate is 8.11%, with build and regression pass rates of 96.40% and 90.99%, but target-test success remains 9.01%. These results show difficulties in both forming target requirement states and implementing supplied targets in existing projects. Based on these findings, we introduce ReqMemBench, which combines requirement diagnostics with delivery validation to support systematic evaluation of project takeover.
Figures & tables
Figure 1: Project takeover experiments. (a) Direct delivery from history, the current request, and the initial project, followed by three delivery checks. (b) Independent requirement diagnostics (RQ1: C1 only). (c) Oracle Pre-State predicts the target from the Gold pre-state, whereas Oracle Post-State supplies the Gold target directly, followed by delivery evaluation. Diagnostic responses are not inputs to the main experiment.
Diagnostic
What is evaluated
Core metrics
RQ1: Relevant information identification
Historical requirements and evidence relevant to the current task
Precision, Recall, and F1 computed separately for requirements and evidence
RQ2: Pre-request state reconstruction
Pre-request states of relevant requirements, Gq−
Attribute Reconstruction Score for matched requirements; Requirement Coverage
RQ3: Request update or clarification
ACT/CLARIFY decision, and Gq+ or material missing information and clarification questions
Table 1: What the requirement diagnostics evaluate.
Harness
Model / summary
Full History (C1)
Relevant History (C2)
Codex
GPT-6 Astra
10.81%
5.41%
GPT-6 Sol
5.41%
2.70%
GPT-5.6 Sol
2.70%
5.41%
GPT-5.6 Terra
5.41%
0.00%
Claude Code
Opus 5.5
5.41%
2.70%
Sonnet 5
0.00%
0.00%
Table 2: Complete-delivery rates (%) under C1/C2 on the same 37 targets. The mean weights all six configurations equally; aggregation rules are in Appendix C.2 .
Configuration
Evidence Precision
Evidence Recall
Requirement Coverage
Attribute Reconstruction Score
GPT-6 Astra
0.2254
0.9211
0.6689
0.2813
GPT-6 Sol
0.3395
0.8620
0.6777
0.2671
GPT-5.6 Sol
0.3162
0.8182
0.6157
0.2451
GPT-5.6 Terra
0.4254
0.6670
0.4950
0.2424
Claude Opus 5.5
0.3189
0.8364
0.5934
0.2811
Claude Sonnet 5
0.3060
0.7461
0.5704
0.2548
Table 3: Selected RQ1–RQ2 configuration means under C1 (0–1 scale). Scoring rules are in Appendix C.1 , with full results and applicable target counts in Appendix E.2 .
Figure 2: Post-request state diagnostics under Full History. Post-State Scores are reported over all Gold-ACT targets, the correct-ACT subset, and the subset with a complete requirement-set match, with target counts shown. The three series describe different evaluation subsets.
Configuration
Build
Target test
Regression
Complete delivery
C1
C2
Pre
Post
C1
C2
Pre
Post
C1
C2
Pre
Post
C1
C2
Pre
Post
GPT-6 Astra
56.76
43.24
48.65
94.59
16.22
8.11
5.41
13.51
45.95
37.84
40.54
81.08
10.81
5.41
5.41
10.81
GPT-6 Sol
48.65
32.43
43.24
94.59
8.11
2.70
5.41
16.22
45.95
24.32
35.14
81.08
5.41
2.70
2.70
13.51
GPT-5.6 Sol
59.46
45.95
62.16
97.30
2.70
5.41
10.81
13.51
51.35
37.84
56.76
94.59
2.70
5.41
10.81
13.51
GPT-5.6 Terra
54.05
37.84
51.35
91.89
5.41
0.00
2.70
5.41
48.65
27.03
45.95
89.19
5.41
0.00
2.70
5.41
Codex mean
54.73
39.86
51.35
94.59
8.11
4.05
6.08
12.16
47.97
31.76
44.59
86.49
6.08
3.38
5.41
10.81
Table 4: Pass rates for individual checks and complete delivery (%; 37 targets per configuration and condition). C1/C2 denote Full/Relevant History, and Pre/Post denote Oracle Pre-State/Post-State. Means weight their configurations equally; counts and aggregation rules are in Appendix E.1 .
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Field
Representation and meaning
Attributes, attributes
Store currently established parameters, behavior, and constraints. Each event records only what it establishes or changes; replay carries forward other attributes that remain valid.
Lifecycle status, lifecycle_status
ACTIVE means currently applicable, DEFERRED means implementation is postponed, and REMOVED means explicitly withdrawn. The value remains null when history is insufficient to determine it.
Scope, scope
Contains persistence , components , and contexts , which specify duration, applicable components, and contexts. A modification overwrites only dimensions explicitly changed.
Ambiguity, ambiguity
Records unresolved issues by source event, including the VALUE, SCOPE, or LIFECYCLE dimension, a description, and provenance. The value is null when no issue remains open.
Implementation evidence, execution
Stores the latest concrete implementation claim, failure, or successful validation, labeled CLAIMED_WORKING, FAILED, or VERIFIED_WORKING. The value is null when no evidence has been established.
Appendix
Table 5: Requirement-state dimensions.
Event
Main replay operation
INTRODUCE
Write initial attributes and scope, and set lifecycle status to ACTIVE. If only an incomplete observation baseline existed, clear its old implementation evidence.
MODIFY
Remove specified attributes, then apply attribute and scope updates. Carry forward unchanged content and clear implementation evidence to null .
DEFER
Set lifecycle status to DEFERRED, preserving existing attributes, scope, and implementation evidence.
RESUME
Set lifecycle status to ACTIVE without attribute or scope modifications.
REMOVE
Set lifecycle status to REMOVED, preserving historical attributes, scope, and implementation evidence.
AMBIGUOUS
Add an open issue, retaining the last established requirement without guessing alternative values or lifecycle status.
Each contributes at least one complete requirement-understanding target.
After target selection, before filtering for the complete diagnostic sequence
350 targets
Output of the target-point selection procedure.
Formal requirement-understanding targets
349 targets
Each target has RQ1, RQ2, and RQ3 files, with 349 files for each RQ.
Gold-ACT / Gold-CLARIFY
312 / 37 targets
Request-update and clarification branches after review and adjudication.
Formal delivery-validation targets
37 targets from 7 projects
The same target set is used under each history condition.
Appendix
Table 7: Dataset scope and target counts.
Statistical unit
Count
Mean
Median
Q1
Q3
Min–max
Complete project interactions
44 projects
181.43
110
74.25
204.50
18–1,075
All takeover targets, C1
349 targets
189.80
118
54
216
2–909
All takeover targets, C2
349 targets
9.74
6
3
10
1–90
Delivery targets, C1
37 targets
186.86
95
55
257
5–745
Delivery targets, C2
37 targets
8.70
7
4
12
1–40
Appendix
Table 8: Interaction and history lengths, measured in messages.
Figure 3: Data construction and input isolation. Requirement atoms, change events, and source evidence are annotated from project interactions, reviewed, and replayed chronologically to obtain requirement states. Selected takeover points yield 349 diagnostic targets, of which 37 additionally have initial project environments and executable validation. Public inputs are separated from Gold and hidden validators. C2 provides relevant historical messages rather than correct requirement states.
Review item
Counting unit
Gold items
Requirement boundaries
Requirement–target item
733
Evidence assignment and coverage
Required evidence group
2,108
Pre-request states
Requirement–target state item
733
ACT post-request states
Requirement–target state item
763
ACT/CLARIFY decisions
Takeover target
349
Missing information and clarification
Blocking issue
43
Appendix
Table 9: Gold items covered by human verification.
Label
Meaning
SAME_ATOM
The same independently evolving requirement, possibly worded differently.
MERGED_ATOMS
The prediction combines this Gold requirement with other requirements.
SUBPART_OF_ATOM
The prediction expresses only an attribute or rule fragment of the Gold requirement.
RELATED_DIFFERENT_ATOM
Related content, but different independently evolving requirements.
UNRELATED
No requirement-level correspondence.
UNCERTAIN
Insufficient information for a reliable judgment.
Appendix
Table 10: Semantic relation labels for requirement matching.
Comparator
English rule
NULL_EXACT
Requires explicit null; omission scores zero.
BOOLEAN_EXACT
Requires the same Boolean value and type.
NUMBER_EXACT
Requires an equal numeric value and excludes Booleans.
NORMALIZED_EXACT
Case-folds strings and collapses whitespace; other values use canonical JSON with sorted keys.
SEMANTIC_FACT
Only EQUIVALENT between text values scores one.
SET_F1
Uses F1 over normalized unique elements; mutually empty sets are N/A.
Appendix
Table 11: Field comparators and scoring rules.
Run outcome
Formal score
Handling
Agent incomplete, no valid deliverable submitted, or coding process failed
0
Record an agent delivery failure and retain the target in the denominator.
Build, task-specific test, and regression checks all PASS
1
Record delivery PASS.
Any validly executed check returns FAIL, with no HARNESS_ERROR
0
Record the primary failure in build, target, then regression order.
A check reports HARNESS_ERROR
No Delivery Pass Rate reported yet
Resolve the infrastructure or evaluator error, then reevaluate the frozen submission.
Checks unfinished or components incomplete
Pending evaluation
Retain pending status; do not count as PASS or reduce the denominator without justification.
Appendix
Table 12: Treatment of run outcomes in delivery scoring.
Agent Harness
Harness Version
Model Identifier
Requirement Diagnostic Budget
Coding Budget
Claude Code
2.1.283
claude-opus-5-5
900 s
1,800 s
Claude Code
2.1.283
claude-sonnet-5
900 s
1,800 s
Codex CLI
0.155.1
gpt-5.6-sol
900 s
1,800 s
Codex CLI
0.155.1
gpt-5.6-terra
900 s
1,800 s
Codex CLI
0.155.1
gpt-6-astra
900 s
1,800 s
Codex CLI
0.155.1
gpt-6-sol
900 s
1,800 s
Appendix
Table 13: Evaluated configurations and per-target time budgets.
Table 16: Semantic-judge request and response contracts.
Configuration
C1 (Full History)
C2 (Relevant History)
Oracle Pre-State
Oracle Post-State
Codex / GPT-6 Astra
4/37 (10.81%)
2/37 (5.41%)
2/37 (5.41%)
4/37 (10.81%)
Codex / GPT-6 Sol
2/37 (5.41%)
1/37 (2.70%)
1/37 (2.70%)
5/37 (13.51%)
Codex / GPT-5.6 Sol
1/37 (2.70%)
2/37 (5.41%)
4/37 (10.81%)
5/37 (13.51%)
Codex / GPT-5.6 Terra
2/37 (5.41%)
0/37 (0.00%)
1/37 (2.70%)
2/37 (5.41%)
Claude Code / Opus 5.5
2/37 (5.41%)
1/37 (2.70%)
2/37 (5.41%)
2/37 (5.41%)
Claude Code / Sonnet 5
0/37 (0.00%)
0/37 (0.00%)
0/37 (0.00%)
0/37 (0.00%)
Appendix
Table 17: Complete-delivery counts and rates.
Configuration
Condition
Build
Target
Regression
All three
GPT-6 Astra
Pre
18/37 (48.65%)
2/37 (5.41%)
15/37 (40.54%)
2/37 (5.41%)
GPT-6 Astra
Post
35/37 (94.59%)
5/37 (13.51%)
30/37 (81.08%)
4/37 (10.81%)
GPT-6 Sol
Pre
16/37 (43.24%)
2/37 (5.41%)
13/37 (35.14%)
1/37 (2.70%)
GPT-6 Sol
Post
35/37 (94.59%)
6/37 (16.22%)
30/37 (81.08%)
5/37 (13.51%)
GPT-5.6 Sol
Pre
23/37 (62.16%)
4/37 (10.81%)
21/37 (56.76%)
4/37 (10.81%)
GPT-5.6 Sol
Post
36/37 (97.30%)
5/37 (13.51%)
35/37 (94.59%)
5/37 (13.51%)
Appendix
Table 18: Oracle component counts and rates by configuration.
Configuration
Condition
Build
Target
Regression
All three
GPT-6 Astra
C1
21
6
17
4
GPT-6 Astra
C2
16
3
14
2
GPT-6 Sol
C1
18
3
17
2
GPT-6 Sol
C2
12
1
9
1
GPT-5.6 Sol
C1
22
1
19
1
GPT-5.6 Sol
C2
17
2
14
2
Appendix
Table 19: Component pass counts under C1 and C2 (37 targets per condition).
Configuration
Pre records
Pre submissions
Pre no submission
Post records
Post submissions
Post no submission
GPT-6 Astra
19
19
18
37
37
0
GPT-6 Sol
17
17
20
37
37
0
GPT-5.6 Sol
23
23
14
37
37
0
GPT-5.6 Terra
19
19
18
34
34
3
Opus 5.5
30
30
7
37
37
0
Sonnet 5
28
28
9
37
37
0
Appendix
Table 20: Oracle process and submission counts.
Configuration
Requirement
Evidence
P
R
F1
P
R
F1
GPT-6 Astra
0.2577
0.6821
0.3423
0.2254
0.9211
0.3166
GPT-6 Sol
0.3196
0.6704
0.3980
0.3395
0.8620
0.4207
GPT-5.6 Sol
0.3768
0.6302
0.4331
0.3162
0.8182
0.3939
GPT-5.6 Terra
0.4428
0.5232
0.4507
0.4254
0.6670
0.4416
Claude Opus 5.5
0.3567
0.6232
0.4193
0.3189
0.8364
0.4048
Appendix
Table 21: Historical information identification and pre-request state reconstruction. (a) Requirement and evidence identification under Full History (C1). (b) Attribute reconstruction and requirement coverage; within each numeric cell, C1 is above C2. Coverage uses 349 tasks per configuration. Attribute scores exclude legitimate N/A cases and retain zero scores for agent failures. Attribute-score sample sizes are given in Table 22 .
Configuration
N C1
N C2
GPT-6 Astra
265
266
GPT-6 Sol
277
252
GPT-5.6 Sol
251
249
GPT-5.6 Terra
221
251
Claude Opus 5.5
253
246
Claude Sonnet 5
238
241
Appendix
Table 22: Numbers of targets included in the Attribute Reconstruction Score means.
Configuration
Decision Accuracy
Balanced Accuracy
ACT Recall
CLARIFY Recall
GPT-6 Astra
0.5931 0.4670
0.6771 0.6066
0.5705 0.4295
0.7838 0.7838
GPT-6 Sol
0.5043 0.3954
0.6156 0.5666
0.4744 0.3494
0.7568 0.7838
GPT-5.6 Sol
0.5358 0.4327
0.6094 0.5874
0.5160 0.3910
0.7027 0.7838
GPT-5.6 Terra
0.5415 0.4413
0.6483 0.6041
0.5128 0.3974
0.7838 0.8108
Claude Opus 5.5
0.7880 0.7335
0.6670 0.6961
0.8205 0.7436
0.5135 0.6486
Claude Sonnet 5
0.6361 0.5673
0.6774 0.6270
0.6250 0.5513
0.7297 0.7027
Appendix
Table 23: Action-selection performance under C1/C2. Paired entries show C1 above C2.
Configuration
Post-State
Blocking-Issue F1
Question Validity
Strict Clarification Success Rate
GPT-6 Astra
0.0741 0.0744
0.0874 0.0667
0.1622 0.1351
0.0000 0.0000
GPT-6 Sol
0.0535 0.0600
0.1051 0.0667
0.2162 0.1351
0.0000 0.0000
GPT-5.6 Sol
0.0538 0.0569
0.0739 0.0802
0.1622 0.1622
0.0000 0.0000
GPT-5.6 Terra
0.0653 0.0870
0.0676 0.0901
0.1351 0.1351
0.0000 0.0270
Claude Opus 5.5
0.1670 0.1710
0.0423 0.1523
0.0811 0.2703
0.0000 0.0000
Claude Sonnet 5
0.1041 0.1064
0.0694 0.0874
0.1081 0.1351
0.0270 0.0270
Appendix
Table 24: Post-request state and clarification scores under C1/C2. Paired entries show C1 above C2.
Configuration
Correct ACT count C1
Conditional Post-State C1
Correct ACT count C2
Conditional Post-State C2
GPT-6 Astra
178
0.1298
134
0.1733
GPT-6 Sol
148
0.1129
109
0.1719
GPT-5.6 Sol
161
0.1043
122
0.1456
GPT-5.6 Terra
160
0.1273
124
0.2189
Claude Opus 5.5
256
0.2036
232
0.2300
Claude Sonnet 5
195
0.1665
172
0.1929
Appendix
Table 25: Post-State Scores conditional on correct ACT decisions.
Configuration
Matched-set count C1 / C2
Matched-set Post C1 / C2
Shared ACT count
Shared-target Post C1 / C2
GPT-6 Astra
50 41
0.2882 0.3715
127
0.1384 0.1710
GPT-6 Sol
36 38
0.2422 0.3492
99
0.1317 0.1591
GPT-5.6 Sol
39 40
0.2299 0.3215
107
0.1051 0.1528
GPT-5.6 Terra
46 41
0.2899 0.4563
100
0.1471 0.2358
Claude Opus 5.5
79 68
0.3656 0.4468
221
0.2082 0.2320
Claude Sonnet 5
54 57
0.3320 0.3604
138
0.1709 0.2029
Appendix
Table 26: Requirement-set matching and shared-target Post-State Scores under C1/C2. Paired entries show C1 above C2.
Evaluation
Prediction reference
Gold requirement
Relation
Target metric
RQ1
reward_payout
Prize Claim Flow
MERGED_ATOMS
Requirement Recall = 0
RQ2
reward_payout
Prize Claim Flow
SAME_ATOM
Requirement Coverage = 0.25
Appendix
Table 27: Cross-diagnostic label disagreement before human adjudication.
Configuration
No correct ACT
Set mismatch
State not fully exact
Removal list only
Exact success
GPT-6 Astra
134 / 178
128 / 93
50 / 41
0 / 0
0 / 0
GPT-6 Sol
164 / 203
112 / 71
36 / 38
0 / 0
0 / 0
GPT-5.6 Sol
151 / 190
122 / 82
39 / 40
0 / 0
0 / 0
GPT-5.6 Terra
152 / 188
114 / 83
46 / 41
0 / 0
0 / 0
Claude Opus 5.5
56 / 80
177 / 164
79 / 68
0 / 0
0 / 0
Claude Sonnet 5
117 / 140
141 / 115
54 / 57
0 / 0
0 / 0
Appendix
Table 28: Decomposition of exact post-request state failures under C1/C2.
Opus diagnostic record
Saved field and result
Pre-request mechanical requirement
requirement_ref = req_power_board_outline; mounting_holes = ”8 at the corners”; mounting_hole_size = ”M2.5”
Post-request references
only req_battery_connector_placement; no req_power_board_outline
Coding-agent benchmarks evaluate whether a single uninterrupted agent can resolve a repository issue. Real software work is messier: tasks are interrupted, reassigned, reviewed, and resumed from partial states left by another agent or engineer. We study this missing dimension through handoff debt: the rediscovery cost imposed when a predecessor's work is opaque or incomplete. Our takeover protocol interrupts a coding agent at deterministic handoff points, freezes the repository, and evaluates successor agents under four handoff views: repository state only, raw trace, summary notes, and structured notes. Across 75 source tasks, the protocol generates 181 handoff-point tasks and 724 takeover runs per successor model. Across three successor models, context-bearing handoffs reduce median agent events by 20-59% and cumulative prompt tokens by 42-63% relative to repository-only takeover. Solved-rate effects are smaller and model-dependent, but efficiency gains are consistent. These findings suggest that coding-agent evaluation should report not only whether a task is solved, but also how costly that work is for another agent to resume.
Coding agents are increasingly used as iterative development partners, but most benchmarks still evaluate one specification followed by one final assessment. This leaves out a basic question: can an agent keep its own codebase working as requirements change? We introduce EvoCode-Bench, a benchmark of 26 stateful coding tasks and 227 evaluated rounds. Each task preserves the agent's workspace for 5-15 rounds, states requirements through observable behavior, and uses cumulative executable tests to check new requirements and still-active prior ones. We evaluate 13 coding agents with two metrics: MT@4, a four-attempt fail-stop multi-round score, and SR, a single-round score from a reference-completed prior state. For most agents, SR exceeds MT@4 by 22-40 points. The gap also changes rankings: the highest-SR agent (78.9) ranks only third in persistent execution (44.0 MT@4). Even the strongest agents achieve only about 50% success on multi-turn metrics, and aggregate pass rate drops below half of round-1 performance by round 5. Failure analysis shows tier-dependent behavior: weaker agents fail early, while stronger agents survive long enough to expose specification-tracking and regression failures. We release the benchmark data and Harbor multi-turn infrastructure.
Haiyang Shen, Xuanzhong Chen, Wendong Xu +3
1UniPat AI · 2Peking University · 3Tsinghua University +1
Coding agents are autonomous developers. Given a request, they change the code and report back when done. As such, they can handle not only independent requests but whole chains of them, each request built on the last. Agents are known to report wrong work as done, yet benchmarks still test them one request at a time, on human-written code, checked once. We build BOBA-Bench, 280 chains of 20 consecutive requests drawn from 140 real open-source projects, and evaluate ten coding agents on both. All ten do far worse on chains. On average, they complete 95% of requests taken one at a time, but when requests are chained together, they complete 51%. Worse code is not the cause. Once checked, agent code is nearly as good a base for later requests as human code. The cause is the mistakes agents miss. Coding agents declare a request done once their own tests pass, even when those tests miss a mistake. The next request then builds on broken code and fails about twice as often. The mistake is rarely found later. Nor can agents catch such mistakes by reviewing code. Asked whether an implementation is correct, they are right 53% of the time, barely better than a coin flip. Coding agents are therefore heavily dependent on access to verification. A single external ``that's wrong'' after a failed request and one repair round lift chain completion from 51% to 59%, and a repaired mistake no longer hurts the requests that follow. In simpler terms, coding agents are only as reliable as the checks they can reach: without verification, every request that follows is built on broken assumptions.