Software development with coding agents often begins with a rough request that users refine after trying the implementation. Final correctness alone does not capture the repeated guidance this process requires. We introduce FeatureLoop, a benchmark of 61 human-reviewed tasks from 34 repositories across five primary programming languages and seven software forms. Each task follows one development goal in a persistent workspace through evolution feedback, which adds or clarifies requirements, and completion feedback, which reports failures of stated requirements. Cumulative executable checks track new behavior, repair, and preservation of earlier functionality. We evaluate 12 frontier models and find that the strongest completes 72.1% of tasks with both feedback types. Across models, however, only 15.4% of observed failed stages recover within two corrective revisions. Further analysis reveals incomplete repairs, regressions, and model-specific differences in feedback use and token efficiency, identifying obstacles to reliable, low-intervention development.
Figures & tables
Figure 1: An Excalidraw interaction excerpt. The canvases show successive implementations; the faded shape marks the position before dragging.
Figure 2: Three development scenarios. Green: new requests; orange: corrections.
Figure 3: Construction stages and their output artifacts.
Figure 4: Dataset composition and cumulative implementation size. (a) Task counts by programming language and software form, including one mixed Go/TypeScript task. (b) Means and interquartile ranges of cumulative test groups, nonblank test lines, and reference-code additions plus deletions. Stage i includes tasks that reach it; the final column includes all 61 tasks.
StrictPass ↑
Evo+Comp path
Model
Full
Evo
Evo+Comp
FDAC ↑
NE
NC
GPT-6 Astra
59.0
68.9
72.1
46.5
2.25
2.00
GPT-5.6 Sol
52.5
62.3
70.5
45.4
2.23
1.89
Kimi K3
39.3
54.1
70.5
44.2
2.25
1.87
Hy4-Preview
47.5
52.5
68.9
42.7
2.28
2.08
DeepSeek V4 Flash
37.7
57.4
70.5
41.2
2.25
1.97
Table 1: Main results, ordered by Evo+Comp FDAC. StrictPass and FDAC are percentages; NE and NC are mean messages per task, including failed runs. Each model is evaluated on the same 61 tasks. Best observed outcome values and the lowest correction count are bold.
Figure 5: Model-level recovery and retention under both feedback types. Recovery uses failed states; retention uses previously passing requirements with later observations.
Figure 6: Primary diagnoses within each task set’s final failures. “Other” groups the remaining diagnoses.
Figure 7: Primary diagnoses within each model’s final failures; cell values are percentages.
Figure 8: Model-level completion and token expenditure under both feedback types. Cost includes tokens spent on failed runs.
Appendix figures & tables58 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Later information
Behavioral evaluation
Interaction measure
SWE-Together
Session-conditioned requests and corrections
Final rubric with execution evidence
User Correction
Asuka-Bench
Feedback from browser checks
Repeated UI criteria
Per-round fix rates
EvoCode-Bench
Fixed instruction sequence
Cumulative tests
Per-round outcomes
LoopsBench
Upfront dependent units
Retained regression tests
Scheduling and loop diagnostics
FeatureLoop
State-conditioned disclosure and correction
Group, prefix, and baseline checks
Message counts and FDAC
Appendix
Table 2: Interaction and evaluation mechanisms in closely related benchmarks. Cumulative checks retain earlier requirements as active correctness obligations.
Formal object
Stored task component
Role during evaluation
W0
Repository checkpoint, frozen environment, and baseline
Gives every evaluated agent the same starting state.
Q1
First active set of requirement groups
Produces the initial request.
Ai+1=Qi+1∖Qi
Groups activated at the next state and their use scenario
Provides candidate evolution feedback.
Public observations for groups in Qi
Observation-to-check mapping
Grounds completion feedback when expressed behavior fails.
Zg(W)
Frozen behavioral checks for group g
Determines whether a requirement is already satisfied.
b(W)
Baseline suite for the initial repository
Detects regressions in retained repository behavior.
Appendix
Table 3: Mapping the task formulation to the stored task components used during evaluation.
Figure 9: The complete six-stage Excalidraw case: successive user requests, illustrative code edits, and cumulative canvas behavior.
State
Behavior made explicit
What must hold in the cumulative workspace
Q1
Grow menu
A selected fillable shape exposes one contextual Grow entry with child-creation and conversion choices, while the existing context menu retains its entries, predicates, and ordering.
Q2
Bound child
A child choice creates exactly one non-overlapping shape of the requested type and one arrow bound at both ends; the new elements remain valid after serialization and restoration.
Q3
Drag-aware routing
The grown connector visibly bends or curves, follows both bound shapes through dragging and a change in relative direction, and leaves manually drawn arrows unchanged.
Q4
In-place conversion
Converting a shape preserves element count, geometry, style, bound text, and existing arrow bindings; dragging the converted shape still updates and reroutes its connectors.
Q5
Blur effects
Gaussian and pixelate modes with an adjustable radius survive serialization and appear on the interactive canvas and in image exports; scenes with blur disabled retain their prior rendering.
Q6
Clipping and composition
A fillable shape clips higher-layer content to its rendered outline, including rotated shapes, while clipping composes with blur, grown children, and bound rerouting connectors.
Appendix
Table 4: The six cumulative states in the Excalidraw task.
Figure 10: Complete author–critic workflow. Solid paths link construction artifacts and execution stages; dashed paths return review findings for revision. The Test and Reference views mark their separate information access.
Requirement semantics, dependency, or public observation
New Contract revision and all later semantic components and executable checks
Wording only, with unchanged semantics
Presentation revision, evaluator mapping, and Dialogue review
Environment, dependency, or runner
Baseline, historical screening, Reference execution, and all Cross evidence that uses the environment
Test package
Historical screening and, for retained candidates, all Test–Reference executions
Reference checkpoint Wm∗
Reference suffix beginning at m and its Cross evidence
Appendix
Table 8: Task components that must be updated after a substantive revision.
Figure 11: Acceptance-check distributions. (a) Checks per task by state count K ; dots are tasks, ticks mark medians. (b) Checks per requirement group.
States K
Tasks
States
Transitions
Acceptance checks
2
33
66
33
153
3
5
15
10
15
4
6
24
18
51
5
4
20
16
27
6
10
60
50
163
7
2
14
12
14
Appendix
Table 9: Distribution of the number of cumulative states per task.
Form
Tasks
States
Mean K
Median K
K≥5
Checks
Browser
7
27
3.86
4
3
121
CLI
21
86
4.10
4
10
132
Embedded
1
2
2.00
2
0
4
Framework
2
12
6.00
6
2
12
Library
7
24
3.43
3
1
32
Service
13
29
2.23
2
0
52
Appendix
Table 10: Software form, number of states, and executable checks.
State
Tasks
Test groups
Test LOC
Reference Δ LOC
1
61
1.15
101.8
82.9
2
61
2.28
212.2
180.5
3
28
4.04
408.1
394.0
4
23
5.61
598.8
563.5
5
17
6.47
732.8
557.2
6
13
8.08
804.8
608.9
Appendix
Table 11: Executable checks and Reference edits by requirement-state position. Each row averages tasks that reach the corresponding state. Test LOC counts nonblank lines in the cumulative Test suite; Reference Δ LOC counts additions plus deletions from the base checkpoint.
Figure 12: Final-state executable evidence across 61 tasks. (a) Task-level values on a log scale, with interquartile ranges and medians. (b) Final Test LOC versus Reference Δ LOC, colored by state count K .
Figure 13: Tasks per repository. Both columns use the same count scale; blue marks indicate multiple-task repositories and teal marks indicate singletons.
Repository
Tasks
Language
Form
States
Checks
micro-editor/micro
6
Go
TUI
6,2,2,2,2,2
62
sharkdp/bat
5
Rust
CLI
5,6,2,2,2
28
casey/just
3
Rust
CLI
2,3,2
9
gitleaks/gitleaks
3
Go
CLI
2,7,6
19
moment/moment
3
JavaScript
Library
4,4,2
14
prometheus/prometheus
3
Go
Service
4,3,2
12
Appendix
Table 12: Repository inventory for the 61 FeatureLoop tasks. “States” lists the number of cumulative requirement states for each task; “Checks” is the repository-level total of executable acceptance checks.
Axis
Question
Levels used in the audit
N
Is the need foreseeable?
0: new direction; 1: weakly suggested; 2: a small set of natural continuations; 3: already entailed.
X
Is exact behavior determined?
0: new information; 1: several plausible contracts; 2: key behavior constrained; 3: essentially fixed.
G
How incomplete is prior delivery?
0: no gap in stated scope; 1: limited gap; 2: material gap; 3: blocked core use.
W
How far is the new workflow?
0: same behavior; 1: same feature; 2: adjacent workflow; 3: only repository/product-level adjacency.
D
What reveals the need?
0: static reasoning; 1: basic trial; 2: realistic workflow; 3: edge cases or lifecycle use; 4: new user information.
T
What kind of evolution?
Ten categories covering completion, extension, composition, compatibility, robustness, lifecycle, observability, policy, and independent capability.
Appendix
Table 13: Semantic annotation dimensions. The primary evolution type T is categorical; the other axes use ordered levels.
Cohort
Rule
Groups
Interpretation
Direction foreseeable
N≥2
42
Prior support for the need’s direction
Strict proactive
N≥2,X≥2,C≤1
16
Enough support for exact, low-risk coverage
Directional, underdetermined
N≥2,X≤1,C≤1
25
Direction supported, exact contract unclear
Later-information
N=0 or D=4
85
New direction or information unavailable earlier
Appendix
Table 14: Requirement cohorts used to analyze prior information. The strict proactive cohort is a subset of the direction-foreseeable cohort. Later-information identifies a type of requirement and applies across development scenarios.
Figure 14: Need foreseeability ( N ) and exact-contract determinacy ( X ) across 147 later requirement groups. Cells are group counts. The outlined region has N≥2 and X≥2 ; the additional commitment-risk filter C≤1 selects the 16-group strict proactive cohort.
ID
Repository
Q/G
N
X
G
W
D
B
C
T
Mix
Conf.
1
1Panel
2/2
0
0
0
3
4
1
1
OC
–
H
2
1Panel
2/2
0
0
0
3
4
1
1
OC
–
H
3
NextChat
2/2
1
0
0
1
4
2
1
IE
–
M
4
NextChat
3/3
1
0
0
1
4
3
1
IE
–
M
5
NextChat
4/4
1
0
0
2
4
3
1
WE
–
M
6
NextChat
5/5
3
1
3
0
3
2
0
EO
+
M
Appendix
Table 15: Complete semantic labels for all 147 later requirement groups. See Table 13 for axis meanings.
Model
Provider
Release date
Source
DeepSeek V4 Flash
DeepSeek
2026-07-31
Model record
DeepSeek V4 Pro
DeepSeek
2026-08-13
Model record
DeepSeek V4.1 Flash
DeepSeek
2026-09-10
Model record
GLM 5.3
Zhipu AI
2026-08-19
Model record
GLM 5.3 Flash
Zhipu AI
2026-08-26
Release note
Hy3
Tencent Hunyuan
2026-07-06
Release note
Appendix
Table 16: Evaluated models and their release or update dates. Dates use year–month–day or year–month. The linked pages identify the model. DeepSeek V4 dates refer to updates after the earlier preview.
Setting
Value
Coding harness
Claude Code 2.1.74
Context window
200,000 tokens per active context
Reasoning effort
High in the recorded configuration
Temperature / top- p
Provider defaults
Full budget
One complete coding-agent session
Evo budget
One coding-agent response per disclosed state
Appendix
Table 17: Execution settings. A response can contain multiple model calls and tool interactions.
Figure 15: Full protocol information flow. Green paths carry requirement disclosures; orange paths carry reports of observed failures. The skip branches bypass behavior already implemented.
Protocol
Initial information
Subsequent information
Agent responses
Full
Complete final specification
None
One session
Evo
Initial request
Missing later requirements
One per disclosed state
Evo+Comp
Initial request
Missing later requirements and observed failures of active requirements
First response plus at most two revisions per state
Appendix
Table 18: Information and response opportunities. Earlier requirements remain final acceptance obligations in all protocols.
Messages per state
Recovery-eligible
States
Delivered messages
0
No
1,564
0
1
Yes
95
95
2
Yes
638
1,276
1
No
22
22
2
No
165
330
Total
2,484
1,723
Appendix
Table 19: Aggregate feedback-event accounting for Evo+Comp. Recovery eligibility requires an observed first-response failure; message counts record delivery regardless of eligibility.
Model
Usable
Omitted
DeepSeek V4 Flash
205
2
DeepSeek V4 Pro
201
6
DeepSeek V4.1 Flash
207
0
GLM 5.3
202
5
GLM 5.3 Flash
197
10
Hy3
184
23
Appendix
Table 20: FDAC state coverage. Every model has 207 possible requirement states.
Observation
Population and window
Count
Rate (%)
New requirement implemented
First responses to later-information groups after disclosure
717/927
77.3
Other requirements still unmet
Locally successful first responses above that do not satisfy all stated requirements
76/717
10.6
Complete state repaired
Observed failed states that receive feedback and pass within two revisions
113/733
15.4
Individual requirement repaired
Failed later groups at stages receiving correction, within two revisions
91/455
20.0
Other requirements still unmet after repair
Successful individual repairs above without cumulative completion within the same window
19/91
20.9
Previously correct behavior lost
Passing groups with later-stage observations that subsequently fail
73/642
11.4
Appendix
Table 21: Observation populations behind the main findings. A requirement group can appear in several model runs; a state may contain several groups. Each row uses its own explicitly defined observation set.
StrictPass ↑
Evo+Comp scores ↑
Model
Full
Evo
Evo+Comp
NodeCredit
FDAC
FDAC-soft
GPT-6 Astra
36/61 (59.0)
42/61 (68.9)
44/61 (72.1)
82.2
46.5
52.6
GPT-5.6 Sol
32/61 (52.5)
38/61 (62.3)
43/61 (70.5)
79.3
45.4
51.9
Kimi K3
24/61 (39.3)
33/61 (54.1)
43/61 (70.5)
84.6
44.2
51.6
Hy4-Preview
29/61 (47.5)
32/61 (52.5)
42/61 (68.9)
81.2
42.7
49.6
DeepSeek V4 Flash
23/61 (37.7)
35/61 (57.4)
43/61 (70.5)
82.0
41.2
49.3
Appendix
Table 22: Complete results, ordered by FDAC. Model StrictPass cells give successes/61 and percentages (the pooled denominator is 732); NodeCredit, FDAC, and FDAC-soft are percentages. Best observed model values in each column are bold.
Model
Full
Evo
Evo+Comp
DeepSeek V4 Flash
26.6–50.3
44.9–69.0
58.1–80.4
DeepSeek V4 Pro
25.2–48.6
37.1–61.4
48.1–71.9
DeepSeek V4.1 Flash
31.0–55.1
44.9–69.0
58.1–80.4
GLM 5.3
32.5–56.7
38.6–62.9
53.0–76.3
GLM 5.3 Flash
25.2–48.6
37.1–61.4
49.7–73.4
Hy3
12.9–33.1
22.3–45.3
25.2–48.6
Appendix
Table 23: Per-model 95% Wilson intervals for StrictPass (%).
Code
Cause
Decision criterion
G1
Relevant code not found
The workspace exists, but the agent does not locate or use the relevant implementation area.
G2
User-path connection
Logic exists, but the real API, CLI, UI, registry, lifecycle, or consumer does not use it.
G3
Disclosed branch omitted
A behavior or compatibility branch already expressed to the agent remains unimplemented.
I1
Behavior or API mismatch
The implementation targets the right area but violates the required type, ordering, state, protocol, representation, or timing.
I2
Build, type, or runtime break
The agent’s patch prevents compilation, startup, test entry, or execution through the user-facing path.
I3
Inconsistent shared state
Parallel registries, shims, generated interfaces, or state sources become inconsistent.
Appendix
Table 24: Primary-cause taxonomy for final failure attribution.
Primary cause
Failures
Share (%)
Behavior or API mismatch (I1)
70
26.6
Build, type, or runtime break (I2)
69
26.2
User-path connection (G2)
57
21.7
Inconsistent shared state (I3)
39
14.8
Disclosed branch omitted (G3)
13
4.9
Earlier behavior regressed (R3)
13
4.9
Appendix
Table 25: Primary causes across all final failures under Evo+Comp.
Outcome across models
Tasks
Failed runs
All models pass
14
0
Some models pass (1–11)
38
155
No model passes
9
108
Appendix
Table 26: Task groups under Evo+Comp.
Task group
G1
G2
G3
I1
I2
I3
R3
V3
Some models pass ( n=155 )
1
33
8
46
54
4
8
1
No model passes ( n=108 )
0
24
5
24
15
35
5
0
Appendix
Table 27: Detailed primary-cause counts by task group.
Model
Failures
None / some
Behavior/API
Build/runtime
User path
Shared state
Other
GPT-6 Astra
17
9 / 8
4
3
5
4
1
GPT-5.6 Sol
18
9 / 9
4
6
4
2
2
Kimi K3
18
9 / 9
7
1
4
4
2
Hy4-Preview
19
9 / 10
6
4
6
2
1
DeepSeek V4 Flash
18
9 / 9
5
4
3
4
2
DeepSeek V4.1 Flash
18
9 / 9
6
0
3
4
5
Appendix
Table 28: Grouped primary-cause counts by model. “None / some” separates failures on tasks solved by no model from failures on tasks solved by at least one model.
Figure 16: Grok 4.6’s NextChat trajectory. Columns follow successive requirement stages; the lower track records the runtime status.
Model
IUD
ICD
DeepSeek V4 Flash
47/61 (77.0)
2/61 (3.3)
DeepSeek V4 Pro
45/61 (73.8)
1/61 (1.6)
DeepSeek V4.1 Flash
49/61 (80.3)
1/61 (1.6)
GLM 5.3
48/61 (78.7)
0/61 (0.0)
GLM 5.3 Flash
45/61 (73.8)
1/61 (1.6)
Hy3
39/61 (63.9)
1/61 (1.6)
Appendix
Table 29: Initial Usable Delivery (IUD) and Initial Complete Delivery (ICD) under Evo. Each model has 61 runs.
Figure 17: Local success and complete requirement-state success. The new-request population contains 927 first responses to later-information groups; the corrective population contains 455 failed later-group opportunities. The populations differ, while each row partitions its own observations into mutually exclusive outcomes.
Cohort
Pre-disclosure
First-response local
First-response cumulative
All groups
75/1660 (4.5)
1078/1533 (70.3)
942/1533 (61.4)
Strict proactive
44/179 (24.6)
67/134 (50.0)
44/134 (32.8)
Directional, unclear
5/291 (1.7)
170/276 (61.6)
146/276 (52.9)
Later-information
1/956 (0.1)
717/927 (77.3)
641/927 (69.1)
Appendix
Table 30: Early coverage and first-response outcomes by semantic cohort under Evo+Comp. Entries are successes/opportunities (percent).
Cohort
Local Repair@2
Prefix Closure@2
All groups
91/455 (20.0)
72/455 (15.8)
Strict proactive
6/67 (9.0)
4/67 (6.0)
Directional, unclear
15/106 (14.2)
13/106 (12.3)
Later-information
50/210 (23.8)
38/210 (18.1)
Appendix
Table 31: Local repair and cumulative closure within two revisions. The two columns use the same observed failed-group opportunities within each cohort.
Figure 18: Final-outcome changes over 732 matched model–task pairs per comparison. Bars distinguish newly successful and newly unsuccessful runs; intervals resample repositories. Each comparison uses independent executions of the complete workflows.
Model
Full → Evo
Evo → Evo+Comp
Rescue / loss
Net change
Rescue / loss
Net change
DeepSeek V4 Flash
14 / 2
+19.7
12 / 4
+13.1
DeepSeek V4 Pro
12 / 4
+13.1
10 / 3
+11.5
DeepSeek V4.1 Flash
14 / 5
+14.8
11 / 3
+13.1
GLM 5.3
10 / 6
+6.6
10 / 1
+14.8
GLM 5.3 Flash
15 / 7
+13.1
12 / 4
+13.1
Appendix
Table 32: Matched protocol transitions by model. Rescue/loss gives counts on the same 61 tasks; net change is in percentage points.
Figure 19: Paired Full → Evo transitions over 61 tasks per model. Bars count lost successes and rescued failures; the right column gives net percentage-point gain.
Figure 20: Paired Evo → Evo+Comp transitions over 61 tasks per model. Bars count lost successes and rescued failures; the right column gives net percentage-point gain.
Stratum
n
r
Full
Evo
Evo+Comp
Δ [95% CI]
Go
19
9
57.0
70.2
81.6
+11.4 [6.1, 19.7]
Rust
15
7
24.4
35.0
48.3
+13.3 [6.5, 20.5]
JavaScript
10
5
45.8
73.3
85.8
+12.5 [1.2, 22.4]
TypeScript
9
8
12.0
10.2
17.6
+7.4 [0.0, 18.8]
Python
7
5
58.3
66.7
78.6
+11.9 [1.7, 26.7]
CLI
21
10
36.5
43.3
57.5
+14.3 [9.3, 21.4]
Appendix
Table 33: Final completion by primary language and software form. n is tasks, r repositories, and the denominator for each protocol is 12n runs. Δ is the paired Evo-to-Evo+Comp change in percentage points with a repository-clustered 95% interval.
Stratum
Recover@2
SER@2
FDAC (%)
Tokens
Telemetry
Go
36/100
186/224
52.2
8.64
227/228
Rust
30/159
85/139
28.2
27.11
180/180
JavaScript
15/42
103/120
53.7
7.28
120/120
TypeScript
14/354
14/106
9.8
37.86
108/108
Python
11/63
65/84
47.1
13.47
84/84
CLI
42/224
137/209
32.4
25.56
252/252
Appendix
Table 34: Evo+Comp diagnostics for the same primary strata. Recovery and SER cells give successes/eligible opportunities. Tokens are millions per reported run; telemetry reports available/total runs. Each stratum groups the sampled tasks by one attribute.
Figure 21: Matched Evo → Evo+Comp change by implementation language and software form. Intervals use repository-clustered bootstrap samples; sparse descriptive-only strata are separated from the primary slices.
Figure 22: Recovery after a failed first response. Segments partition the 733 observed failed states by their first successful revision, with unresolved states shown separately.
Model
Opportunities
Recover@1
Recover@2
Observed retention
DeepSeek V4 Flash
57
11 (19.3)
14 (24.6)
52/62 (83.9)
DeepSeek V4 Pro
68
8 (11.8)
11 (16.2)
46/53 (86.8)
DeepSeek V4.1 Flash
58
5 (8.6)
7 (12.1)
61/62 (98.4)
GLM 5.3
55
8 (14.5)
9 (16.4)
51/59 (86.4)
GLM 5.3 Flash
55
5 (9.1)
7 (12.7)
46/51 (90.2)
Hy3
92
12 (13.0)
12 (13.0)
27/33 (81.8)
Appendix
Table 35: Completion-feedback conversion and observed requirement retention by model. Recover@1 and Recover@2 use feedback opportunities as their denominator. Retention uses only previously satisfied requirement groups with at least one later-state observation.
Figure 23: Completion-feedback recovery by model. Bars show fixes obtained after the first revision and those added by a second; the right columns give Recover@2 and opportunity counts.
Figure 24: State outcomes under Evo, over 207 possible states per model; Pooled aggregates all 12 models. Hatching denotes unobserved records.
Figure 25: State outcomes under Evo+Comp, over 207 possible states per model; Pooled aggregates all 12 models. After-feedback denotes delayed resolution; hatching denotes unobserved records.
Model
Mean NE
Mean NC
Triggered runs
Success (%)
Stalled (%)
Regressions
DeepSeek V4 Flash
2.25
1.97
31
41.9
43.9
9
DeepSeek V4 Pro
2.30
2.44
32
25.0
52.6
7
DeepSeek V4.1 Flash
2.28
2.07
29
37.9
44.3
6
GLM 5.3
2.23
2.02
28
25.0
45.2
8
GLM 5.3 Flash
2.28
2.48
32
28.1
49.8
6
Hy3
2.36
4.03
46
15.2
67.1
4
Appendix
Table 36: Feedback and path diagnostics under Evo+Comp. NE and NC are mean delivered messages per task. Triggered runs receive completion feedback; Success is their final StrictPass rate. Stalled is the percentage of responses with no observed cumulative progress; regressions are events, not distinct tasks.
Protocol
All pass
Some pass
None pass
Full
6
35
20
Evo
10
37
14
Evo+Comp
14
38
9
Appendix
Table 37: How many tasks are solved by all, some, or none of the models under each protocol.
Model
SER@0
SER@1
SER@2
Runs
DeepSeek V4 Flash
59.3
69.5
71.2
59
DeepSeek V4 Pro
54.2
59.3
62.7
59
DeepSeek V4.1 Flash
57.4
65.6
67.2
61
GLM 5.3
62.1
67.2
67.2
58
GLM 5.3 Flash
56.4
61.8
65.5
55
Hy3
33.3
45.8
45.8
48
Appendix
Table 38: Trajectory reliability at the first-response, first-revision, and second-revision checkpoints of the same runs. “Runs” counts trajectories with a usable result at every later state.
Figure 26: Trajectory reliability by model when both feedback types are available. Segments give the fraction of eligible runs that complete every cumulative stage by the first response, first corrective revision, or second corrective revision within that stage.
Full
Evo
Evo+Comp
Model
Mean
/ success
Mean
/ success
Mean
/ success
GPT-6 Astra
1.55
2.63
8.22
11.94
8.88
12.32
GPT-5.6 Sol
5.19
9.89
14.18
22.76
15.56
22.07
Kimi K3
6.63
16.85
13.31
24.60
16.82
23.86
Hy4-Preview
7.34
15.44
13.19
25.14
21.48
31.20
DeepSeek V4 Flash
11.47
29.91
14.18
24.71
21.40
30.36
Appendix
Table 39: Token consumption in millions using only runs with reported telemetry. Each protocol separates mean tokens per reported run from total reported tokens per reported StrictPass success, including tokens spent on failures. Coverage is 726/732 for Full, 732/732 for Evo, and 731/732 for Evo+Comp. Minimum reported token costs within each column are bold.
Figure 27: SER@2 versus reported tokens per successful final delivery across all models. Cost includes tokens from reported failed runs; the dashed line marks the observed Pareto frontier.
Model
PCC
Uptake@1
Repair@2
Closure@2
Retention
DeepSeek V4 Flash
5/16
61/78
9/33
9/33
52/62
DeepSeek V4 Pro
2/15
63/79
9/43
7/43
46/53
DeepSeek V4.1 Flash
3/16
64/80
7/31
5/31
61/62
GLM 5.3
6/15
64/79
5/30
3/30
51/59
GLM 5.3 Flash
5/14
56/75
8/39
6/39
46/51
Hy3
0/14
36/67
13/65
9/65
27/33
Appendix
Table 40: Conditional model profiles under Evo+Comp. Each entry is successes/opportunities. Uptake uses the later-information cohort; repair and closure use all eligible failed later groups. Percentages can be recovered directly from the displayed counts.
A software feature usually evolves over time: developers extend its first version as new use cases and requirements arrive, and each extension builds on code written earlier. Most feature benchmarks evaluate a single request on a fixed codebase. Feature evolution requires evaluating successive extensions on an agent's own earlier code and checking whether the feature remains correct. We introduce FeatureArc to evaluate coding agents on evolving features, together with a pipeline that constructs its tasks from repository histories. The benchmark combines dependencies between successive stages of the same feature, explicit requirements and interface contracts, and executable checks across four programming languages. The pipeline validates fail-to-pass (F2P) tests for new behavior and pass-to-pass (P2P) tests for existing behavior on the reference code before and after each stage. We evaluated 10 coding agents on 73 manually reviewed tasks of 2 to 10 stages. Each agent works in one persistent session and workspace without external test feedback or rollback. The evaluation reveals a gap between first-stage and whole-feature success: pooled success falls from 62.7% at the first stage to 28.5% when the whole feature is tested on the final code. Trajectory analysis identifies inherited defects as a common failure mechanism, present in 72.9% of failed later stages. These earlier errors often remain undetected: in stages following a failure, agents leave the earlier error unnoticed about 95% of the time, and their self-tests rarely revisit earlier stages. Evolving a feature autonomously, without human checks between stages, therefore remains a substantial challenge for current coding agents. At Tencent, the pipeline extends internal evaluations to this setting and guides the development of Hunyuan, Tencent's family of large language models.
Yunlong Wang, Alyssa Wu, Cuiyun Gao +3
Hunyuan Team, Tencent · The Chinese University of Hong Kong
Coding agents are increasingly used as iterative development partners, but most benchmarks still evaluate one specification followed by one final assessment. This leaves out a basic question: can an agent keep its own codebase working as requirements change? We introduce EvoCode-Bench, a benchmark of 26 stateful coding tasks and 227 evaluated rounds. Each task preserves the agent's workspace for 5-15 rounds, states requirements through observable behavior, and uses cumulative executable tests to check new requirements and still-active prior ones. We evaluate 13 coding agents with two metrics: MT@4, a four-attempt fail-stop multi-round score, and SR, a single-round score from a reference-completed prior state. For most agents, SR exceeds MT@4 by 22-40 points. The gap also changes rankings: the highest-SR agent (78.9) ranks only third in persistent execution (44.0 MT@4). Even the strongest agents achieve only about 50% success on multi-turn metrics, and aggregate pass rate drops below half of round-1 performance by round 5. Failure analysis shows tier-dependent behavior: weaker agents fail early, while stronger agents survive long enough to expose specification-tracking and regression failures. We release the benchmark data and Harbor multi-turn infrastructure.
Haiyang Shen, Xuanzhong Chen, Wendong Xu +3
1UniPat AI · 2Peking University · 3Tsinghua University +1
Coding agent infrastructure is shifting from harness engineering toward loop engineering as coding agents are deployed for sustained long-horizon software development. Existing benchmarks often center on localized tasks or end-state outcomes, offering limited insight into sustained execution. We introduce LOOPSBENCH, a long-horizon benchmark for loop engineering in coding agent evaluation. Each task is a dependency DAG over separately testable development units with source-evidenced prerequisite edges. LOOPSBENCH comprises 112 tasks from authentic sources spanning 8 programming languages and 9 domains. Its flow-aware runtime releases tests along the ready frontier and retains completed nodes as regression obligations. We evaluate frontier coding agents paired with widely used loop implementations. The strongest configuration, Opus-4.7 with Claude Code and outer continuation, resolves 25.00% of tasks. Recorded plans recover only part of the source-recovered prerequisite DAG, and regression events remain visible across the evaluated loop profiles. We open source the benchmark data and code, including all tasks, more than 5,300 development units, and executable tests, at microsoft/Loopsbench.
Han Li, Zhemin Fang, Rili Feng +8
Microsoft · Nanjing University · Shanghai Jiao Tong University +1