Benchmarking Coding Agents under Evolving Requirements

Jia Feng; Alyssa Wu; Cuiyun Gao; Yunlong Wang; Songyuanyi Lu; Zhichao Hu

cs.SEOct 26, 2026

Benchmarking Coding Agents under Evolving Requirements

Authors: Jia Feng, Alyssa Wu, Cuiyun Gao, Yunlong Wang, Songyuanyi Lu, Zhichao Hu

Organizations: Hunyuan Team, Tencent · The Chinese University of Hong Kong

Abstract

Software development with coding agents often begins with a rough request that users refine after trying the implementation. Final correctness alone does not capture the repeated guidance this process requires. We introduce FeatureLoop, a benchmark of 61 human-reviewed tasks from 34 repositories across five primary programming languages and seven software forms. Each task follows one development goal in a persistent workspace through evolution feedback, which adds or clarifies requirements, and completion feedback, which reports failures of stated requirements. Cumulative executable checks track new behavior, repair, and preservation of earlier functionality. We evaluate 12 frontier models and find that the strongest completes 72.1% of tasks with both feedback types. Across models, however, only 15.4% of observed failed stages recover within two corrective revisions. Further analysis reveals incomplete repairs, regressions, and model-specific differences in feedback use and token efficiency, identifying obstacles to reliable, low-intervention development.

Figures & tables

Figure 1: An Excalidraw interaction excerpt. The canvases show successive implementations; the faded shape marks the position before dragging.

Figure 2: Three development scenarios. Green: new requests; orange: corrections.

Figure 3: Construction stages and their output artifacts.

Panel of Figure 4 — Figure 4: Dataset composition and cumulative implementation size. (a) Task counts by programming language and software form, including one mixed Go/TypeScript task. (b) Means and interquartile ranges of cumulative test groups, nonblank test lines, and reference-code additions plus deletions. Stage $i$ includes tasks that reach it; the final column includes all 61 tasks.

	StrictPass $\uparrow$			Evo+Comp path
Model	Full	Evo	Evo+Comp	FDAC $\uparrow$	$N_{E}$	$N_{C}$
GPT-6 Astra	59.0	68.9	72.1	46.5	2.25	2.00
GPT-5.6 Sol	52.5	62.3	70.5	45.4	2.23	1.89
Kimi K3	39.3	54.1	70.5	44.2	2.25	1.87
Hy4-Preview	47.5	52.5	68.9	42.7	2.28	2.08
DeepSeek V4 Flash	37.7	57.4	70.5	41.2	2.25	1.97

Table 1: Main results, ordered by Evo+Comp FDAC. StrictPass and FDAC are percentages;

N_{E}

and

N_{C}

are mean messages per task, including failed runs. Each model is evaluated on the same 61 tasks. Best observed outcome values and the lowest correction count are bold.

Figure 5: Model-level recovery and retention under both feedback types. Recovery uses failed states; retention uses previously passing requirements with later observations.

Figure 6: Primary diagnoses within each task set’s final failures. “Other” groups the remaining diagnoses.

Figure 7: Primary diagnoses within each model’s final failures; cell values are percentages.

Figure 8: Model-level completion and token expenditure under both feedback types. Cost includes tokens spent on failed runs.

Appendix figures & tables58 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Later information	Behavioral evaluation	Interaction measure
SWE-Together	Session-conditioned requests and corrections	Final rubric with execution evidence	User Correction
Asuka-Bench	Feedback from browser checks	Repeated UI criteria	Per-round fix rates
EvoCode-Bench	Fixed instruction sequence	Cumulative tests	Per-round outcomes
LoopsBench	Upfront dependent units	Retained regression tests	Scheduling and loop diagnostics
FeatureLoop	State-conditioned disclosure and correction	Group, prefix, and baseline checks	Message counts and FDAC

Appendix

Table 2: Interaction and evaluation mechanisms in closely related benchmarks. Cumulative checks retain earlier requirements as active correctness obligations.

Formal object	Stored task component	Role during evaluation
$W_{0}$	Repository checkpoint, frozen environment, and baseline	Gives every evaluated agent the same starting state.
$Q_{1}$	First active set of requirement groups	Produces the initial request.
$A_{i+1}=Q_{i+1}\setminus Q_{i}$	Groups activated at the next state and their use scenario	Provides candidate evolution feedback.
Public observations for groups in $Q_{i}$	Observation-to-check mapping	Grounds completion feedback when expressed behavior fails.
$Z_{g}(W)$	Frozen behavioral checks for group $g$	Determines whether a requirement is already satisfied.
$b(W)$	Baseline suite for the initial repository	Detects regressions in retained repository behavior.

Appendix

Table 3: Mapping the task formulation to the stored task components used during evaluation.

Figure 9: The complete six-stage Excalidraw case: successive user requests, illustrative code edits, and cumulative canvas behavior.

State	Behavior made explicit	What must hold in the cumulative workspace
$Q_{1}$	Grow menu	A selected fillable shape exposes one contextual Grow entry with child-creation and conversion choices, while the existing context menu retains its entries, predicates, and ordering.
$Q_{2}$	Bound child	A child choice creates exactly one non-overlapping shape of the requested type and one arrow bound at both ends; the new elements remain valid after serialization and restoration.
$Q_{3}$	Drag-aware routing	The grown connector visibly bends or curves, follows both bound shapes through dragging and a change in relative direction, and leaves manually drawn arrows unchanged.
$Q_{4}$	In-place conversion	Converting a shape preserves element count, geometry, style, bound text, and existing arrow bindings; dragging the converted shape still updates and reroutes its connectors.
$Q_{5}$	Blur effects	Gaussian and pixelate modes with an adjustable radius survive serialization and appear on the interactive canvas and in image exports; scenes with blur disabled retain their prior rendering.
$Q_{6}$	Clipping and composition	A fillable shape clips higher-layer content to its rendered outline, including rotated shapes, while clipping composes with blur, grown children, and bound rerouting connectors.

Appendix

Table 4: The six cumulative states in the Excalidraw task.

Figure 10: Complete author–critic workflow. Solid paths link construction artifacts and execution stages; dashed paths return review findings for revision. The Test and Reference views mark their separate information access.

Role	May read	Must not read
Source/Feature	Frozen source records, checkpoint, repository evidence	Private Test or Reference material, or evaluated-agent solution content
Contract	Sources, feature specification, requirement sequence, environment report	Private Test or Reference material, or evaluated-agent solution content
Dialogue	Frozen Contract and user-facing narrative context	Hidden tests, Reference, or future evaluated-agent behavior
Test Author/Critic	Frozen Contract, $W_{0}$ , environment, complete state map, and same-side Test files	Reference code, diffs, or execution results
Reference Author/Critic	Frozen Contract, $W_{0}$ , environment, complete state map, and same-side workspace	Tests, fixtures, assertions, or Test results
Adjudicator	Contract, structured execution outcomes, suite summaries, and changed-file summaries	Joint write access or unrecorded transfer of one side’s private implementation to the other

Appendix

Table 5: Construction roles and their information boundaries.

Tier	Models	Routing role
Low latency	Hy3; DeepSeek V4 Flash	High-throughput generation, organization, and low-risk revision
Medium	DeepSeek V4 Pro; GPT-5.6 Terra	Routine Author work and workspace-based revision
Strong	GLM 5.3; Kimi K3	Difficult Author work, independent semantic checks, and root-cause analysis
Flagship	GPT-5.6 Sol; Claude Opus 5	High-complexity Author work, final independent review, and support for disputed decisions

Appendix

Table 6: Model pool used for agent-assisted benchmark construction.

Evidence	Validation role	Use in task acceptance
Historical screening	Checks later public repository history within a recorded cutoff and coverage	Required after Test is frozen and before Reference construction
Reference pass	Establishes a feasible implementation under the frozen Contract and environment	Required at every state
Adjacent-state check	Confirms that newly activated behavior distinguishes adjacent Reference checkpoints	Required for every state transition
Earlier-state suite	Checks earlier active behavior on later Reference checkpoints	Required through the Reference suffix
Baseline suite	Checks retained repository behavior at each Reference checkpoint	Required at every state
Targeted mutation	Probes failure sensitivity for a selected behavioral defect	Executed for constructed targeted faulty versions

Appendix

Table 7: Validation roles of the construction evidence.

Modified component	Required regeneration or re-execution
Source or feature specification	Requirement sequence, Contract, Dialogue, Test, historical screening, Reference, Cross, and review
Requirement semantics, dependency, or public observation	New Contract revision and all later semantic components and executable checks
Wording only, with unchanged semantics	Presentation revision, evaluator mapping, and Dialogue review
Environment, dependency, or runner	Baseline, historical screening, Reference execution, and all Cross evidence that uses the environment
Test package	Historical screening and, for retained candidates, all Test–Reference executions
Reference checkpoint $W_{m}^{*}$	Reference suffix beginning at $m$ and its Cross evidence

Appendix

Table 8: Task components that must be updated after a substantive revision.

Figure 11: Acceptance-check distributions. (a) Checks per task by state count

K

; dots are tasks, ticks mark medians. (b) Checks per requirement group.

States $K$	Tasks	States	Transitions	Acceptance checks
2	33	66	33	153
3	5	15	10	15
4	6	24	18	51
5	4	20	16	27
6	10	60	50	163
7	2	14	12	14

Appendix

Table 9: Distribution of the number of cumulative states per task.

Form	Tasks	States	Mean $K$	Median $K$	$K\geq 5$	Checks
Browser	7	27	3.86	4	3	121
CLI	21	86	4.10	4	10	132
Embedded	1	2	2.00	2	0	4
Framework	2	12	6.00	6	2	12
Library	7	24	3.43	3	1	32
Service	13	29	2.23	2	0	52

Appendix

Table 10: Software form, number of states, and executable checks.

State	Tasks	Test groups	Test LOC	Reference $\Delta$ LOC
1	61	1.15	101.8	82.9
2	61	2.28	212.2	180.5
3	28	4.04	408.1	394.0
4	23	5.61	598.8	563.5
5	17	6.47	732.8	557.2
6	13	8.08	804.8	608.9

Appendix

Table 11: Executable checks and Reference edits by requirement-state position. Each row averages tasks that reach the corresponding state. Test LOC counts nonblank lines in the cumulative Test suite; Reference

\Delta

LOC counts additions plus deletions from the base checkpoint.

Figure 12: Final-state executable evidence across 61 tasks. (a) Task-level values on a log scale, with interquartile ranges and medians. (b) Final Test LOC versus Reference

\Delta

LOC, colored by state count

K

Figure 13: Tasks per repository. Both columns use the same count scale; blue marks indicate multiple-task repositories and teal marks indicate singletons.

Repository	Tasks	Language	Form	States	Checks
micro-editor/micro	6	Go	TUI	6,2,2,2,2,2	62
sharkdp/bat	5	Rust	CLI	5,6,2,2,2	28
casey/just	3	Rust	CLI	2,3,2	9
gitleaks/gitleaks	3	Go	CLI	2,7,6	19
moment/moment	3	JavaScript	Library	4,4,2	14
prometheus/prometheus	3	Go	Service	4,3,2	12

Appendix

Table 12: Repository inventory for the 61 FeatureLoop tasks. “States” lists the number of cumulative requirement states for each task; “Checks” is the repository-level total of executable acceptance checks.

Axis	Question	Levels used in the audit
$N$	Is the need foreseeable?	0: new direction; 1: weakly suggested; 2: a small set of natural continuations; 3: already entailed.
$X$	Is exact behavior determined?	0: new information; 1: several plausible contracts; 2: key behavior constrained; 3: essentially fixed.
$G$	How incomplete is prior delivery?	0: no gap in stated scope; 1: limited gap; 2: material gap; 3: blocked core use.
$W$	How far is the new workflow?	0: same behavior; 1: same feature; 2: adjacent workflow; 3: only repository/product-level adjacency.
$D$	What reveals the need?	0: static reasoning; 1: basic trial; 2: realistic workflow; 3: edge cases or lifecycle use; 4: new user information.
$T$	What kind of evolution?	Ten categories covering completion, extension, composition, compatibility, robustness, lifecycle, observability, policy, and independent capability.

Appendix

Table 13: Semantic annotation dimensions. The primary evolution type

T

is categorical; the other axes use ordered levels.

Cohort	Rule	Groups	Interpretation
Direction foreseeable	$N\geq 2$	42	Prior support for the need’s direction
Strict proactive	$N\geq 2,\ X\geq 2,\ C\leq 1$	16	Enough support for exact, low-risk coverage
Directional, underdetermined	$N\geq 2,\ X\leq 1,\ C\leq 1$	25	Direction supported, exact contract unclear
Later-information	$N=0$ or $D=4$	85	New direction or information unavailable earlier

Appendix

Table 14: Requirement cohorts used to analyze prior information. The strict proactive cohort is a subset of the direction-foreseeable cohort. Later-information identifies a type of requirement and applies across development scenarios.

Figure 14: Need foreseeability (

N

) and exact-contract determinacy (

X

) across 147 later requirement groups. Cells are group counts. The outlined region has

N\geq 2

and

X\geq 2

; the additional commitment-risk filter

C\leq 1

selects the 16-group strict proactive cohort.

ID	Repository	$Q/G$	$N$	$X$	$G$	$W$	$D$	$B$	$C$	$T$	Mix	Conf.
1	1Panel	2/2	0	0	0	3	4	1	1	OC	–	H
2	1Panel	2/2	0	0	0	3	4	1	1	OC	–	H
3	NextChat	2/2	1	0	0	1	4	2	1	IE	–	M
4	NextChat	3/3	1	0	0	1	4	3	1	IE	–	M
5	NextChat	4/4	1	0	0	2	4	3	1	WE	–	M
6	NextChat	5/5	3	1	3	0	3	2	0	EO	+	M

Appendix

Table 15: Complete semantic labels for all 147 later requirement groups. See Table 13 for axis meanings.

Model	Provider	Release date	Source
DeepSeek V4 Flash	DeepSeek	2026-07-31	Model record
DeepSeek V4 Pro	DeepSeek	2026-08-13	Model record
DeepSeek V4.1 Flash	DeepSeek	2026-09-10	Model record
GLM 5.3	Zhipu AI	2026-08-19	Model record
GLM 5.3 Flash	Zhipu AI	2026-08-26	Release note
Hy3	Tencent Hunyuan	2026-07-06	Release note

Appendix

Table 16: Evaluated models and their release or update dates. Dates use year–month–day or year–month. The linked pages identify the model. DeepSeek V4 dates refer to updates after the earlier preview.

Setting	Value
Coding harness	Claude Code 2.1.74
Context window	200,000 tokens per active context
Reasoning effort	High in the recorded configuration
Temperature / top- $p$	Provider defaults
Full budget	One complete coding-agent session
Evo budget	One coding-agent response per disclosed state

Appendix

Table 17: Execution settings. A response can contain multiple model calls and tool interactions.

Figure 15: Full protocol information flow. Green paths carry requirement disclosures; orange paths carry reports of observed failures. The skip branches bypass behavior already implemented.

Protocol	Initial information	Subsequent information	Agent responses
Full	Complete final specification	None	One session
Evo	Initial request	Missing later requirements	One per disclosed state
Evo+Comp	Initial request	Missing later requirements and observed failures of active requirements	First response plus at most two revisions per state

Appendix

Table 18: Information and response opportunities. Earlier requirements remain final acceptance obligations in all protocols.

Messages per state	Recovery-eligible	States	Delivered messages
0	No	1,564	0
1	Yes	95	95
2	Yes	638	1,276
1	No	22	22
2	No	165	330
Total		2,484	1,723

Appendix

Table 19: Aggregate feedback-event accounting for Evo+Comp. Recovery eligibility requires an observed first-response failure; message counts record delivery regardless of eligibility.

Model	Usable	Omitted
DeepSeek V4 Flash	205	2
DeepSeek V4 Pro	201	6
DeepSeek V4.1 Flash	207	0
GLM 5.3	202	5
GLM 5.3 Flash	197	10
Hy3	184	23

Appendix

Table 20: FDAC state coverage. Every model has 207 possible requirement states.

Observation	Population and window	Count	Rate (%)
New requirement implemented	First responses to later-information groups after disclosure	717/927	77.3
Other requirements still unmet	Locally successful first responses above that do not satisfy all stated requirements	76/717	10.6
Complete state repaired	Observed failed states that receive feedback and pass within two revisions	113/733	15.4
Individual requirement repaired	Failed later groups at stages receiving correction, within two revisions	91/455	20.0
Other requirements still unmet after repair	Successful individual repairs above without cumulative completion within the same window	19/91	20.9
Previously correct behavior lost	Passing groups with later-stage observations that subsequently fail	73/642	11.4

Appendix

Table 21: Observation populations behind the main findings. A requirement group can appear in several model runs; a state may contain several groups. Each row uses its own explicitly defined observation set.

	StrictPass $\uparrow$			Evo+Comp scores $\uparrow$
Model	Full	Evo	Evo+Comp	NodeCredit	FDAC	FDAC-soft
GPT-6 Astra	36/61 (59.0)	42/61 (68.9)	44/61 (72.1)	82.2	46.5	52.6
GPT-5.6 Sol	32/61 (52.5)	38/61 (62.3)	43/61 (70.5)	79.3	45.4	51.9
Kimi K3	24/61 (39.3)	33/61 (54.1)	43/61 (70.5)	84.6	44.2	51.6
Hy4-Preview	29/61 (47.5)	32/61 (52.5)	42/61 (68.9)	81.2	42.7	49.6
DeepSeek V4 Flash	23/61 (37.7)	35/61 (57.4)	43/61 (70.5)	82.0	41.2	49.3

Appendix

Table 22: Complete results, ordered by FDAC. Model StrictPass cells give successes/61 and percentages (the pooled denominator is 732); NodeCredit, FDAC, and FDAC-soft are percentages. Best observed model values in each column are bold.

Model	Full	Evo	Evo+Comp
DeepSeek V4 Flash	26.6–50.3	44.9–69.0	58.1–80.4
DeepSeek V4 Pro	25.2–48.6	37.1–61.4	48.1–71.9
DeepSeek V4.1 Flash	31.0–55.1	44.9–69.0	58.1–80.4
GLM 5.3	32.5–56.7	38.6–62.9	53.0–76.3
GLM 5.3 Flash	25.2–48.6	37.1–61.4	49.7–73.4
Hy3	12.9–33.1	22.3–45.3	25.2–48.6

Appendix

Table 23: Per-model 95% Wilson intervals for StrictPass (%).

Code	Cause	Decision criterion
G1	Relevant code not found	The workspace exists, but the agent does not locate or use the relevant implementation area.
G2	User-path connection	Logic exists, but the real API, CLI, UI, registry, lifecycle, or consumer does not use it.
G3	Disclosed branch omitted	A behavior or compatibility branch already expressed to the agent remains unimplemented.
I1	Behavior or API mismatch	The implementation targets the right area but violates the required type, ordering, state, protocol, representation, or timing.
I2	Build, type, or runtime break	The agent’s patch prevents compilation, startup, test entry, or execution through the user-facing path.
I3	Inconsistent shared state	Parallel registries, shims, generated interfaces, or state sources become inconsistent.

Appendix

Table 24: Primary-cause taxonomy for final failure attribution.

Primary cause	Failures	Share (%)
Behavior or API mismatch (I1)	70	26.6
Build, type, or runtime break (I2)	69	26.2
User-path connection (G2)	57	21.7
Inconsistent shared state (I3)	39	14.8
Disclosed branch omitted (G3)	13	4.9
Earlier behavior regressed (R3)	13	4.9

Appendix

Table 25: Primary causes across all final failures under Evo+Comp.

Outcome across models	Tasks	Failed runs
All models pass	14	0
Some models pass (1–11)	38	155
No model passes	9	108

Appendix

Table 26: Task groups under Evo+Comp.

Task group	G1	G2	G3	I1	I2	I3	R3	V3
Some models pass ( $n=155$ )	1	33	8	46	54	4	8	1
No model passes ( $n=108$ )	0	24	5	24	15	35	5	0

Appendix

Table 27: Detailed primary-cause counts by task group.

Model	Failures	None / some	Behavior/API	Build/runtime	User path	Shared state	Other
GPT-6 Astra	17	9 / 8	4	3	5	4	1
GPT-5.6 Sol	18	9 / 9	4	6	4	2	2
Kimi K3	18	9 / 9	7	1	4	4	2
Hy4-Preview	19	9 / 10	6	4	6	2	1
DeepSeek V4 Flash	18	9 / 9	5	4	3	4	2
DeepSeek V4.1 Flash	18	9 / 9	6	0	3	4	5

Appendix

Table 28: Grouped primary-cause counts by model. “None / some” separates failures on tasks solved by no model from failures on tasks solved by at least one model.

Figure 16: Grok 4.6’s NextChat trajectory. Columns follow successive requirement stages; the lower track records the runtime status.

Model	IUD	ICD
DeepSeek V4 Flash	47/61 (77.0)	2/61 (3.3)
DeepSeek V4 Pro	45/61 (73.8)	1/61 (1.6)
DeepSeek V4.1 Flash	49/61 (80.3)	1/61 (1.6)
GLM 5.3	48/61 (78.7)	0/61 (0.0)
GLM 5.3 Flash	45/61 (73.8)	1/61 (1.6)
Hy3	39/61 (63.9)	1/61 (1.6)

Appendix

Table 29: Initial Usable Delivery (IUD) and Initial Complete Delivery (ICD) under Evo. Each model has 61 runs.

Figure 17: Local success and complete requirement-state success. The new-request population contains 927 first responses to later-information groups; the corrective population contains 455 failed later-group opportunities. The populations differ, while each row partitions its own observations into mutually exclusive outcomes.

Cohort	Pre-disclosure	First-response local	First-response cumulative
All groups	75/1660 (4.5)	1078/1533 (70.3)	942/1533 (61.4)
Strict proactive	44/179 (24.6)	67/134 (50.0)	44/134 (32.8)
Directional, unclear	5/291 (1.7)	170/276 (61.6)	146/276 (52.9)
Later-information	1/956 (0.1)	717/927 (77.3)	641/927 (69.1)

Appendix

Table 30: Early coverage and first-response outcomes by semantic cohort under Evo+Comp. Entries are successes/opportunities (percent).

Cohort	Local Repair@2	Prefix Closure@2
All groups	91/455 (20.0)	72/455 (15.8)
Strict proactive	6/67 (9.0)	4/67 (6.0)
Directional, unclear	15/106 (14.2)	13/106 (12.3)
Later-information	50/210 (23.8)	38/210 (18.1)

Appendix

Table 31: Local repair and cumulative closure within two revisions. The two columns use the same observed failed-group opportunities within each cohort.

Figure 18: Final-outcome changes over 732 matched model–task pairs per comparison. Bars distinguish newly successful and newly unsuccessful runs; intervals resample repositories. Each comparison uses independent executions of the complete workflows.

Model	Full $\rightarrow$ Evo		Evo $\rightarrow$ Evo+Comp
	Rescue / loss	Net change	Rescue / loss	Net change
DeepSeek V4 Flash	14 / 2	+19.7	12 / 4	+13.1
DeepSeek V4 Pro	12 / 4	+13.1	10 / 3	+11.5
DeepSeek V4.1 Flash	14 / 5	+14.8	11 / 3	+13.1
GLM 5.3	10 / 6	+6.6	10 / 1	+14.8
GLM 5.3 Flash	15 / 7	+13.1	12 / 4	+13.1

Appendix

Table 32: Matched protocol transitions by model. Rescue/loss gives counts on the same 61 tasks; net change is in percentage points.

Figure 19: Paired Full

\rightarrow

Evo transitions over 61 tasks per model. Bars count lost successes and rescued failures; the right column gives net percentage-point gain.

Figure 20: Paired Evo

\rightarrow

Evo+Comp transitions over 61 tasks per model. Bars count lost successes and rescued failures; the right column gives net percentage-point gain.

Stratum	$n$	$r$	Full	Evo	Evo+Comp	$\Delta$ [95% CI]
Go	19	9	57.0	70.2	81.6	+11.4 [6.1, 19.7]
Rust	15	7	24.4	35.0	48.3	+13.3 [6.5, 20.5]
JavaScript	10	5	45.8	73.3	85.8	+12.5 [1.2, 22.4]
TypeScript	9	8	12.0	10.2	17.6	+7.4 [0.0, 18.8]
Python	7	5	58.3	66.7	78.6	+11.9 [1.7, 26.7]
CLI	21	10	36.5	43.3	57.5	+14.3 [9.3, 21.4]

Appendix

Table 33: Final completion by primary language and software form.

n

is tasks,

r

repositories, and the denominator for each protocol is

12n

runs.

\Delta

is the paired Evo-to-Evo+Comp change in percentage points with a repository-clustered 95% interval.

Stratum	Recover@2	SER@2	FDAC (%)	Tokens	Telemetry
Go	36/100	186/224	52.2	8.64	227/228
Rust	30/159	85/139	28.2	27.11	180/180
JavaScript	15/42	103/120	53.7	7.28	120/120
TypeScript	14/354	14/106	9.8	37.86	108/108
Python	11/63	65/84	47.1	13.47	84/84
CLI	42/224	137/209	32.4	25.56	252/252

Appendix

Table 34: Evo+Comp diagnostics for the same primary strata. Recovery and SER cells give successes/eligible opportunities. Tokens are millions per reported run; telemetry reports available/total runs. Each stratum groups the sampled tasks by one attribute.

Figure 21: Matched Evo

\rightarrow

Evo+Comp change by implementation language and software form. Intervals use repository-clustered bootstrap samples; sparse descriptive-only strata are separated from the primary slices.

Figure 22: Recovery after a failed first response. Segments partition the 733 observed failed states by their first successful revision, with unresolved states shown separately.

Model	Opportunities	Recover@1	Recover@2	Observed retention
DeepSeek V4 Flash	57	11 (19.3)	14 (24.6)	52/62 (83.9)
DeepSeek V4 Pro	68	8 (11.8)	11 (16.2)	46/53 (86.8)
DeepSeek V4.1 Flash	58	5 (8.6)	7 (12.1)	61/62 (98.4)
GLM 5.3	55	8 (14.5)	9 (16.4)	51/59 (86.4)
GLM 5.3 Flash	55	5 (9.1)	7 (12.7)	46/51 (90.2)
Hy3	92	12 (13.0)	12 (13.0)	27/33 (81.8)

Appendix

Table 35: Completion-feedback conversion and observed requirement retention by model. Recover@1 and Recover@2 use feedback opportunities as their denominator. Retention uses only previously satisfied requirement groups with at least one later-state observation.

Figure 23: Completion-feedback recovery by model. Bars show fixes obtained after the first revision and those added by a second; the right columns give Recover@2 and opportunity counts.

Figure 24: State outcomes under Evo, over 207 possible states per model; Pooled aggregates all 12 models. Hatching denotes unobserved records.

Figure 25: State outcomes under Evo+Comp, over 207 possible states per model; Pooled aggregates all 12 models. After-feedback denotes delayed resolution; hatching denotes unobserved records.

Model	Mean $N_{E}$	Mean $N_{C}$	Triggered runs	Success (%)	Stalled (%)	Regressions
DeepSeek V4 Flash	2.25	1.97	31	41.9	43.9	9
DeepSeek V4 Pro	2.30	2.44	32	25.0	52.6	7
DeepSeek V4.1 Flash	2.28	2.07	29	37.9	44.3	6
GLM 5.3	2.23	2.02	28	25.0	45.2	8
GLM 5.3 Flash	2.28	2.48	32	28.1	49.8	6
Hy3	2.36	4.03	46	15.2	67.1	4

Appendix

Table 36: Feedback and path diagnostics under Evo+Comp.

N_{E}

and

N_{C}

are mean delivered messages per task. Triggered runs receive completion feedback; Success is their final StrictPass rate. Stalled is the percentage of responses with no observed cumulative progress; regressions are events, not distinct tasks.

Protocol	All pass	Some pass	None pass
Full	6	35	20
Evo	10	37	14
Evo+Comp	14	38	9

Appendix

Table 37: How many tasks are solved by all, some, or none of the models under each protocol.

Model	SER@0	SER@1	SER@2	Runs
DeepSeek V4 Flash	59.3	69.5	71.2	59
DeepSeek V4 Pro	54.2	59.3	62.7	59
DeepSeek V4.1 Flash	57.4	65.6	67.2	61
GLM 5.3	62.1	67.2	67.2	58
GLM 5.3 Flash	56.4	61.8	65.5	55
Hy3	33.3	45.8	45.8	48

Appendix

Table 38: Trajectory reliability at the first-response, first-revision, and second-revision checkpoints of the same runs. “Runs” counts trajectories with a usable result at every later state.

Figure 26: Trajectory reliability by model when both feedback types are available. Segments give the fraction of eligible runs that complete every cumulative stage by the first response, first corrective revision, or second corrective revision within that stage.

	Full		Evo		Evo+Comp
Model	Mean	/ success	Mean	/ success	Mean	/ success
GPT-6 Astra	1.55	2.63	8.22	11.94	8.88	12.32
GPT-5.6 Sol	5.19	9.89	14.18	22.76	15.56	22.07
Kimi K3	6.63	16.85	13.31	24.60	16.82	23.86
Hy4-Preview	7.34	15.44	13.19	25.14	21.48	31.20
DeepSeek V4 Flash	11.47	29.91	14.18	24.71	21.40	30.36

Appendix

Table 39: Token consumption in millions using only runs with reported telemetry. Each protocol separates mean tokens per reported run from total reported tokens per reported StrictPass success, including tokens spent on failures. Coverage is 726/732 for Full, 732/732 for Evo, and 731/732 for Evo+Comp. Minimum reported token costs within each column are bold.

Figure 27: SER@2 versus reported tokens per successful final delivery across all models. Cost includes tokens from reported failed runs; the dashed line marks the observed Pareto frontier.

Model	PCC	Uptake@1	Repair@2	Closure@2	Retention
DeepSeek V4 Flash	5/16	61/78	9/33	9/33	52/62
DeepSeek V4 Pro	2/15	63/79	9/43	7/43	46/53
DeepSeek V4.1 Flash	3/16	64/80	7/31	5/31	61/62
GLM 5.3	6/15	64/79	5/30	3/30	51/59
GLM 5.3 Flash	5/14	56/75	8/39	6/39	46/51
Hy3	0/14	36/67	13/65	9/65	27/33

Appendix

Table 40: Conditional model profiles under Evo+Comp. Each entry is successes/opportunities. Uptake uses the later-information cohort; repair and closure use all eligible failed later groups. Percentages can be recovered directly from the displayed counts.

Coding Agent Benchmarks

See all 47 papers

FeatureArc: Benchmarking Coding Agents on Evolving Features
Oct 27, 2026Yunlong Wang, Alyssa Wu, Cuiyun Gao +3Coding Agent Benchmarks Software Engineering
Hunyuan Team, Tencent · The Chinese University of Hong Kong
EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions
May 22, 2026Haiyang Shen, Xuanzhong Chen, Wendong Xu +3AI Coding Agents Software Engineering Benchmarks
1UniPat AI · 2Peking University · 3Tsinghua University +1
LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation
Jul 31, 2026Han Li, Zhemin Fang, Rili Feng +8AI Coding Agents Software Engineering Benchmarks
Microsoft · Nanjing University · Shanghai Jiao Tong University +1

Oct 27, 2026J/K move · Enter open · S save

FeatureArc: Benchmarking Coding Agents on Evolving Features

Yunlong Wang, Alyssa Wu, Cuiyun Gao, Jia Feng+2

Hunyuan Team, Tencent · The Chinese University of Hong Kong

Coding Agent Benchmarks Software Engineering Software Engineering Agents Long-Horizon Agent Evaluation

Open paper AskPDF arXiv

A software feature usually evolves over time: developers extend its first version as new use cases and requirements arrive, and each extension builds on code written earlier. Most feature benchmarks evaluate a single request on a fixed codebase. Feature evolution requires evaluating successive extensions on an agent's own earlier code and checking whether the feature remains correct. We introduce FeatureArc to evaluate coding agents on evolving features, together with a pipeline that constructs its tasks from repository histories. The benchmark combines dependencies between successive stages of the same feature, explicit requirements and interface contracts, and executable checks across four programming languages. The pipeline validates fail-to-pass (F2P) tests for new behavior and pass-to-pass (P2P) tests for existing behavior on the reference code before and after each stage. We evaluated 10 coding agents on 73 manually reviewed tasks of 2 to 10 stages. Each agent works in one persistent session and workspace without external test feedback or rollback. The evaluation reveals a gap between first-stage and whole-feature success: pooled success falls from 62.7% at the first stage to 28.5% when the whole feature is tested on the final code. Trajectory analysis identifies inherited defects as a common failure mechanism, present in 72.9% of failed later stages. These earlier errors often remain undetected: in stages following a failure, agents leave the earlier error unnoticed about 95% of the time, and their self-tests rarely revisit earlier stages. Evolving a feature autonomously, without human checks between stages, therefore remains a substantial challenge for current coding agents. At Tencent, the pipeline extends internal evaluations to this setting and guides the development of Hunyuan, Tencent's family of large language models.

Abstract

Coding Agent Benchmarks

Explore similar work

FeatureArc: Benchmarking Coding Agents on Evolving Features

EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions

LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation