When Coding Agents Outpace Review: Evaluating PRD-Driven Development in Mature Enterprise Repositories

Tianqi Liu; Yandong Guan; Songze Li; Wenkai Wang; Gang Yu; Chengfei Lv

cs.SEOct 19, 2026

When Coding Agents Outpace Review: Evaluating PRD-Driven Development in Mature Enterprise Repositories

Authors: Tianqi Liu, Yandong Guan, Songze Li, Wenkai Wang, Gang Yu, Chengfei Lv

Organizations: Zhejiang University · Alibaba Taobao & Tmall Group

Abstract

Coding agents increasingly undertake development tasks in mature enterprise repositories, yet existing benchmarks offer limited insight into whether they fulfill complete Product Requirement Documents (PRDs) and where delivery breaks down. Issue resolution and development from scratch leave incremental PRD fulfillment underexplored, while isolated tests provide incomplete evidence of business and integration requirements. We introduce PACE, a benchmark of 43 enterprise tasks with 750 expert-audited Code Points. It combines requirement reconstruction, criterion construction, and evidence-based evaluation to measure PRD fulfillment and connect delivery gaps to agent behavior. Across 11 model--scaffold configurations, most scores fall between 52.1 and 78.1; GPT-5.6-Sol achieves the highest complete-delivery rate of 14.0%. Incomplete implementation and missing integration are the principal bottlenecks: producing individual components does not ensure a complete delivery. Evaluation achieves 95.2% agreement with expert consensus, with lower case-level variation across judging records than repeated executions. Beyond model comparison, the benchmark provides actionable feedback for harness refinement: diagnosis guides targeted improvements, and successive harness releases raise mean performance from 59.4 to 63.0 using the same model and task suite. Code is available at https://github.com/TianqiLiu1/PACE-Bench.

Figures & tables

Benchmark	Task input	New / edited code (LOC / files)	Existing codebase	Generated acceptance criteria	Evaluation method
SWE-bench ( 2024 )	Issue description	$\sim$ 15 / 1	✓	✕	Unit + regression tests
DevEval ( 2025a )	PRD	$\sim$ 413 / $\sim$ 4.4 †	✕	✕	Unit + acceptance tests
FEA-Bench ( 2025b )	Feature request	128.5 / 2.6 †	✓	✕	Unit tests
SWE-bench Pro ( 2026 )	Issue description	107.4 / 4.1 †	✓	✕	Unit + regression tests
PRDBench ( 2026 )	PRD	$1{,}279^{\ddagger}$ / n/r	✕	✓	Unit / CLI / file tests with an agent judge
FeatureBench ( 2026 )	Feature request	790.2 / 15.7 †	✓	✕	Unit + regression tests

Table 1: Comparison of task inputs, code scale, and evaluation design. Generated acceptance criteria are synthesized requirements for judging candidate implementations. Code size uses medians;

\dagger

marks means, and

\ddagger

denotes an annotation scaffold rather than patch edits. Measurement scopes appear in Appendix M.1 .

Figure 1: PACE-Bench turns production requirements into frozen acceptance criteria, evidence-grounded judgments, and traceable diagnoses for agent improvement.

Figure 2: Application domains, PRD objectives, and language combinations across the 43 production cases.

Figure 3: Task-scale distributions: repository size, PRD length, acceptance scope, and reference edits. Dashed lines mark medians.

Model	Score (mean $\pm$ SE)	Success rate
Unified minimal scaffold
GPT-5.6-Sol	78.08 $\pm$ 2.71	14.0%
GLM-5.2	68.80 $\pm$ 3.72	9.3%
Qwen3.7-Max	66.70 $\pm$ 3.45	7.0%
Kimi-K2.6	63.09 $\pm$ 3.79	4.7%
Opus-4.8	61.68 $\pm$ 4.39	7.0%

Table 2: Functional score (mean

\pm

SE) and complete-delivery success over 43 cases.

Figure 4: Performance by derived/implicit Code Point share. Lines show scores; pale bars show mean tool use.

Figure 5: Weighted acceptance loss across 43 cases, decomposed into four outcome symptoms.

Figure 6: Case-level co-occurrence of outcome symptoms (rows) and process signals (columns) for three models, using a shared color scale.

	Code quality (0–100)				Execution process
Model	Conv.	Arch.	Name	Test	Verify	Self	Post	Tools
Unified minimal scaffold
GPT-5.6-Sol	58.2	68.4	94.3	47.2	10.4	8.4	5	62.0
GLM-5.2	57.7	66.9	93.2	51.1	5.9	4.0	11	122.8
Qwen3.7-Max	52.5	62.8	94.0	45.4	4.7	4.3	4	75.9
Kimi-K2.6	57.4	65.1	95.5	43.2	8.0	6.9	10	110.1

Table 3: Auxiliary code-quality and process observations. Verify, Self, and Tools are mean counts per case; Post counts cases with successful verification after the last edit (of 43). Test scores use the nine cases requiring tests. None enters the functional score.

Control	Score / 100
Positive control
Accepted reference patch	98.35
Negative controls
A. Empty patch	0.00
B. Same-repository unrelated patch	0.00
C. Applicable cross-case patch	0.00

Table 4: Reference and negative-control scores.

Method	Agr. $\uparrow$	Weighted $\kappa$ $\uparrow$	Severe dis. $\downarrow$
Full pipeline	95.2%	0.945	0.0%
w/o contract grounding	70.3%	0.595	9.7%
w/o tiered judgment	87.7%	0.843	1.5%
w/o checklist aggregation	76.3%	0.677	5.9%
Vanilla LLM judge	77.6%	0.686	4.3%
Vanilla LLM with contract	76.0%	0.652	8.3%

Table 5: Expert-consensus agreement, grouped into the full pipeline, component ablations, and judge baselines.

Figure 7: Qwen3.7-Max repetition: any-run versus every-run success. Annotations show changes in percentage points.

Knowledge input	Mean score $\pm$ SE	$\Delta$
No knowledge	$58.54\pm 5.11$	—
Locators only	$57.80\pm 4.84$	$-0.74$
Repository-matched knowledge	$59.23\pm 4.77$	$+0.69$
Shared long context	$57.03\pm 5.22$	$-1.51$

Table 6: Paired knowledge comparison on 28 cases selected for available repository knowledge.

Figure 8: A representative retrieval diagnosis and scores of three successive harness releases.

Figure 9: Monthly coding-agent platform observations from February to August.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Configuration
Cases / Code Points	43 / 750
Common scaffold	OpenHands 1.21.0
Agent iteration limit	200
Tool concurrency	4
Qwen generation effort	xhigh
Semantic judge	Opus-4.8

Appendix

Table 7: Common evaluation settings; native-scaffold configurations are recorded separately.

Final PRD source	Cases
Direct or reconstructed task document, without later user clarification	8
First user request alone	3
Initial task plus later user clarification	32
Total	43

Appendix

Table 8: Source paths contributing to the 43 final PRDs.

	Missing	Partial	Covered
Missing	1	0.5	0
Partial	0.5	1	0.5
Covered	0	0.5	1

Appendix

Table 9: Linear agreement weights for Cohen’s

\kappa

Figure 10: Outcome symptoms and process signals across the eight primary models.

Figure 11: Task-size associations with eight-model mean score. Each point is a case; intervals use case-level bootstrap sampling.

Release	Main change
v1	Frontend task flow and backend simple-task guide.
v2	Fuller tool guidance, complex-task decisions, and subtask tracking.
v3	Updated tool metadata, clarification rules, and plan-completion events.

Appendix

Table 10: Changes across the three evaluated harness releases.

Measure	Value
Mean score, executions 1 / 2 / 3	66.70 / 65.44 / 65.37
Mean score, judging records 1 / 2 / 3	66.70 / 66.47 / 66.40
Complete deliveries, judging records 1 / 2 / 3	3/43 / 2/43 / 3/43
Covered-point rate, judging records 1 / 2 / 3	53.3% / 52.5% / 52.4%
Linearly weighted Cohen’s $\kappa$ (mean)	0.885
Severe disagreement rate	0.22%

Appendix

Table 11: Three executions and three fixed-patch judging records over the same 43 cases.

Descriptor	$\rho$	$p$	95% CI
Code Point count	$-.040$	.801	$[-.36,.26]$
Changed lines	$-.058$	.710	$[-.40,.27]$
Changed files	$-.037$	.814	$[-.34,.29]$
PRD length	$.083$	.598	$[-.24,.39]$

Appendix

Table 12: Task-size correlations and case-bootstrap intervals over 43 cases.

AI Coding Agents

AI coding agents use language models and software tools to plan, edit, test, and maintain code with varying levels of autonomy.

See all 329 papers

Aug 10, 2026

A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents

Large language model-powered coding agents are increasingly used to modify existing code repositories, for example, by adding features or fixing bugs. Yet existing repository-level benchmarks typically evaluate only whether the final patch passes tests. Satisfying a user request requires a long chain of interdependent reasoning and decisions: an agent must recover explicit and implicit requirements, formulate a repository-grounded implementation plan, and translate it into correct code. A pass/fail outcome cannot characterize how an unsuccessful trajectory diverges from the requirements and implementation process needed for a correct patch. To address this gap, we introduce SWE-RPG, a repository-level benchmark that combines executable patch evaluation with validated ground-truth references (GTs) for (1) Requirement Clarification and (2) Implementation Planning. These intermediate GTs support retrospective, GT-aligned diagnosis of complete coding-agent trajectories across clarification, planning, code generation, and artifact submission. SWE-RPG comprises 163 tasks from 31 Python and Java repositories, including 113 bug fixes and 50 feature additions. We evaluate 3 coding agents, including Claude Code, Codex, and OpenCode, with 6 large language model backends, including Claude-Sonnet-5 and GPT-5.6-Terra. Results show that the evaluated popular coding agents still struggle to implement user requests in existing repositories, achieving an average resolved rate of only 31.5% on SWE-RPG. Intermediate-GT diagnosis further identifies implicit requirement recovery as the main bottleneck, accounting for 24.5%--46.0% of agent runs. This result suggests implicit-requirement recovery as a key candidate direction for improving coding agents. The benchmark data and evaluation code are available at https://github.com/Xin-Zhou-smu/SWE-RPG-Bench.

Xin Zhou, Chun Yong Chong, Kisub Kim +11

Singapore Management University, Singapore · Monash University Malaysia, Malaysia · Daegu Gyeongbuk Institute of Science and Technology (DGIST), Republic of Korea +5

Requirements Engineering AI Coding Agents Code Generation Benchmarks

Ask PDF

Oct 26, 2026

Benchmarking Coding Agents under Evolving Requirements

Software development with coding agents often begins with a rough request that users refine after trying the implementation. Final correctness alone does not capture the repeated guidance this process requires. We introduce FeatureLoop, a benchmark of 61 human-reviewed tasks from 34 repositories across five primary programming languages and seven software forms. Each task follows one development goal in a persistent workspace through evolution feedback, which adds or clarifies requirements, and completion feedback, which reports failures of stated requirements. Cumulative executable checks track new behavior, repair, and preservation of earlier functionality. We evaluate 12 frontier models and find that the strongest completes 72.1% of tasks with both feedback types. Across models, however, only 15.4% of observed failed stages recover within two corrective revisions. Further analysis reveals incomplete repairs, regressions, and model-specific differences in feedback use and token efficiency, identifying obstacles to reliable, low-intervention development.

Jia Feng, Alyssa Wu, Cuiyun Gao +3

Hunyuan Team, Tencent · The Chinese University of Hong Kong

Coding Agent Benchmarks Code Generation Process-Based Agent Evaluation LLM Agent Reliability

Ask PDF

Apr 3, 2026

Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution

Existing datasets for coding agents evaluate performance on isolated, single pull request (PR) tasks in a stateless manner, failing to capture the reality of real-world software development where code changes accumulate, technical debt accrues, and test suites grow over time. To bridge this gap, we introduce an automated coding task generation framework, which helps generate our dataset SWE-STEPS, that evaluates coding agents on long-horizon tasks through two realistic settings mirroring actual developer workflows: Conversational coding with iterative requests, and single-shot Project Requirement document (PRD)-based coding. Unlike existing datasets that evaluate agents on disjointed Pull Requests (PRs), our framework assesses performance across chains of dependent PRs, enabling evaluation of sequential execution, regression verification, and long-term repository health. We discover that widely used isolated PR evaluations yield inflated success rates, w.r.t. our settings - overshooting performance by as much as 20 percentage points - because they ignore the ``spillover'' effects of previous inefficient or buggy code. Furthermore, our analysis reveals that even when agents successfully resolve issues, they degrade repository health by generating code with higher cognitive complexity and technical debt compared to human developers, underscoring the necessity for multidimensional evaluation.

KN Ajay Shastry, Ganesh Senrayan, Shrey Satapara +2

Fujitsu Research

Code Generation Evaluation Software Engineering Long-Horizon Agent Evaluation Software Engineering Benchmarks

Ask PDF

Abstract

AI Coding Agents

Explore similar work

A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents

Benchmarking Coding Agents under Evolving Requirements

Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution