Software development is iterative, yet agentic coding benchmarks hide design issues through their single-shot setup. Recent iterative benchmarks attempt to remedy this but heavily constrain an agent's design decision space, making it impossible to faithfully measure how their decisions shape future extensions. We introduce SlopCodeBench, a benchmark of 36 problems and 196 checkpoints where agents repeatedly extend their own solutions. Unlike prior iterative benchmarks, our evolving specifications demand architectural decisions but leave internal structure to the agent. We measure two forms of degradation: structural erosion (concentrated complexity) and verbosity (redundant code). Evaluating 15 coding agents across open and closed models, we find that no agent fully solves any problem end-to-end, and the best agent passes 14.8% of checkpoints. Quality degrades across checkpoints, with structural erosion rising in 77% of trajectories and verbosity in 75.5%. Compared to 473 open-source Python repositories, agent code is 2.3x more verbose and 2.0x more eroded, and the human repositories degrade less often and by smaller margins across their git histories. Explicit quality guidance reduces initial verbosity and erosion by up to a third, without affecting degradation rates. SlopCodeBench provides the first measurement of code degradation under iterative extension, revealing that agents pass checkpoints while producing code that erodes and bloats with each turn.
Figures & tables
Figure 1: Iterative evaluation in SCBench. A task unfolds over checkpoints. At each step, the agent extends its prior workspace to satisfy an updated spec (orange). Correctness, erosion, and verbosity are scored externally and withheld from the agent (gray). Local shortcuts compound: correctness falls from 1.00 to 0.10 while erosion and verbosity grow.
Solve Rate (%)
Cost & Time
Quality
Model
Strict
Iso.
Core
Partial
$/CKPT
Net $
Min/CKPT
Erosion
Verbosity
Opus 4.5
9.2
17.3
57.7
25.0
2.53 ± 1.57
492.40
7.2 ± 3.7
0.70 ± 0.20
0.43 ± 0.18
Opus 4.6
9.7
20.9
67.3
27.8
3.17 ± 2.82
620.80
13.1 ± 12.9
0.75 ± 0.15
0.44 ± 0.19
Opus 4.7
8.2
20.9
65.8
36.1
2.17 ± 1.83
425.34
6.8 ± 5.0
0.76 ± 0.17
0.48 ± 0.21
Sonnet 4.6
7.1
16.8
57.7
27.8
1.96 ± 1.97
383.83
14.2 ± 14.2
0.75 ± 0.20
0.44 ± 0.19
GPT 5.2 Codex
9.7
21.9
56.1
36.1
0.85 ± 0.80
167.29
12.3 ± 9.7
0.73 ± 0.18
0.50 ± 0.16
Table 1: Per-model performance on SlopCodeBench. For each model we report its best just-solve run, ranked by isolated solve rate with core, strict, and $/checkpoint as tiebreakers. Solve rates use a fixed 196-checkpoint denominator; missing checkpoints count as unsolved. Other metrics are only for ran checkpoints. Bold marks the best value per column. See Table 8 for the full set of runs.
Figure 2: Performance degrades as problems progress and cost per checkpoint steadily rise. Left: Checkpoint solve rate by type. Right: Mean cost per checkpoint as problems progress.
Figure 3: Models from both frontier labs produce similar degradation trends as the problem progresses . Erosion and verbosity across problem progress for six representative models.
Figure 4: Agents produce lower quality code and degrade it faster than real developers. Verbosity and erosion across normalized trajectory progress, with 95% confidence intervals. GitHub-star bins over 473 repositories for a total of 13667 checkpoints. Table 2 has HEAD commit details.
Group
n
Verbosity
Erosion
Hobby ( <100★ )
114
0.21±0.14
0.33±0.25
Niche ( 100–1k★ )
124
0.19±0.11
0.32±0.22
Established ( 1k–10k★ )
120
0.18±0.08
0.33±0.20
Major ( ≥10k★ )
115
0.19±0.10
0.37±0.19
Human panel (all)
473
0.19±0.11
0.34±0.22
Agent checkpoints
2869
0.44±0.18
0.68±0.20
Table 2: Verbosity and erosion at HEAD commit, by GitHub-star tier and against agent checkpoints.
Figure 5: Prompting can improve initial quality, but does not remove the iterative issues. Mean structural erosion (top) and verbosity (bottom) at each progress bin, per model.
Model
Prompt
Strict
Iso.
Core
Erosion
Verbosity
Cost
GPT 5.3 Codex
Just-Solve
11.2
26.0
60.7
0.64
0.46
0.66
Anti-Slop
11.2 (+0.0)
23.5 ( -2.6 )
57.1 ( -3.6 )
0.42 ( -0.22 )
0.31 ( -0.15 )
0.73 ( +0.07 )
Plan-First
8.2 ( -3.1 )
21.4 ( -4.6 )
62.2 ( +1.5 )
0.55 ( -0.10 )
0.40 ( -0.06 )
0.68 ( +0.02 )
GPT 5.4
Just-Solve
10.7
23.5
62.8
0.51
0.33
0.72
Anti-Slop
9.2 ( -1.5 )
25.5 ( +2.0 )
61.2 ( -1.5 )
0.28 ( -0.23 )
0.24 ( -0.09 )
0.90 ( +0.18 )
Plan-First
9.7 ( -1.0 )
25.0 ( +1.5 )
63.3 ( +0.5 )
0.39 ( -0.11 )
0.30 ( -0.02 )
0.83 ( +0.11 )
Table 3: Raw values per prompt with the lift over just-solve in parentheses. Green means improvements while red indicates regressions.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Problem
CPs
Design decisions stressed
cfgpipe
6
Source-priority resolution, typed parsing boundaries, nested schema representation, watch and event architecture, redaction, and store-prefix composition.
circuit_eval
8
Circuit IR, parser separation across formats, two-valued versus three-valued logic, optimization pass pipeline, and equivalence checking.
code_search
5
Regex versus AST matching, metavariable representation, rewrite conflict handling, selector semantics, and language adapter boundaries.
dag_execution
3
DAG DSL parsing, task parameter model, execution ordering, cache key design, and per-task cache overrides.
database_migration
5
Migration operation model, rollback representation, dependency sorting, cycle errors, and constraint or index handling.
datagate
7
Dataset ingestion abstraction, query and filter pipeline, pagination and export contracts, cache invalidation, runtime config, and access control.
Appendix
Table 4: SlopCodeBench problems and the design decisions stressed by iterative checkpoints, part 1 of 2. CPs gives the number of checkpoints.
Problem
CPs
Design decisions stressed
forge
8
Resource models for blueprints, allocations, units, modules, revision gates, strict JSON contracts, and admin access boundaries.
Layer merge semantics, conflict resolution, fragment expansion, environment interpolation, multi-run manifests, and JSON Schema validation.
log_query
5
SQL-like grammar design, NDJSON streaming execution, aggregation and joins, schema label mapping, and subquery scope.
meshctl
8
Declarative resource specs, persisted state shape, lifecycle and topology operations, security model, migrations, network exposure, and multi-region policy.
metric_transform_lang
5
DSL parser and interpreter design, stream processing, aggregation and window state, temporal joins, and resumable execution.
Appendix
Table 5: SlopCodeBench problems and the design decisions stressed by iterative checkpoints, part 2 of 2.
Metric
Pass
Cost
Reported erosion
−0.018
0.167
Best variant (CC > 12 + Linear SLOC)
−0.136
0.095
LOC
−0.212
0.502
Mean CC
−0.046
0.218
Max CC
−0.117
0.356
Clone Ratio
−0.042
0.114
Appendix
Table 6: Erosion metric sensitivity sweep.
Model
Harness
Version
Reasoning
Opus 4.5
Claude Code
2.0.51
high
Opus 4.6
Claude Code
2.1.32
high
Sonnet 4.6
Claude Code
2.1.44
high
GPT 5.2 Codex
Codex CLI
0.93.0
high
GPT 5.3 Codex
Codex CLI
0.98.0
high
GPT 5.3 Spark
Codex CLI
0.100.0
high
Appendix
Table 7: Harness version and reasoning effort used for each model in the main evaluation.
Solve Rate (%)
Cost
Quality
Model
Harness
Version
Think
Prompt
Strict
Iso.
Core
Partial
$/CKPT
Net $
Erosion
Verbosity
Opus 4.5 [ 5 ]
Claude Code
2.0.51
High
Just Solve
9.2
17.3
57.7
25.0
2.53 ± 1.57
492.40
0.70 ± 0.20
0.43 ± 0.18
Opus 4.6 [ 6 ]
Claude Code
2.1.32
High
Just Solve
9.7
20.9
67.3
27.8
3.17 ± 2.82
620.80
0.75 ± 0.15
0.44 ± 0.19
Opus 4.7 [ 7 ]
Claude Code
2.1.111
High
Just Solve
8.2
20.9
65.8
36.1
2.17 ± 1.83
425.34
0.76 ± 0.17
0.48 ± 0.21
Sonnet 4.6 [ 8 ]
Claude Code
2.1.44
High
Just Solve
7.1
16.8
57.7
27.8
1.96 ± 1.97
383.83
0.75 ± 0.20
0.44 ± 0.19
GPT 5.2 Codex [ 45 ]
Codex CLI
0.93.0
High
Just Solve
9.7
21.9
56.1
36.1
0.85 ± 0.80
167.29
0.73 ± 0.18
0.50 ± 0.16
Appendix
Table 8: Full set of runs on SlopCodeBench across all harnesses, versions, reasoning levels, and prompt strategies. Bold marks the best value per column.
Coding agents are increasingly used as iterative development partners, but most benchmarks still evaluate one specification followed by one final assessment. This leaves out a basic question: can an agent keep its own codebase working as requirements change? We introduce EvoCode-Bench, a benchmark of 26 stateful coding tasks and 227 evaluated rounds. Each task preserves the agent's workspace for 5-15 rounds, states requirements through observable behavior, and uses cumulative executable tests to check new requirements and still-active prior ones. We evaluate 13 coding agents with two metrics: MT@4, a four-attempt fail-stop multi-round score, and SR, a single-round score from a reference-completed prior state. For most agents, SR exceeds MT@4 by 22-40 points. The gap also changes rankings: the highest-SR agent (78.9) ranks only third in persistent execution (44.0 MT@4). Even the strongest agents achieve only about 50% success on multi-turn metrics, and aggregate pass rate drops below half of round-1 performance by round 5. Failure analysis shows tier-dependent behavior: weaker agents fail early, while stronger agents survive long enough to expose specification-tracking and regression failures. We release the benchmark data and Harbor multi-turn infrastructure.
Haiyang Shen, Xuanzhong Chen, Wendong Xu +3
1UniPat AI · 2Peking University · 3Tsinghua University +1
Coding agents are increasingly deployed in real software development, where a single version iteration requires months of coordinated work across many files. However, most existing benchmarks focus predominantly on single-issue bug fixes from Python repositories, with coarse pass/fail evaluation outcomes, and thus fail to capture long-horizon, multi-target development at real engineering scale. To address this gap, we present RoadmapBench, a benchmark of 115 long-horizon coding tasks grounded in real open-source version upgrades across 17 repositories and 5 programming languages. Each task places the agent on a source-version code snapshot and provides a multi-target roadmap instruction requiring it to implement the functionality introduced in the target version, with a median modification of 3,700 lines across 51 files. We conduct a systematic evaluation on thirteen frontier models and find that even the strongest, Claude-Opus-4.7, resolves only 39.1% of tasks, while the weakest achieves merely 5.2%, in stark contrast to existing bug-fix benchmarks, suggesting that long-horizon software development remains a largely unsolved problem.
Xinbo Xu, Ruihan Yang, Haiyang Shen +13
UniPat AI · Peking University · Fudan University +4
We introduce StaminaBench, a benchmark that measures the stamina of coding agents: how many consecutive interaction turns (change requests) they can handle before failing. Unlike the prevailing fraction-of-tasks-solved metric, this matches real vibe-coding where sessions run dozens or hundreds of turns. In StaminaBench, agents implement a REST API server and modify it across a tunable number of procedurally generated follow-up change requests - 100 in our experiments, resulting in codebases of up to 6,000 lines. Tests are generated fully programmatically without LLM involvement, ensuring reproducibility and reliability; change sequences are drawn from either a hardcoded or LLM-driven sampler, both constrained to a structured action space to ensure changes are valid. The agent and the server run in an isolated environment and communicate with the benchmark through HTTP, making testing fully black-box and language-agnostic. We evaluate six agent harnesses paired with seven open-source LLMs across 20 scenarios of 100 turns each and find that: (1) all the tested models fail within 5-6 turns, confirming that vibe-coding-style programming without thorough testing produces bugs; (2) passing test feedback back to the agent and allowing it to retry improves passed turn count by up to 12x; and (3) a good harness is required for strong performance: stronger models exhibit up to a 6x gap between their best and worst harness, while weaker models fail with any harness. We release the benchmark and the generated tasks to enable further research into multi-turn coding agent behavior. Benchmark code and data: github.com/amazon-science/StaminaBench.