Coding agents now write whole codebases in long autonomous runs, guided mainly by whether their tests pass. Yet passing tests alone does not reveal accumulating complexity or gaps in test coverage, which then persist as agents extend their own code. We introduce metrics-in-the-loop, an inference-time intervention that gives agents in-flight feedback: cheap, automatically computed code metrics injected into the agent's context during the trajectory. After each code-changing agent turn, we insert short feedback messages naming changed functions whose cyclomatic complexity exceeds a limit (complexity feedback) or changed lines that are skipped by tests (coverage feedback), giving the agent the opportunity to refactor code or strengthen tests before continuing. We run extensive experiments and ablations comparing several interventions across three long-horizon coding benchmarks and four strong models. We find that complexity feedback significantly reduces erosion, a measure of complexity concentration of code, without affecting held-out correctness of the code. Broadly, our experiments investigate a practical form of agent "re-railing", where timely, actionable measurements redirect agents before weaknesses become the foundation they keep building on. As agents scale to codebases with millions of lines, such interventions become critical to sustaining effective development and long-term code maintainability.
Figures & tables
Figure 1: Metrics-in-the-loop. (a) One round: after a turn that changes code, the changed functions are measured and the report (cyclomatic complexity (C) and/or test coverage (T)) enters the agent’s context before its next turn. (b) Adding both reports to the reminder baseline (R → CT) lowers erosion for all four models on Commit0 (circles) and SlopCodeBench (squares), with no detectable effect on held-out correctness for any model. Each arrow starts at R (hollow) and shows the estimated paired change to CT (filled), with whiskers for the 95% intervals. The axes show held-out pass fraction and one minus the share of complexity in functions above 10 (higher is better for both axes).
Figure 2: CT − R in E10 during the run, by model, with pointwise 95% t bands. (a) Commit0: paired differences within libraries at each token budget (log scale, up to 2M), with at least 15 paired libraries. (b) SlopCodeBench: paired differences within problems at each checkpoint, with at least 5 pairs; not all problems reach 8 checkpoints.
E10
change in E10
correctness
Model
under R
C–R
CT–R
CT–R
Commit0, 25 libraries
DeepSeek
0.251
−0.123 [−0.185, −0.061]
−0.070 [−0.110, −0.030]
+4.4 [−1.2, +10.0]
GLM
0.231
−0.120 [−0.177, −0.064]
−0.096 [−0.145, −0.046]
+4.7 [−1.6, +11.0]
Haiku
0.122
−0.070 [−0.108, −0.031]
−0.054 [−0.082, −0.026]
−2.3 [−10.2, +5.6]
Luna
0.311
−0.266 [−0.346, −0.186]
−0.261 [−0.339, −0.182]
+3.3 [−4.8, +11.5]
Table 1: Erosion under complexity feedback and correctness under combined feedback, by model. E10 ( ↓ better) is the share of complexity mass in functions with cyclomatic complexity above 10. Contrasts are paired mean differences with 95% intervals. Correctness ( ↑ better) is the held-out pass fraction or, on SWE-EVO, the fix rate; differences are in percentage points. Commit0 erosion estimates use seeds 1–5 for R and CT and 1–3 for C. Seeds 4 and 5 were added after the initial analysis; correctness uses the planned seeds 1–3. SWE-EVO intervals are clustered by six projects, with CT using 29 of 35 transitions. Pooled estimates are post hoc. I2 ( Higgins and Thompson, 2002 ) estimates the proportion of variation among models attributable to heterogeneity; high I2 indicates that they average effects that differ across models. App. B reports all contrasts and paired samples.
Check
Threat it guards against
What it changed here
Match instructions and delivery in the reminder arm
Instructions or delivery frequency affect behavior
For Haiku on SlopCodeBench, the reminder alone reduces E10 by 0.383 ; complexity feedback reduces it by a further 0.045
Outcomes withheld from the agent
Evaluation rewards the optimized metric ( Manheim and Garrabrant, 2018 )
On SlopCodeBench, the benchmark’s erosion score, verbosity and hidden-suite coverage mostly support the E10 result; on Commit0, every model’s C − R coverage interval includes zero
Audit the changes behind the score
Function splitting lowers E10 without simplifying code
On Commit0, 3 to 46% of each erosion difference remains after inlining in both arms; DeepSeek’s and GLM’s total branching changes little
Specify the scale; report models before pooling
A pooled estimate obscures model differences
Absolute E10 effects on SlopCodeBench differ fifteenfold ( I2 100%), yet every model removes nearly all erosion present under the reminder arm
For SWE-EVO, transition-level intervals exclude zero for seven of eight model estimates across C − R (35 transitions) and CT − R (29); project clustering reduces this to four
Use operating-system isolation for held-out material
Agents access hidden tests or finished code
On Commit0, agents routinely bypassed a filter on path words, before proper exclusion measures were taken; on SlopCodeBench and SWE-EVO, operating-system isolation (bubblewrap in Dockers) blocked every attempt
Table 2: List of checks of interventions made during a coding agent run (Section 3 , Appendix A ).
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Arm
Message after a turn that changes code
B0
none
R
general feedback instructions only (“placebo”)
C
complexity table and general feedback instructions
T
coverage table and general feedback instructions
CT
both tables, shown one after the other, and general feedback instructions
Appendix
Table A1: Experimental arms. Every arm except B0 receives the same R-arm feedback instructions in addition to its own report.
Commit0
SlopCodeBench
SWE-EVO
Task
reconstruct a Python library from its specification
build a program over 3–8 checkpoints, each a specification that extends the previous one
update a published project to its next release using that release’s problem statement
Tasks
25 libraries
35 problems
35 transitions in 6 projects
Seeds
3; 5 for R and CT
2
2
Cells launched / analyzed
1,900 / 1,850
1,400 / 1,318
1,400 / 1,400
Budget
2M processed tokens per cell
1M processed tokens per checkpoint
2M processed tokens and 90 minutes per cell
Context
one conversation
fresh at each checkpoint; workspace, own tests and notes kept; every checkpoint runs even if the previous one failed
one conversation
Appendix
Table A2: The three experiments. Each runs the five arms of Table A1 with the same feedback wording under DeepSeek V4.1 Flash (2026-09-10 release), GLM-5.3 Flash (2026-08-26 release), Claude Haiku 5.5 and GPT-6 Luna. A cell is one combination of task, arm, model and seed. Counts are summed over the four models.
Figure B1: Arrow diagrams for the other three contrasts, drawn as in Figure 1 (b): (a) R → C, complexity feedback; (b) R → T, coverage feedback; (c) B0 → R, the reminder alone; four models, circles Commit0 and squares SlopCodeBench. Hollow markers denote the starting arm and filled markers the ending arm (Commit0: seeds 1–5 for R and 1–3 for C, T and B0; SlopCodeBench: complete-case pairing of problems within each model). Each arrow starts at the starting arm’s mean for the paired tasks and shows the estimated paired change. Whiskers show the paired 95% t interval for that change, shifted to the arrowhead. They do not represent uncertainty in the ending arm’s mean or a joint two-dimensional region. Each axis uses its own complete-case task set, so an arrow’s pass fraction and E10 may use different tasks. Absolute levels are not comparable across models.
Figure B2: SWE-EVO arrow diagrams for all four contrasts, drawn as in Figure B1 , with the fix rate on the horizontal axis and the fall in E10 from the starting code, in ‰, on the vertical one; higher is better on both. Hollow and filled triangles denote the starting and ending arms, averaged over the paired transitions: 35 for R → C and B0 → R, and 29 for R → CT and R → T. Arrows show the reported paired estimates. Whiskers show the paired 95% t intervals over transitions, shifted to the arrowhead. An interval containing zero therefore reaches the hollow triangle’s coordinate on that axis. These plotted intervals treat transitions as independent. The main analysis instead clusters intervals by project (Section 2 ), giving wider intervals for most models and contrasts. Transition-level intervals exclude zero for seven of eight C − R and CT − R erosion estimates; project-clustered intervals exclude zero for four (Table 2 ). Vertical scales differ by panel.
Model
R
C–R
CT–R
T–R
R–B0
n
Commit0, 25 libraries
DeepSeek
0.251 (25)
−0.123 [−0.185, −0.061]
−0.070 [−0.110, −0.030]
+0.007 [−0.009, +0.024]
−0.002 [−0.019, +0.015]
25
GLM
0.231 (24)
−0.120 [−0.177, −0.064]
−0.096 [−0.145, −0.046]
+0.015 [−0.020, +0.049]
−0.021 [−0.056, +0.014]
24
Haiku
0.122 (25)
−0.070 [−0.108, −0.031]
−0.054 [−0.082, −0.026]
+0.021 [+0.002, +0.040]
−0.078 [−0.121, −0.035]
25
Luna
0.311 (25)
−0.266 [−0.346, −0.186]
−0.261 [−0.339, −0.182]
0.000 [−0.038, +0.038]
−0.036 [−0.056, −0.015]
25
Pool (post hoc)
−0.112 [−0.138, −0.087]
−0.078 [−0.097, −0.059]
+0.012 [+0.001, +0.023]
−0.020 [−0.032, −0.009]
Appendix
Table B1: Change in E10 at the final tree, per model, for all four contrasts. Lower is better. R gives the reminder-arm mean over its included tasks, with the task count in parentheses. Contrasts may use different paired task sets, so adding a contrast to R need not recover the other arm’s mean. n is the number of tasks paired for CT–R. Cells with no measured outcome are dropped. Commit0 pairs arm means within library, over seeds 1–5 for R and CT and seeds 1–3 for B0, C and T; SlopCodeBench pairs arm means within problem, over two seeds. E10 was planned as a secondary outcome on all three benchmarks; correctness was the primary one. On SWE-EVO, contrasts involving T or CT use 29 of the 35 transitions. On six dvc transitions, a test module that fails to import stopped the project’s suite before any test ran, so the coverage feedback that defines T and CT measured nothing. We excluded those six before reading any T or CT result (App. A ); the other contrasts use all 35. Intervals are clustered by the six projects, as planned. The small-cluster correction was chosen later: CR2 with Bell–McCaffrey degrees of freedom ( Bell and McCaffrey, 2002 ; Pustejovsky and Tipton, 2018 ) , about two per model. The pooled estimate uses a t interval with Satterthwaite degrees of freedom ( Satterthwaite, 1946 ) . The pooled estimate is a post hoc fixed-effect inverse-variance average across four models. Its standard error assumes independent models despite their shared tasks. I2 estimates the proportion of variation among the four estimates attributable to heterogeneity; high values indicate that the pooled estimate averages differing effects.
Model
R
C–R
CT–R
T–R
R–B0
n
Commit0, seeds 1–3 (planned)
DeepSeek
68.9 (25)
+2.7 [−1.8, +7.3]
+4.4 [−1.2, +10.0]
+0.6 [−5.2, +6.3]
+5.9 [−1.9, +13.7]
25
GLM
35.9 (25)
+1.4 [−6.3, +9.1]
+4.7 [−1.6, +11.0]
+1.0 [−7.1, +9.0]
−0.8 [−8.2, +6.6]
25
Haiku
58.1 (25)
−2.7 [−9.9, +4.6]
−2.3 [−10.2, +5.6]
−1.5 [−7.0, +3.9]
−1.5 [−6.8, +3.9]
25
Luna
51.0 (25)
−0.4 [−8.8, +8.0]
+3.3 [−4.8, +11.5]
−1.6 [−7.8, +4.6]
−0.5 [−6.5, +5.5]
25
Pool (post hoc)
+1.0 [−2.0, +4.0]
+3.1 [−0.1, +6.3]
−0.6 [−3.5, +2.4]
+0.2 [−2.9, +3.2]
Appendix
Table B2: Change in held-out pass fraction on Commit0 and SlopCodeBench, per model, in percentage points, for all four contrasts. Commit0 uses the withheld official suite at the 2M-token cap. SlopCodeBench uses the strict hidden-test pass fraction at the final checkpoint. R is expressed as a percentage. R gives the reminder-arm mean over its included tasks, with the task count in parentheses. Contrasts may use different paired task sets, so adding a contrast to R need not recover the other arm’s mean. n is the number of tasks paired for CT–R. Cells with no measured outcome are dropped. We added seeds 4 and 5 for R and CT after inspecting results from seeds 1–3. The other arms use seeds 1–3. The pooled estimate is a post hoc fixed-effect inverse-variance average across four models. Its standard error assumes independent models despite their shared tasks. I2 estimates the proportion of variation among the four estimates attributable to heterogeneity; high values indicate that the pooled estimate averages differing effects.
Model
R
C–R
CT–R
T–R
R–B0
n
Fix rate (primary)
DeepSeek
45.5 (35)
+5.9 [−7.6, +19.4]
+5.1 [−2.3, +12.5]
−3.4 [−19.0, +12.3]
−0.6 [−12.4, +11.2]
29
GLM
31.2 (35)
+1.7 [−14.3, +17.6]
+1.5 [−31.2, +34.1]
+3.7 [−16.6, +24.1]
+1.0 [−11.7, +13.7]
29
Haiku
31.6 (35)
−0.9 [−9.1, +7.3]
+0.4 [−8.1, +9.0]
−0.8 [−11.4, +9.9]
−2.3 [−8.6, +4.1]
29
Luna
35.7 (35)
+2.8 [−18.8, +24.5]
+2.2 [−5.4, +9.7]
+4.5 [−20.0, +28.9]
−4.9 [−25.6, +15.9]
29
Pool (post hoc)
+1.2 [−2.6, +4.9]
+2.7 [−0.2, +5.6]
−0.2 [−5.3, +4.8]
−1.6 [−4.8, +1.5]
Appendix
Table B3: SWE-EVO correctness per model, in percentage points, for all four contrasts. Fix rate is the fraction of initially failing tests that pass on the final code. It is set to zero if the agent breaks a test that passes both before and after the reference change. Resolution means every official test passes. R is in percent. On SWE-EVO, the plan named three main comparisons, all on fix rate: C–R, T–R and CT–R. Each has its own 95% interval, with no correction for multiple comparisons. Contrasts involving T or CT use 29 of the 35 transitions. On six dvc transitions, a test module that fails to import stopped the project’s suite before any test ran, so the coverage feedback that defines T and CT measured nothing. We excluded those six before reading any T or CT result (App. A ); the other contrasts use all 35. Intervals are clustered by the six projects, as planned. The small-cluster correction was chosen later: CR2 with Bell–McCaffrey degrees of freedom ( Bell and McCaffrey, 2002 ; Pustejovsky and Tipton, 2018 ) , about two per model. The pooled estimate uses a t interval with Satterthwaite degrees of freedom ( Satterthwaite, 1946 ) . The pooled estimate is a post hoc fixed-effect inverse-variance average across four models. Its standard error assumes independent models despite their shared tasks. I2 estimates the proportion of variation among the four estimates attributable to heterogeneity; high values indicate that the pooled estimate averages differing effects.
complexity: edited / under limit
coverage rose
Benchmark
Model
R
C
CT
R
T
CT
Commit0
DeepSeek
20% / 1%
54% / 46%
36% / 23%
64%
57%
57%
Commit0
GLM
30% / 3%
60% / 58%
46% / 35%
40%
42%
34%
Commit0
Haiku
31% / 9%
83% / 86%
67% / 66%
47%
51%
49%
Commit0
Luna
49% / 3%
88% / 94%
85% / 90%
59%
86%
81%
SCB
DeepSeek
38% / 5%
94% / 94%
84% / 82%
17%
95%
92%
Appendix
Table B4: Agent responses to feedback, using included cells and Commit0 seeds 1–3. For complexity, we report the proportions of over-limit functions edited again and under the limit at the end of the run. C and CT count functions flagged in the report; R counts over-limit functions edited by the agent. For coverage, all arms use the same rule: among functions edited while coverage was below 1, we report the proportion whose coverage later increased. The proportions pool function-turn observations, so a function counts once for each turn on which it was flagged or edited. No uncertainty intervals are reported.
change in E10
change in branches and lines
Model
pairs
as measured
after inlining
branches
lines
per 1k lines
Commit0, seeds 1–3
DeepSeek
C–R
62
−0.134 [−0.200, −0.073]
−0.023 [−0.043, −0.004]
+1%
+3%
−7.8 [−14.0, −1.9]
CT–R
61
−0.066 [−0.103, −0.035]
−0.016 [−0.037, +0.004]
+1%
+2%
−5.6 [−10.3, −0.8]
GLM
C–R
56
−0.124 [−0.190, −0.060]
−0.029 [−0.085, +0.020]
0%
+4%
−4.6 [−13.2, +5.6]
CT–R
55
−0.096 [−0.166, −0.025]
−0.003 [−0.047, +0.051]
−4%
+1%
−5.3 [−17.3, +7.9]
Appendix
Table B5: What lies behind the drop in E10 . A branch is a point where the code can take one of two paths, such as an if , a loop, an and or an or ; a function’s cyclomatic complexity is its number of branches plus one. Splitting a complex function into helpers lowers E10 without removing a single branch, so we check how much of that drop survives undoing it. Each pair is one task and seed with both arms’ final code on record (seeds 1–3 on Commit0). As measured is the change in E10 on these pairs; it differs slightly from Table B1 , which averages seeds within each task first. After inlining repeats the measurement as if every function called from only one place had been written inside its caller. We do this in both arms, including helpers written under R or by the libraries’ original authors, so it is a sensitivity check rather than an estimate of what splitting causes. The last three columns describe the whole program. Branches is the percent change in the number of branches, which splitting does not remove; lines is the percent change in lines of code; and per 1k lines is the change in branches per thousand lines of code. Branches and per 1k lines can have opposite signs because the code’s length changes too: a few more branches spread over many more lines means fewer branches per line. The branch count also grows with how much of the library an agent builds, and a few pairs where the C or CT run built much more than the R run make it rise in some rows. Intervals are 95% bootstrap intervals that resample whole tasks; the two percent changes have none.
Figure C1: E10 for every arm and model. (a) Commit0: paired change in E10 from R at each processed-token budget, with libraries as pairs and pointwise 95% t bands. Points require at least 15 paired libraries. R and CT use seeds 1–5; the other arms use seeds 1–3. (b) SlopCodeBench: E10 by arm at each checkpoint, averaged first within problems and then across problems, with a pointwise 95% t band. We use included cells from the main runs, with two seeds. Points require at least 5 problems at the checkpoint. The second row of tick labels gives the number reaching it under any arm for that model.
Figure C2: E6 , the share of complexity mass in functions above the report’s limit of 6, for every arm and model, drawn as Figure C1 draws E10 . (a) Commit0: paired change in E6 from R at each processed-token budget, with libraries as pairs and pointwise 95% t bands. Points require at least 15 paired libraries. R and CT use seeds 1–5; the other arms use seeds 1–3. (b) SlopCodeBench: E6 at each checkpoint by arm, built as in Figure C1 (b).
Software development with coding agents often begins with a rough request that users refine after trying the implementation. Final correctness alone does not capture the repeated guidance this process requires. We introduce FeatureLoop, a benchmark of 61 human-reviewed tasks from 34 repositories across five primary programming languages and seven software forms. Each task follows one development goal in a persistent workspace through evolution feedback, which adds or clarifies requirements, and completion feedback, which reports failures of stated requirements. Cumulative executable checks track new behavior, repair, and preservation of earlier functionality. We evaluate 12 frontier models and find that the strongest completes 72.1% of tasks with both feedback types. Across models, however, only 15.4% of observed failed stages recover within two corrective revisions. Further analysis reveals incomplete repairs, regressions, and model-specific differences in feedback use and token efficiency, identifying obstacles to reliable, low-intervention development.
Jia Feng, Alyssa Wu, Cuiyun Gao +3
Hunyuan Team, Tencent · The Chinese University of Hong Kong
Software development is iterative, yet agentic coding benchmarks hide design issues through their single-shot setup. Recent iterative benchmarks attempt to remedy this but heavily constrain an agent's design decision space, making it impossible to faithfully measure how their decisions shape future extensions. We introduce SlopCodeBench, a benchmark of 36 problems and 196 checkpoints where agents repeatedly extend their own solutions. Unlike prior iterative benchmarks, our evolving specifications demand architectural decisions but leave internal structure to the agent. We measure two forms of degradation: structural erosion (concentrated complexity) and verbosity (redundant code). Evaluating 15 coding agents across open and closed models, we find that no agent fully solves any problem end-to-end, and the best agent passes 14.8% of checkpoints. Quality degrades across checkpoints, with structural erosion rising in 77% of trajectories and verbosity in 75.5%. Compared to 473 open-source Python repositories, agent code is 2.3x more verbose and 2.0x more eroded, and the human repositories degrade less often and by smaller margins across their git histories. Explicit quality guidance reduces initial verbosity and erosion by up to a third, without affecting degradation rates. SlopCodeBench provides the first measurement of code degradation under iterative extension, revealing that agents pass checkpoints while producing code that erodes and bloats with each turn.
Gabriel Orlanski, Devjeet Roy, Alexander Yun +7
University of Wisconsin–Madison · Washington State University · MIT
Coding agent infrastructure is shifting from harness engineering toward loop engineering as coding agents are deployed for sustained long-horizon software development. Existing benchmarks often center on localized tasks or end-state outcomes, offering limited insight into sustained execution. We introduce LOOPSBENCH, a long-horizon benchmark for loop engineering in coding agent evaluation. Each task is a dependency DAG over separately testable development units with source-evidenced prerequisite edges. LOOPSBENCH comprises 112 tasks from authentic sources spanning 8 programming languages and 9 domains. Its flow-aware runtime releases tests along the ready frontier and retains completed nodes as regression obligations. We evaluate frontier coding agents paired with widely used loop implementations. The strongest configuration, Opus-4.7 with Claude Code and outer continuation, resolves 25.00% of tasks. Recorded plans recover only part of the source-recovered prerequisite DAG, and regression events remain visible across the evaluated loop profiles. We open source the benchmark data and code, including all tasks, more than 5,300 development units, and executable tests, at microsoft/Loopsbench.
Han Li, Zhemin Fang, Rili Feng +8
Microsoft · Nanjing University · Shanghai Jiao Tong University +1