Metrics-in-the-Loop: Complexity Feedback Reshapes Agent-Written Code
Organizations: Nera Software · Mila – Quebec Artificial Intelligence Institute · Cohere Labs · ELLIS Institute Tübingen
Abstract
Coding agents now write whole codebases in long autonomous runs, guided mainly by whether their tests pass. Yet passing tests alone does not reveal accumulating complexity or gaps in test coverage, which then persist as agents extend their own code. We introduce metrics-in-the-loop, an inference-time intervention that gives agents in-flight feedback: cheap, automatically computed code metrics injected into the agent's context during the trajectory. After each code-changing agent turn, we insert short feedback messages naming changed functions whose cyclomatic complexity exceeds a limit (complexity feedback) or changed lines that are skipped by tests (coverage feedback), giving the agent the opportunity to refactor code or strengthen tests before continuing. We run extensive experiments and ablations comparing several interventions across three long-horizon coding benchmarks and four strong models. We find that complexity feedback significantly reduces erosion, a measure of complexity concentration of code, without affecting held-out correctness of the code. Broadly, our experiments investigate a practical form of agent "re-railing", where timely, actionable measurements redirect agents before weaknesses become the foundation they keep building on. As agents scale to codebases with millions of lines, such interventions become critical to sustaining effective development and long-term code maintainability.
Figures & tables
| change in | correctness | |||
| Model | under R | C–R | CT–R | CT–R |
| Commit0, 25 libraries | ||||
| DeepSeek | 0.251 | −0.123 [−0.185, −0.061] | −0.070 [−0.110, −0.030] | +4.4 [−1.2, +10.0] |
| GLM | 0.231 | −0.120 [−0.177, −0.064] | −0.096 [−0.145, −0.046] | +4.7 [−1.6, +11.0] |
| Haiku | 0.122 | −0.070 [−0.108, −0.031] | −0.054 [−0.082, −0.026] | −2.3 [−10.2, +5.6] |
| Luna | 0.311 | −0.266 [−0.346, −0.186] | −0.261 [−0.339, −0.182] | +3.3 [−4.8, +11.5] |
| Check | Threat it guards against | What it changed here |
|---|---|---|
| Match instructions and delivery in the reminder arm | Instructions or delivery frequency affect behavior | For Haiku on SlopCodeBench, the reminder alone reduces by ; complexity feedback reduces it by a further |
| Outcomes withheld from the agent | Evaluation rewards the optimized metric ( Manheim and Garrabrant, 2018 ) | On SlopCodeBench, the benchmark’s erosion score, verbosity and hidden-suite coverage mostly support the result; on Commit0, every model’s C R coverage interval includes zero |
| Audit the changes behind the score | Function splitting lowers without simplifying code | On Commit0, 3 to 46% of each erosion difference remains after inlining in both arms; DeepSeek’s and GLM’s total branching changes little |
| Specify the scale; report models before pooling | A pooled estimate obscures model differences | Absolute effects on SlopCodeBench differ fifteenfold ( 100%), yet every model removes nearly all erosion present under the reminder arm |
| Identify independent units | Ignored task corre- lation gives too-narrow intervals ( Miller, 2024 ) | For SWE-EVO, transition-level intervals exclude zero for seven of eight model estimates across C R (35 transitions) and CT R (29); project clustering reduces this to four |
| Use operating-system isolation for held-out material | Agents access hidden tests or finished code | On Commit0, agents routinely bypassed a filter on path words, before proper exclusion measures were taken; on SlopCodeBench and SWE-EVO, operating-system isolation (bubblewrap in Dockers) blocked every attempt |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Arm | Message after a turn that changes code |
|---|---|
| B0 | none |
| R | general feedback instructions only (“placebo”) |
| C | complexity table and general feedback instructions |
| T | coverage table and general feedback instructions |
| CT | both tables, shown one after the other, and general feedback instructions |
| Commit0 | SlopCodeBench | SWE-EVO | |
|---|---|---|---|
| Task | reconstruct a Python library from its specification | build a program over 3–8 checkpoints, each a specification that extends the previous one | update a published project to its next release using that release’s problem statement |
| Tasks | 25 libraries | 35 problems | 35 transitions in 6 projects |
| Seeds | 3; 5 for R and CT | 2 | 2 |
| Cells launched / analyzed | 1,900 / 1,850 | 1,400 / 1,318 | 1,400 / 1,400 |
| Budget | 2M processed tokens per cell | 1M processed tokens per checkpoint | 2M processed tokens and 90 minutes per cell |
| Context | one conversation | fresh at each checkpoint; workspace, own tests and notes kept; every checkpoint runs even if the previous one failed | one conversation |
| Model | R | C–R | CT–R | T–R | R–B0 | |
| Commit0, 25 libraries | ||||||
| DeepSeek | 0.251 (25) | −0.123 [−0.185, −0.061] | −0.070 [−0.110, −0.030] | +0.007 [−0.009, +0.024] | −0.002 [−0.019, +0.015] | 25 |
| GLM | 0.231 (24) | −0.120 [−0.177, −0.064] | −0.096 [−0.145, −0.046] | +0.015 [−0.020, +0.049] | −0.021 [−0.056, +0.014] | 24 |
| Haiku | 0.122 (25) | −0.070 [−0.108, −0.031] | −0.054 [−0.082, −0.026] | +0.021 [+0.002, +0.040] | −0.078 [−0.121, −0.035] | 25 |
| Luna | 0.311 (25) | −0.266 [−0.346, −0.186] | −0.261 [−0.339, −0.182] | 0.000 [−0.038, +0.038] | −0.036 [−0.056, −0.015] | 25 |
| Pool (post hoc) | −0.112 [−0.138, −0.087] | −0.078 [−0.097, −0.059] | +0.012 [+0.001, +0.023] | −0.020 [−0.032, −0.009] | ||
| Model | R | C–R | CT–R | T–R | R–B0 | |
| Commit0, seeds 1–3 (planned) | ||||||
| DeepSeek | 68.9 (25) | +2.7 [−1.8, +7.3] | +4.4 [−1.2, +10.0] | +0.6 [−5.2, +6.3] | +5.9 [−1.9, +13.7] | 25 |
| GLM | 35.9 (25) | +1.4 [−6.3, +9.1] | +4.7 [−1.6, +11.0] | +1.0 [−7.1, +9.0] | −0.8 [−8.2, +6.6] | 25 |
| Haiku | 58.1 (25) | −2.7 [−9.9, +4.6] | −2.3 [−10.2, +5.6] | −1.5 [−7.0, +3.9] | −1.5 [−6.8, +3.9] | 25 |
| Luna | 51.0 (25) | −0.4 [−8.8, +8.0] | +3.3 [−4.8, +11.5] | −1.6 [−7.8, +4.6] | −0.5 [−6.5, +5.5] | 25 |
| Pool (post hoc) | +1.0 [−2.0, +4.0] | +3.1 [−0.1, +6.3] | −0.6 [−3.5, +2.4] | +0.2 [−2.9, +3.2] | ||
| Model | R | C–R | CT–R | T–R | R–B0 | |
| Fix rate (primary) | ||||||
| DeepSeek | 45.5 (35) | +5.9 [−7.6, +19.4] | +5.1 [−2.3, +12.5] | −3.4 [−19.0, +12.3] | −0.6 [−12.4, +11.2] | 29 |
| GLM | 31.2 (35) | +1.7 [−14.3, +17.6] | +1.5 [−31.2, +34.1] | +3.7 [−16.6, +24.1] | +1.0 [−11.7, +13.7] | 29 |
| Haiku | 31.6 (35) | −0.9 [−9.1, +7.3] | +0.4 [−8.1, +9.0] | −0.8 [−11.4, +9.9] | −2.3 [−8.6, +4.1] | 29 |
| Luna | 35.7 (35) | +2.8 [−18.8, +24.5] | +2.2 [−5.4, +9.7] | +4.5 [−20.0, +28.9] | −4.9 [−25.6, +15.9] | 29 |
| Pool (post hoc) | +1.2 [−2.6, +4.9] | +2.7 [−0.2, +5.6] | −0.2 [−5.3, +4.8] | −1.6 [−4.8, +1.5] | ||
| complexity: edited / under limit | coverage rose | ||||||
|---|---|---|---|---|---|---|---|
| Benchmark | Model | R | C | CT | R | T | CT |
| Commit0 | DeepSeek | 20% / 1% | 54% / 46% | 36% / 23% | 64% | 57% | 57% |
| Commit0 | GLM | 30% / 3% | 60% / 58% | 46% / 35% | 40% | 42% | 34% |
| Commit0 | Haiku | 31% / 9% | 83% / 86% | 67% / 66% | 47% | 51% | 49% |
| Commit0 | Luna | 49% / 3% | 88% / 94% | 85% / 90% | 59% | 86% | 81% |
| SCB | DeepSeek | 38% / 5% | 94% / 94% | 84% / 82% | 17% | 95% | 92% |
| change in | change in branches and lines | ||||||
| Model | pairs | as measured | after inlining | branches | lines | per 1k lines | |
| Commit0, seeds 1–3 | |||||||
| DeepSeek | C–R | 62 | −0.134 [−0.200, −0.073] | −0.023 [−0.043, −0.004] | +1% | +3% | −7.8 [−14.0, −1.9] |
| CT–R | 61 | −0.066 [−0.103, −0.035] | −0.016 [−0.037, +0.004] | +1% | +2% | −5.6 [−10.3, −0.8] | |
| GLM | C–R | 56 | −0.124 [−0.190, −0.060] | −0.029 [−0.085, +0.020] | 0% | +4% | −4.6 [−13.2, +5.6] |
| CT–R | 55 | −0.096 [−0.166, −0.025] | −0.003 [−0.047, +0.051] | −4% | +1% | −5.3 [−17.3, +7.9] | |
Software Engineering
Software engineering studies how to design, build, test, deploy, and maintain reliable software systems and the processes behind them.
See all 296 papers