Coding assistants built on Large Language Models (LLMs) have become a staple of modern software development, increasingly trusted to execute workflows and modernize code end-to-end. Much of this trust rests on passing tests, which are often written by the coding agents themselves. In this work, we examine the ability of these agents to both write reliable tests and translate them between programming languages. We benchmark three popular coding agents, each evaluated with a native frontier model and high reasoning effort. Verifying across a broad suite of metrics, we find that coding agents far exceed human written suites on traditional measures of coverage. This advantage, however, is uneven: agent test quality degrades sharply with project size, and much of the advantage lies in a greater number of written tests. We also show that agentic test translation---while seemingly strong in aggregate---remains inconsistent and unreliable.
Figures & tables
Figure 1: Experimental Design Overview . (Top): We use Cursor, Claude Code, and Codex to generate unit tests for 240 projects across three languages, measuring unit testing capabilities. (Bottom) Additionally, we assess test translation from C++ to Rust, evaluating the model’s ability to translate existing C++ test suites.
Figure 2: Coding agents outperform human-written test suites on average, but the lead narrows on larger projects. (a) Mutation score by suite author; all agents exceed human baseline. (b) Mutant kill probability by language. Measured suites perform best in Python and worst in C++ across the board. (c) Mutation score by project-size. Agent scores decay faster than human scores; on large projects only Claude is far ahead. Error bands are 95% CIs.
Language
N
Small
Medium
Large
LOC med. [IQR]
Files med.
Python
80
30
30
20
1,656 [ 738 , 5,111 ]
8
C++
80
30
30
20
1,408 [ 615 , 5,122 ]
5
Rust
80
30
30
20
2,526 [ 734 , 5,012 ]
8.5
Total
240
90
90
60
1,688 [ 700 , 5,015 ]
8
Table 1: Unit Test generation corpus: 240 frozen, anonymized repositories, stratified by language and size.
Figure 3: Agents achieve similar test quality at very different costs. (a) Production lines covered per test written: Codex is the most economical author; Claude reaches comparable coverage with roughly five times as many tests. (b) Vacuous-test rate by language, rising from Python to Rust; Claude and Cursor fare worst throughout. (c) Spend per mutant killed, relative to Codex: Claude uses far more tokens per mutant killed than other suites. Error bars are 95% CIs.
Metric
C++ human
Claude Opus 5
Codex gpt-5.6
Cursor grok-4.6
Mutation score
0.559 ± 0.033
0.514 ± 0.033
0.489 ± 0.035
0.513 ± 0.033
Edge case coverage
0.340 ± 0.035
0.297 ± 0.031
0.287 ± 0.032
0.289 ± 0.031
Line coverage
0.747 ± 0.034
0.651 ± 0.032
0.612 ± 0.036
0.638 ± 0.034
Table 2: Aggregate metrics for human source test suites and transpiled Rust across all three coding agents. Transpiled test suites lag behind C++ source suites in every major indicator, in particular in line coverage. Values are mean ± sample standard deviation across three runs.
Figure 4: Edge case coverage scores for human source test suites, transpiled Rust test suites, and test suites independently generated from transpiled Rust source.
Figure 5: Mutation scores across human, transpiled, and independently generated test suites. Transpilation performance is upper bounded by independently generated performance, and transpiled test suites lag meaningfully behind their human sources on several projects.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Mean by author
Metric
Claude
Cursor
Codex
Human
Agent mean − Human [95% CI]
Mutation score
0.818
0.776
0.724
0.625
0.147 [0.120, 0.174]
Line coverage
0.926
0.913
0.873
0.762
0.142 [0.116, 0.168]
Coverage at pass
0.926
0.913
0.873
0.762
0.142 [0.116, 0.168]
Mutants killed per test
0.33
0.48
1.59
8.40
− 7.63 [ − 10.01, − 5.25]
Assertion density
0.240
0.289
0.280
0.170
0.099 [0.082, 0.116]
Appendix
Table 3: Overall projects: mean of each test-suite author on every metric, and the paired gap between the agents and the human baseline. “Agent mean” averages Claude, Cursor, and Codex on the same project.
Mean by author
Metric
Claude
Cursor
Codex
Human
Agent mean − Human [95% CI]
Mutation score
0.891
0.841
0.791
0.706
0.134 [0.082, 0.187]
Line coverage
0.981
0.963
0.931
0.844
0.115 [0.072, 0.158]
Coverage at pass
0.981
0.963
0.931
0.844
0.115 [0.072, 0.158]
Mutants killed per test
0.23
0.39
1.11
1.41
− 0.84 [ − 1.42, − 0.25]
Assertion density
0.261
0.318
0.279
0.199
0.087 [0.060, 0.115]
Appendix
Table 4: Python projects: mean of each test-suite author on every metric, and the paired gap between the agents and the human baseline. “Agent mean” averages Claude, Cursor, and Codex on the same project.
Mean by author
Metric
Claude
Cursor
Codex
Human
Agent mean − Human [95% CI]
Mutation score
0.746
0.682
0.651
0.537
0.156 [0.106, 0.207]
Line coverage
0.894
0.869
0.847
0.742
0.129 [0.077, 0.180]
Coverage at pass
0.894
0.869
0.847
0.742
0.129 [0.077, 0.180]
Mutants killed per test
0.38
0.53
1.96
21.28
− 20.56 [ − 26.87, − 14.25]
Assertion density
0.257
0.311
0.314
0.145
0.149 [0.117, 0.182]
Appendix
Table 5: C++ projects: mean of each test-suite author on every metric, and the paired gap between the agents and the human baseline. “Agent mean” averages Claude, Cursor, and Codex on the same project.
Mean by author
Metric
Claude
Cursor
Codex
Human
Agent mean − Human [95% CI]
Mutation score
0.816
0.804
0.730
0.632
0.151 [0.114, 0.188]
Line coverage
0.904
0.906
0.841
0.701
0.182 [0.141, 0.224]
Coverage at pass
0.904
0.906
0.841
0.701
0.182 [0.141, 0.224]
Mutants killed per test
0.39
0.51
1.71
2.51
− 1.64 [ − 2.36, − 0.92]
Assertion density
0.200
0.239
0.248
0.168
0.061 [0.036, 0.087]
Appendix
Table 6: Rust projects: mean of each test-suite author on every metric, and the paired gap between the agents and the human baseline. “Agent mean” averages Claude, Cursor, and Codex on the same project.
Mean by author
Metric
Claude
Cursor
Codex
Human
Agent mean − Human [95% CI]
Mutation score
0.863
0.852
0.837
0.674
0.176 [0.134, 0.218]
Line coverage
0.930
0.927
0.935
0.808
0.123 [0.088, 0.158]
Coverage at pass
0.930
0.927
0.935
0.808
0.123 [0.088, 0.158]
Mutants killed per test
0.36
0.44
1.66
11.28
− 10.56 [ − 14.96, − 6.17]
Assertion density
0.234
0.277
0.253
0.187
0.068 [0.038, 0.097]
Appendix
Table 7: Small projects: mean of each test-suite author on every metric, and the paired gap between the agents and the human baseline. “Agent mean” averages Claude, Cursor, and Codex on the same project.
Mean by author
Metric
Claude
Cursor
Codex
Human
Agent mean − Human [95% CI]
Mutation score
0.817
0.786
0.731
0.616
0.162 [0.116, 0.208]
Line coverage
0.937
0.923
0.890
0.738
0.179 [0.132, 0.225]
Coverage at pass
0.937
0.923
0.890
0.738
0.179 [0.132, 0.225]
Mutants killed per test
0.36
0.54
1.72
8.83
− 7.95 [ − 12.22, − 3.68]
Assertion density
0.243
0.303
0.284
0.165
0.112 [0.083, 0.141]
Appendix
Table 8: Medium projects: mean of each test-suite author on every metric, and the paired gap between the agents and the human baseline. “Agent mean” averages Claude, Cursor, and Codex on the same project.
Mean by author
Metric
Claude
Cursor
Codex
Human
Agent mean − Human [95% CI]
Mutation score
0.752
0.647
0.546
0.566
0.083 [0.029, 0.137]
Line coverage
0.905
0.875
0.755
0.730
0.115 [0.057, 0.172]
Coverage at pass
0.905
0.875
0.755
0.730
0.115 [0.057, 0.172]
Mutants killed per test
0.24
0.44
1.29
3.43
− 2.78 [ − 5.31, − 0.24]
Assertion density
0.243
0.286
0.317
0.154
0.128 [0.100, 0.157]
Appendix
Table 9: Large projects: mean of each test-suite author on every metric, and the paired gap between the agents and the human baseline. “Agent mean” averages Claude, Cursor, and Codex on the same project.
Event
Other recorded source reads (analyst summary)
item_3
Further public-header and dispatcher reads; false_positives.h ; project, source, and test CMake files.
Table 10: Additional source inspection in the minisketch trajectory, summarized from the recorded commands.
Event
Failed
Passed
Exit code
item_22
16
77
1
item_28
3
90
1
item_30
1
92
1
item_34
0
93
0
Appendix
Table 11: Recorded test-execution outcomes for geometer, repetition C.
Recorded coverage signal
Trajectories
Of all 675
No measurement attempt
376
55.7%
Measurement attempted
211
31.3%
Successful measurement documented
99
14.7%
Appendix
Table 12: Selected coverage observations, including inherited instrumentation. Rows are not an exhaustive partition; successful measurement is a subset of attempted measurement. No weak-outcome classification is used.
Symbol
Meaning
U
Production lines we measure.
Cover(P)
Those lines that passing tests execute.
K , V
Killed mutants and viable mutants.
R
Viable mutants the suite actually reaches.
n , nloc
Tests written, and test lines written.
Tt
Lines test t covers, after removing shared setup.
Appendix
Table 13: Notation.
Metric
Formula
Line coverage
↑
∣Cover(P)∣/∣U∣
Mutation score
↑
∣K∣/∣V∣
Kill rate (reached)
↑
∣K∣/∣R∣
Mutants not reached
↓
1−∣R∣/∣V∣
Tests written
–
n
Test lines
–
nloc
Appendix
Table 14: Reported metrics. ↑ higher is better; ↓ lower is better.
Metric
Standard prompt
Failure-aware prompt
Difference
Line coverage
92.21%
96.43%
+4.22 pp
Mutation score
69.98%
73.61%
+3.63 pp
Redundancy
18.01%
20.80%
+2.79 pp
Vacuous-test rate
3.86%
3.62%
−0.24 pp
Appendix
Table 15: Test-quality metrics on the matched 41-project cohort. The standard-prompt value is the mean of three runs; the failure-aware prompt uses one run.
Project
Repository
Commit
License
Citation
C++ projects
AHRS
wgrand/AHRS
a9f989d07bf1acc75774f5e595c2652eed9385f9
MIT
William Grand. AHRS. https://github.com/wgrand/AHRS (commit a9f989d07bf1acc75774f5e595c2652eed9385f9).
Large Language Model (LLM) code agents increasingly resolve repository-level issues by iteratively editing code, invoking tools, and validating candidate patches. In these workflows, agents often write tests on the fly, but the value of this behavior remains unclear. This raises a central question: how does agent-written testing relate to task outcomes and resource use during repository-level issue resolution? To better understand the role of agent-written tests, we analyze 5,000 recorded agent runs produced by six strong LLMs on SWE-bench Verified using the lightweight mini-SWE-agent scaffold. Our results show that test writing is common for most models. GPT-5.2 is an exception: it rarely writes new test files while achieving a resolution rate close to the top-ranked model. Test-writing frequency varies widely across models. Within a given model, it differs less between resolved and unresolved tasks. When tests are written, their files contain substantially more statements that print runtime values than assertions that check conditions. We then compare baseline prompts with revised prompts that encourage or discourage writing new test files for four models. Agents start or stop writing test files as requested on 36.8%-93.2% of tasks. Resolved-task totals differ by 0-13 out of 500 per model, with no statistically significant difference, while API calls or token use differ substantially for three models. Similar resolution totals can therefore hide substantial differences in how agents test and how many resources they use.
Zhi Chen, Zhensu Sun, Yuling Shi +4
Singapore Management University, Singapore, Singapore · Shanghai Jiao Tong University, Shanghai, China · ByteDance, Beijing, China
Large language model (LLM) agents are rapidly reshaping software engineering, accompanied by an explosion of new code benchmarks. Yet nearly all existing benchmarks still rely on the same decades-old criterion: a solution is correct if it passes a fixed set of unit tests. Such tests are often insufficient: they check only part of what the task requires, so agents can reward hack them or silently miss required behavior while still passing every test. As a result, higher benchmark scores may partly reflect better adaptation to the evaluator rather than better problem solving. Existing works focus on static test augmentation: they strengthen each task's tests once, before any trial is seen, and thus overlook how real trials actually fail. We introduce TestJack, a scalable framework for evaluating patches beyond fixed tests. For each trial, TestJack generates tests targeting prompt requirements the patch may violate, retains only tests passed by the ground-truth patch, and re-examines any trial failures. Each confirmed failure is thus supported by a replayable test. To reduce evaluation cost, we also introduce a lightweight variant which audits a random sample of trials in depth and reuses the resulting tests across all trials for the same task. Across 6 frontier model backends and 5 benchmarks such as DeepSWE and SWE Marathon, we find that about 34.4% of the model trials currently judged correct violate the task requirements, lowering the overall resolution rate from 50.6% to 33.2%. Our results reveal a fundamental limitation of current coding-agent evaluation: as LLMs become better at optimizing against fixed evaluators, those evaluators themselves must become more adaptive.
Shuangjie Yao, Hao Wang, Koushik Sen +3
Columbia University · University of California, Berkeley · George Mason University
Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue. Agent-generated tests can encode incomplete or incorrect behavioral targets; when the same trajectory writes both the patch and the test, their errors can agree and create false confidence. We introduce ExecCritic, combining a test--verify--revise scaffold with a role-specific reinforcement learning recipe for training agents within it. The scaffold separates test construction from source-code repair: a Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code from their execution feedback without changing the tests. Both roles use Qwen-3.5-35B-A3B as the backbone and are trained separately. In Learn to Test, the Test agent learns to produce behaviorally valid tests that distinguish correct from incorrect patches. In Test to Improve, the Repair agent learns both direct task resolution and feedback-guided revision. On SWE-bench Verified, test quality determines whether feedback helps: holding the base Repair agent fixed, tests from the base Test agent reduce resolved rate from a no-test baseline of 61.2% to 57.3%, whereas tests from GPT-5.6-sol raise it to 65.3%. Role-specific post-training raises the Qwen Test agent's Base-to-Gold success from 22.2% to 62.2%; composing the two post-trained Qwen agents reaches 72.6%, an 11.4-point gain over the original no-test baseline without stronger-model or Oracle feedback at evaluation time. Code is publicly available at https://github.com/MSR-Orchard/execcritic.
Leitian Tao, Baolin Peng, Haorui Wang +7
University of Wisconsin–Madison · Microsoft Research · Georgia Tech