Can Coding Agents Write and Transpile Unit Tests
Organizations: Code Metal
Abstract
Coding assistants built on Large Language Models (LLMs) have become a staple of modern software development, increasingly trusted to execute workflows and modernize code end-to-end. Much of this trust rests on passing tests, which are often written by the coding agents themselves. In this work, we examine the ability of these agents to both write reliable tests and translate them between programming languages. We benchmark three popular coding agents, each evaluated with a native frontier model and high reasoning effort. Verifying across a broad suite of metrics, we find that coding agents far exceed human written suites on traditional measures of coverage. This advantage, however, is uneven: agent test quality degrades sharply with project size, and much of the advantage lies in a greater number of written tests. We also show that agentic test translation---while seemingly strong in aggregate---remains inconsistent and unreliable.
Figures & tables
| Language | Small | Medium | Large | LOC med. [IQR] | Files med. | |
|---|---|---|---|---|---|---|
| Python | 80 | 30 | 30 | 20 | [ , ] | 8 |
| C++ | 80 | 30 | 30 | 20 | [ , ] | 5 |
| Rust | 80 | 30 | 30 | 20 | [ , ] | 8.5 |
| Total | 240 | 90 | 90 | 60 | [ , ] | 8 |
| Metric | C++ human | Claude Opus 5 | Codex gpt-5.6 | Cursor grok-4.6 |
|---|---|---|---|---|
| Mutation score | 0.559 0.033 | 0.514 0.033 | 0.489 0.035 | 0.513 0.033 |
| Edge case coverage | 0.340 0.035 | 0.297 0.031 | 0.287 0.032 | 0.289 0.031 |
| Line coverage | 0.747 0.034 | 0.651 0.032 | 0.612 0.036 | 0.638 0.034 |
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
| Mean by author | |||||
| Metric | Claude | Cursor | Codex | Human | Agent mean Human [95% CI] |
| Mutation score | 0.818 | 0.776 | 0.724 | 0.625 | 0.147 [0.120, 0.174] |
| Line coverage | 0.926 | 0.913 | 0.873 | 0.762 | 0.142 [0.116, 0.168] |
| Coverage at pass | 0.926 | 0.913 | 0.873 | 0.762 | 0.142 [0.116, 0.168] |
| Mutants killed per test | 0.33 | 0.48 | 1.59 | 8.40 | 7.63 [ 10.01, 5.25] |
| Assertion density | 0.240 | 0.289 | 0.280 | 0.170 | 0.099 [0.082, 0.116] |
| Mean by author | |||||
| Metric | Claude | Cursor | Codex | Human | Agent mean Human [95% CI] |
| Mutation score | 0.891 | 0.841 | 0.791 | 0.706 | 0.134 [0.082, 0.187] |
| Line coverage | 0.981 | 0.963 | 0.931 | 0.844 | 0.115 [0.072, 0.158] |
| Coverage at pass | 0.981 | 0.963 | 0.931 | 0.844 | 0.115 [0.072, 0.158] |
| Mutants killed per test | 0.23 | 0.39 | 1.11 | 1.41 | 0.84 [ 1.42, 0.25] |
| Assertion density | 0.261 | 0.318 | 0.279 | 0.199 | 0.087 [0.060, 0.115] |
| Mean by author | |||||
| Metric | Claude | Cursor | Codex | Human | Agent mean Human [95% CI] |
| Mutation score | 0.746 | 0.682 | 0.651 | 0.537 | 0.156 [0.106, 0.207] |
| Line coverage | 0.894 | 0.869 | 0.847 | 0.742 | 0.129 [0.077, 0.180] |
| Coverage at pass | 0.894 | 0.869 | 0.847 | 0.742 | 0.129 [0.077, 0.180] |
| Mutants killed per test | 0.38 | 0.53 | 1.96 | 21.28 | 20.56 [ 26.87, 14.25] |
| Assertion density | 0.257 | 0.311 | 0.314 | 0.145 | 0.149 [0.117, 0.182] |
| Mean by author | |||||
| Metric | Claude | Cursor | Codex | Human | Agent mean Human [95% CI] |
| Mutation score | 0.816 | 0.804 | 0.730 | 0.632 | 0.151 [0.114, 0.188] |
| Line coverage | 0.904 | 0.906 | 0.841 | 0.701 | 0.182 [0.141, 0.224] |
| Coverage at pass | 0.904 | 0.906 | 0.841 | 0.701 | 0.182 [0.141, 0.224] |
| Mutants killed per test | 0.39 | 0.51 | 1.71 | 2.51 | 1.64 [ 2.36, 0.92] |
| Assertion density | 0.200 | 0.239 | 0.248 | 0.168 | 0.061 [0.036, 0.087] |
| Mean by author | |||||
| Metric | Claude | Cursor | Codex | Human | Agent mean Human [95% CI] |
| Mutation score | 0.863 | 0.852 | 0.837 | 0.674 | 0.176 [0.134, 0.218] |
| Line coverage | 0.930 | 0.927 | 0.935 | 0.808 | 0.123 [0.088, 0.158] |
| Coverage at pass | 0.930 | 0.927 | 0.935 | 0.808 | 0.123 [0.088, 0.158] |
| Mutants killed per test | 0.36 | 0.44 | 1.66 | 11.28 | 10.56 [ 14.96, 6.17] |
| Assertion density | 0.234 | 0.277 | 0.253 | 0.187 | 0.068 [0.038, 0.097] |
| Mean by author | |||||
| Metric | Claude | Cursor | Codex | Human | Agent mean Human [95% CI] |
| Mutation score | 0.817 | 0.786 | 0.731 | 0.616 | 0.162 [0.116, 0.208] |
| Line coverage | 0.937 | 0.923 | 0.890 | 0.738 | 0.179 [0.132, 0.225] |
| Coverage at pass | 0.937 | 0.923 | 0.890 | 0.738 | 0.179 [0.132, 0.225] |
| Mutants killed per test | 0.36 | 0.54 | 1.72 | 8.83 | 7.95 [ 12.22, 3.68] |
| Assertion density | 0.243 | 0.303 | 0.284 | 0.165 | 0.112 [0.083, 0.141] |
| Mean by author | |||||
| Metric | Claude | Cursor | Codex | Human | Agent mean Human [95% CI] |
| Mutation score | 0.752 | 0.647 | 0.546 | 0.566 | 0.083 [0.029, 0.137] |
| Line coverage | 0.905 | 0.875 | 0.755 | 0.730 | 0.115 [0.057, 0.172] |
| Coverage at pass | 0.905 | 0.875 | 0.755 | 0.730 | 0.115 [0.057, 0.172] |
| Mutants killed per test | 0.24 | 0.44 | 1.29 | 3.43 | 2.78 [ 5.31, 0.24] |
| Assertion density | 0.243 | 0.286 | 0.317 | 0.154 | 0.128 [0.100, 0.157] |
| Event | Other recorded source reads (analyst summary) |
|---|---|
| item_3 | Further public-header and dispatcher reads; false_positives.h ; project, source, and test CMake files. |
| item_6 | Remaining shared sketch code; fielddefines.h , util.h , and int_utils.h . |
| item_7 | Public header read from HEAD . |
| Event | Failed | Passed | Exit code |
|---|---|---|---|
| item_22 | 16 | 77 | 1 |
| item_28 | 3 | 90 | 1 |
| item_30 | 1 | 92 | 1 |
| item_34 | 0 | 93 | 0 |
| Recorded coverage signal | Trajectories | Of all 675 |
|---|---|---|
| No measurement attempt | 376 | 55.7% |
| Measurement attempted | 211 | 31.3% |
| Successful measurement documented | 99 | 14.7% |
| Symbol | Meaning |
|---|---|
| Production lines we measure. | |
| Those lines that passing tests execute. | |
| , | Killed mutants and viable mutants. |
| Viable mutants the suite actually reaches. | |
| , | Tests written, and test lines written. |
| Lines test covers, after removing shared setup. |
| Metric | Formula | |
| Line coverage | ||
| Mutation score | ||
| Kill rate (reached) | ||
| Mutants not reached | ||
| Tests written | – | |
| Test lines | – |
| Metric | Standard prompt | Failure-aware prompt | Difference |
|---|---|---|---|
| Line coverage | 92.21% | 96.43% | pp |
| Mutation score | 69.98% | 73.61% | pp |
| Redundancy | 18.01% | 20.80% | pp |
| Vacuous-test rate | 3.86% | 3.62% | pp |
| Project | Repository | Commit | License | Citation |
|---|---|---|---|---|
| C++ projects | ||||
| AHRS | wgrand/AHRS | a9f989d07bf1acc75774f5e595c2652eed9385f9 | MIT | William Grand. AHRS. https://github.com/wgrand/AHRS (commit a9f989d07bf1acc75774f5e595c2652eed9385f9). |
| artificial-neural-network | camchambers/artificial-neural-network | f74dc79cda62ff7f049488cb3cf00c25fdf63e0e | MIT | Cam. artificial-neural-network. https://github.com/camchambers/artificial-neural-network (commit f74dc79cda62ff7f049488cb3cf00c25fdf63e0e). |
| base64 | tobiaslocker/base64 | 8d96a2a737ac1396304b1de289beb3a5ea0cb752 | MIT | Tobias Locker. base64. https://github.com/tobiaslocker/base64 (commit 8d96a2a737ac1396304b1de289beb3a5ea0cb752). |
| biscuit | lioncash/biscuit | 805c1756dd602e50bd4f57e772998ab3ad91df7f | MIT | Lioncash/Lioncache. biscuit. https://github.com/lioncash/biscuit (commit 805c1756dd602e50bd4f57e772998ab3ad91df7f). |
AI Coding Agents
AI coding agents use language models and software tools to plan, edit, test, and maintain code with varying levels of autonomy.
See all 353 papers