Revisiting the Survival of Agent-Attributed Code: How Measurement Choices Decide the Verdict
Organizations: Concordia University, Montréal, Canada
Abstract
Studies of how long code from AI coding agents survives after it is merged reach conflicting verdicts: agent code is reported to be retained longer than human code, about as long, or modified sooner. Each study makes defensible measurement choices, but how much these choices decide the verdict is unknown. We revisit the survival of agent-attributed code in our earlier study: 854,139 lines that 2,802 merged pull requests (PRs) added to 215 repositories of the AIDev dataset. We vary nine measurement choices in five research questions and build a specification curve of 88 survival analyses, tested under three variance estimators, several observation windows, and permuted labels. For code from human-supervised agents, mostly GitHub Copilot, the choices change the size of the effect but not its direction. For code from the fully-autonomous agent Devin, they decide the verdict. Across repositories, 15 of 22 specifications report it modified sooner when lines are treated as independent, but only 3 when standard errors are clustered by repository, no more than permuted labels produce. Within repositories, no specification detects a difference. LLM intent classifiers agree with human labels about as well as two humans agree with each other, whereas keyword rules do not (kappa = 0.69 to 0.72 against 0.38), and the two reach different verdicts on why agent code is modified. Humans make most modifications to agent code under every unit and window. Re-run on our data, five published designs each change their result with a single choice. Whether agent code survives longer than human code is therefore decided as much by the measurement as by the code. We derive an operationalization checklist for post-merge studies.
Figures & tables
| Choice | Popescu et al. ( Popescu et al., 2026 ) | Sawada et al. ( Sawada et al., 2026 ) | Xia and Miller ( Xia and Miller, 2026 ) | Kraishan ( Kraishan, 2026 ) | Takerngsaksiri et al. ( Takerngsaksiri et al., 2026 ) | This study |
|---|---|---|---|---|---|---|
| Data, window | Agent and human commits; 3 days, 1 week, 3 weeks | AIDev; first 6 months | 182 repositories; May 2025 to May 2026 | Agent and same-repository human PRs; 90 days | 218 repositories; 30 days | AIDev, 215 repositories; at least 5 months |
| Unit | Commit (all added lines), per agent | File | Line, agents pooled | PR, per agent | PR, agents pooled | Line, hunk, PR; two tiers |
| Tracking | Line content in the nearest later commit | File history | Diff replay, verified with blame | Most-changed file | File co-edited by a later PR | Line identity (reverse blame with -M -C ), compared with two other trackers |
| Modification | Line changed | Later commit to the file | Any diff touching the line | Follow-up commit to the file | Verified fix PR | Syntactic and AST-structural |
| Intent | None | Conventional Commits | Rules, LLM fallback | None | LLM and human fix verification | Keyword, 3 LLMs, combinations; 618 human labels |
| Comparison | Not stated (sample stratified by repository stars) | Same-repository sample, pooled | Within repositories | Same-repository baseline, pooled tests | Within repositories (Mantel-Haenszel) | Across and within repositories |
| Design | Lines | Mod. % A / H | Agents | Supervised | Devin |
|---|---|---|---|---|---|
| Within repositories | |||||
| Primary: no vendored, PR-weighted | 648,959 | 28.0 / 34.5 | 0.83 | 0.76 | 0.95 |
| With vendored, PR-weighted | 854,139 | 28.1 / 34.8 | 0.83 | 0.76 | 0.95 |
| No vendored, unweighted | 648,959 | 23.4 / 30.1 | 0.91 | 0.80 | 1.06 |
| With vendored, unweighted | 854,139 | 23.3 / 50.3 | 0.92 | 0.86 | 1.00 |
| Sampled, PR-weighted | 277,283 | 28.1 / 34.7 | 0.83 | 0.76 | 0.95 |
| Part of the design | Earlier study | This paper |
|---|---|---|
| Cohort | 2,802 PRs in 215 repositories | Reused |
| Line tracker | Reverse blame, next commit touching the file | Reused as the first step; new: identity tracking, line-number tracking, and their comparison (RQ1) |
| Vendored and generated code | Excluded from the primary analysis | Reused; new: 4,102 further generated lines removed, and the removal as an explicit step (RQ1) |
| Unit and weighting | Line, PR-weighted; hunk and file for comparison | Reused; new: line, PR-weighted line, and hunk compared across all other choices (RQ1) |
| Modification definition | Any change; substantial share of hunk or file | New: structural definition based on the abstract syntax tree (AST), with a manual check (RQ2) |
| Intent classifier | Keyword rules | New: three LLMs, three ways of combining them, 618 human labels, corrected shares (RQ4) |
| Tier | Agents | Collaboration Mode |
|---|---|---|
| Human-supervised | Copilot, Cursor, | Variably human-supervised; |
| Claude Code, Codex | degree of oversight unverifiable | |
| from PR-level metadata | ||
| Fully-autonomous | Devin | Whole tasks delegated to the |
| agent, which submits PRs as | ||
| an independent contributor |
| Line number | Identity | Earlier | ||||
|---|---|---|---|---|---|---|
| Group | Modified | HR | Modified | HR | Modified | HR |
| Human-supervised | 46.8% | 0.770 | 18.3% | 0.673 | 21.4% | 0.707 |
| Fully-autonomous (Devin) | 63.8% | 0.992 | 34.9% | 1.294 | 37.7% | 1.260 |
| Human baseline | 64.5% | 1.000 | 29.0% | 1.000 | 32.4% | 1.000 |
| Tier | Single-line | Small block | Large block |
|---|---|---|---|
| (1 line) | (2 to 10 lines) | (11+ lines) | |
| Human-supervised | 0.776 [0.55, 1.10] (10,657) | 0.893 [0.57, 1.39] (5,384) | 0.963 [0.64, 1.45] (1,867) |
| Fully-autonomous (Devin) | 1.385 [0.86, 2.22] (2,932) | 1.736 [1.11, 2.71] (1,595) | 1.440 [1.01, 2.05] (968) |
| Human baseline (N) | 21,051 | 11,646 | 3,934 |
| The pooled Devin hunk estimate (1.56) is also significant with repository-clustered errors. | |||
| Group | Syntactic | AST-structural |
|---|---|---|
| Human-supervised | 20.1% | 15.7% |
| Fully-autonomous (Devin) | 21.5% | 15.3% |
| Human baseline | 25.0% | 18.9% |
| Group | Content changed | Line deleted | File deleted | Whitespace-only a |
|---|---|---|---|---|
| Human-supervised | 63.7 | 20.8 | 15.5 | 18.9 |
| Fully-autonomous (Devin) | 60.5 | 10.3 | 29.2 | 21.0 |
| Human baseline | 59.3 | 12.9 | 27.8 | 18.4 |
| Repositories containing | Count (%) |
|---|---|
| Human-supervised code | 150 (69.8%) |
| Fully-autonomous (Devin) code | 31 (14.4%) |
| Human-baseline code | 193 (89.8%) |
| Both agent tiers | 10 (4.7%) |
| Only human-supervised | 140 (65.1%) |
| Only fully-autonomous | 21 (9.8%) |
| Tier | Comparison | Clustered by repository | Clustered by PR | Model-based |
|---|---|---|---|---|
| Human-supervised | Across | 13 / 9 / 0 (8 / 6 / 0) | 13 / 9 / 0 | 21 / 0 / 1 |
| Within | 11 / 10 / 1 (7 / 6 / 1) | 12 / 9 / 1 | 21 / 0 / 1 | |
| Fully-autonomous (Devin) | Across | 1 / 18 / 3 (0 / 13 / 1) | 0 / 13 / 9 | 5 / 2 / 15 |
| Within | 0 / 22 / 0 (0 / 14 / 0) | 1 / 20 / 1 | 11 / 4 / 7 |
| Agent | Lines | PRs | Repositories a | HR [95% CI] |
|---|---|---|---|---|
| GitHub Copilot | 67,398 | 489 | 53 | 0.653 [0.459, 0.954] |
| OpenAI Codex | 48,867 | 491 | 63 | 0.831 [0.468, 1.218] |
| Cursor | 13,340 | 44 | 16 | 0.379 [0.150, 3.695] |
| Claude Code | 48,182 | 31 | 13 | 0.934 [0.136, 1.950] |
| Devin | 87,634 | 412 | 31 | 1.059 [0.409, 2.076] |
| Approach | Human-supervised | Fully-autonomous | Human baseline | Human-supervised vs. human | Devin vs. human | |
|---|---|---|---|---|---|---|
| Keyword a | 0.38 | 36.2 / 12.8 / 30.6 / 10.3 | 30.6 / 15.6 / 36.0 / 13.5 | 34.2 / 13.7 / 28.7 / 11.2 | 0.10 (0.04) | 0.001 (0.11) |
| DeepSeek | 0.70 | 22.7 / 22.2 / 49.6 / 5.4 | 22.4 / 25.8 / 47.3 / 4.5 | 22.1 / 23.7 / 49.4 / 4.8 | 0.62 (0.02) | 0.67 (0.02) |
| gpt-oss | 0.69 | 21.2 / 22.4 / 51.5 / 4.9 | 22.2 / 24.0 / 48.6 / 5.2 | 20.9 / 23.2 / 51.8 / 4.1 | 0.59 (0.02) | 0.37 (0.03) |
| MiniMax | 0.71 | 24.8 / 21.6 / 48.2 / 5.4 | 24.5 / 22.7 / 48.5 / 4.4 | 24.3 / 22.6 / 48.8 / 4.2 | 0.34 (0.03) | 0.998 (0.003) |
| Majority vote b | 0.71 | 23.1 / 21.7 / 50.0 / 5.3 | 23.3 / 23.7 / 48.8 / 4.2 | 22.5 / 23.0 / 50.1 / 4.4 | 0.43 (0.02) | 0.93 (0.01) |
| Weighted vote | 0.71 | 23.3 / 21.8 / 49.6 / 5.4 | 23.3 / 23.7 / 48.6 / 4.4 | 22.7 / 23.0 / 49.9 / 4.4 | 0.40 (0.03) | 0.95 (0.01) |
| Approach | Human-supervised | Fully-autonomous | Human baseline |
|---|---|---|---|
| One confusion matrix | |||
| DeepSeek | 23.6 [19.4, 27.4] | 23.4 [18.4, 28.5] | 23.0 [19.6, 26.1] |
| gpt-oss | 25.6 [22.1, 29.0] | 26.8 [21.7, 31.8] | 25.2 [22.6, 28.4] |
| MiniMax | 23.7 [19.4, 27.5] | 23.6 [19.0, 28.7] | 23.5 [20.7, 26.4] |
| Majority vote | 24.0 [19.8, 27.6] | 24.4 [19.5, 29.3] | 23.6 [20.6, 26.7] |
| One confusion matrix per tier | |||
| Code written by | Commits | Human | Agent | Most frequent agent modifier |
|---|---|---|---|---|
| Human-supervised agents | 1,453 | 88.6% | 11.4% | Copilot, 151 of 166 |
| Fully-autonomous (Devin) | 617 | 85.7% | 14.3% | Devin, 85 of 88 |
| Human developers | 3,230 | 96.9% | 3.1% | Copilot and Devin, 40 each of 100 |
| Unit | Window | Human-supervised | Fully-autonomous (Devin) | Human baseline |
|---|---|---|---|---|
| Commit | 7 days | 14.5 (12.9) | 24.6 (24.6) | 2.1 |
| Commit | 30 days | 12.3 (11.3) | 22.5 (22.5) | 2.3 |
| Commit | Full | 11.4 (9.9) | 14.3 (13.8) | 3.1 |
| Line | 7 days | 6.3 (5.5) | 11.4 (11.4) | 0.3 |
| Line | 30 days | 8.1 (6.2) | 9.6 (9.6) | 1.1 |
| Line | Full | 9.8 (8.4) | 5.7 (5.6) | 1.2 |
| Window | PRs | Across, clustered | Across, model-based | Within, clustered |
|---|---|---|---|---|
| Full ( 5 months) | 412 | 1 / 18 / 3 (1.16) | 5 / 2 / 15 | 0 / 22 / 0 (0.91) |
| 8 months | 270 | 0 / 14 / 8 (1.28) | 0 / 0 / 22 | 0 / 22 / 0 (1.06) |
| 8 months, censored at 240 days | 270 | 0 / 18 / 4 (1.24) | 0 / 1 / 21 | 0 / 22 / 0 (1.03) |
| 10 months, censored at 300 days | 125 | 0 / 17 / 5 (1.43) | 0 / 1 / 21 | 0 / 22 / 0 (0.98) |
| Full, without pdfme #1059 | 411 | 0 / 19 / 3 (1.18) | 3 / 1 / 18 | 0 / 22 / 0 (0.91) |
| Full, without the 142 PRs, censored at 150 days | 270 | 0 / 17 / 5 (1.29) | 3 / 1 / 18 | 0 / 21 / 1 (1.09) |
| Study | Design as re-run | Reported | On our data | Single change that moves it |
|---|---|---|---|---|
| Xia and Miller ( Xia and Miller, 2026 ) | Agents pooled, line units, within-repository Cox | HR 1.11, not significant; corrective HR 1.49 | HR 0.92, : we agree | Corrective outcome as a cause-specific hazard, as in their study: HR 0.68 with keyword-first labels, 1.37 with LLM labels ( and ) |
| Sawada et al. ( Sawada et al., 2026 ) | File level, months 1 to 6, human share of later edits | Humans do about 83% of maintenance | Humans make 88.2%; Devin files 0.156 modifying commits per month, human 0.140, human-supervised 0.102 | Direction agrees; the share depends on the unit and window |
| Popescu et al. ( Popescu et al., 2026 ) | Per agent, survival at 3 days, 1 week, 3 weeks, pooled across repositories; at the line level, since our data do not identify each PR’s first commit | Human commits keep their added lines more often than every agent’s (small effects) | At 3 weeks: Cursor 77.2%, Claude Code 81.8%, Devin 88.6%, human 91.7%, Codex 93.2%, Copilot 96.1%: of the four agents we share with them (Jules is not in AIDev), we agree for Claude Code and Devin; Cursor, which they do not study, is also below | Within repositories, only Copilot differs, and it is retained longer (HR 0.65, ) |
| Kraishan ( Kraishan, 2026 ) | Per agent, PR churn on the most-changed file within 90 days, against same-repository humans | Claude Code lowest churn; Copilot lower; Codex and Devin at human level | Cliff’s ( Cliff, 1993 ) : Codex , Copilot , Devin , Claude Code , Cursor : we agree on Copilot | With a repository model, no agent differs ( ) |
| Takerngsaksiri et al. ( Takerngsaksiri et al., 2026 ) | PR units, 30 days, Mantel-Haenszel OR ( Mantel and Haenszel, 1959 ) over repositories with both groups; our outcome is any modification, not a verified fix | OR 1.62 for a follow-up fix; most fixes by the same agent | Mantel-Haenszel OR, agents pooled 0.86 ( ); Devin 0.77, human-supervised 0.92 | Not comparable (different outcome). The raw Devin gap (51.7% vs. 46.0% of PRs modified) reverses within the 31 shared repositories (crude, unstratified OR 0.86 for Devin) |
| Choice | Recommendation | Effect of the choice on our data |
|---|---|---|
| 1. Line tracking | Follow line identity rather than line location (for example, reverse blame with rename and copy detection along the first-parent chain), verify each modification in its diff, and report the estimate under a second tracker. | Recorded modifications about double under line-number tracking, and only 35.9% of its “modified” claims correct in a manual check, against 98.7% under identity tracking; Devin HR 1.29 with identity tracking, 0.99 with line-number tracking. |
| 2. Files, sampling, and weighting | Remove vendored and generated files. Do not cap or sample lines within a PR. Report line-weighted and PR-weighted estimates side by side, and the share of lines from the largest PRs. | HR 0.38 to 0.39 with vendored code, about 0.9 without; Devin HR 0.90 per line, 1.19 per PR, 1.56 per hunk, 1.29 with caps; one PR holds 41.7% of Devin lines. |
| 3. Modification definition | Report modification events by type (content change, line deleted, file deleted, whitespace only), with an AST-structural definition as a sensitivity check. | Devin HR 0.90 syntactic, 0.85 structural (across); human-supervised within 0.81 (not significant) and 0.72 (significant); 18.8% of content modifications are whitespace only. |
| 4. Intent classifier | Prefer LLM classifiers over keyword rules when the commits are too many to label by hand. Validate the classifier against a stratified human sample, check that its accuracy does not differ by tier, report with a CI, label by hand the commits on which the models disagree, correct shares for classification error, and check any human-label difference on the unlabeled commits. | 0.38 for keyword rules, 0.69 to 0.72 for LLMs with the diff, 0.68 to 0.74 with the message alone; unanimous LLM labels 86% correct, two-to-one majorities 54%; keyword rules alone find that Devin code differs. |
| 5. Comparison | Compare within repositories: stratify by repository, use cluster-bootstrap intervals, and report the number of contributing repositories and the minimum detectable effect. | Human-supervised HR 0.89 across, 0.81 within (0.72 structural); Devin HR 0.90 across, 1.06 within, and 0.62 pooled on the 31 shared repositories; within repositories, only hazard ratios beyond 0.32 and 3.13 are detectable for Devin. |
| 6. Standard errors | Cluster standard errors by repository in every model, also across repositories, and test a specification curve against permuted labels. | Human-supervised across: 21 of 22 “longer” with model-based errors, 13 clustered by repository. Devin across: 15 of 22 “sooner” with model-based errors, 9 clustered by PR, 3 clustered by repository; line-level CI [0.882, 0.910] becomes [0.425, 1.889]. |
| Claim | Applies to | Not claimed |
|---|---|---|
| Sensitivity of verdicts | Line-level post-merge designs in general; one dataset suffices to show that a choice can change a verdict. | That every dataset is equally sensitive to every choice. |
| Agent effects | Merged AIDev PRs in 215 repositories, merged before August 2025 and followed until December 2025, without vendored and generated files. | Effects of agents in general, or of fully-autonomous agents other than Devin. |
| No difference | The specification for which it is reported, with its interval. | That agent code and human code are retained equally long. |
| Intent | Relative differences between tiers. | Exact intent shares of any group. |
AI Coding Agents
AI coding agents use language models and software tools to plan, edit, test, and maintain code with varying levels of autonomy.
See all 354 papers