cs.SENov 16, 2026

Revisiting the Survival of Agent-Attributed Code: How Measurement Choices Decide the Verdict

Authors: Musfiqur Rahman, Mahi Begoug, Emad Shihab

Organizations: Concordia University, Montréal, Canada

Abstract

Studies of how long code from AI coding agents survives after it is merged reach conflicting verdicts: agent code is reported to be retained longer than human code, about as long, or modified sooner. Each study makes defensible measurement choices, but how much these choices decide the verdict is unknown. We revisit the survival of agent-attributed code in our earlier study: 854,139 lines that 2,802 merged pull requests (PRs) added to 215 repositories of the AIDev dataset. We vary nine measurement choices in five research questions and build a specification curve of 88 survival analyses, tested under three variance estimators, several observation windows, and permuted labels. For code from human-supervised agents, mostly GitHub Copilot, the choices change the size of the effect but not its direction. For code from the fully-autonomous agent Devin, they decide the verdict. Across repositories, 15 of 22 specifications report it modified sooner when lines are treated as independent, but only 3 when standard errors are clustered by repository, no more than permuted labels produce. Within repositories, no specification detects a difference. LLM intent classifiers agree with human labels about as well as two humans agree with each other, whereas keyword rules do not (kappa = 0.69 to 0.72 against 0.38), and the two reach different verdicts on why agent code is modified. Humans make most modifications to agent code under every unit and window. Re-run on our data, five published designs each change their result with a single choice. Whether agent code survives longer than human code is therefore decided as much by the measurement as by the code. We derive an operationalization checklist for post-merge studies.

Figures & tables

Explore similar work

CardsList
  1. Will It Survive? Deciphering the Fate of AI Agent-Authored Code in Open Source

    Jan 23, 2026Musfiqur Rahman, Mahi Begoug, Emad ShihabAI Coding AgentsOpen-Source Software Development

  2. Agent or Human? Detecting Commits by AI Coding Agents in the Absence of Authorship Signatures

    Oct 26, 2026Kosei Horikawa, Hao Li, Yutaro Kashiwa +5Authorship AttributionAI Coding Agents

  3. Beyond Simpson's Paradox: A Cascade of Confounders in AI Agent Pull-Request Co-Authorship

    Jun 21, 2026Haoran Yu, Xiaochong Jiang, Lifei Liu +3Software EngineeringAI Coding Agents