cs.SEApr 1, 2026
SaveInvestigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Time
Organizations: Delft University of Technology, Delft, The Netherlands · University of California, Davis, Davis, USA · GitHub, Denver, USA
Abstract
The rise of large language models for code has reshaped software development. Autonomous coding agents, able to create branches, open pull requests, and perform code reviews, now actively contribute to real-world projects. Their growing role offers a unique and timely opportunity to investigate AI-driven contributions and their effects on code quality, team dynamics, and software maintainability. In this work, we construct a novel dataset of approximately open-source pull requests, including associated commits, comments, reviews, issues, and file changes, collectively representing millions of lines of source code. We compare five popular coding agents, including OpenAI Codex, Claude Code, GitHub Copilot, Google Jules, and Devin, examining how their usage differs in various development aspects such as merge frequency, edited file types, and developer interaction signals, including comments and reviews. Furthermore, we emphasize that code authoring and review are only a small part of the larger software engineering process, as the resulting code must also be maintained and updated over time. Hence, we offer several longitudinal estimates of survival and churn rates for agent-generated versus human-authored code. Ultimately, our findings indicate an increasing agent activity in open-source projects, although their contributions are associated with more churn over time compared to human-authored code.
Figures & tables
| Coding Agent | Search Signal |
|---|---|
| OpenAI Codex | head:codex/ |
| GitHub Copilot | head:copilot/ |
| Google Jules | author:google-labs-jules[bot] |
| Devin | author:devin-ai-integration[bot] |
| Claude Code | (“Co-Authored-By: Claude” OR “Generated with Claude Code”) |
Table 1. Search signals used to filter PRs per coding agent. Search signals used to filter PRs per coding agent.
| Coding Agent | Search Signal |
|---|---|
| OpenAI Codex | head:codex/ |
| GitHub Copilot | head:copilot/ |
| Cursor | head:cursor/ |
| Cosine | head:cosine/ |
| Tembo | head:tembo/ |
| OpenHands | head:openhands |
Table 2. Search signals for distinguishing agent- and human-generated PRs. Search signals for distinguishing agent- and human-generated PRs
| PR Author | #PR | #Repository | #Commit | #Comment | #Review | #Issue | #Changed File |
|---|---|---|---|---|---|---|---|
| OpenAI Codex | 20,835 | 41,669 | 27,530 | 3,693 | 1,957 | 45 | 90,822 |
| Claude Code | 19,148 | 38,260 | 82,755 | 22,329 | 12,728 | 4,052 | 255,275 |
| GitHub Copilot | 18,563 | 37,125 | 69,896 | 26,664 | 20,665 | 9,744 | 158,404 |
| Google Jules | 18,468 | 36,936 | 41,032 | 5,700 | 3,249 | 2,185 | 138,610 |
| Devin | 14,045 | 28,090 | 51,641 | 27,518 | 6,901 | 294 | 131,454 |
| Human | 20,910 | 41,542 | 102,037 | 18,559 | 21,401 | 1,973 | 194,861 |
Table 3. Dataset summary of pull requests and related contributions. The first three columns indicate the number of PRs created by each author, along with their base and head repositories (hence the large number), while the other four reflect activity associated with those PRs. Dataset summary of pull requests and related contributions. The first three columns indicate the number of PRs created by each author, along with their base and head repositories (hence the large number), while the other four reflect activity associated with those PRs.
Figure 1. Overview of dataset structure. Solid lines indicate relationships between entities. Dotted lines denote nested objects. Overview of dataset structure. Solid lines indicate relationships between entities. Dotted lines denote nested objects.
Figure 2. Fraction of PRs linked to issues. Wilson 95% CIs shown. Fraction of PRs linked to issues. Wilson 95% CIs shown.
| PR Author | PR’s Repository Star Count | ||||
| %=0 | % 10 | % 100 | Median | Med(if 0) | |
| Claude | 51.7% | 19.7% | 10.4% | 0 | 5 |
| Codex | 75.3% | 5.2% | 2.4% | 0 | 1 |
| Copilot | 59.6% | 15.2% | 9.1% | 0 | 3 |
| Devin | 64.1% | 23.0% | 14.3% | 0 | 39 |
| Jules | 75.7% | 4.0% | 1.4% | 0 | 1 |
Table 4. Distribution of pull requests across repository stars, with most PRs originating from repositories with stars. Human PRs tend to appear in higher-starred repositories. Distribution of pull requests across repository stars, with most PRs originating from repositories with stars. Human PRs tend to appear in higher-starred repositories.
Figure 3. The plot at the top shows the median PR change size for each agent, while the bottom one shows the median fraction of changes that are additions. Bootstrapped 95% CIs shown. The plot at the top shows the median PR change size for each agent, while the bottom one shows the median fraction of changes that are additions. Bootstrapped 95% CIs shown.
Figure 4. Percentage of pull requests containing the corresponding file type. The five most common types are shown (related types grouped, e.g., TS/JS, YAML/TOML); less common types are grouped as “Other” with right-margin annotations listing the next four most frequent types. Wilson 95% confidence intervals are shown. Percentage of pull requests containing the corresponding file type. The five most common types are shown (related types grouped, e.g., TS/JS, YAML/TOML); less common types are grouped as “Other” with right-margin annotations listing the next four most frequent types. Wilson 95% confidence intervals are shown.
Figure 5. Percentage of pull requests merged per agent, grouped by repository star count (Wilson 95% CIs shown). Lower panel shows each agent’s PR fraction within star categories (p33 and p66 of human data). Some agents show an “inverted-u” trend across star counts. Percentage of pull requests merged per agent, grouped by repository star count (Wilson 95% CIs shown). Lower panel shows each agent's PR fraction within star categories (p33 and p66 of human data). Some agents show an “inverted-u” trend across star counts.
Figure 6. Distribution of merge times across agents (log scale) as boxplots with p10–p90 whiskers. Cliff’s and Mann-Whitney U test p-values comparing to humans are labeled. Merge times generally increase with repository star count; Codex and Jules PRs merge faster, while shorter times may reflect prior human interaction. Star boundaries are based on human PR tertiles of merged PRs. Distribution of merge times across agents (log scale) as boxplots with p10–p90 whiskers. Cliff's and Mann-Whitney U test p-values comparing to humans are labeled. Merge times generally increase with repository star count; Codex and Jules PRs merge faster, while shorter times may reflect prior human interaction. Star boundaries are based on human PR tertiles of merged PRs.
Figure 7. Developer interactions overview with pull requests, showing average counts of user comments (i.e., non-bot comments), reviews (main bar, left axis), and the proportion of PRs with at least 1 comment or review (dots, right axis). Bootstrapped 95% CIs shown. Developer interactions overview with pull requests, showing average counts of user comments (i.e., non-bot comments), reviews (main bar, left axis), and the proportion of PRs with at least 1 comment or review (dots, right axis). Bootstrapped 95% CIs shown.
Figure 8. Left plot denotes the fraction of commits with complete line survival across agents measured at 3-day, 1-week, and 3-week intervals. The right plot shows the average net additions per agent across the same time intervals. Bootstrapped 95% CIs shown. Left plot denotes the fraction of commits with complete line survival across agents measured at 3-day, 1-week, and 3-week intervals. The right plot shows the average net additions per agent across the same time intervals. Bootstrapped 95% CIs shown.
Figure 9. Per-agent distribution of commit churn rates measured over 3-day, 1-week, and 3-week intervals. Per-agent distribution of commit churn rates measured over 3-day, 1-week, and 3-week intervals.
Figure 10. Per-agent distribution of commit deletion rates measured over 3-day, 1-week, and 3-week intervals. Per-agent distribution of commit deletion rates measured over 3-day, 1-week, and 3-week intervals.