cs.SEApr 1, 2026

Investigating Autonomous Agent Contributions in the Wild: Activity Patterns and Code Change over Time

Authors: Razvan Mihai Popescu, David Gros, Andrei Botocan, Rahul Pandita, Prem Devanbu, Maliheh Izadi

Organizations: Delft University of Technology, Delft, The Netherlands · University of California, Davis, Davis, USA · GitHub, Denver, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

The rise of large language models for code has reshaped software development. Autonomous coding agents, able to create branches, open pull requests, and perform code reviews, now actively contribute to real-world projects. Their growing role offers a unique and timely opportunity to investigate AI-driven contributions and their effects on code quality, team dynamics, and software maintainability. In this work, we construct a novel dataset of approximately 110,000110,000 open-source pull requests, including associated commits, comments, reviews, issues, and file changes, collectively representing millions of lines of source code. We compare five popular coding agents, including OpenAI Codex, Claude Code, GitHub Copilot, Google Jules, and Devin, examining how their usage differs in various development aspects such as merge frequency, edited file types, and developer interaction signals, including comments and reviews. Furthermore, we emphasize that code authoring and review are only a small part of the larger software engineering process, as the resulting code must also be maintained and updated over time. Hence, we offer several longitudinal estimates of survival and churn rates for agent-generated versus human-authored code. Ultimately, our findings indicate an increasing agent activity in open-source projects, although their contributions are associated with more churn over time compared to human-authored code.

Figures & tables

Coding AgentSearch Signal
OpenAI Codexhead:codex/
GitHub Copilothead:copilot/
Google Julesauthor:google-labs-jules[bot]
Devinauthor:devin-ai-integration[bot]
Claude Code(“Co-Authored-By: Claude” OR “Generated with Claude Code”)
Table 1. Search signals used to filter PRs per coding agent. Search signals used to filter PRs per coding agent.
Coding AgentSearch Signal
OpenAI Codexhead:codex/
GitHub Copilothead:copilot/
Cursorhead:cursor/
Cosinehead:cosine/
Tembohead:tembo/
OpenHandshead:openhands
Table 2. Search signals for distinguishing agent- and human-generated PRs. Search signals for distinguishing agent- and human-generated PRs
PR Author#PR#Repository#Commit#Comment#Review#Issue#Changed File
OpenAI Codex20,83541,66927,5303,6931,9574590,822
Claude Code19,14838,26082,75522,32912,7284,052255,275
GitHub Copilot18,56337,12569,89626,66420,6659,744158,404
Google Jules18,46836,93641,0325,7003,2492,185138,610
Devin14,04528,09051,64127,5186,901294131,454
Human20,91041,542102,03718,55921,4011,973194,861
Table 3. Dataset summary of pull requests and related contributions. The first three columns indicate the number of PRs created by each author, along with their base and head repositories (hence the large number), while the other four reflect activity associated with those PRs. Dataset summary of pull requests and related contributions. The first three columns indicate the number of PRs created by each author, along with their base and head repositories (hence the large number), while the other four reflect activity associated with those PRs.
Figure 1. Overview of dataset structure. Solid lines indicate relationships between entities. Dotted lines denote nested objects. Overview of dataset structure. Solid lines indicate relationships between entities. Dotted lines denote nested objects.
Figure 2. Fraction of PRs linked to issues. Wilson 95% CIs shown. Fraction of PRs linked to issues. Wilson 95% CIs shown.
PR AuthorPR’s Repository Star Count
 %=0% ≥\geq 10% ≥\geq 100MedianMed(if >> 0)
Claude51.7%19.7%10.4%05
Codex75.3%5.2%2.4%01
Copilot59.6%15.2%9.1%03
Devin64.1%23.0%14.3%039
Jules75.7%4.0%1.4%01
Table 4. Distribution of pull requests across repository stars, with most PRs originating from repositories with 00 stars. Human PRs tend to appear in higher-starred repositories. Distribution of pull requests across repository stars, with most PRs originating from repositories with 00 stars. Human PRs tend to appear in higher-starred repositories.
Figure 3. The plot at the top shows the median PR change size for each agent, while the bottom one shows the median fraction of changes that are additions. Bootstrapped 95% CIs shown. The plot at the top shows the median PR change size for each agent, while the bottom one shows the median fraction of changes that are additions. Bootstrapped 95% CIs shown.
Figure 4. Percentage of pull requests containing the corresponding file type. The five most common types are shown (related types grouped, e.g., TS/JS, YAML/TOML); less common types are grouped as “Other” with right-margin annotations listing the next four most frequent types. Wilson 95% confidence intervals are shown. Percentage of pull requests containing the corresponding file type. The five most common types are shown (related types grouped, e.g., TS/JS, YAML/TOML); less common types are grouped as “Other” with right-margin annotations listing the next four most frequent types. Wilson 95% confidence intervals are shown.
Figure 5. Percentage of pull requests merged per agent, grouped by repository star count (Wilson 95% CIs shown). Lower panel shows each agent’s PR fraction within star categories (p33 and p66 of human data). Some agents show an “inverted-u” trend across star counts. Percentage of pull requests merged per agent, grouped by repository star count (Wilson 95% CIs shown). Lower panel shows each agent's PR fraction within star categories (p33 and p66 of human data). Some agents show an “inverted-u” trend across star counts.
Figure 6. Distribution of merge times across agents (log scale) as boxplots with p10–p90 whiskers. Cliff’s δ\delta and Mann-Whitney U test p-values comparing to humans are labeled. Merge times generally increase with repository star count; Codex and Jules PRs merge faster, while shorter times may reflect prior human interaction. Star boundaries are based on human PR tertiles of merged PRs. Distribution of merge times across agents (log scale) as boxplots with p10–p90 whiskers. Cliff's δ\delta and Mann-Whitney U test p-values comparing to humans are labeled. Merge times generally increase with repository star count; Codex and Jules PRs merge faster, while shorter times may reflect prior human interaction. Star boundaries are based on human PR tertiles of merged PRs.
Figure 7. Developer interactions overview with pull requests, showing average counts of user comments (i.e., non-bot comments), reviews (main bar, left axis), and the proportion of PRs with at least 1 comment or review (dots, right axis). Bootstrapped 95% CIs shown. Developer interactions overview with pull requests, showing average counts of user comments (i.e., non-bot comments), reviews (main bar, left axis), and the proportion of PRs with at least 1 comment or review (dots, right axis). Bootstrapped 95% CIs shown.
Figure 8. Left plot denotes the fraction of commits with complete line survival across agents measured at 3-day, 1-week, and 3-week intervals. The right plot shows the average net additions per agent across the same time intervals. Bootstrapped 95% CIs shown. Left plot denotes the fraction of commits with complete line survival across agents measured at 3-day, 1-week, and 3-week intervals. The right plot shows the average net additions per agent across the same time intervals. Bootstrapped 95% CIs shown.
Figure 9. Per-agent distribution of commit churn rates measured over 3-day, 1-week, and 3-week intervals. Per-agent distribution of commit churn rates measured over 3-day, 1-week, and 3-week intervals.
Figure 10. Per-agent distribution of commit deletion rates measured over 3-day, 1-week, and 3-week intervals. Per-agent distribution of commit deletion rates measured over 3-day, 1-week, and 3-week intervals.

Explore similar work

CardsList
Jul 23, 2026

How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests

Recent advances in large language models and their rapid adoption across software engineering tasks have made Artificial Intelligence (AI) coding agents an integral component of modern software development workflows. While developers increasingly benefit from these coding agents, their impact on software quality remains insufficiently understood. In particular, how agentic contributions evolve across the software development lifecycle has not been thoroughly investigated. This study aims to characterize agentic pull requests (PR) in comparison to human generated PRs and to examine how their properties change across different stages of the development lifecycle. Using the AIDev dataset, we first analyze how differences in merge rates between agentic and human generated PRs vary over time. We then identify the types of development tasks where AI coding agents are predominantly applied and investigate how these task distributions evolve across development quarters. Finally, we compare a set of key characteristics of agentic and human generated PRs, focusing on their implications for software quality and their temporal dynamics. Overall, our findings provide an empirical and longitudinal perspective on the role of AI coding agents in software development, offering a more nuanced understanding of their benefits and limitations in real-world practices.

Iren Mazloomzadeh, Mohammad Mehdi Morovati, Foutse Khomh

Polytechnique Montréal, Canada

Software Engineering AgentsAI Coding AgentsCode Generation EvaluationAI-Assisted Software Development
AskPDF
Aug 29, 2026

AgentLogs: A Dataset for Opening the Black Box of GitHub's Cloud Agent

Generative AI-based software engineering agents are becoming routine contributors to real-world software projects. On GitHub, developers can assign tasks to the Copilot cloud agent, which autonomously explores the repository, edits code, runs commands, and opens or reviews pull requests, producing a detailed log of every step along the way. While existing datasets capture outcomes of agent contributions, such as agent-authored pull requests, the process by which agents produce these contributions remains largely unexplored. To address this gap, we introduce AgentLogs, a large-scale dataset of agent activity on GitHub. AgentLogs comprises 307,416 agent tasks and 549,239 agent sessions in 35,810 of the 1,812,362 popular public repositories that we scanned, together with 64,255,174 session log entries that record each agent run step by step, including prompts, intermediate reasoning, tool calls (e.g., file edits, git operations, and GitHub interactions), and token usage. By exposing not only what agents contribute but also how they work, AgentLogs enables research on agent behavior, efficiency and cost, task formulation, failure modes, and human-agent collaboration in agentic software engineering.

Jonan Richards, Kosei Horikawa, Youmei Fan +2

Radboud University, the Netherlands · Nara Institute of Science and Technology, Japan

Software Engineering AgentsAI Coding AgentsHuman-AI Collaboration
AskPDF
Aug 14, 2026

Engineering Signals of Human-AI Collaboration in the Agentic Coding Era: A Longitudinal Analysis of 33,228 Pull Requests from vLLM and SGLang with Implications for Biomedical AI Agents and Bioinformatics Pipeline Developmen

The rapid adoption of AI coding assistants and autonomous agentic development systems has coincided with major changes in the pace and structure of open-source software engineering. Yet empirical longitudinal evidence of these changes at the team level remains limited. We present a descriptive longitudinal analysis of seven engineering metrics: pull request (PR) throughput, cycle time, contributor diversity, PR comment density, merge rate, new-author participation, and PR size. Metrics were computed from all merged PRs in two high-velocity AI infrastructure repositories, vLLM (February 2023-June 2026; 18,290 PRs) and SGLang (January 2024-June 2026; 14,938 PRs). We segment development into four eras aligned with major changes in AI-assisted software development and examine human- and bot-authored activities. Both projects show substantial increases in development velocity and AI-developer collaboration signals. PR throughput increased 21x in vLLM and 17.9x in SGLang, while bot-authored PRs accounted for less than 0.2% of this growth, indicating that the increase was overwhelmingly human-driven. In the latest era, median cycle time was 1.04 days for vLLM and 0.62 days for SGLang, while P90 cycle times reached 16.8 and 14.3 days, respectively. Monthly unique authors increased steadily in both projects, suggesting broader contributor participation. PR comment density increased 4.2x in vLLM and 3.8x in SGLang, with bot comments contributing an estimated 15-20% of the increase. In contrast, PR size remained relatively stable across eras. Overall, AI-assisted development is associated with higher throughput, broader contributor participation, and increased AI-developer collaboration signals in high-velocity open-source software development.

Jiada Li, Xuesong Ye, Olamide Olowoniyi

Georgia Institute of Technology, Atlanta, GA, 30332 · Independent Researcher, Albany, NY, 12210

Human-AI CollaborationAI-Assisted Software DevelopmentSoftware Engineering
AskPDF