Agent or Human? Detecting Commits by AI Coding Agents in the Absence of Authorship Signatures
Authors: Kosei Horikawa, Hao Li, Yutaro Kashiwa, Youssef Esseddiq Ouatiti, Mohammed Sayagh, Bram Adams, Hajimu Iida, Ahmed E. Hassan
Organizations: Nara Institute of Science and Technology, Ikoma, Japan · Queen’s University, Kingston, Canada · École de technologie supérieure - Québec University, Montréal, Canada
AI coding agents such as Claude Code, GitHub Copilot coding agent, and Cursor now commit directly to open-source repositories. Empirical studies of these agents identify their commits through authorship signatures, such as bot accounts and Co-authored-by trailers. These signatures are optional, can be removed, and are often attached to pull requests rather than to individual commits. Hence, it is unclear whether Agent commits can be told apart from Human commits once the signatures are gone. We investigate the signals and limits of signature-free detection using 7,837,946 commits from 1,226 repositories, covering five coding agents and Human commits. After signatures are masked, detectors rank Agent commits above Human commits with a ROC AUC of up to 0.859, and most of this signal comes from message verbosity and change size rather than from the code itself. On the shared sample, gpt-6-sol without fine-tuning reaches AUCs similar to XGBoost and TF-IDF. At the observed Agent share, however, the best detector on the full test set reaches a precision of only 0.332. Detection transfers poorly to held-out Claude Code and Copilot, with TF-IDF reaching an AUC of only 0.524 for Claude Code. Temporal changes also depend on the agent, e.g., Copilot AUC drops and later recovers, while more training history helps Claude Code but not Copilot. These results indicate that signature-free detection can prioritize commits for review but cannot replace recorded attribution, and that studies relying on signature-based labels should treat them as a lower bound on agent activity.
Figures & tables
Topics studied
Analysis unit
Agent identification
Agent use and contribution characteristics ( Li et al., 2025 ; Robbes et al., 2026 )
Table 1. Prior studies of AI coding agents and their methods for identifying agent contributions. The identification methods refer to dataset labels, and detector inputs may differ.
Figure 1. Maintainer comment closing a PR suspected of being generated by an AI agent ( https://github.com/Fredolx/open-tv/pull/397 ). A maintainer closes a PR as suspected AI-generated work. The contributor reports using AI only for autocompletion; the exchange illustrates disagreement about authorship when commits lack agent signatures.
Figure 2. Overview of the study design.
Era
Group
# Repos
# Commits
Median words
# Files
# LOC
Median LOC
Pre-LLM
Human
1,224
2,504,139
7
15,795,739
1,715,849,223
16
LLM
Human
1,224
2,509,950
8
18,206,607
1,559,030,746
16
Agent
Copilot
697
27,179
14
190,486
21,634,462
26
Claude Code
906
141,034
65
1,147,521
89,580,602
60
Cursor
27
1,160
36
4,855
280,170
41
Devin
10
75
60
156
6,755
47
Table 2. Repositories, commits, and changes for the Agent and Human classes across the three eras: Pre-LLM (2020-03-29 to 2021-06-28), LLM (2023-09-24 to 2024-12-23), and Agent (2025-06-01 to 2026-08-31). Medians pool commits within each row; the repository-level medians in RQ1 give each repository equal weight.
Extension diversity ( Dey et al., 2020 ) and file-type fractions
8
Code Diff
Addition fraction, changes per file, single-file and large-change indicators
4
Code Style
Added-line count ( Shi et al., 2025 ) , layout ( Caliskan-Islam et al., 2015 ) , identifiers, and annotation proxies
13
Code Vocabulary
Token counts, Zipf and Heaps exponents ( Shi et al., 2025 )
4
Table 3. Overview of detector features.
Family
Feature
Claude Code
Copilot
Cursor
Commit Message
Characters
0.698 ∗
0.508 ∗
0.764 ∗
Words
0.693 ∗
0.422 ∗
0.692 ∗
Lines
0.641 ∗
0.544 ∗
0.786 ∗
Conventional prefix
0.021 ∗
0.000
0.144
Code Diff
Addition share
0.150 ∗
0.123 ∗
0.153
Changed lines per file
0.193 ∗
0.063 ∗
0.089
Table 4. Cliff’s δ between Agent and Agent-era Human commits, as the median over repositories. Positive values mean higher values for Agent commits, and ∗ marks an adjusted p<0.05 . Only features with ∣δ∣≥0.100 for at least one agent are shown. Orange marks positive and blue negative effects.
Family
Feature
pre-LLM to LLM
LLM to Agent
pre-LLM to Agent
Commit Message
Characters
0.399 ∗
0.563 ∗
0.663 ∗
Words
0.372 ∗
0.567 ∗
0.629 ∗
Lines
0.509 ∗
0.575 ∗
0.689 ∗
Conventional prefix
0.572 ∗
0.680 ∗
0.827 ∗
Code Diff
Addition share
0.009
0.632 ∗
0.524 ∗
Changed lines per file
0.030
0.738 ∗
0.628 ∗
Table 5. Changes in Human commits between eras in the same 1,224 repositories, as matched-pairs rank-biserial correlations ( Kerby, 2014 ) . Positive values mean higher values in the later era, and ∗ marks an adjusted p<0.05 . Rows are the features of Table 4 . The correlation measures how consistently repositories change in one direction, not the size of the change.
Test set
Detector
ROC AUC
PR-AUC
Lift
Precision
Recall
MCC
Weighted F1
Full
XGBoost, 32 features
0.792
0.187
3.152
0.184
0.473
0.224
0.875
LightGBM, 32 features
0.792
0.187
3.161
0.182
0.476
0.223
0.874
CatBoost, 32 features
0.790
0.188
3.175
0.186
0.461
0.223
0.877
TF-IDF, message and diff
0.859
0.305
5.151
0.332
0.497
0.361
0.919
TF-IDF, message only
0.859
0.314
5.306
0.320
0.534
0.366
0.916
TF-IDF, diff only
0.716
0.136
2.299
0.146
0.388
0.158
0.863
Table 6. Detector performance on masked commits, using the full test set (2,704,768 commits) or a balanced sample (4,000 commits). Detectors within each set use the same commits. Lift is PR-AUC divided by the Agent share. Weighted F1 averages Agent and Human F1 by class size. The sample’s PR-AUC, lift, and weighted F1 are not comparable with those of the full test set. Its precision and MCC are re-weighted to the observed Agent share of 0.059. CodeBERT is trained on 100,000 commits per fold.
Figure 3. XGBoost ROC AUC when one feature family is removed, with 95% repository-bootstrap intervals. The dashed line marks the model with all five families. Removing Commit Message features lowers overall binary ROC AUC from \AUCXGB{} to \AUCNoMessage{}. Other family removals leave AUC between \AUCAblationMin{} and \AUCAblationMax{}.
Full test set
Size-matched
Agent subset
Agent share
ROC AUC
PR-AUC
Lift
Precision
Recall
MCC
ROC AUC
Claude Code
0.050
0.786
0.157
3.123
0.156
0.458
0.201
0.543
Copilot
0.009
0.824
0.040
4.390
0.037
0.549
0.116
0.692
Cursor
< 0.001
0.879
0.004
8.292
0.002
0.648
0.032
0.655
Devin
< 0.001
0.904
< 0.001
15.121
< 0.001
0.730
0.010
0.703
Google Jules
< 0.001
0.853
0.001
5.429
0.001
0.623
0.016
0.675
Table 7. XGBoost detection of each agent against the same 2,544,682 Human commits. Lift is PR-AUC divided by the Agent share. The last column is ROC AUC on size-matched test pairs, which pair each Agent commit with a Human commit from the same repository, language, month, and size range.
Unseen agent
Training Other agents
Training Human
Test Unseen agent
Test Human
Claude Code
6,527
446,694
141,034
2,207,402
Copilot
53,251
1,139,036
27,179
1,515,060
Cursor
146,865
2,543,735
1,160
110,361
Devin
168,366
2,630,883
75
23,213
Google Jules
166,019
2,526,866
313
127,230
Table 8. Commits in each unseen-agent experiment. Training and test repositories are disjoint.
Figure 4. ROC AUC on unseen agents. Boxes and whiskers summarize repository-bootstrap draws (median, interquartile range, and 95% interval), and diamonds show the test-set AUC. Two horizontal boxplots per unseen agent show repository-bootstrap AUC draws for XGBoost: repository-disjoint training (filled) and repository overlap allowed (hatched). These distributions condition on fixed fitted models.
Unseen agent
Agent share
Detector
ROC AUC
PR-AUC
Lift
Precision
Recall
MCC
Claude Code
0.060
XGBoost
0.702
0.109
1.809
0.069
0.005
0.003
TF-IDF
0.524
0.068
1.131
0.086
0.024
0.014
Copilot
0.018
XGBoost
0.706
0.039
2.209
0.047
0.250
0.072
TF-IDF
0.630
0.028
1.608
0.034
0.143
0.036
Cursor
0.010
XGBoost
0.898
0.067
6.396
0.069
0.552
0.174
TF-IDF
0.739
0.023
2.224
0.029
0.293
0.064
Table 9. Detection of unseen agents, with both detectors on the same test commits and thresholds selected on known agents.
Figure 5. ROC AUC of Claude Code and Copilot by training cutoff (rows) and test month (columns). White cells are not evaluated. Two heatmaps show Claude Code and Copilot AUCs for all \NMonthPairs{} pairs of training cutoff and later test month. Each cell prints its AUC.
Agent
Training cutoff
ROC AUC
PR-AUC
Precision
Recall
F1
Claude Code
June 2025
0.705
0.222
0.320
0.075
0.121
July 2026
0.752
0.249
0.312
0.185
0.233
Copilot
June 2025
0.779
0.019
0.027
0.092
0.042
July 2026
0.768
0.015
0.016
0.138
0.029
Table 10. Detection in August 2026 by training cutoff. Training starts in June 2025. Test commits: 22,545 Claude Code (Agent share 0.108) and 1,101 Copilot (0.006), each against the same 185,832 Human commits.
Generative AI coding agents are entering the open-source supply chain, yet their diverse and often invisible traces leave their prevalence poorly understood. We introduce a multi-layered detection framework that integrates configuration-file scanning, commit-message analysis, author-identity matching, and bot-signature lookup across World of Code (180M+ Git repositories), classifying agent traces into four behavioral types. No single method captures more than a fraction of activity: multi-method detection identifies 850,157 Claude Code commits in one snapshot, of which bot-account lookup_the signal most adoption studies rely on_recovers only 28,154 (3.3%), a 30x relative-recall gap, so single-signal prevalence estimates are biased low by at least this factor. Every detection pattern is hand-validated (495 labels) with per-cell precision and Wilson confidence intervals. Across snapshots from December 2024 to April 2026, commit-attributed agents generate over 320,000 commits per month; Claude Code leads (886,122 commits across 17,295 projects) and dominates silent, configuration-file-only adoption (21,078 projects). Compared against an independent pull-request census (AIDev), the two channels capture nearly disjoint agent populations_a PR census misses 79% of commit-detected Claude Code adopters and essentially all Codex adopters_and different kinds of work: PR-deployed cloud agents (Codex, Cursor) surface as feature work, while commit-deployed in-editor agents (Claude Code, OpenHands, Aider) surface as maintenance. The observed work profile follows deployment and detection mode rather than the tool itself, so no single channel is representative.
Coding agents can be guided with instruction files in a repository. We examine this emerging practice through a census of 22,854 popular, active GitHub repositories in thirteen languages, treating each repository's root 'AGENTS.md' and 'CLAUDE.md' as one instruction set. Agent instructions appear in 26.3% of repositories and 59.6% of the most-starred; where both files exist, 'CLAUDE.md' usually points to 'AGENTS.md'. We analyse 341,852 statements using model-assisted annotation and human review. The annotations reveal a practical mix of project knowledge and rules about code, building and workflow. Collaboration rules name concrete requirements, from test plans to commit formats, yet 74.0% of directives give no reason. Projects disagree on AI attribution: commit-message rules more often prohibit it than require it, while pull-request rules more often require disclosure. A third of instruction sets have a generator-associated signature, a tool-inserted block, shared text or an AI-marked creation commit; four in five are edited after creation. Prespecified tests associate repository popularity with rules on pull requests and AI contributions. Exploratory analyses link activity to process-oriented content and show that LLM and agent projects are over-represented among adopters. The models agree much more closely with each other than with the human reviewer's topic assignments, a caution for automated annotation. We release the annotations, taxonomies and pointers to the analysed file versions.
Open source communities have been flooded with AI-generated contributions. In defense, they have written contribution rules to regulate coding agents' behavior, spanning from a total ban, mandatory disclosure, to verification gates and human sign-offs. Yet, whether coding agents read and follow those rules, and behave in open source repositories, remains unknown. To estimate real-world rule compliance of coding agents, we curate 106 issues from 49 repositories containing AI contribution rules into RepoComplianceBench. We judge the trajectory of each run against the repository's rules, measuring whether the agent refuses to contribute, discloses its assistance truthfully, clears the required verification gates, or escalates critical steps to a human. We also test if extra prompts, rule disclosure, or feedback from the compliance verifier help with the situation. Our experiments on four frontier models show that today's agents almost never proactively retrieve the contribution rules. Agents pick up disclosure and verification with reminder prompts, rule quotes, and verifier feedback; however, they never refuse to contribute in AI-banned repositories under any condition we tested. The status reveals that verification and disclosure issues are solvable with existing mechanisms, yet enforcing bans and human escalations remains an open problem.