Agent or Human? Detecting Commits by AI Coding Agents in the Absence of Authorship Signatures
Authors: Kosei Horikawa, Hao Li, Yutaro Kashiwa, Youssef Esseddiq Ouatiti, Mohammed Sayagh, Bram Adams, Hajimu Iida, Ahmed E. Hassan
Organizations: Nara Institute of Science and Technology, Ikoma, Japan · Queen’s University, Kingston, Canada · École de technologie supérieure - Québec University, Montréal, Canada
AI coding agents such as Claude Code, GitHub Copilot coding agent, and Cursor now commit directly to open-source repositories. Empirical studies of these agents identify their commits through authorship signatures, such as bot accounts and Co-authored-by trailers. These signatures are optional, can be removed, and are often attached to pull requests rather than to individual commits. Hence, it is unclear whether Agent commits can be told apart from Human commits once the signatures are gone. We investigate the signals and limits of signature-free detection using 7,837,946 commits from 1,226 repositories, covering five coding agents and Human commits. After signatures are masked, detectors rank Agent commits above Human commits with a ROC AUC of up to 0.859, and most of this signal comes from message verbosity and change size rather than from the code itself. On the shared sample, gpt-6-sol without fine-tuning reaches AUCs similar to XGBoost and TF-IDF. At the observed Agent share, however, the best detector on the full test set reaches a precision of only 0.332. Detection transfers poorly to held-out Claude Code and Copilot, with TF-IDF reaching an AUC of only 0.524 for Claude Code. Temporal changes also depend on the agent, e.g., Copilot AUC drops and later recovers, while more training history helps Claude Code but not Copilot. These results indicate that signature-free detection can prioritize commits for review but cannot replace recorded attribution, and that studies relying on signature-based labels should treat them as a lower bound on agent activity.
Figures & tables
Topics studied
Analysis unit
Agent identification
Agent use and contribution characteristics ( Li et al., 2025 ; Robbes et al., 2026 )
Table 1. Prior studies of AI coding agents and their methods for identifying agent contributions. The identification methods refer to dataset labels, and detector inputs may differ.
Figure 1. Maintainer comment closing a PR suspected of being generated by an AI agent ( https://github.com/Fredolx/open-tv/pull/397 ). A maintainer closes a PR as suspected AI-generated work. The contributor reports using AI only for autocompletion; the exchange illustrates disagreement about authorship when commits lack agent signatures.
Figure 2. Overview of the study design.
Era
Group
# Repos
# Commits
Median words
# Files
# LOC
Median LOC
Pre-LLM
Human
1,224
2,504,139
7
15,795,739
1,715,849,223
16
LLM
Human
1,224
2,509,950
8
18,206,607
1,559,030,746
16
Agent
Copilot
697
27,179
14
190,486
21,634,462
26
Claude Code
906
141,034
65
1,147,521
89,580,602
60
Cursor
27
1,160
36
4,855
280,170
41
Devin
10
75
60
156
6,755
47
Table 2. Repositories, commits, and changes for the Agent and Human classes across the three eras: Pre-LLM (2020-03-29 to 2021-06-28), LLM (2023-09-24 to 2024-12-23), and Agent (2025-06-01 to 2026-08-31). Medians pool commits within each row; the repository-level medians in RQ1 give each repository equal weight.
Extension diversity ( Dey et al., 2020 ) and file-type fractions
8
Code Diff
Addition fraction, changes per file, single-file and large-change indicators
4
Code Style
Added-line count ( Shi et al., 2025 ) , layout ( Caliskan-Islam et al., 2015 ) , identifiers, and annotation proxies
13
Code Vocabulary
Token counts, Zipf and Heaps exponents ( Shi et al., 2025 )
4
Table 3. Overview of detector features.
Family
Feature
Claude Code
Copilot
Cursor
Commit Message
Characters
0.698 ∗
0.508 ∗
0.764 ∗
Words
0.693 ∗
0.422 ∗
0.692 ∗
Lines
0.641 ∗
0.544 ∗
0.786 ∗
Conventional prefix
0.021 ∗
0.000
0.144
Code Diff
Addition share
0.150 ∗
0.123 ∗
0.153
Changed lines per file
0.193 ∗
0.063 ∗
0.089
Table 4. Cliff’s δ between Agent and Agent-era Human commits, as the median over repositories. Positive values mean higher values for Agent commits, and ∗ marks an adjusted p<0.05 . Only features with ∣δ∣≥0.100 for at least one agent are shown. Orange marks positive and blue negative effects.
Family
Feature
pre-LLM to LLM
LLM to Agent
pre-LLM to Agent
Commit Message
Characters
0.399 ∗
0.563 ∗
0.663 ∗
Words
0.372 ∗
0.567 ∗
0.629 ∗
Lines
0.509 ∗
0.575 ∗
0.689 ∗
Conventional prefix
0.572 ∗
0.680 ∗
0.827 ∗
Code Diff
Addition share
0.009
0.632 ∗
0.524 ∗
Changed lines per file
0.030
0.738 ∗
0.628 ∗
Table 5. Changes in Human commits between eras in the same 1,224 repositories, as matched-pairs rank-biserial correlations ( Kerby, 2014 ) . Positive values mean higher values in the later era, and ∗ marks an adjusted p<0.05 . Rows are the features of Table 4 . The correlation measures how consistently repositories change in one direction, not the size of the change.
Test set
Detector
ROC AUC
PR-AUC
Lift
Precision
Recall
MCC
Weighted F1
Full
XGBoost, 32 features
0.792
0.187
3.152
0.184
0.473
0.224
0.875
LightGBM, 32 features
0.792
0.187
3.161
0.182
0.476
0.223
0.874
CatBoost, 32 features
0.790
0.188
3.175
0.186
0.461
0.223
0.877
TF-IDF, message and diff
0.859
0.305
5.151
0.332
0.497
0.361
0.919
TF-IDF, message only
0.859
0.314
5.306
0.320
0.534
0.366
0.916
TF-IDF, diff only
0.716
0.136
2.299
0.146
0.388
0.158
0.863
Table 6. Detector performance on masked commits, using the full test set (2,704,768 commits) or a balanced sample (4,000 commits). Detectors within each set use the same commits. Lift is PR-AUC divided by the Agent share. Weighted F1 averages Agent and Human F1 by class size. The sample’s PR-AUC, lift, and weighted F1 are not comparable with those of the full test set. Its precision and MCC are re-weighted to the observed Agent share of 0.059. CodeBERT is trained on 100,000 commits per fold.
Figure 3. XGBoost ROC AUC when one feature family is removed, with 95% repository-bootstrap intervals. The dashed line marks the model with all five families. Removing Commit Message features lowers overall binary ROC AUC from \AUCXGB{} to \AUCNoMessage{}. Other family removals leave AUC between \AUCAblationMin{} and \AUCAblationMax{}.
Full test set
Size-matched
Agent subset
Agent share
ROC AUC
PR-AUC
Lift
Precision
Recall
MCC
ROC AUC
Claude Code
0.050
0.786
0.157
3.123
0.156
0.458
0.201
0.543
Copilot
0.009
0.824
0.040
4.390
0.037
0.549
0.116
0.692
Cursor
< 0.001
0.879
0.004
8.292
0.002
0.648
0.032
0.655
Devin
< 0.001
0.904
< 0.001
15.121
< 0.001
0.730
0.010
0.703
Google Jules
< 0.001
0.853
0.001
5.429
0.001
0.623
0.016
0.675
Table 7. XGBoost detection of each agent against the same 2,544,682 Human commits. Lift is PR-AUC divided by the Agent share. The last column is ROC AUC on size-matched test pairs, which pair each Agent commit with a Human commit from the same repository, language, month, and size range.
Unseen agent
Training Other agents
Training Human
Test Unseen agent
Test Human
Claude Code
6,527
446,694
141,034
2,207,402
Copilot
53,251
1,139,036
27,179
1,515,060
Cursor
146,865
2,543,735
1,160
110,361
Devin
168,366
2,630,883
75
23,213
Google Jules
166,019
2,526,866
313
127,230
Table 8. Commits in each unseen-agent experiment. Training and test repositories are disjoint.
Figure 4. ROC AUC on unseen agents. Boxes and whiskers summarize repository-bootstrap draws (median, interquartile range, and 95% interval), and diamonds show the test-set AUC. Two horizontal boxplots per unseen agent show repository-bootstrap AUC draws for XGBoost: repository-disjoint training (filled) and repository overlap allowed (hatched). These distributions condition on fixed fitted models.
Unseen agent
Agent share
Detector
ROC AUC
PR-AUC
Lift
Precision
Recall
MCC
Claude Code
0.060
XGBoost
0.702
0.109
1.809
0.069
0.005
0.003
TF-IDF
0.524
0.068
1.131
0.086
0.024
0.014
Copilot
0.018
XGBoost
0.706
0.039
2.209
0.047
0.250
0.072
TF-IDF
0.630
0.028
1.608
0.034
0.143
0.036
Cursor
0.010
XGBoost
0.898
0.067
6.396
0.069
0.552
0.174
TF-IDF
0.739
0.023
2.224
0.029
0.293
0.064
Table 9. Detection of unseen agents, with both detectors on the same test commits and thresholds selected on known agents.
Figure 5. ROC AUC of Claude Code and Copilot by training cutoff (rows) and test month (columns). White cells are not evaluated. Two heatmaps show Claude Code and Copilot AUCs for all \NMonthPairs{} pairs of training cutoff and later test month. Each cell prints its AUC.
Agent
Training cutoff
ROC AUC
PR-AUC
Precision
Recall
F1
Claude Code
June 2025
0.705
0.222
0.320
0.075
0.121
July 2026
0.752
0.249
0.312
0.185
0.233
Copilot
June 2025
0.779
0.019
0.027
0.092
0.042
July 2026
0.768
0.015
0.016
0.138
0.029
Table 10. Detection in August 2026 by training cutoff. Training starts in June 2025. Test commits: 22,545 Claude Code (Agent share 0.108) and 1,101 Copilot (0.006), each against the same 185,832 Human commits.