Organizations: The University of Hong Kong, Hong Kong, Hong Kong · Huazhong University of Science and Technology, China · Xi’an Jiaotong University, China · Independent Researcher
AI-generated bug reports have drawn criticism for unreliable claims and faulty reproducers that burden maintainers. However, as LLM capabilities improve, some maintainers have begun to express more positive views on AI use in bug reporting and maintenance. Understanding this evolution requires examining whether AI-involved reports are becoming more reliable, which uses of AI are associated with valid findings, and how maintainers investigate these reports and act on them. To this end, we conduct the first large-scale study of 120,918 bug reports from 101 open-source projects between January 2024 and September 2026, tracing AI involvement from discovery and reporting through validation and resolution. Our findings show that AI-involved reports become more complete over time, with a growing share of reports confirmed as valid. In bug reporting, AI for reproduction and diagnosis are associated with higher validity, whereas report writing is not. Extends into maintenance, maintainers increasingly use AI to investigate code and recommend fixes. However, they devote more effort to AI-assisted reports of human-discovered bugs than to reports of AI-discovered bugs. These findings highlight both AI's growing participation in bug investigation and the continuing need for verification and follow-up. Drawing on these findings, we provide practical guidance for turning AI-generated findings into verifiable reports: emphasizing reproducibility, verification, and accountability. We followed this guidance when submitting 53 reports on AI-discovered bugs, of which 69.81% were accepted by maintainers as of writing.
Figures & tables
Fig. 1: Evolving challenges of AI-generated reports.
Fig. 2: Measurement framework for AI usage, report reliability, and maintainer practices.
TABLE I: Overview of our dataset.
Fig. 3: Monthly AI-involved reports by role and distinct reporting accounts from January 2024 to September 2026.
Discovery Method
AI-disc.
AI-assist.
Suspect.
Other
Source Code Review
51.52%
6.66%
11.46%
5.62%
Static Analysis
5.05%
0.44%
0.00%
0.31%
Fuzzing
4.29%
0.44%
0.00%
0.67%
Dynamic Analysis
0.25%
0.24%
2.08%
0.60%
Testing
3.79%
4.23%
5.21%
4.70%
Build / Installation
0.51%
3.39%
4.17%
5.65%
TABLE II: Distribution of bug discovery methods across AI roles. AI-disc.: AI-discovered; AI-assist.: AI-assisted; Suspect.: suspected AI involvement.
AI Role
Quality
Missing Elements
Avg. Missing
High
Med.
Low
Summary
Comp.
Trigger
Repro.
Observed
Expected
Env.
Other
33.77%
51.91%
14.32%
9.69%
7.60%
22.09%
37.72%
16.39%
42.77%
29.19%
1.65
AI-disc.
56.31%
37.12%
6.57%
1.52%
1.52%
7.32%
18.69%
9.60%
16.67%
30.05%
0.85
AI-assist.
50.00%
42.86%
7.14%
3.11%
2.23%
9.81%
26.36%
6.22%
27.19%
21.33%
0.96
Suspect.
45.83%
42.71%
11.46%
4.17%
3.12%
13.54%
30.21%
9.38%
29.17%
28.12%
1.18
AI-invol.
50.70%
42.10%
7.20%
2.93%
2.17%
9.60%
25.47%
6.77%
25.87%
22.70%
0.95
TABLE III: Information completeness by AI role.
Fig. 4: Co-occurrence of AI tasks in assisted bug reports. Diagnosis : problem diagnosis; Fix : fix recommendation; Writing : report writing; Reproduction : reproduction and testing; Retrieval : information retrieval; Debugging : debugging support.
Fig. 5: Quarterly report quality: (a) high-quality share among all reports; (b) valid share among decided reports. Shading shows 95% project-cluster bootstrap confidence intervals.
Fig. 6: Rejection-reason prevalence among invalid reports: (a) AI-involved versus Other; (b) AI roles versus Other. EB : Expected Behavior; OS : Out of Scope; M : Misuse; U : Unreproducible; IC : Incorrect Claims. Reasons are non-exclusive; the catch-all Others reason is omitted.
Factor
Valid (%)
OR [95% CI]
q
Main Model: Discovery Method ( q<0.001 )
Operational Usage (ref.)
45.43%
1
–
Source Code Review
79.13%
2.82 [2.26, 3.52]
<0.001
Fuzzing
72.65%
2.78 [1.43, 5.42]
0.005
Testing
71.49%
2.25 [1.93, 2.63]
<0.001
Dynamic Analysis
65.55%
1.98 [1.18, 3.34]
0.019
TABLE IV: Factors associated with maintainer-confirmed validity.
Fig. 7: Observed cumulative closure rates within 90 days.
Fig. 8: Share of reports with at least k maintainer replies within 30 days.
AI Role
Metric
Triage
Validation
Resolution
Total
Info.
Dup.
Total
Repro.
Code
Total
Fix
Patch
Total
Other
Mean
0.28
0.13
0.41
0.30
0.70
1.01
0.53
0.05
0.58
2.00
AI-disc.
Mean
0.06
0.06
0.12
0.10
0.37
0.47
0.29
0.05
0.34
0.93
Adj. Δ
−0.12 *
−0.03
−0.15 *
−0.17 *
−0.27
−0.44 *
−0.30 *
−0.03
−0.34 *
−0.93 *
AI-assist.
Mean
0.32
0.11
0.44
0.34
0.73
1.06
0.61
0.07
0.68
2.18
Adj. Δ
+0.10 *
+0.01
+0.11 *
+0.08 *
+0.15 *
+0.24 *
+0.09
+0.02
+0.11
+0.46 *
TABLE V: Maintainer actions per report within 30 days, by AI role.
Material / Action
AI-Gen
Other
OR [95% CI]
q
Reproducer
Reject
23.53%
5.14%
8.16 [3.15, 18.86] *
< .001
Req. supp.
38.24%
58.17%
0.52 [0.24, 1.09]
0.147
Use
38.24%
27.47%
1.31 [0.60, 2.76]
0.650
Technical claim
Reject
59.46%
54.72%
1.57 [1.11, 2.23] *
0.034
TABLE VI: Maintainer actions on AI-generated versus other report materials.
Fig. 9: AI-involved maintainer replies by author type and distinct accounts, grouped by reply month.
Fig. 10: Action co-occurrence in AI-involved maintainer replies. Cells count reports with each action on the diagonal and both actions elsewhere.
AI Role
Metric
AI-Involved Replies
Actions in AI-Involved Replies
Any
AI Bot
Maint. AI
Triage
Validation
Resolution
Other
Value
1.26
0.86
0.47
0.39
1.10
0.79
AI-disc.
Value
2.28
1.27
1.01
0.00
1.77
1.77
Adj. Δ
−2.09
−2.13
−0.26
−0.76
−2.35
−1.83
AI-assist.
Value
4.89
2.42
3.04
1.15
4.44
3.41
Adj. Δ
+1.56∗
+0.18
+1.58∗
+0.33
+1.64∗
+1.05
TABLE VII: AI-involved maintainer replies and associated actions by report AI role.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
AI Role
Stat.
Valid
Invalid
Uncertain
Valid/Dec.
AI-disc.
Rate (%)
25.51
12.12
62.37
67.79
Δ (pp)
−10.92∗
−2.32
+13.25
−3.73
95% CI
[−16.8,−2.5]
[−8.3,4.5]
[1.2,22.9]
[−13.3,6.3]
q
0.033
0.819
0.102
0.819
AI-assist.
Rate (%)
32.46
20.57
46.97
61.20
Δ (pp)
+0.12
−0.31
+0.19
+0.39
Appendix
TABLE VIII: Bug report validity by AI role.
AI Role
Metric
Triage
Validation
Resolution
Total
Info.
Dup.
Total
Repro.
Code
Total
Fix
Patch
Total
AI-disc.
95% CI
[−0.18, −0.04]
[−0.08, 0.02]
[−0.24, −0.05]
[−0.27, −0.06]
[−0.53, 0.00]
[−0.79, −0.08]
[−0.48, −0.08]
[−0.08, 0.03]
[−0.53, −0.08]
[−1.54, −0.25]
q -value
0.018
0.322
0.018
0.018
0.090
0.040
0.018
0.322
0.018
0.018
AI-assist.
95% CI
[0.05, 0.18]
[−0.01, 0.03]
[0.05, 0.20]
[0.04, 0.15]
[0.07, 0.29]
[0.11, 0.44]
[0.02, 0.20]
[0.00, 0.05]
[0.03, 0.25]
[0.21, 0.85]
q -value
0.018
0.598
0.018
0.018
0.023
0.018
0.085
0.218
0.085
0.018
Suspect.
95% CI
[−0.19, 0.12]
[−0.13, 0.02]
[−0.28, 0.10]
[−0.25, 0.12]
[−0.36, 0.49]
[−0.57, 0.55]
[−0.42, 0.31]
[−0.06, 0.12]
[−0.46, 0.36]
[−1.18, 0.79]
Appendix
TABLE IX: Adjusted differences in maintainer actions: 95% bootstrap CIs in actions/report and BH-corrected q -values.
Fig. 11: Report length distributions.
Category
Definition
Discovery
Source Code Review . Discovered by inspecting, reviewing, or auditing source code directly, by a human, LLM/AI agent, or another code-review process; excludes automated static analyzers.
Static Analysis . Discovered by automated static analysis without executing the target, such as CodeQL, Infer, Coverity, Cppcheck, or static taint analysis.
Fuzzing . Discovered by fuzzing or automatically generated/mutated inputs, such as AFL, libFuzzer, Syzkaller, or another fuzzing system.
Dynamic Analysis . Discovered primarily through a runtime bug-detection tool such as AddressSanitizer, UBSan, TSan, MSan, Valgrind, or a similar dynamic analysis tool.
Testing . Discovered while running unit, regression, self-, integration, or other deliberately designed tests.
Build / Installation . Discovered during configuration, compilation, linking, packaging, building, or installation.
Appendix
TABLE X: Definitions of study measures: Study Scope and AI Use in Bug Reporting.