Copilot or Culprit? How AI Is Changing Open-Source Bug Reporting and Maintenance
Organizations: The University of Hong Kong, Hong Kong, Hong Kong · Huazhong University of Science and Technology, China · Xi’an Jiaotong University, China · Independent Researcher
Abstract
AI-generated bug reports have drawn criticism for unreliable claims and faulty reproducers that burden maintainers. However, as LLM capabilities improve, some maintainers have begun to express more positive views on AI use in bug reporting and maintenance. Understanding this evolution requires examining whether AI-involved reports are becoming more reliable, which uses of AI are associated with valid findings, and how maintainers investigate these reports and act on them. To this end, we conduct the first large-scale study of 120,918 bug reports from 101 open-source projects between January 2024 and September 2026, tracing AI involvement from discovery and reporting through validation and resolution. Our findings show that AI-involved reports become more complete over time, with a growing share of reports confirmed as valid. In bug reporting, AI for reproduction and diagnosis are associated with higher validity, whereas report writing is not. Extends into maintenance, maintainers increasingly use AI to investigate code and recommend fixes. However, they devote more effort to AI-assisted reports of human-discovered bugs than to reports of AI-discovered bugs. These findings highlight both AI's growing participation in bug investigation and the continuing need for verification and follow-up. Drawing on these findings, we provide practical guidance for turning AI-generated findings into verifiable reports: emphasizing reproducibility, verification, and accountability. We followed this guidance when submitting 53 reports on AI-discovered bugs, of which 69.81% were accepted by maintainers as of writing.
Figures & tables
| Discovery Method | AI-disc. | AI-assist. | Suspect. | Other |
|---|---|---|---|---|
| Source Code Review | 51.52% | 6.66% | 11.46% | 5.62% |
| Static Analysis | 5.05% | 0.44% | 0.00% | 0.31% |
| Fuzzing | 4.29% | 0.44% | 0.00% | 0.67% |
| Dynamic Analysis | 0.25% | 0.24% | 2.08% | 0.60% |
| Testing | 3.79% | 4.23% | 5.21% | 4.70% |
| Build / Installation | 0.51% | 3.39% | 4.17% | 5.65% |
| AI Role | Quality | Missing Elements | Avg. Missing | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| High | Med. | Low | Summary | Comp. | Trigger | Repro. | Observed | Expected | Env. | ||
| Other | 33.77% | 51.91% | 14.32% | 9.69% | 7.60% | 22.09% | 37.72% | 16.39% | 42.77% | 29.19% | 1.65 |
| AI-disc. | 56.31% | 37.12% | 6.57% | 1.52% | 1.52% | 7.32% | 18.69% | 9.60% | 16.67% | 30.05% | 0.85 |
| AI-assist. | 50.00% | 42.86% | 7.14% | 3.11% | 2.23% | 9.81% | 26.36% | 6.22% | 27.19% | 21.33% | 0.96 |
| Suspect. | 45.83% | 42.71% | 11.46% | 4.17% | 3.12% | 13.54% | 30.21% | 9.38% | 29.17% | 28.12% | 1.18 |
| AI-invol. | 50.70% | 42.10% | 7.20% | 2.93% | 2.17% | 9.60% | 25.47% | 6.77% | 25.87% | 22.70% | 0.95 |
| Factor | Valid (%) | OR [95% CI] | |
| Main Model: Discovery Method ( ) | |||
| Operational Usage (ref.) | 45.43% | 1 | – |
| Source Code Review | 79.13% | 2.82 [2.26, 3.52] | |
| Fuzzing | 72.65% | 2.78 [1.43, 5.42] | 0.005 |
| Testing | 71.49% | 2.25 [1.93, 2.63] | |
| Dynamic Analysis | 65.55% | 1.98 [1.18, 3.34] | 0.019 |
| AI Role | Metric | Triage | Validation | Resolution | Total | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Info. | Dup. | Total | Repro. | Code | Total | Fix | Patch | Total | |||
| Other | Mean | 0.28 | 0.13 | 0.41 | 0.30 | 0.70 | 1.01 | 0.53 | 0.05 | 0.58 | 2.00 |
| AI-disc. | Mean | 0.06 | 0.06 | 0.12 | 0.10 | 0.37 | 0.47 | 0.29 | 0.05 | 0.34 | 0.93 |
| Adj. | −0.12 * | −0.03 | −0.15 * | −0.17 * | −0.27 | −0.44 * | −0.30 * | −0.03 | −0.34 * | −0.93 * | |
| AI-assist. | Mean | 0.32 | 0.11 | 0.44 | 0.34 | 0.73 | 1.06 | 0.61 | 0.07 | 0.68 | 2.18 |
| Adj. | +0.10 * | +0.01 | +0.11 * | +0.08 * | +0.15 * | +0.24 * | +0.09 | +0.02 | +0.11 | +0.46 * | |
| Material / Action | AI-Gen | Other | OR [95% CI] | |
|---|---|---|---|---|
| Reproducer | ||||
| Reject | 23.53% | 5.14% | 8.16 [3.15, 18.86] * | .001 |
| Req. supp. | 38.24% | 58.17% | 0.52 [0.24, 1.09] | 0.147 |
| Use | 38.24% | 27.47% | 1.31 [0.60, 2.76] | 0.650 |
| Technical claim | ||||
| Reject | 59.46% | 54.72% | 1.57 [1.11, 2.23] * | 0.034 |
| AI Role | Metric | AI-Involved Replies | Actions in AI-Involved Replies | ||||
|---|---|---|---|---|---|---|---|
| Any | AI Bot | Maint. AI | Triage | Validation | Resolution | ||
| Other | Value | 1.26 | 0.86 | 0.47 | 0.39 | 1.10 | 0.79 |
| AI-disc. | Value | 2.28 | 1.27 | 1.01 | 0.00 | 1.77 | 1.77 |
| Adj. | |||||||
| AI-assist. | Value | 4.89 | 2.42 | 3.04 | 1.15 | 4.44 | 3.41 |
| Adj. | |||||||
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| AI Role | Stat. | Valid | Invalid | Uncertain | Valid/Dec. |
|---|---|---|---|---|---|
| AI-disc. | Rate (%) | 25.51 | 12.12 | 62.37 | 67.79 |
| (pp) | |||||
| 95% CI | |||||
| 0.033 | 0.819 | 0.102 | 0.819 | ||
| AI-assist. | Rate (%) | 32.46 | 20.57 | 46.97 | 61.20 |
| (pp) |
| AI Role | Metric | Triage | Validation | Resolution | Total | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Info. | Dup. | Total | Repro. | Code | Total | Fix | Patch | Total | |||
| AI-disc. | 95% CI | [−0.18, −0.04] | [−0.08, 0.02] | [−0.24, −0.05] | [−0.27, −0.06] | [−0.53, 0.00] | [−0.79, −0.08] | [−0.48, −0.08] | [−0.08, 0.03] | [−0.53, −0.08] | [−1.54, −0.25] |
| -value | 0.018 | 0.322 | 0.018 | 0.018 | 0.090 | 0.040 | 0.018 | 0.322 | 0.018 | 0.018 | |
| AI-assist. | 95% CI | [0.05, 0.18] | [−0.01, 0.03] | [0.05, 0.20] | [0.04, 0.15] | [0.07, 0.29] | [0.11, 0.44] | [0.02, 0.20] | [0.00, 0.05] | [0.03, 0.25] | [0.21, 0.85] |
| -value | 0.018 | 0.598 | 0.018 | 0.018 | 0.023 | 0.018 | 0.085 | 0.218 | 0.085 | 0.018 | |
| Suspect. | 95% CI | [−0.19, 0.12] | [−0.13, 0.02] | [−0.28, 0.10] | [−0.25, 0.12] | [−0.36, 0.49] | [−0.57, 0.55] | [−0.42, 0.31] | [−0.06, 0.12] | [−0.46, 0.36] | [−1.18, 0.79] |
| Category | Definition |
|---|---|
| Discovery | Source Code Review . Discovered by inspecting, reviewing, or auditing source code directly, by a human, LLM/AI agent, or another code-review process; excludes automated static analyzers. |
| Static Analysis . Discovered by automated static analysis without executing the target, such as CodeQL, Infer, Coverity, Cppcheck, or static taint analysis. | |
| Fuzzing . Discovered by fuzzing or automatically generated/mutated inputs, such as AFL, libFuzzer, Syzkaller, or another fuzzing system. | |
| Dynamic Analysis . Discovered primarily through a runtime bug-detection tool such as AddressSanitizer, UBSan, TSan, MSan, Valgrind, or a similar dynamic analysis tool. | |
| Testing . Discovered while running unit, regression, self-, integration, or other deliberately designed tests. | |
| Build / Installation . Discovered during configuration, compilation, linking, packaging, building, or installation. |
AI-Assisted Software Development
AI-assisted software development uses language models and coding agents to help people design, write, test, review, and maintain software.
See all 117 papers