A Survey of Bugs in AI-Generated Code
Organizations: School of Mathematical and Computational Sciences, Massey University, New Zealand · School of Computer Science, Wuhan University, China · SWAT Laboratory, Polytechnique Montréal, Canada
Abstract
AI coding systems are increasingly used to help developers write code at scale, improving software development efficiency and boosting productivity. While these models and agents have shown strong performance, concerns persist regarding the quality of the AI-generated code. Although prior work has reported a range of quality issues associated with AI-generated code, these findings remain scattered and lack a systematic synthesis. A comprehensive review is lacking that demonstrates the bug types, patterns, distribution, their correlation with specific AI coding systems, and the remediation strategies proposed to address them. To fill this gap, we conduct a systematic literature review of existing studies on AI-generated code to develop a comprehensive understanding of bugs in the generated code and provide a foundation for future improvements in AI-assisted coding. Our results show that bugs are widespread in AI-generated code, with functional and stability bugs emerging as the most frequently reported categories across the reviewed studies, a substantial proportion of which is attributed to model hallucination. Model size, instruction-following capability, and the quality and diversity of training data are key factors influencing bug occurrence. To mitigate bugs, prompt engineering techniques are widely used; however, certain domain-specific or semantically complex issues remain difficult to resolve automatically and still require human intervention.
Figures & tables
| Component | Keywords and corresponding synonyms, abbreviations, and other alternatives |
| AI Tools | LLM, AI, Artificial Intelligence, automatic, LLMs, large language models, ChatGPT, Copilot |
| Generation Tasks | generated code, generated programs, code generation, code generated |
| Issue Focus | bug, buggy code, bug patterns, incorrect code, code quality issues, defects, fault, flaw |
| Search Engine | Search String |
| Scopus | TITLE-ABS-KEY ( ( “AI” OR “Artificial Intelligence” OR “automatic” OR “LLMs” OR “large language models” OR “ChatGPT” OR “Copilot” ) AND ( “generated programs” OR “code generation” OR “generated code” OR “code generated” ) AND ( “bug*” OR “incorrect code” OR “code quality” OR “defects” OR “fault” OR “flaw” ) ) AND PUBYEAR > 2019 AND PUBYEAR < 2027 AND ( LIMIT-TO ( DOCTYPE , “cp” ) OR LIMIT-TO ( DOCTYPE , “ar” ) ) |
| Google Scholar | “AI-generated code” OR “LLM-generated code” OR “large language models generated code” OR “ChatGPT-generated code” OR “Copilot-generated code” OR “buggy code” OR “bug patterns” OR “incorrect code” OR “code quality issues” OR “defects” OR “fault” OR “flaw” after:2019 |
| Bug Category | Bug Subcategory | ODC Bug Type |
| Functionality | Semantic/Logical Bugs | Algorithm |
| API/Package/Library-related Bugs | Interface | |
| Variable/Object/Parameter/Property-related Bugs | Assignment | |
| Type Bugs | Assignment | |
| Function/Method-related Bugs | Function | |
| Other Functionality Bugs | Function |
| Category | Studies | References |
| Functional Bugs | 106 | ( Tian et al., 2024 ; Barbàra et al., 2024 ; Wang et al., 2024a ; Zhong and Wang, 2024 ; Zhou et al., 2025 ; Baralla et al., 2024 ; Fan et al., 2023 ; Pan et al., 2025 ; Siddiq et al., 2024c ; Khan et al., 2024 ; Jesse et al., 2023 ; Haindl and Weinberger, 2024 ; Song et al., 2023 ; Corso et al., 2024 ; Tang et al., 2024a ; Spinellis, 2024 ; Wang et al., 2025c ; Karanjai et al., 2023 ; Yu et al., 2024 ; Tambon et al., 2025 ; Patel et al., 2024 ; Liu et al., 2024b ; Liu et al., 2024a ; Rabbi et al., 2024 ; Pan et al., 2024 ; Dong et al., 2024a ; Nguyen and Nadi, 2022 ; Yetistiren et al., 2022 ; Ságodi et al., 2024 ; Zheng et al., 2023 ; Mastropaolo et al., 2023 ; Billah et al., 2024 ; Cipriano and Alves, 2024 ; Ramos et al., 2024 ; Siddiq et al., 2022 ; Bai et al., 2025 ; Inala et al., 2022 ; Cassano et al., 2023 ; Zhang et al., 2023 ; Mathews and Nagappan, 2024 ; Ouyang et al., 2025 ; Chen and Huang, 2023 ; Ho et al., 2024 ; Weisz et al., 2022 ; Kou et al., 2024 ; Kabir et al., 2024 ; Fan et al., 2024 ; Kouemo Ngassom et al., 2024 ; Almanasra and Suwais, 2025 ; Omidvar Tehrani et al., 2024 ; Wu et al., 2023 ; Du et al., 2024 ; Tang et al., 2024b ; Feng et al., 2023 ; Jain et al., 2022 ; Zhang et al., 2024a ; Liu et al., 2023 ; Vieira et al., 2025 ; Du et al., 2025 ; Yu et al., 2025 ; Midolo and Tramontana, 2025 ; Ye et al., 2025 ; Zhu et al., 2026 ; Liu et al., 2026 ; Wang et al., 2026a ; Huang et al., 2026 ; Ma et al., 2026 ; Zhuo et al., 2026 ; Wang et al., 2026b ; Fayed and Fayed, 2026 ; Shehab et al., 2026 ; Swaraj and Kumar, 2026 ; Eagal et al., 2025 ; Liang et al., 2025 ; Bayram et al., 2025 ; Liu et al., 2025 ; Ron et al., 2026 ; Zhao et al., 2025 ; Nekrasov et al., 2026 ; Feischl and Kern, 2026 ; Bui et al., 2025 ; Pendyala and Thakur, 2025 ; Bui et al., 2025 ; Yang et al., 2025 ; Nogueira et al., 2025 ; Zhang et al., 2026b ; Afrin et al., 2025 ; Song et al., 2025 ; Honarvar et al., 2025 ; Olivieri et al., 2025 ; Wang, 2025 ; Nouri et al., 2025 ; Almanasra et al., 2026 ; Jahan and Wang, 2025 ; Khojah et al., 2025 ; Spracklen et al., 2025a ; Cotroneo et al., 2025 ; Abbassi et al., 2025 ; Yuen et al., 2025 ; Wang et al., 2025b ; Kabir and Hassan, 2025 ; Dolcetti et al., 2026 ; Dou et al., 2026a ; Calò et al., 2026 ; Bai et al., 2025 ; Sun et al., 2026 ) |
| Stability Bugs | 75 | ( Ren et al., 2023 ; Zhou et al., 2025 ; Fan et al., 2023 ; Pan et al., 2025 ; Siddiq et al., 2024c ; Khan et al., 2024 ; Haindl and Weinberger, 2024 ; Corso et al., 2024 ; Tang et al., 2024a ; Spinellis, 2024 ; Karanjai et al., 2023 ; Patel et al., 2024 ; Liu et al., 2024a ; Rabbi et al., 2024 ; Dong et al., 2024a ; Nguyen and Nadi, 2022 ; Ságodi et al., 2024 ; Billah et al., 2024 ; Cipriano and Alves, 2024 ; Siddiq et al., 2022 ; Zhang et al., 2024c ; Siddiq et al., 2024a ; Inala et al., 2022 ; Weisz et al., 2022 ; Wu et al., 2023 ; Feng et al., 2023 ; Cassano et al., 2023 ; Zhang et al., 2023 ; Mathews and Nagappan, 2024 ; Ho et al., 2024 ; Almanasra and Suwais, 2025 ; Du et al., 2024 ; Tang et al., 2024b ; Zhang et al., 2024a ; DePalma et al., 2024 ; Vieira et al., 2025 ; Du et al., 2025 ; Midolo and Tramontana, 2025 ; He et al., 2026 ; Liu et al., 2026 ; Wang et al., 2026a ; Huang et al., 2026 ; Ma et al., 2026 ; Zhuo et al., 2026 ; Wang et al., 2026b ; Fayed and Fayed, 2026 ; Shehab et al., 2026 ; Swaraj and Kumar, 2026 ; Eagal et al., 2025 ; Liang et al., 2025 ; Bayram et al., 2025 ; Ron et al., 2026 ; Feischl and Kern, 2026 ; Bui et al., 2025 ; Improta et al., 2025 ; Pendyala and Thakur, 2025 ; Bui et al., 2025 ; Zhang et al., 2026b ; Borg et al., 2026 ; Afrin et al., 2025 ; Song et al., 2025 ; Honarvar et al., 2025 ; Olivieri et al., 2025 ; Wang, 2025 ; Nouri et al., 2025 ; Almanasra et al., 2026 ; Li et al., 2026 ; Cotroneo et al., 2025 ; Abbassi et al., 2025 ; Yuen et al., 2025 ; Kabir and Hassan, 2025 ; Dolcetti et al., 2026 ; Dou et al., 2026a ; Bai et al., 2025 ; Sun et al., 2026 ) |
| Syntax Bugs | 61 | ( Tian et al., 2024 ; Wang et al., 2024a ; Zhou et al., 2025 ; Fan et al., 2023 ; Pan et al., 2025 ; Song et al., 2023 ; Corso et al., 2024 ; Spinellis, 2024 ; Wang et al., 2025c ; Tambon et al., 2025 ; Liu et al., 2024b ; Liu et al., 2024a ; Rabbi et al., 2024 ; Hendrycks et al., 2021 ; Pan et al., 2024 ; Dong et al., 2024a ; Ságodi et al., 2024 ; Zheng et al., 2023 ; Mastropaolo et al., 2023 ; Cipriano and Alves, 2024 ; Ramos et al., 2024 ; Siddiq et al., 2022 ; Fakih et al., 2024 ; Siddiq et al., 2024a ; Inala et al., 2022 ; Cassano et al., 2023 ; Zhang et al., 2023 ; Chen and Huang, 2023 ; Weisz et al., 2022 ; Kabir et al., 2024 ; Feng et al., 2023 ; Zhang et al., 2024a ; Du et al., 2025 ; Midolo and Tramontana, 2025 ; Ye et al., 2025 ; Wang et al., 2026a ; Ma et al., 2026 ; Wang et al., 2026b ; Fayed and Fayed, 2026 ; Shehab et al., 2026 ; Swaraj and Kumar, 2026 ; Liang et al., 2025 ; Bayram et al., 2025 ; Liu et al., 2025 ; Ron et al., 2026 ; Nekrasov et al., 2026 ; Feischl and Kern, 2026 ; Improta et al., 2025 ; Pendyala and Thakur, 2025 ; Nogueira et al., 2025 ; Afrin et al., 2025 ; Song et al., 2025 ; Honarvar et al., 2025 ; Wang, 2025 ; Nouri et al., 2025 ; Almanasra et al., 2026 ; Abbassi et al., 2025 ; Wang et al., 2025b ; Dolcetti et al., 2026 ; Dou et al., 2026a ; Bai et al., 2025 ) |
| System Bugs | 37 | ( Wang et al., 2024a ; Fan et al., 2023 ; Pan et al., 2025 ; Song et al., 2023 ; Wang et al., 2025c ; Liu et al., 2024b ; Liu et al., 2024a ; Yetistiren et al., 2022 ; Zheng et al., 2023 ; Billah et al., 2024 ; Inala et al., 2022 ; Cassano et al., 2023 ; Zhang et al., 2023 ; Mathews and Nagappan, 2024 ; Nettur et al., 2024 ; Ho et al., 2024 ; Almanasra and Suwais, 2025 ; Du et al., 2024 ; Zhang et al., 2024a ; Vieira et al., 2025 ; Wang et al., 2026a ; Nekrasov et al., 2026 ; Pendyala and Thakur, 2025 ; Yang et al., 2025 ; Nogueira et al., 2025 ; Afrin et al., 2025 ; Honarvar et al., 2025 ; Olivieri et al., 2025 ; Nouri et al., 2025 ; Almanasra et al., 2026 ; Cotroneo et al., 2025 ; Abbassi et al., 2025 ; Yuen et al., 2025 ; Kabir and Hassan, 2025 ; Dou et al., 2026a ) |
| Test Bugs | 12 | ( Wang et al., 2024a ; Tu et al., 2024 ; Karanjai et al., 2023 ; Dakhel et al., 2024 ; Zilberman and Cheng, 2024 ; Nettur et al., 2024 ; Siddiq et al., 2024b ; Yuan et al., 2024 ; Yang et al., 2024 ; Chu et al., 2025 ; Alves et al., 2025 ; Zhang et al., 2026a ) |
| Others/Unspecified Bugs | 10 | ( Zhou et al., 2025 ; Tang et al., 2024a ; Tambon et al., 2025 ; Pan et al., 2024 ; Inala et al., 2022 ; Cassano et al., 2023 ; Weisz et al., 2022 ; Liu et al., 2026 ; Swaraj and Kumar, 2026 ; Nouri et al., 2025 ) |
| Category | Studies | References |
| Prompt Engineering and Strategies | 45 | ( Tian et al., 2024 ; Ren et al., 2023 ; Moratis et al., 2024 ; Zhong and Wang, 2024 ; Khan et al., 2024 ; Jesse et al., 2023 ; Wang et al., 2025c ; Karanjai et al., 2023 ; Tambon et al., 2025 ; Liu et al., 2024b ; Liu et al., 2024a ; Pan et al., 2024 ; Billah et al., 2024 ; Zhang et al., 2024c ; Bai et al., 2025 ; Chen and Huang, 2023 ; Nettur et al., 2024 ; Yuan et al., 2024 ; Yang et al., 2024 ; Zhu et al., 2026 ; Liu et al., 2026 ; Wang et al., 2026a ; Fayed and Fayed, 2026 ; Liang et al., 2025 ; Feischl and Kern, 2026 ; Pendyala and Thakur, 2025 ; Yang et al., 2025 ; Nogueira et al., 2025 ; Zhang et al., 2026a ; Borg et al., 2026 ; Song et al., 2025 ; Wang, 2025 ; Li et al., 2026 ; Khojah et al., 2025 ; Spracklen et al., 2025a ; Yuen et al., 2025 ; Dolcetti et al., 2026 ; Dou et al., 2026a ; Sun et al., 2026 ; Wang et al., 2026b ; Nekrasov et al., 2026 ) |
| Code Enhancement Modules / Frameworks | 14 | ( Dakhel et al., 2024 ; Ramos et al., 2024 ; Siddiq et al., 2024a ; Inala et al., 2022 ; Zhang et al., 2023 ; Wu et al., 2023 ; Siddiq et al., 2024b ; Jain et al., 2022 ; Yang et al., 2024 ; Zhuo et al., 2026 ; Shehab et al., 2026 ; Liu et al., 2025 ; Ron et al., 2026 ; Bui et al., 2025 ) |
| Autonomous Coding Agents | 5 | ( Wang et al., 2024a ; Bai et al., 2025 ; Zhang et al., 2026b ; Bai et al., 2025 ; Wang et al., 2026b ) |
| Program-analysis-based Methods | 9 | ( Wang et al., 2024a ; Fan et al., 2023 ; Liu et al., 2024a ; Fakih et al., 2024 ; Mathews and Nagappan, 2024 ; Kouemo Ngassom et al., 2024 ; Yang et al., 2024 ; Huang et al., 2026 ; Nekrasov et al., 2026 ) |
| Mitigation Approaches for Task-Specific Code Generation | 8 | ( Karanjai et al., 2023 ; Chen and Huang, 2023 ; Yuan et al., 2024 ; Dakhel et al., 2024 ; Siddiq et al., 2024b ; Tang et al., 2024b ; Chu et al., 2025 ; Zhang et al., 2026a ) |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Studies | Datasets | Models/Tools | Findings |
| Zilberman et al. ( Zilberman and Cheng, 2024 ) | HumanEval, APPS, and ReCA | GPT-3, GPT-3.5, and GPT-4 | GPT-3 produced the highest proportion of functional programs (likely due to the simplicity of HumanEval), while GPT-4 improved over GPT-3.5. Under black-box settings, all models generated longer test suites with irrelevant statements, whereas white-box prompts improved coverage by providing more context. The authors also analyzed test smells, finding that Magic Number (Hard-coding specific values in test code without using constant names or explanations) and Assertion Roulette (multiple assertions in the same test with no description, which makes it difficult to determine which assertion statement is the cause of the test failure) are the most common. GPT-3 produced the fewest smells under white-box conditions, while GPT-3.5 and GPT-4 performed worse on APPS and ReCA. |
| Siddiq et al. ( Siddiq et al., 2024b ) | HumanEval | Codex (2K/4K), GPT-3.5-Turbo, and StarCoder | Less than half of their HumanEval unit tests compiled, though StarCoder reached 70%. On SF110, only % compiled. The most frequent bugs were unknown symbols, incompatible conversions, and abstract instantiation, with unknown symbols causing over 62% of the failures. For HumanEval, StarCoder produced the most correct tests, while GPT-3.5-Turbo generated the most tests with at least one passing case. Codex (4K) achieved the highest line and branch coverage, though still below manual tests. Familiar test smells included Magic Number and Assertion Roulette, AR, Lazy Tests, and Eager Tests; GPT-3.5-Turbo generated the fewest, while StarCoder’s tests almost always contained at least one smell. |
| Du et al. ( Du et al., 2024 ) | ClassEval: class-level code generation benchmark | Llama 3, Gemma, CodeLlama, CodeGemma, DeepSeek-V3, GPT-4o and Claude-3.5-sonnet | The authors’ analysis found that most incorrect code encountered AttributeError and TypeError , indicating that the model understood the error message and satisfied the syntactic or semantic constraints of the code context. Furthermore, in a few cases, KeyError was raised due to incorrect operations on dictionary variables. |
| Yang et al. ( Yang et al., 2024 ) | 17 Java projects | CodeLlama, ChatGPT and DeepSeekCoder | CodeLlama-7B-Instruct generated many duplicate tests, while CodeLlama-13B-Instruct produced fewer but not better results. Many generated unit tests were syntactically invalid, mainly due to unresolved symbols, parameter mismatches, and abstract instantiations - all linked to hallucinations. CodeLlama-7B/13B-Instruct mostly produced bugs from runtime exceptions, reflecting a weaker ability in generating correct assertions. |
| Alves et al. ( Alves et al., 2025 ) | HumanEval | GPT-4o, Amazon Q, and LLama 3.3 | GPT-4o exhibited the fewest errors, Amazon Q Developer the most, and LLaMA an intermediate level. Some errors were model-specific; for example, IndentationError and NameError occurred only in tests generated by Amazon Q Developer, whereas AssertionError was observed across all three LLMs. Amazon Q Developer exhibited the highest number and diversity of test smells, with some issues occurring exclusively in its generated tests. LLaMA exhibited only three types of test smells, while GPT-4o exhibited four, with a notable concentration of Suboptimal Assertion. |
| Studies | Prompting strategy | Models/Tools | Findings |
| Tambon et al. ( Tambon et al., 2025 ) | Docstring-based prompts and human-labeled prompts | CodeGen, PanGu-Coder, and Codex | For robust models like Codex, the most common bug pattern is Missing Corner Case. In contrast, for relatively weak and open-source models like PanGu-Coder or CodeGen, the most common category for both prompt types is Misinterpretation. Among all models, CodeGen has the highest number of bug samples in Silly Mistake and Syntax bug categories, which may be due to CodeGen’s tendency to regenerate code (including signatures) from scratch. Codex has the highest number of samples in the Non-Prompted Consideration category across all prompt types. For both prompt types, the second and third-most-common bug patterns in PanGu-Coder-generated code are Missing Corner Case and Hallucinated Object. For the code generated by PanGu-Coder, changing the prompt type reduces the occurrence of Hallucinated Object but increases the number of Prompt-biased code. Considering all three studied models together, Misinterpretation is the most common bug pattern, followed by Missing Corner Cases, Hallucinated Object, Incomplete Generation, and Wrong Attribute. The incorrect code generated with human-labeled prompts appears to contain fewer “trivial” bugs, such as Syntax Bugs and Silly Mistakes, than the code generated with docstring-based prompts. |
| Khojah et al. ( Khojah et al., 2025 ) | Few-shot, persona, chain-of-thought, function signature, list of packages | GPT-4o, Llama3, Mistral | Including a function signature or few-shot examples significantly improves code correctness and reduces common errors, particularly TypeError, AttributeError, and ImportError. Providing package information further decreases import-related errors, although it may introduce hallucinated imports. Combining multiple prompt techniques offers little additional benefit. Llama 3 tends to generate simpler solutions, while signatures and few-shot examples often produce more complex code with additional code smells. Adding a relevant persona slightly improves code quality but comes with a small reduction in correctness. |
| Yuen et al. ( Yuen et al., 2025 ) | Zero-Shot, Few-Shot, Chain-of-Thought, and Chain-of-Thought-Few-Shot | GPT-4o, Llama3-70B | CoT and CoT Few-Shot consistently achieved higher correctness for both Llama 3 and GPT-4o than Zero-Shot prompting. They also generated code that was more similar to human-written code while maintaining comparable complexity and structure. In contrast, Zero-Shot prompting produced substantially more errors across both models, with Logic and Control Flow errors being the most prevalent error type across prompting strategies. |
LLM Code Generation
Large language model code generation studies how models translate instructions and context into executable programs and how to make them reliable.
See all 332 papers