Investigating The Smells of LLM Generated Code

Debalina Ghosh Paul; Hong Zhu; Ian Bayley

cs.SEOct 3, 2025

Investigating The Smells of LLM Generated Code

Authors: Debalina Ghosh Paul, Hong Zhu, Ian Bayley

Organizations: School of Engineering, Computing and Mathematics, Oxford Brookes University, Oxford, OX3 0BP, UK

Abstract

Context: Large Language Models (LLMs) are increasingly being used to generate program code. Much research has been reported on the functional correctness of generated code, but there is far less on code quality. Objectives: In this study, we propose a scenario-based method of evaluating the quality of LLM-generated code to identify the weakest scenarios, for which the quality of LLM-generated code should be improved. Methods: The method measures code smells, an important indicator of code quality, and compares them with a baseline formed from reference solutions of professionally written code. The test dataset is divided into various subsets according to the topics of the code and complexity of the coding tasks to represent different scenarios of using LLMs for code generation. We will also present an automated test system for this purpose and report experiments with the Java programs generated in response to prompts given to four state-of-the-art LLMs: Gemini Pro, ChatGPT, Codex, and Falcon. Results: We find that LLM-generated code has a higher incidence of code smells compared to reference solutions. Falcon performed the least badly, with a smell increase of 42.28%, followed by Gemini Pro (62.07%), ChatGPT (65.05%) and finally Codex (84.97%). The average smell increase across all LLMs was 63.34%, comprising 73.35% for implementation smells and 21.42% for design smells. We also found that the increase in code smells is greater for more complex coding tasks and for more advanced topics, such as those involving object-orientated concepts. Conclusion: In terms of code smells, LLMs' performances on various coding task complexities and topics are highly correlated to the quality of human-written code in the corresponding scenarios. However, the quality of LLM-generated code is noticeably poorer than human-written code.

Figures & tables

Work	Aims	Tools	Language	Usage	Dataset (Size)	LLM	Smell Types
Siddiq, et al. 2022	Investigating the impact of code smells in training datasets on the quality of generated code	Pylint	Python	Detect code smells in training datasets and generated codes	HumanEval (164)	GPT-Neo, GitHub Copilot	Implementation smells
		Bandit		Detect security smells in training datasets and generated codes			Security smells
Moratis, et al. 2024	Assessing the quality of code generated via iterative conversations in the Write this code scenario	PMD	JavaScript	Detect code smell and measure code quality	DevGPT (47)	ChatGPT	Best practice, code style, error prone
	Assessing the quality improvement of code generated via iterative conversations in the Improve this code scenario			Comparing code smells before and after refactoring	DevGPT (334)
DePalma, et al. 2024	Evaluating LLM capability for code refactoring	PMD	Java	Assessing the quality of the code before and after refactoring	Ma et al. [ 29 ] (40)	ChatGPT	Best practice, code style, design, documentation, error prone, multi-threading, performance, security
Liu, at el. 2024	Evaluating LLM capability for fixing code quality issues	PMD, CheckStyle	Java	Assessing the quality of generated code	LMDefect + (2033)	ChatGPT	Implementation and design smells

Table 1 : Summary of Related Works

Smell Name	Detection Rule(s)	Tool(s) Used
Inconsistent Naming Convention	Local Variable Naming Convention	PMD
	/Local Variable Name	/CheckStyle ${}^{(\ref{ftn:OnCheckStyle})}$
	Formal Parameter Naming Convention	PMD
	Method Naming Convention	PMD
	/Method Name	/CheckStyle ${}^{(\ref{ftn:OnCheckStyle})}$
	Class Naming Convention	PMD

Table 2 : Smell Detection Rules Used for Implementation Smells

Smell Name	Detection Rules	Tool Used
Modularity	God Class	PMD
	Data Class	PMD
	Too Many Methods	PMD
	Too Many Fields	PMD
	Use Utility Class	PMD
	Hide Utility Class Constructor	CheckStyle

Table 3 : Smell Detection Rules Used for Design Smells

Figure 1 : Structure of The Test System

Name	Year	Version	Size
Gemini Pro	2023	Gemini Pro 1.0	Unknown
Falcon	2023	Falcon-7B	7B
ChatGPT	2023	GPT-3.5-turbo	Unknown
Codex	2021	GPT-3 (Codex)	12B

Table 4 : Large Language Models Evaluated

Feature	#Textbook Tasks	#Real Tasks
# Topics	25	18
# Tasks per Topic (average)	20.00	27.78
# Tasks per Topic (max)	79	61
# Tasks per Topic (min)	8	5
Input Length (average)	18.55	21.54
Input Length (max)	35	31

Table 5 : Statistical Characteristics of the Test Dataset

Figure 2 : The Experiment Setup

LLM Model	Avg VS	Inc.Rate (%)	MoE(%)	p-value
Reference	19.378	N/A	6.045	N/A
Falcon	27.571	42.28%	4.654	1.0749E-19
Gemini Pro	31.406	62.07%	4.284	6.0923E-38
ChatGPT	31.789	64.05%	4.029	3.0557E-42
Codex	35.844	84.97%	3.708	3.9380E-68

Table 6 : Overall Code Smells in LLM-generated Code

Figure 3 : All Code Smells Detected on the Whole Test Dataset

Table 7 : Code Smell by Code Topics

Model	Best Topics (VS)	Worst Topics (VS)	Most Improved Toipics (Inc %)	Most Worsened Topics (Inc %)
Baseline	Basic Exercise (1.96)	Searching & Sorting (5.67)	N/A	N/A
	DateTime (2.18)	Polymorphism (5.72)	N/A	N/A
	String (2.23)	Inheritance (6.91)	N/A	N/A
Falcon	Basic Exercise (1.99)	Polymorphism (6.91)	Interfaces (-23.34)	Array (58.16)
	Lambda (2.41)	Inheritance (7.59)	Collections (-12.03)	OOP (96.78)
	Collections (2.56)	Encapsulation (9.91)	Lambda (-11.07)	Encapsulation (172.25)

Table 8 : The Best/Worst Topics And The Most Improved/Worsened Topics

Complexity	Baseline	Falcon	GeminiPro	ChatGPT	Codex	Average
Cyclomatic	0.9344	0.9555	0.9916	0.9485	0.9962	0.9653
Cognitive	0.8800	0.9754	0.6312	0.3025	0.6578	0.6894
Lines of Code	0.9952	0.8694	0.9532	0.6347	0.8900	0.8685

Table 9 : Correlations Bwt VS and Coding Task Complexities

Figure 4 : Variation of Code Smells by Complexity

Figure 5 : Increase Rates of Code Smells by Complexity

Table 10 : Code Smells by Smell Type

Table 11 : Pearson Correlation Coefficients btw Baseline and LLMs’ VS Scores on Various Smells Types

Table 12 : Pearson Correlation Coefficients btw Baseline VS Scores and LLMs’ Increase Rates on Various Smells Types

Table 13 : Pearson Correlation Coefficients btw LLMs’ Increase Rates of Implementation Smells

Table 14 : The Most Prevalent Smells in Each Type

Figure 6 : Numbers of LLM-generated Solutions Passed Test

LLM	Avg of Code Smells		Increase	Margin of Error (%)		p-value
LLM	Correct	Incorrect	Rate (%)	Correct	Incorrect	of T-Test
ChatGPT	31.3279	32.9344	5.1278	5.0862	6.5124	0.2391
Gemini Pro	30.6299	33.8833	10.6216	5.0835	7.8972	0.0400
Falcon	32.0511	23.6830	-26.1085	6.5659	6.2138	0.0000
Codex	35.6416	36.4379	2.2342	4.5385	6.4342	0.5839

Table 15 : Smells of Correct and Incorrect Codes

Explore similar work

LLM Code Smells: A Taxonomy and Detection Approach
May 21, 2026Zacharie Chenail-Larcher, Brahim Mahmoudi, Naouel Moha +2Code Quality LLM Inference Optimization
Ecole de technologie sup´erieure, Montr´eal, Qu´ebec, Canada. · Universit´e du Qu´ebec `a Montr´eal, Montr´eal, Qu´ebec, Canada.
The Readability Spectrum: Patterns, Issues, and Prompt Effects in LLM-Generated Code
May 13, 2026Hengzhi Ye, Fengyuan Ran, Weiwei Xu +1Code Generation Readability
Peking university · Beijing, Beijing, China · Wuhan University +1
Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code
May 6, 2026Kaifeng He, Xiaojun Zhang, Peiliang Cai +7Code Generation Code Quality
Sun Yat-sen University, China · Nanyang Technological University, Singapore, Singapore · Tongji University, Shanghai, China +1

May 21, 2026, cs.SEJ/K move · Enter open · S save

LLM Code Smells: A Taxonomy and Detection Approach

Zacharie Chenail-Larcher, Brahim Mahmoudi, Naouel Moha, Quentin Stiévenart+1

Ecole de technologie sup´erieure, Montr´eal, Qu´ebec, Canada. · Universit´e du Qu´ebec `a Montr´eal, Montr´eal, Qu´ebec, Canada.

Code Quality LLM Inference Optimization Empirical Study Taxonomy

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly integrated into software systems for diverse purposes, due to their versatility, flexibility, and ability to simulate human reasoning to some extent. However, poor integration of LLM inference in source code can undermine software system quality. Therefore, inadequate LLM integration coding practices must be documented to help developers mitigate such issues. Following our earlier work on LLM code smells, this paper consolidates and refines the concept by presenting a self-contained taxonomy and a catalog of nine LLM code smells. We also create SpecDetect4LLM, a static source code analysis tool for their detection, and conduct extensive empirical evaluations of its detection effectiveness (precision and recall) as well as the prevalence of LLM code smells across 692 open-source software projects (171,194 source files). Our results show that LLM code smells affect 73.5% of the analyzed systems, with a detection precision of 91.3% and a recall of 71.8%.

Abstract

Explore similar work

LLM Code Smells: A Taxonomy and Detection Approach

The Readability Spectrum: Patterns, Issues, and Prompt Effects in LLM-Generated Code

Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code