Operational Robustness of LLMs on Code Generation

Debalina Ghosh Paul; Hong Zhu; Ian Bayley

cs.SEFeb 21, 2026

Operational Robustness of LLMs on Code Generation

Authors: Debalina Ghosh Paul, Hong Zhu, Ian Bayley

Organizations: School of Engineering, Computing and Mathematics, Oxford Brookes University Oxford, OX3 0BP, United Kingdom

Abstract

It is now common practice in software development for large language models (LLMs) to be used to generate program code. It is desirable to evaluate the robustness of LLMs for this usage. This paper is concerned in particular with how sensitive LLMs are to variations in descriptions of the coding tasks. However, existing techniques for evaluating this robustness are unsuitable for code generation because the input data space of natural language descriptions is discrete. To address this problem, we propose a robustness evaluation method called scenario domain analysis, which aims to find the expected minimal change in the natural language descriptions of coding tasks that would cause the LLMs to produce incorrect outputs. We have formally proved the theoretical properties of the method and also conducted extensive experiments to evaluate the robustness of four state-of-the-art art LLMs: Gemini-pro, Codex, Llamma2 and Falcon 7B, and have found that we are able to rank these with confidence from best to worst. Moreover, we have also studied how robustness varies in different scenarios, including the variations with the topic of the coding task and with the complexity of its sample solution, and found that robustness is lower for more complex tasks and also lower for more advanced topics, such as multi-threading and data structures.

Figures & tables

Rank ( $n$ )	write	a	program	to	replace	specified	character	with	another
1	writing	another	programs	take	replacing	applicable	characters	and	one
2	read	an	project	would	replacement	parameter	protagonist	while	a
3	publish	one	funding	instead	replaced	specifying	portrayed	made	an
4	notes	same	programme	could	replaces	depending	portrays	both	same

Table 1: Examples of The Neighbourhoods of Words

Original : Write a Java program to replace a specified character with another character.
Paraphrase	$k$	$n$
Writing a Java program to replace a specified character with another character.	1	1
Write a Java program to replace a applicable character with another character.	1	1
Write a Java program to replace a specified character with one character.	1	1
Write a Java program to replace a specified character with a character.	1	2
Write a Java program to replace a parameter character with another character.	1	2

Table 2: Examples of Paraphrases

Figure 1: Structure of Test System for Evaluation of Robustness

Metric	Implementation
RoBERTa	Query the STSb-RoBERTa-base ML model installed on local machine, which is downloaded from Hugging Face
Euclidean	Invoke the euclidean function of the scipy.spatial.distance module
Cosine	Invoke the cosine_similarity function of the sklearn.metrics.pairwise module
BLEU	Invoke the corpus_bleu function in the sacreBLEU library, with parameter $n$ = 4
ROUGE-N	Use the RougeScorer class of the rouge_score library
ROUGE-L	Use the RougeScorer class of the rouge_score library

Table 3: Implementation of Similarity Metrics

Feature	#Textbook Tasks	#Real Tasks
Number of Topics Covered	9	7
Average Num of Tasks in A Topic	66.67	42.85
Max Num of Tasks in A Topic	156	61
Min Num of Tasks in A Topic	8	5
Average Complexity	3.448	3.200
Max Complexity	13	6

Table 4: Statistical Characteristics of the Test Dataset

D_{1}

Similarity Metric	Gemini-Pro		Codex		Llama2		Falcon
Similarity Metric	$R^{o}$	$R^{*}$	$R^{o}$	$R^{*}$	$R^{o}$	$R^{*}$	$R^{o}$	$R^{*}$
RoBERTa	1.0365	1.1117	1.0879	1.1432	1.1945	1.2478	1.2046	1.2550
Euclidean	0.5307	0.4991	0.5103	0.4814	0.4021	0.3748	0.4065	0.3781
Cosine Similarity	0.9291	0.9302	0.9301	0.9381	0.9801	0.9811	0.9703	0.9712
BLEU	0.4273	0.4565	0.4164	0.4388	0.5336	0.5628	0.5295	0.5583
ROUGE-N	0.6224	0.6428	0.6354	0.6587	0.7414	0.7662	0.7381	0.7606

Table 5: Robustness Measured by

R^{o}

and

R^{*}

with Different Similarity Metrics on Paraphrases

Panel of Figure 3 — Figure 3: Comparison of LLMs on Robustness of Code Generation

Panel of Figure 4 — Figure 4: Variation of Robustness with Cyclomatic Complexity

Similarity Metrics	Codex	GeminiPro	Falcon	Llama2
RoBERTa	0.9924	0.9594	0.9368	0.9404
BLEU	0.8577	0.8860	0.8989	0.9007
ChrF	0.8287	0.7063	0.8950	0.8796
Cosine	0.7661	0.7610	0.7198	0.7706
Meteor	0.8171	0.8057	0.8188	0.8116
RougeL	0.8483	0.8838	0.8477	0.8498

Table 6: Pearson Correlation Coefficients btw Cyclomatic Complexity and Robustness Measure

R^{o}

Figure 5: Variation of Robustness with Topics

Figure 6: Differences btw the Edit Distances to the Original Description before and after the Tipping Points

Metric	Codex		GeminiPro		Llama2		Falcon
Metric	$n$	$k$	$n$	$k$	$n$	$k$	$n$	$k$
RoBERTa	3.67	2.03	3.86	2.73	2.65	2.43	2.67	2.03
ChrF	2.67	2.62	2.68	2.03	2.46	2.11	2.64	2.22
Euclidean	2.76	2.73	2.67	2.48	2.47	2.32	2.68	2.03
Cosine	2.86	2.23	2.67	2.33	2.48	2.41	2.38	2.33
BLEU	2.63	2.09	2.22	2.63	2.63	2.03	2.63	2.04

Table 7: Average Values of

n

and

k

of the First Failure Paraphrases

Metric	GeminiPro	Codex	Falcon	Llama2
BLEU	14.82	14.13	13.79	13.65
ChrF	16.02	15.34	14.97	14.85
Levenshtein-Char	16.63	15.96	15.56	15.44
Levenshtein-Word	14.98	14.35	13.97	13.87
ROUGE-N	15.27	14.59	14.23	14.08
ROUGE-L	15.46	14.45	14.09	13.94

Table 8: Average Number of LLM Queries Per Test Case

Figure 7: Distribution of the Number of Generated Paraphrases

Metric	Average	StDev	Max	Min
BLEU	0.2399	0.0078	0.2922	0.2304
ChrF	0.2489	0.0078	0.297	0.2394
Levenshtein-Char	0.4730	0.0079	0.5229	0.4634
Levenshtein-Word	0.4799	0.0079	0.5248	0.4704
RougeN	0.8812	0.0082	0.9333	0.8714
RougeL	0.9119	0.0078	0.9629	0.9024

Table 9: Times to Calculate the Distance bwt The Original Task Description And A Paraphrase

Metric	Uniqueness (%)	Distinctness	Differentness
RoBERTa	100.00	1.0000	0.2334
ChrF	93.22	0.9655	0.2169
Euclidean	73.94	0.8721	0.2010
Cosine	35.35	0.6104	0.0789
BLEU	14.45	0.3242	0.1291
METEOR	0.00	0.0804	0.1454

Table 10: Comparison of Distance Metrics

Similarity Metric	GeminiPro		Codex		Llama2		Falcon		Average
Similarity Metric	Difference	Ratio	Difference	Ratio	Difference	Ratio	Difference	Ratio	Difference	Ratio
RoBERTa	0.0752	0.0700	0.0553	0.0496	0.0533	0.0436	0.0504	0.0410	0.0586	0.0511
Euclidean	0.0316	0.0614	0.0289	0.0583	0.0273	0.0703	0.0284	0.0724	0.0291	0.0656
Cosine Similarity	0.0011	0.0012	0.0080	0.0086	0.0010	0.0010	0.0009	0.0009	0.0027	0.0029
BLEU	0.0292	0.0661	0.0224	0.0524	0.0292	0.0533	0.0288	0.0530	0.0274	0.0562
ROUGE-N	0.0204	0.0322	0.0233	0.0360	0.0248	0.0329	0.0225	0.0300	0.0228	0.0328

Table 11: Absolute Differences between

R^{o}

and

R^{*}

and The Ratios to Their Means

Explore similar work

Investigating The Smells of LLM Generated Code
Oct 3, 2025Debalina Ghosh Paul, Hong Zhu, Ian BayleySoftware Engineering Code Generation Evaluation
School of Engineering, Computing and Mathematics, Oxford Brookes University, Oxford, OX3 0BP, UK
Characterizing Overconfident Failure in LLM-Based Code Generation
Oct 8, 2026Ravishka Rathnasuriya, Wei YangLLM Code Generation Confidence Estimation in Language Models
Institute of Systems for Advanced Computing, Fudan University, Shanghai, China
Code Is More Than Text: Uncertainty Estimation for Code Generation
Jun 8, 2026Yuling Shi, Caiqi Zhang, Yuexian Li +4Confidence Estimation in Language Models LLM Reliability
Shanghai Jiao Tong University · University of Cambridge

Oct 3, 2025J/K move · Enter open · S save

Investigating The Smells of LLM Generated Code

Debalina Ghosh Paul, Hong Zhu, Ian Bayley

School of Engineering, Computing and Mathematics, Oxford Brookes University, Oxford, OX3 0BP, UK

Software Engineering Code Generation Evaluation LLM Code Generation

Open paper AskPDF arXiv

Context: Large Language Models (LLMs) are increasingly being used to generate program code. Much research has been reported on the functional correctness of generated code, but there is far less on code quality. Objectives: In this study, we propose a scenario-based method of evaluating the quality of LLM-generated code to identify the weakest scenarios, for which the quality of LLM-generated code should be improved. Methods: The method measures code smells, an important indicator of code quality, and compares them with a baseline formed from reference solutions of professionally written code. The test dataset is divided into various subsets according to the topics of the code and complexity of the coding tasks to represent different scenarios of using LLMs for code generation. We will also present an automated test system for this purpose and report experiments with the Java programs generated in response to prompts given to four state-of-the-art LLMs: Gemini Pro, ChatGPT, Codex, and Falcon. Results: We find that LLM-generated code has a higher incidence of code smells compared to reference solutions. Falcon performed the least badly, with a smell increase of 42.28%, followed by Gemini Pro (62.07%), ChatGPT (65.05%) and finally Codex (84.97%). The average smell increase across all LLMs was 63.34%, comprising 73.35% for implementation smells and 21.42% for design smells. We also found that the increase in code smells is greater for more complex coding tasks and for more advanced topics, such as those involving object-orientated concepts. Conclusion: In terms of code smells, LLMs' performances on various coding task complexities and topics are highly correlated to the quality of human-written code in the corresponding scenarios. However, the quality of LLM-generated code is noticeably poorer than human-written code.

Abstract

Explore similar work

Investigating The Smells of LLM Generated Code

Characterizing Overconfident Failure in LLM-Based Code Generation

Code Is More Than Text: Uncertainty Estimation for Code Generation