Beyond Passing Tests: A Systematic Review and Controlled Re-evaluation of Techniques for Improving the Quality of LLM-Generated Code

Prashank Kadam

cs.SEOct 28, 2026

Beyond Passing Tests: A Systematic Review and Controlled Re-evaluation of Techniques for Improving the Quality of LLM-Generated Code

Authors: Prashank Kadam

Organizations: Cerby Inc., USA

Abstract

Large language models pass most functional-correctness benchmarks, but passing tests says little about whether their code is readable, efficient or secure. Many techniques claim to improve these qualities, yet their evidence is scattered, usually covers one quality dimension and is rarely compared at an equal compute budget. We contribute a systematic review and a controlled re-evaluation. A systematic literature review of 628 primary studies (July 2021 to October 2026), with disclosed, author-reviewed LLM assistance, organizes the techniques by stage, signal source and quality target. Most studies measure only the dimension they target, gains often come with unchanged or unreported correctness, single automated quality signals are gamed, and compute, variance and contamination are rarely reported. A controlled re-evaluation runs ten techniques on four models at matched budgets on correctness, maintainability, efficiency and security benchmarks, extends it to thirteen current models and repository-level tasks, and compares 66 highlighted techniques head-to-head on Gemma 4 31B. At one call, quality prompting raises analyzer issue counts and leaves complexity and judge scores almost unchanged. At matched budgets, best-of-n sampling with an analyzer-weighted ranker matches or beats prompting, planning, self-refinement, a reviewer agent and retrieval on measures the ranker does not optimize, while an analyzer feedback loop and fine-tuning beat it only on analyzer issue counts. Large correctness losses on HumanEval+, a likely contaminated benchmark near its ceiling, do not appear on contests held after the models' training cutoffs. In the head-to-head, 1 of 66 techniques beats best-of-n on its primary target beyond the 95% interval, a rate consistent with chance. Gains should be reported against a budget-matched sampling baseline and on measures independent of the technique's own signal.

Figures & tables

Panel of Figure 1 — Figure 1. PRISMA 2020 flow diagram. Screening, full-text assessment and extraction were LLM-assisted and author-reviewed. Flow diagram of the review: records identified by database searches, other methods and snowballing, then screened, sought for retrieval, assessed for eligibility and included, with the number excluded at each step.

Family	Stage	Typical signal sources	Section
Prompting	pre-generation	none; self (planning, chain of thought)	§ 8
Tool feedback	post-generation loop	static analyzer, compiler, tests, profiler	§ 9
Self-refinement	post-generation loop	self; tests (self-debugging)	§ 10
Multi-agent	post-generation loop	other LLM; tools invoked by agents	§ 11
Retrieval	pre-generation	retrieved knowledge	§ 12
Decoding	in-decoding	grammar or type constraints, learned verifier, tests	§ 13

Table 1. How the conventional families map onto the facets. Typical values only; individual techniques are classified one by one in the extraction data.

Figure 2. Faceted taxonomy: number of included studies by stage (columns) and signal source (rows), one panel per targeted quality dimension. A study counts once per combination it uses. Counts from LLM-assisted extraction reviewed by the author. Grids of study counts, one panel per quality dimension, with technique stages as columns and signal sources as rows.

Panel of Figure 3 — Figure 3. Evidence maps. Left: technique family $\times$ targeted dimension; each cell gives the number of studies and is shaded by their mean QA score (0–10). Right: signal source $\times$ targeted dimension (number of studies). Empty and thin cells are the gaps discussed in § 17 and § 19 . Two heat maps of study counts: technique family by targeted dimension, shaded by mean quality-assessment score, and signal source by targeted dimension.

(a) Model (abbr.)	Platform; architecture	Sampling control	Role
Claude Opus 5.5 (Op)	Claude API; closed	effort only; always thinks	strong anchor
gpt-oss-120b (oss)	Fireworks serverless; MoE	temperature, seed, effort	weak anchor
Gemma 4 31B IT (G31)	Fireworks dedicated; dense	temperature, seed	dense arm, tuned
Gemma 4 26B-A4B IT (G26)	Fireworks dedicated; MoE	temperature, seed	MoE arm, tuned
GPT-6.1 Sol, Gemini 4 Argon	Vercel, one pinned provider; closed	native effort only	extended grid
Kimi K3, GLM-5.3, Qwen3.8-Max, MiMo-V2.6-Pro, DeepSeek V4.1-Flash, V4-Flash, MiniMax M3	Fireworks; MoE	temperature, seed	extended grid

Table 2. E1. Setup. (a) Models, (b) benchmarks, (c) techniques with family, stage and signal source, and their use of the call budget

B

	Op			oss			G31			G26
Technique	HE+	LCB	LCBP	HE+	LCB	LCBP	HE+	LCB	LCBP	HE+	LCB	LCBP
T0 single shot	96.40	–	69.82	81.27	61.61	21.94	96.06	69.91	27.68	89.16	77.10	22.99
T1 quality instr.	99.10	–	68.71	83.17	59.64	22.12	97.96	80.71	28.07	89.68	80.97	23.61
T2 SCoT	91.41	–	69.77	85.31	60.38	22.49	85.56	84.17	30.60	89.80	82.34	24.51
T3 analyzer loop	90.23	–	72.17	84.86	58.99	22.71	86.12	79.48	28.48	84.11	73.91	22.34
T4 Self-Refine	89.50	–	67.93	81.58	63.57	24.14	84.98	78.90	26.16	91.25	80.99	23.16

Table 3. E2. Correctness ( COR ), pass@1 (%) at

B=1

(T0) and

B=4

(others). HE+ = HumanEval+ (likely contaminated anchor); LCB = LiveCodeBench release v6, problems up to April 2025, reported only for models whose training cutoff precedes it; LCBP = LiveCodeBench Pro protocol on Codeforces contests held after 2026-07-01, pass@1 over all difficulty tiers.

Model	T0	T1	T2	T3	T4	T5	T6	T7	T8	T9	T10
Claude Opus 5.5	75.17	71.78	68.18	66.24	68.48	76.55	67.03	71.88	–	–	–
GPT-6.1 Sol	76.59	76.49	75.29	73.59	77.19	80.25	72.18	79.90	–	–	–
Gemini 4 Argon	76.38	75.86	75.85	71.15	76.43	78.05	73.71	80.26	–	–	–
Kimi K3	65.47	70.80	66.63	62.62	68.58	68.28	64.85	70.68	–	–	–
GLM-5.3	71.95	72.14	66.92	67.83	69.27	72.77	67.54	73.10	–	–	–
Qwen3.8-Max	59.19	60.92	54.32	56.61	56.91	56.27	55.27	58.43	–	–	–

Table 4. E2b. Repository-level correctness ( COR ): DeepSWE v1.1 pass@1 (%) ( Huang et al., 2026d ) , at

B=1

(T0) and

B=4

(others), for all thirteen models of the main and extended grids (§ 16.2 );

B

counts complete agent runs on the mini-SWE-agent harness. Technique labels as in Table 2 c. Each technique acts on the agent: T1 and T2 change its task prompt, T3–T5 add analyzer, self-critique or reviewer feedback between runs, T6 retrieves into its context, T7 keeps the best of four runs by the T7 ranker on the repository’s visible tests, and T8–T10 use the fine-tuned Gemma weights.

(a) API-served models
	Op				oss
Technique	C%	Lint	CC	J	C%	Lint	CC	J
T0 single shot	86.99	21.37	7.68	4.30	85.73	29.22	10.72	3.55
T1 quality instr.	82.76	20.01	7.53	4.42	79.20	27.14	10.25	3.73
T2 SCoT	79.78	18.71	7.32	4.59	75.11	26.97	9.97	4.01
T3 analyzer loop	38.35	7.81	7.75	4.53	38.31	11.31	9.79	3.75

Table 5. E3. Readability and maintainability ( R/M ),

B=4

(T0:

B=1

). C% = share of PythonSecurityEval solutions with at least one maintainability issue (lower is better); Lint = maintainability issues per kLOC on RACE R/M and NoFunEval (lower is better); CC = mean cognitive complexity per function (lower is better); J = judge score (Claude Opus 5.5), 1–5 (higher is better); the Op column’s J scores the judge’s own model and may include self-preference (§ 20 ).

(a) API-served models
	Op			oss
Technique	DPS	ET	Bey	DPS	ET	Bey
T0 single shot	78.06	47.79	75.03	79.41	61.33	69.70
T1 quality instr.	79.90	49.70	76.48	75.01	62.24	70.29
T2 SCoT	79.20	48.64	79.38	75.11	63.71	74.18
T3 analyzer loop	75.46	44.11	79.72	72.17	60.53	70.55

Table 6. E4. Efficiency ( EFF ),

B=4

(T0:

B=1

). DPS = EvalPerf DPS

{}_{\text{norm}}

from CPU instruction counts; ET = EffiBench-X execution-time score relative to the human-expert solution, averaged over six languages (%); Bey = Mercury Beyond (%). Higher is better for all three.

(a) API-served models
	Op				oss
Technique	FS	SP	Ins	B/k	FS	SP	Ins	B/k
T0 single shot	74.85	58.54	42.67	3.40	45.02	26.38	44.93	5.14
T1 quality instr.	66.87	55.13	37.80	3.34	46.07	28.61	44.79	4.96
T2 SCoT	74.30	54.74	40.46	3.37	46.68	27.47	46.72	5.14
T3 analyzer loop	79.89	61.53	13.79	1.20	53.07	30.47	16.07	1.84

Table 7. E5. Security ( SEC ),

B=4

(T0:

B=1

). FS = CWEval func-sec@1 (%, higher is better); SP = BaxBench sec_pass@1, secure and correct (%, higher is better); Ins = share of PythonSecurityEval solutions with at least one security issue (lower is better); B/k = security issues per kLOC on PythonSecurityEval (lower is better).

	(a) better/worse than T0				(b) better/worse than T7
Technique	COR	R/M	EFF	SEC	COR	R/M	EFF	SEC
T1 quality instructions	2/0 (15)	$\star$ 6/1 (16)	$\star$ 0/3 (12)	$\star$ 4/1 (16)	2/1 (15)	0/13 (16)	3/2 (12)	0/12 (16)
T2 SCoT / plan first	$\star$ 1/3 (15)	$\star$ 15/0 (16)	4/2 (12)	0/0 (16)	1/3 (15)	3/11 (16)	4/1 (12)	0/13 (16)
T3 analyzer loop	1/4 (15)	$\star$ 12/1 (16)	1/2 (12)	$\star$ 10/0 (16)	0/3 (15)	9/5 (16)	2/2 (12)	8/0 (16)
T4 Self-Refine	$\star$ 1/3 (15)	$\star$ 8/0 (16)	$\star$ 3/3 (12)	$\star$ 7/0 (16)	1/2 (15)	1/12 (16)	2/2 (12)	0/10 (16)
T5 coder + reviewer	$\star$ 2/1 (15)	$\star$ 15/0 (16)	$\star$ 2/2 (12)	$\star$ 7/0 (16)	1/2 (15)	3/9 (16)	3/0 (12)	0/7 (16)

Table 8. E6. Trade-off matrix. For each technique and dimension, the number of metric–model cells of Tables 3 – 7 ( COR includes LiveCodeBench Pro and DeepSWE) in which the technique is better / worse than the reference by more than the 95% half-width of Table 10 , with the number of cells compared in brackets. (a) Against T0, the single shot (

B=1

; techniques at

B=4

, so this view is not budget-matched). (b) Against T7, best-of-

n

with the same ranker at the same budget (

B=4

\star

marks a targeted dimension. Cells where neither count is reached are differences within the interval.

Technique	CC	J	FS	SP	$\geq$ T7	J2	H
T1 quality instructions	0/1	0/4	0/1	0/3	16 of 16	0/3	$-0.19$
T2 SCoT / plan first	2/0	1/3	0/1	0/4	13 of 16	1/2	$-0.08$
T3 analyzer loop	1/1	0/4	0/0	0/0	15 of 16	0/3	$-0.14$
T4 Self-Refine	1/1	0/3	0/2	0/2	15 of 16	0/3	$-0.16$
T5 coder + reviewer	2/0	1/2	0/0	0/2	13 of 16	1/2	$-0.04$
T6 RAG	2/1	1/3	0/0	0/0	13 of 16	1/2	$-0.06$

Table 9. E6c. Measures the ranker does not optimize ,

B=4

, against T7. The T7 ranker and the analyzer loop (T3) act on Pylint, Bandit and Safety findings, which SonarQube’s issue counts also ingest; the measures here do not use them: CC = cognitive complexity, J = judge score (Claude Opus 5.5), FS = CWEval func-sec@1 and SP = BaxBench sec_pass@1 (both from tests and exploits). Cells: better/worse than T7 by more than the half-width of Table 10 , over the four models (two for T8–T10).

\geq

T7: cells in which T7 is better or within the interval. J2 = second judge (GPT-6.1 Sol), better/worse cells. H = mean human rating (1–5) minus T7’s, pooled over models, from the two raters’ ratings (half-width about

\pm

0.26).

Figure 4. Fig. E1. Budget curves (Supplementary Table ): primary metric per dimension (rows) and model (columns) at

B\in\{1,4,8,16\}

calls per task. T7, best-of-

n

with the analyzer-weighted ranker, is the budget-matched reference; the dashed line is the single shot. The analyzer loop (T3) and RL (T10) are highlighted; arrows in the row labels give the better direction. Grid of line charts, one row per quality dimension and one column per model, showing each technique's primary metric at 1, 4, 8 and 16 calls per task, with best-of-n as the reference and the single shot as a dashed line.

	Metric	Benchmark	$n$	Op	oss	G31	G26
COR	HE+ pass@1	HumanEval+	164	1.41	4.62	3.16	4.08
	LCB pass@1	LiveCodeBench v6 (dated)	100–400 a	–	4.00	5.84	7.65
	LCBP pass@1	LiveCodeBench Pro, post-cutoff contests	155	5.67	5.81	6.26	5.66
	DS pass@1	DeepSWE v1.1	113	6.66	5.74	5.40	4.46
R/M	C%	PythonSecurityEval	470	3.24	3.40	2.58	3.59
	Lint/kLOC	RACE R/M, NoFunEval	300–600 a	1.82	2.21	2.17	2.10

Table 10. E7. Uncertainty. 95% bootstrap half-width (

\pm

) of the paired per-task effect of a technique against T7 at

B=4

, per metric and model. A difference in Tables 3 – 7 and is called significant only when it exceeds the half-width of its metric and model; the same half-widths are used for comparisons against T0.

Figure 5. Fig. E2. Head-to-head summary (E12, Gemma 4 31B IT): each highlighted technique’s difference from best-of-

n

on the primary metric of its targeted dimension (Pylint score for R/M , EvalPerf DPS

{}_{\text{norm}}

for EFF , CWEval func-sec@1 for SEC ), with its 95% paired bootstrap interval; filled markers lie beyond the interval. All columns, both references and the budget-matched references are in Supplementary Table , and the per-column half-widths in Supplementary Table . Forest plot of the highlighted techniques, grouped by targeted dimension, each with its difference from best-of-n on its primary metric and a 95 percent interval; most intervals cross zero or lie below it.

Family	Studies	Non-targeted dim. measured		COR measured
		$n$	%	$n$	%
Prompting	113	15	13	79	70
Tool feedback	124	12	10	94	76
Self-refinement	17	1	6	10	59
Multi-agent	75	10	13	66	88
Retrieval	37	4	11	28	76

Table 11. How often included studies measured anything beyond their targeted dimension: share of studies per family that measured at least one non-targeted quality dimension, and share that measured correctness. Counts from LLM-assisted extraction reviewed by the author.

COR \ target	$\uparrow$	$\leftrightarrow$	$\downarrow$	mixed	nr	total
$\uparrow$	257 (136–340)	3 (0–7)	3 (0–7)	26 (11–42)	20 (8–34)	309 (223–362)
$\leftrightarrow$	51 (24–78)	2 (0–5)	0 (0–1)	2 (0–5)	1 (0–3)	56 (37–74)
$\downarrow$	27 (11–44)	1 (0–3)	4 (1–9)	23 (9–38)	0 (0–1)	55 (37–73)
mixed	65 (31–98)	1 (0–3)	1 (0–3)	37 (16–58)	3 (0–7)	107 (74–136)
nr	161 (83–225)	3 (0–7)	3 (0–7)	37 (17–59)	6 (1–12)	210 (150–255)
total	561 (409–601)	10 (5–16)	11 (6–17)	125 (88–158)	30 (19–42)	737

Table 12. Direction of effect on correctness (rows) against direction of effect on the targeted quality dimension (columns), counted per technique row of extraction.csv (vote counting by direction, SWiM; no effect sizes are pooled). Counts from LLM-assisted extraction reviewed by the author. In parentheses: how far each count could move if direction fields were wrong at the rate at which the independent second coder disagreed with the extraction (20.9%, 58 of 278 audited fields; the rate is drawn from its posterior, and the range is the 95th percentile of the rows a count can lose or gain).

Family	$n$	Med. QA	Comp. %	Cont. %	Human %	Artif. %
Prompting	113	6.5	38	31	32	73
Tool feedback	124	6.0	69	31	9	65
Self-refinement	17	6.0	24	47	29	65
Multi-agent	75	5.0	65	17	8	53
Retrieval	37	6.0	41	46	30	62
Decoding	18	6.5	67	22	11	78

Table 13. Evidence quality by family: median QA score (0–10, items QA1–QA10 of Supplementary Material §S4) and the share of studies that report compute or cost (QA7

>0

), consider contamination (QA9

>0

), use human evaluation, and release artifacts (QA8

>0

). LLM-assisted extraction reviewed by the author.

Appendix figures & tables42 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Title	Year	Family	Targets	Src
V00036	Planning with Large Language Models for Code Generation	2023	Decoding	R/M	arXiv:2303.05510
R00867	Constrained Decoding for Secure Code Generation	2024	Decoding	SEC	arXiv:2405.00218
S01754	Make Every Move Count: LLM-based High-Quality RTL Code Generation Using MCTS	2024	Decoding	EFF	arXiv:2402.03289
V00117	LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-Based Code Completion	2024	Decoding, Prompting	R/M	arXiv:2406.09834
R02790	A Mixture of Linear Corrections Generates Secure Code	2025	Decoding	SEC	arXiv:2507.09508
R03351	HiLDE: Intentional Code Generation via Human-in-the-Loop Decoding	2025	Decoding	SEC	arXiv:2505.22906

Appendix

Table S1. Included primary studies (LLM-assisted screening and extraction, reviewed by the author). ID = record identifier in the replication package; Targets = targeted dimensions (none: the study targets correctness and reports the quality dimensions listed); Src = arXiv ID or DOI.

ID	Q1	Q2	Q3	Q4	Q5	Q6	Q7	Q8	Q9	Q10	Total
O00001	0.5	1	0.5	1	1	0.5	0	1	0.5	0	6.0
O00002	1	1	0.5	1	1	0.5	1	1	0.5	0	7.5
O00007	0.5	0.5	1	0.5	1	0.5	0	1	0	1	6.0
O00008	1	1	0.5	0	1	1	0.5	1	0	0.5	6.5
O00015	1	1	1	0	1	0	0.5	1	0.5	0.5	6.5
O00021	1	1	0.5	0.5	1	0.5	0	1	1	1	7.5

Appendix

Table S2. Per-study quality assessment (items QA1–QA10 of §S4; 0, 0.5 or 1 per item; LLM-assisted, reviewed by the author).

Release	Released	OW	Source
Claude (Anthropic)
Claude Sonnet 3.7 (claude-3-7-sonnet-20250219)	2025-02	N	( Anthropic, 2026b ; Anthropic, 2026e )
Claude Opus 4	2025-05	N	( Anthropic, 2026b )
Claude Sonnet 4	2025-05	N	( Anthropic, 2026b )
Claude Opus 4.1	2025-08	N	( Anthropic, 2026b )
Claude Sonnet 4.5	2025-09	N	( Anthropic, 2026b )

Appendix

Table S3. Frontier and major open-weight model releases, January 2025 to October 2026 (338 rows), grouped by family in release order. Released: year and month of the first public release, as stated in the cited source ( nr : no verified source states it). p First public release was a preview, experimental or beta version. y The source states day and month only; the year follows from the dated post or the year section of the page in which the entry appears. OW: open weights (Y), API or product only (N), or only some of the models in the row (p); a weights announced, not yet released. Names as the vendor writes them; API model identifiers in parentheses where the vendor names the release by its identifier.

Domain	ID	Title	Year	Family	Cit.
Compiler and IR transforms	R03592	Agentic Auto-Scheduling: An Experimental Study of LLM-Guided Loop Optimization	2025	Tool feedback	6
Compiler and IR transforms	R05239	HintPilot: LLM-based Compiler Hint Synthesis for Code Optimization	2026	Retrieval	4
Compiler and IR transforms	R03953	LOOPRAG: Enhancing Loop Transformation Optimization with Retrieval-Augmented Large Language Models	2025	Retrieval	3
Data science	R05154	How to Build AI Agents by Augmenting LLMs with Codified Human Expert Domain Knowledge? A Software Engineering Framework	2026	Retrieval	3
Data science	R02038	Engineering Systems for Data Analysis Using Interactive Structured Inductive Programming	2025	Prompting	3
Data science	S04182	Diff-SQL: SQL Efficiency Optimization via Patch Generation and Constraint Alignment	2026	Self-refinement	0

Appendix

Table S4. Domain-specific techniques: eligible techniques tied to a narrow artifact or domain, which cannot run on the shared general-purpose benchmarks of the main article’s head-to-head; up to three per domain, by Semantic Scholar citations, October 2026. ID = record identifier in the replication package. LLM-assisted screening and extraction reviewed by the author.

	$n$	%		$n$	%
Year			Publisher
2022	5	1	IEEE	192	28
2023	24	3	ACM	108	16
2024	79	11	Springer	70	10
2025	235	34	Elsevier	43	6
2026	344	50	MDPI	26	4

Appendix

Table S5. The 687 reports sought but not retrieved (no full text), by year, type, publisher, the search that found them and open-access status. Type, publisher and open-access status come from the Unpaywall and Crossref records retrieved during full-text retrieval (no record: no DOI or no Unpaywall entry). Of these, 15 are another version of an included study.

Study	Dim.	Setting	Headline finding (as reported)
Pearce et al. (2022)	SEC	Copilot; scenarios built around high-risk CWEs	about 40% of programs vulnerable
Perry et al. (2023)	SEC	User study; codex-davinci-002 assistant	36% of assisted vs. 7% of control participants wrote SQL-injectable code
Asare et al. (2023)	SEC	Copilot; C/C++ scenarios with known human vulnerabilities	replicates vulnerable code about 33% of the time, the fix at a 25% rate
Fu et al. (2025)	SEC	Copilot, CodeWhisperer and Codeium snippets in GitHub projects	weaknesses in 29.5% of Python and 24.2% of JavaScript snippets
Vero et al. (2025)	SEC , COR	BaxBench backend-generation tasks	best model (o1) 62% correct; exploits succeed on about half of correct programs
Zhao et al. (2026a)	SEC , COR	SusVibes repository tasks; SWE-Agent with Claude 4 Sonnet	57% functionally correct, 11.8% secure

Appendix

Table S6. Context studies of the non-functional quality of LLM-generated code (no technique proposed). Grey literature is marked as such. Values as reported by the original authors; settings differ across rows, so values are not directly comparable.

Benchmark	Year	Dim.	Lang.	Size	Oracle / metric	Contam.
HumanEval+ ( Liu et al., 2023 )	2023	COR	Py	164	amplified tests; pass@ $k$	pub.
MBPP+ ( Liu et al., 2023 )	2023	COR	Py	nr	subset of MBPP (974) ( Austin et al., 2021 ) ; pass@ $k$	pub.
LiveCodeBench ( Jain et al., 2025 )	2024	COR	Py	511	contest tests; pass@1	date
BigCodeBench ( Zhuo et al., 2025 )	2024	COR	Py	1,140	library-call tests; pass@1	–
SWE-bench Verified ( Jimenez et al., 2024 ; Vals AI, 2026 )	2024	COR	Py	500	repo tests; resolve rate	pub.
SWE-bench Pro ( Deng et al., 2026b )	2025	COR	multi	1,865	repo tests; resolve rate	held-out

Appendix

Table S7. Benchmarks for correctness, readability/maintainability and efficiency. Dim.: COR correctness, R/M readability/maintainability, EFF efficiency, SEC security. Year: first public release. Size: tasks (problems, prompts or instances) as defined by each paper. Contam.: pub. = problems public before 2024; date = release-date filter against model cutoffs; held-out = private or copyleft-only split; new = newly written tasks; – = no specific measure reported. Values as reported by the original authors; settings differ across rows, so values are not directly comparable.

Benchmark	Year	Lang.	Size	Oracle / metric	Contam.
Asleep at the Keyboard ( Pearce et al., 2022 )	2021	C/Py	89	CodeQL + manual; CWE	pub.
SecurityEval ( Siddiq and Santos, 2022 )	2022	Py	121	static analyzers; no tests ( Siddiq et al., 2024b )	pub.
LLMSecEval ( Tony et al., 2023 )	2023	C/Py	150	CodeQL	pub.
CyberSecEval ( Bhatt et al., 2023 )	2023	multi	nr	ICD rules (189); v3 ( Wan et al., 2024 )	pub.
CodeLMSec ( Hajipour et al., 2023 )	2023	C/Py	nr	non-secure prompts found by few-shot model inversion	pub.
PythonSecurityEval ( Alrashedy et al., 2023 )	2023	Py	470	Bandit, CodeQL	pub.

Appendix

Table S8. Benchmarks for security ( SEC ). + COR : functional correctness is tested on the same outputs. Columns as in Table S7 . Values as reported by the original authors; settings differ across rows, so values are not directly comparable.

Family (releases)	Released	OW	Arch.	Vendor-reported coding scores	Cutoff	NF
Anthropic. Claude Sonnet 3.7; Opus 4, 4.1, 4.5; Sonnet 4, 4.5; Haiku 4.5	2025-02–11	N	nr	Sonnet 4.5: SWE-V 77.2%	Jul 2025	nr
Opus 4.6, 4.7, 4.8, 5; Sonnet 4.6, 5; Fable 5 (with Mythos 5) ( Anthropic, 2026h )	2026-02–07	N	nr	Fable 5: SWE-V 95%; SWE-P 80%	Jan 2026	nr
Fable 5.1 (with Mythos 5.1); Opus 5.5; Sonnet 5.5; Haiku 5.5 ( Anthropic, 2026d ; Anthropic, 2026f )	2026-09–10	N	nr	Opus 5.5: TB 4.0 66.4%; FC 54.4%	Jun 2026	FC
OpenAI. o3-mini; GPT-4.5; GPT-4.1; o3, o4-mini; codex-1 ( OpenAI, 2025h )	2025-01–05	N	nr	GPT-4.1: SWE-V 54.6%; Aider Polyglot 52.9%	June 2024	Edit
gpt-oss-120b, gpt-oss-20b ( OpenAI et al., 2025 )	2025-08	Y	MoE, 116.8B / 5.1B	120b (high): SWE-V 62.4	June 2024	nr
GPT-5; GPT-5-Codex; GPT-5.1, 5.1-Codex-Max; GPT-5.2, 5.2-Codex	2025-08–12	N	nr	GPT-5.2 Thinking: SWE-P 55.6%; SWE-V 80.0%	Aug 31, 2025	nr

Appendix

Table S9. Frontier and major open-weight models released between January 2025 and October 2026. Vendor sources (release posts and model cards) for every row are listed in § S5 . Each row groups successive releases of one family; where a row has several models, the architecture, scores and cutoff refer to the model named in the cell (otherwise to the newest release). The model table of § S5 gives the release month and source of every release named here. Released: first and last month among the dated releases of the row. All values are vendor-reported (model cards, technical reports, vendor announcements) under the vendor’s own harness and effort setting. OW: open weights (p: some releases). Arch.: total / active parameters. Cutoff: training-data or knowledge cutoff as stated by the vendor. NF: non-functional evaluation of generated code reported by the vendor ( SEC security, EFF efficiency; FC: FrontierCode v1.1, a merge-readiness composite that includes correctness; Hum.: blind human rating of coding quality; Edit: rate of extraneous code edits). SWE-V: SWE-bench Verified; SWE-P: SWE-bench Pro; TB: Terminal-Bench (version); LCB: LiveCodeBench; DS: DeepSWE v1.1. a Weights of Mistral Large 4 announced, not yet released. Values as reported by the original authors; settings differ across rows, so values are not directly comparable.

(a) API-served models
	Op				oss
Technique	$B=1$	$B=4$	$B=8$	$B=16$	$B=1$	$B=4$	$B=8$	$B=16$
COR HumanEval+ pass@1 (%, higher is better)
T0 single shot	96.40	–	–	–	81.27	–	–	–
T1 quality instr.	97.64	99.10	98.75	99.22	80.08	83.17	82.91	85.68
T2 SCoT	92.15	91.41	91.99	90.34	81.31	85.31	85.98	88.45

Appendix

Table S10. E8. Budget curves. Primary metric per dimension at

B\in\{1,4,8,16\}

call-equivalents; T7 is the budget-matched reference and T0 is defined at

B=1

only.

B=4

cells are the measurements of Tables – ; at

B=1

, T3, T4, T5 and T7 equal T0 by construction (no refinement round, one sample).

	Op			oss			G31			G26
Metric	$B{=}8$	$B{=}16$	SD	$B{=}8$	$B{=}16$	SD	$B{=}8$	$B{=}16$	SD	$B{=}8$	$B{=}16$	SD
HE+ pass@1	1.29	1.34	0.45	4.24	4.34	0.46	2.77	2.96	0.42	3.65	4.02	0.42
LCB pass@1	–	–	–	4.29	3.63	0.46	5.83	5.59	0.36	7.67	7.40	0.46
LCBP pass@1	5.39	6.04	0.40	5.68	5.80	0.45	6.24	6.18	0.39	6.15	5.69	0.30
DS pass@1	6.91	6.24	0.39	5.62	5.49	0.28	5.27	6.18	0.45	4.04	4.83	0.47
C%	3.10	2.93	0.32	3.12	3.35	0.44	2.30	2.84	0.40	3.34	3.36	0.38

Appendix

Table S11. Per-budget 95% half-widths (

B=4

as in the main article;

B=8

and

B=16

) and seed dispersion (median SD across seeds /

B=4

half-width), per metric and model.

ID	Technique	EP	LCB	LCBP	CWE	CSE	EB	DPS	RM
H01	SCoT prompting	2.93	5.88	5.42	8.04	2.83	0.16	2.74	0.42
H02	Self-planning	2.41	6.74	5.62	7.04	2.94	0.18	4.04	0.40
H03	LLM4EFFI	2.62	4.74	5.21	5.67	3.81	0.18	2.70	0.34
H04	Code2API	2.73	4.70	5.39	5.95	2.86	0.25	3.42	0.25
H05	MPCO	2.68	6.74	4.49	6.22	3.65	0.21	3.49	0.29
H06	BRACE	2.95	5.95	6.53	5.88	2.92	0.23	3.29	0.36

Appendix

Table S12. E12: 95% paired bootstrap half-widths (

\pm

) of each technique’s difference from best-of-

n

, per column of the head-to-head table (differences from T0 have half-widths within 0.3 points of these).

Technique, model	$B=1$	$B=4$	$B=8$	$B=16$
T7 best-of- $n$ , Op	71.03, 73.89, 79.63 (74.85)	77.21, 78.92, 78.38 (78.17)	70.91, 74.90, 77.48 (74.43)	85.67, 85.28, 87.92 (86.29)
T1 quality instr., Op	63.89, 67.07, 67.10 (66.02)	64.05, 65.03, 71.53 (66.87)	74.54, 78.33, 80.95 (77.94)	84.19, 80.40, 87.35 (83.98)
T7 best-of- $n$ , G31	48.05, 51.13, 49.26 (49.48)	55.03, 52.30, 58.93 (55.42)	52.31, 50.72, 52.37 (51.80)	43.80, 45.67, 50.57 (46.68)
T10 RL, G26	49.42, 42.87, 50.00 (47.43)	49.32, 53.58, 56.73 (53.21)	48.19, 53.03, 52.17 (51.13)	56.41, 51.18, 60.74 (56.11)

Appendix

Table S13. Per-seed CWEval func-sec@1 (%) of four uneven budget-curve cells: the three end-to-end runs, with their mean (the value of Table S10 ) in parentheses. At

B=1

, T7 equals T0 by construction.

oss

G31

G26

Technique

T0 single shot

7.68

4.30

3.66

10.72

3.55

3.20

9.83

3.74

3.35

9.69

3.96

3.44

T1 quality instr.

+0.05

+0.10

†

+0.07

+0.11

+0.06

+0.05

{-}0.29

+0.14

†

+0.09

+0.17

{-}0.03

{-}0.02

T2 SCoT

{-}0.19

+0.05

+0.04

{-}0.36

†

+0.09

†

+0.06

{-}0.40

+0.08

+0.05

{-}0.14

+0.02

+0.00

T6 RAG

+0.03

{-}0.02

{-}0.03

{-}0.17

+0.04

+0.02

+0.21

{-}0.05

{-}0.04

+0.09

+0.03

+0.01

Appendix

Table S14. Single call (

B=1

) on the measures the analyzers do not compute (RACE R/M and NoFunEval). T0 row: values; other rows: paired difference from T0 ( † : 95% bootstrap interval excludes zero). CC = cognitive complexity (lower is better); J = judge (Claude Opus 5.5) and J2 = second judge (GPT-6.1 Sol), 1–5 (higher is better).

(a) Maintainability, RACE R/M and NoFunEval
	Op		oss		G31		G26
Technique	LOC	Mnt	LOC	Mnt	LOC	Mnt	LOC	Mnt
T0 single shot	31.2	0.67	44.7	1.31	38.2	1.01	39.2	1.13
T1 quality instr.	36.1	0.72	49.6	1.35	43.8	1.08	45.4	1.29
T2 SCoT	34.5	0.65	44.9	1.21	38.0	0.90	40.3	1.01
T3 analyzer loop	34.2	0.27	44.8	0.51	37.6	0.38	40.4	0.44

Appendix

Table S15. Solution length and absolute issue counts at

B=4

(T0:

B=1

), next to the densities of Tables and . LOC = mean non-blank, non-comment lines per solution; Mnt = maintainability issues per solution (RACE R/M and NoFunEval; density = Lint/kLOC); Sec = security issues per solution (PythonSecurityEval; density = B/k).

Experiment	Tasks per model	Models	Per task and seed	Seeds	Call-eq.	Requests
Main grid (E2–E8)	2,632 (Op: 2,552, no LCB)	4; T8–T10 on 2	161.5; +85 (T8–T10)	3	6,404	3,121
Extended grid (E9)	2,552	9	23.9 (T0, T1–T7 at $B=4$ )	3	1,647	994
Agent benchmarks (E2b)	113 DeepSWE + 1,037 other	13	25 runs (DeepSWE; +11 on the Gemma pair); 1 run (other)	3	158 runs	5,819
Head-to-head (E12)	3,988	1	280.6 (66 techniques at 3.9 mean calls, T0, best-of- $n$ , best-of-2/3/5/8)	3	3,357	2,887
Judges	417k outputs	2 judges	1	–	834	834
Fidelity checks and reruns	authors’ settings	–	–	–	34	34

Appendix

Table S16. Derivation of the Calls column of Table S17 : tasks, models, call-equivalents per task and seed, seeds, total call-equivalents (agent experiments: runs) and billed requests, in thousands.

Experiment	Calls	In tok.	Out tok.	Claude	Vercel	Fireworks	Total
Main grid (E2–E8)	3,121	3,910	2,747	25,269	–	1,791	27,060
Extended grid (E9, function level)	994	986	746	–	5,304	1,495	6,799
Agent benchmarks (E2b, agent table)	5,819	52,911	1,833	7,288	7,659	5,433	20,380
Head-to-head (E12)	2,887	3,109	1,351	–	–	1,136	1,136
Judges and manual-rating support	834	1,213	172	5,056	2,853	–	7,909
Fidelity checks and reruns	34	59	18	1,046	–	–	1,046

Appendix

Table S17. Per-experiment compute ledger. Calls = billed requests in thousands; tokens in millions (input includes cached prefixes, billed at the cache rate); fees in US$.

		COR	R/M		EFF		SEC
Model	Cutoff	LCBP	Lint	J	DPS	Bey	FS	SP	B/k
Claude Opus 5.5	Jun 2026	69.82	21.37	4.30	78.06	75.03	74.85	58.54	3.40
GPT-6.1 Sol	Apr 2026	66.79	22.56	4.19	76.32	72.76	72.79	57.39	3.59
Gemini 4 Argon	Mar 2026	67.43	21.08	4.34	80.36	76.19	73.72	56.22	3.39
Kimi K3	nr	55.31	24.34	4.12	78.22	76.30	69.43	49.11	3.80
GLM-5.3	nr	57.50	23.51	4.09	77.77	73.07	68.74	50.66	3.81

Appendix

Table S18. E9. Extended model grid, single shot (T0). Function-level measures of Tables – for the current frontier and open-weight models. LCBP = LiveCodeBench Pro protocol on Codeforces contests held after 2026-07-01, pass@1; Lint = maintainability issues per kLOC (lower is better); J = judge score 1–5; DPS = EvalPerf DPS

{}_{\text{norm}}

; Bey = Mercury Beyond; FS = CWEval func-sec@1; SP = BaxBench sec_pass@1; B/k = security issues per kLOC (lower is better). Cutoff = stated training-data cutoff ( nr = not stated). HumanEval+ and LiveCodeBench v6 are omitted because every cutoff after April 2025 postdates their problems.

Technique	LCBP	Lint	DPS	FS
GPT-6.1 Sol
T0 single shot	66.79	22.56	76.32	72.79
T1 quality instr.	63.66	21.84	77.78	72.74
T2 SCoT	70.69	20.72	82.16	69.89
T3 analyzer loop	67.35	8.30	80.15	81.37
T4 Self-Refine	65.24	20.80	80.05	68.92

Appendix

Table S19. E10. Techniques on the extended grid at

B=4

(T0:

B=1

): primary metric per dimension for each technique of Table c on the models of Table S18 . LCBP and FS: higher is better; Lint: lower is better; DPS: EvalPerf DPS

{}_{\text{norm}}

, higher is better. Training-time techniques (T8–T10) apply to the Gemma pair only.

	COR		R/M			EFF		SEC
Model	DS	TB	FC	FC c	Er	GSO	SWf	SV s	SV f
Claude Opus 5.5	75.17	61.79	53.08	59.20	0.33	21.19	53.87	56.52	94.58
GPT-6.1 Sol	76.59	55.51	49.12	56.49	0.36	17.81	51.19	50.11	90.18
Gemini 4 Argon	76.38	57.28	46.86	54.45	0.35	19.54	51.74	47.73	91.29
Kimi K3	65.47	35.29	44.63	49.56	0.41	16.15	45.26	42.18	84.37
GLM-5.3	71.95	41.97	40.07	45.34	0.38	15.68	47.90	44.09	83.21

Appendix

Table S20. E11. Repository- and agent-level benchmarks , single agent run per task with the vendor-recommended harness (mini-SWE-agent for DeepSWE, Terminal-Bench’s own harness, the benchmark authors’ harness elsewhere), all thirteen models. COR : DS = DeepSWE v1.1 pass@1 ( Huang et al., 2026d ) ; TB = Terminal-Bench 4.0 ( Merrill et al., 2026 ) . R/M : FC = FrontierCode v1.1 Main composite score, which adds a code-quality rubric to functional blockers, and FC c = its correct rate, so FC c

-

FC isolates the quality penalty ( Cognition, 2026 ) ; Er = SlopCodeBench structural erosion, 0–1, lower is better ( Orlanski et al., 2026 ) . EFF : GSO = Opt@1 ( Shetty et al., 2025 ) ; SWf = SWE-fficiency speedup relative to the expert patch,

\times 100

( Ma et al., 2025 ) . SEC : SV s and SV f = SusVibes SecPass and FuncPass ( Zhao et al., 2026a ) . All values are percentages unless stated.

ID	Technique	EP	LCB	LCBP	CWE	CSE	EB	DPS	RM	Calls
–	T0 single shot (reference)	93.83	69.91	27.14	47.85	32.59	2.64	79.37	6.83	1
–	best-of- $n$ (reference)	89.97	73.42	30.92	57.07	26.55	2.87	82.42	7.69	4.19
Prompting
H01	SCoT prompting	88.04	72.97	30.03	51.24	28.34	2.78	73.72	7.87	1.93
H02	Self-planning	83.45	71.26	28.82	51.84	29.38	2.85	82.02	6.77	2.10
H03	LLM4EFFI	94.51	75.79	31.59	48.79	31.12	2.44	76.28	6.86	4.10

Appendix

Table S21. E12. Head-to-head of the highlighted techniques (§ ): one model, a single-shot and a best-of-

n

reference, every output scored on all four dimensions. COR : EP = HumanEval+/MBPP+ pass@1, LCB = LiveCodeBench v6 pass@1 on the same post-cutoff problems as Table , LCBP = LiveCodeBench Pro protocol on Codeforces contests held after 2026-07-01, pass@1. SEC : CWE = CWEval func-sec@1, CSE = CyberSecEval insecure-code rate (lower is better). EFF : EB = EffiBench normalized execution time (lower is better), DPS = EvalPerf DPS

{}_{\text{norm}}

. R/M : RM = Pylint score on the EP and LCB outputs. Calls = mean model calls per task. The last panel runs best-of-

n

with the same ranker at two, three, five and eight calls per task; each family is compared with the row nearest its median calls per task, decoding with the main reference. Indicative: point estimates; per-technique half-widths are in Table S12 , and Fig. E2 of the main article plots each technique’s primary target with its interval.

ID	Technique	Setting (metric; benchmark; model)	Original	Reproduced
H01	SCoT prompting	Maintainability, human rating (0-2); 200 programs from HumanEval, MBPP, MBCPP; ChatGPT gpt-3.5-turbo-0301 (closed)	n.c.	–
H02	Self-planning	Readability, human rating (0-4, violin plot only); 50 HumanEval tasks; code-davinci-002 (175B, closed)	n.c.	–
H03	LLM4EFFI	eff@1; ENAMEL (142 HumanEval problems); DeepSeek-V3 (open)	60.41	60.94
H04	Code2API	Method-name expressiveness, human rating (1-4); APIzator set (200 Stack Overflow Java snippets); GPT-3.5-turbo (closed)	n.c.	–
H05	MPCO	Average statistical rank of mean %PI (lower is better); 5 projects, 50 hotspots; GPT-4o, Gemini 2.5 Pro, Claude 3.7 Sonnet (closed)	n.c.	–
H06	BRACE	CWEval Sec pass@10, mean of 6 models; CWEval (119 tasks); Qwen3-8B/14B, Qwen3.6-27B, Gemma-3-4B/12B, Gemma-4-31B (open) †	75.77	72.35

Appendix

Table S22. E12 fidelity check: original versus reproduced number per technique.

COR \ target	$\uparrow$	$\leftrightarrow$	$\downarrow$	mixed	nr	total
$\uparrow$	102	0	0	14	10	126
$\leftrightarrow$	18	1	0	1	0	20
$\downarrow$	12	1	1	9	0	23
mixed	30	0	1	13	1	45
nr	58	1	2	13	6	80
total	220	3	4	50	17	294

Appendix

Table S23. Peer-reviewed studies only (sensitivity re-count of the main article’s table of the same name). Direction of effect on correctness (rows) against direction of effect on the targeted quality dimension (columns), counted per technique row of extraction.csv (vote counting by direction, SWiM; no effect sizes are pooled). Counts from LLM-assisted extraction reviewed by the author.

Family	Studies	Non-targeted dim. measured		COR measured
		$n$	%	$n$	%
Prompting	58	9	16	40	69
Tool feedback	43	4	9	32	74
Self-refinement	6	1	17	5	83
Multi-agent	20	1	5	17	85
Retrieval	15	1	7	12	80

Appendix

Table S24. Peer-reviewed studies only (sensitivity re-count of the main article’s table of the same name). How often included studies measured anything beyond their targeted dimension: share of studies per family that measured at least one non-targeted quality dimension, and share that measured correctness. Counts from LLM-assisted extraction reviewed by the author.

Family	$n$	Med. QA	Comp. %	Cont. %	Human %	Artif. %
Prompting	58	6.5	31	34	34	79
Tool feedback	43	6.5	49	33	23	74
Self-refinement	6	5.8	17	67	33	67
Multi-agent	20	5.5	45	15	10	60
Retrieval	15	6.0	53	53	40	60
Decoding	6	6.2	67	17	33	100

Appendix

Table S25. Peer-reviewed studies only (sensitivity re-count of the main article’s table of the same name). Evidence quality by family: median QA score (0–10, items QA1–QA10 of Appendix S4 ) and the share of studies that report compute or cost (QA7

>0

), consider contamination (QA9

>0

), use human evaluation, and release artifacts (QA8

>0

). LLM-assisted extraction reviewed by the author.

Mechanisms	Studies	Representative studies
retrieval + self-refinement + tool feedback	33	Retrieval Augmented Search ( Anupam et al., 2026 ) ; SCGAgent ( Saul et al., 2025 ) ; ECO ( Kim et al., 2025 ) ; Safety Sidecar ( Wang et al., 2026a )
tool feedback	27	LLM4PLC (R00632); RustMap (R02061); LLMigrate (R02100); DeRep (R02183)
retrieval + tool feedback	26	SBLLM ( Gao et al., 2025b ) ; RAG + multi-tool ( Sriram et al., 2026 ) ; LLM-driven HLS repair framework (R01058); APILOT (R01272)
another LLM + tool feedback	16	Codexity Preshot Repair ( Kim et al., 2024 ) ; Hybrid pipeline P10 ( Sabir et al., 2026 ) ; CodeEvolve ( Borra et al., 2026 ) ; SecureForge ( Liu et al., 2026a )
retrieval + another LLM + tool feedback	15	MARCO ( Rahman et al., 2025 ) ; SysLLMatic ( Peng et al., 2026a ) ; HLSPilot (R01152); VeriOpt (R03540)
self-refinement + tool feedback	13	CodeChain ( Le et al., 2024a ) ; SecCode ( Liu et al., 2024c ) ; LLM4EFFI ( Ye et al., 2025 ) ; ArchCode (R00592)

Appendix

Table S26. Hybrid and other studies by combination of mechanisms (derived from their signal sources and stage), with representative studies.

Study	Strategy	Models; benchmark	Target-dimension effect	COR effect
Tony ( Tony et al., 2025 )	RCI (one critique–improve round)	GPT-3/3.5/4; LLMSecEval, Bandit	SEC $\uparrow$ : weakness density 77.55% lower (GPT-4)	nr
Bruni ( Bruni et al., 2025 )	Security-aware persona prefix	GPT-4o; LLMSecEval + SecurityEval, Semgrep/CodeQL	SEC $\uparrow$ : 56% fewer vulnerabilities	HumanEval pass@1 86.77% $\to$ 86.65% (GPT-4o-mini)
Bruni ( Bruni et al., 2025 )	RCI on baseline code	GPT-4o; as above	SEC $\uparrow$ : 64.7% fewer vulnerabilities	$\downarrow$ pass@1 86.77% $\to$ 76.65% (GPT-4o-mini)
Kharma ( Kharma et al., 2026a )	WA-0CoT (CWE-aware CoT)	5 LLMs; Java, C++, C, Python	SEC $\leftrightarrow$ frequency and density; CWE mix shifts; R/M $\downarrow$ more warnings (C)	nr
Urmi ( Urmi et al., 2026 )	Structured + security guidance	GPT-4o; 424 Python tasks, Bandit/CodeQL	SEC high-severity 20.8% $\to$ 13.6%; low-severity 32% $\to$ 43.5%	nr (requested constructs sometimes removed)
Pereira ( Pereira and Garcia, 2026 )	ISO/IEC 25010-grounded requirements	GPT-5.4; HumanEval/ET	R/M $\uparrow$ unreadability density 0.88 $\to$ 0.69 (performance requirement)	not reliably improved; ET pass rate $\downarrow$ (error handling)

Appendix

Table S27. Reported effects of prompting strategies on quality and correctness.

\uparrow

\downarrow

\leftrightarrow

: improvement/degradation/no significant change in the stated metric relative to the study’s baseline. Values as reported; settings differ, so rows are not directly comparable.

Technique	Year	Cit.	L/R	Core idea	Target	Reported result	COR	Code
SCoT ( Li et al., 2025b )	2025	405	L	Constrains the chain of thought to sequence, branch and loop structures, then implements code from it.	R/M	Maintainability, human rating 0–2, ChatGPT: CoT 1.616 $\to$ 1.873	$\uparrow$	–
Self-planning ( Jiang et al., 2024 )	2024	306	L	Few-shot prompts make the model write a concise step plan, then code guided by that plan.	R/M	Readability, human rating: gain reported without numbers ( nr )	$\uparrow$	–
LLM4EFFI ( Ye et al., 2025 )	2025	15	L	Explores algorithms in a logic domain, optimizes the implementation in a code domain, then verifies with synthetic tests.	EFF	eff@1, ENAMEL, DeepSeek-V3: Effi-Learner 52.22 $\to$ 60.41	$\uparrow$	✓
Code2API ( Mai et al., 2024 )	2024	12	L	Role, chain-of-thought and few-shot prompting turn Stack Overflow snippets into reusable, well-named API methods.	R/M	Method-name expressiveness, human rating 1–4, GPT-3.5: APIzator 2.23 $\to$ 3.65	mixed	✓
MPCO ( Gong et al., 2025a )	2025	10	L	A meta-prompter LLM writes project-, task- and model-aware code-optimization prompts.	EFF	Mean rank of %PI over five projects (lower is better): contextual prompting 2.00 $\to$ 1.00	nr	–
BRACE ( Xu et al., 2026a )	2026	1	R	A separate LLM predicts the task’s likely CWEs; after catalog validation they are given to the target model as a cue.	SEC	Safe Code Rate (judge classifier), CyberNative, open-weight models (avg.): Secure Prompt 33.9% $\to$ 91.2%	mixed	–

Appendix

Table S28. Highlighted prompting techniques: landmarks (L, most cited) and recent leaders (R, 2025–2026, largest reported gain). Year = year of the cited version. Cit. = Semantic Scholar citations, October 2026. COR = direction of the effect on correctness as coded in our extraction. Values as reported by the original authors; settings differ across rows, so values are not directly comparable.

Study	Tool(s)	Feedback format	Max iter.	Gate	COR	Lang.; models
Blyth et al. (2025)	Bandit, Pylint	inline tags, severity order	10	fitness	$\uparrow$	Py; GPT-4o
Wadhwa et al. (2024)	CodeQL, Sonar	rule + fix advice; ranker	one pass, 10 samples	ranker	review	Py, Java; GPT-3.5/4
Abtahi and Azim (2025)	SonarQube	issue table + RAG	two phases	manual	manual	one project; GPT-3.5/4o
Joos et al. (2026)	SonarQube	agent + rejection context	40 cycles	build, tests	manual	Java; GPT-4.1 mini
Islam et al. (2026)	Pylint, Radon, Bandit, …	metric diagnostics	five	none	no	Py; several
Ravi et al. (2025)	javac, PMD, tests	errors, violations	five/loop	tests	$\uparrow$	Java; GPT-4o-mini

Appendix

Table S29. Design space of tool-feedback loops. Gate: acceptance check other than the driving tool. COR : correctness measured, with direction. Author-reported values, not directly comparable across rows.

Study	Setting	Target-dimension effect	COR effect
Blyth et al. (2025)	GPT-4o; PythonSecurityEval	insecure $>$ 40% $\to$ 13%; readability viol. $>$ 80% $\to$ 11%	pass@1 38.16% $\to$ 47.81%
Wadhwa et al. (2024)	GPT-3.5/4; CodeQL, Python	pass checks 81.57%; reviewer rejection 55.11% (47.55% ranked)	review only
Joos et al. (2026)	GPT-4.1 mini; SonarQube	plausible 96.8%; correct 86.3%	gated
Abtahi and Azim (2025)	GPT-3.5 $\to$ 4o; one project	issues 7,599 $\to$ 4,340 $\to$ 1,058	manual
Alrashedy et al. (2023)	GPT-4; PythonSecurityEval	vulnerable: Bandit 40.2% $\to$ 7.4%; CodeQL 17.9% $\to$ 7.7%	nr
PromSec in ( Dai et al., 2026a )	GPT-3.5; BigCodeBench	Secure-Pass@1 0.3219 $\to$ 0.2447	Pass@1 $\downarrow$

Appendix

Table S30. Reported effects on the target dimension and on correctness (

\to

: before to after). Author-reported values, not directly comparable across rows.

Technique	Year	Cit.	L/R	Core idea	Target	Reported result	COR	Code
SBLLM ( Gao et al., 2025b )	2025	76	L	Evolves optimized programs: selects representative executed samples, retrieves optimization patterns, combines them with genetic-operator-style prompts.	EFF	SP@5 (%), GPT-4, PIE Python: ICL 181.53 $\to$ 209.59	$\uparrow$	✓
PerfCodeGen ( Peng et al., 2025b )	2025	58	L	After a correctness phase, returns the most time-consuming unit test and asks for a faster program.	EFF	%Opt Best@8, GPT-4, MBPP: base 43.48 $\to$ 56.52	$\uparrow$	–
PromSec ( Nazzal et al., 2024 )	2024	55	L	A contrastively trained graph GAN repairs the vulnerable code graph; the repaired code is turned into an improved prompt, iteratively.	SEC	nr (CWE counts shown only in plots)	$\uparrow$	✓
DeepDevVuln filter ( Chan et al., 2023 )	2023	41	L	A fine-tuned transformer vulnerability detector flags incomplete code; completions it flags are discarded.	SEC	vulnerable scenarios, text-davinci-003: unfiltered 78.00% $\to$ filtered 8.0%	nr	–
FRANC repair ( Siddiq et al., 2024a )	2024	32	L	Builds a repair prompt from Bandit or SpotBugs messages (generated-knowledge prompting, three prompt structures) and re-queries the model.	SEC , R/M	prompts with a good suggestion after repair: up to 80%; no pre-repair value	nr	✓
Optimo ( Peng et al., 2026b )	2026	0	R	Differential profiling finds bottlenecks, routed to multi-level, pattern-specific optimization prompts with shared and special strategies and correctness reflection.	EFF	Opt%, GPT-4o code, EffiBench: SBLLM 3.58% $\to$ 35.79%	$\uparrow$	✓

Appendix

Table S31. Highlighted tool-feedback loop techniques: landmarks (L, most cited) and recent leaders (R, 2025–2026, largest reported gain). Year = year of the cited version. Cit. = Semantic Scholar citations, October 2026. COR = direction of the effect on correctness as coded in our extraction. Values as reported by the original authors; settings differ across rows, so values are not directly comparable.

Study (model)	Setting and metric	Before $\rightarrow$ after	COR
Self-Refine (GPT-4)	CodeNet rewrite; GPT-4-judged meaningful names ( R/M )	27.4 $\rightarrow$ 56.2	nr
EffiLearner (StarCoder2-15B)	EffiBench; execution time, s ( EFF )	0.93 $\rightarrow$ 0.12	$\downarrow$ 0% to 0.5%
EffiLearner (StarCoder2-15B)	EffiBench; total memory usage, Mb $\cdot$ s ( EFF )	22.02 $\rightarrow$ 2.03	as above
Self-Refine in EffiLearner (CodeLlama-70B)	EffiBench; execution time change ( EFF )	$+$ 51.9%	nr
Reflexion + ICD (five open code LLMs)	Instruct Prime; ICD secure rate ( SEC )	70.74% $\rightarrow$ 79.43%	nr

Appendix

Table S32. Reported effects of self-refinement on non-functional quality (Self-Refine on PIE: Table S33 ). Values as reported by the original authors; settings differ across rows, so values are not directly comparable.

Technique	Year	Cit.	L/R	Core idea	Target	Reported result	COR	Code
Self-Refine ( Madaan et al., 2023 )	2023	5050	L	One LLM alternates between feedback on its own output and refinement using that feedback, for several iterations.	EFF	% programs optimized, PIE, GPT-4: no refinement 27.3 $\to$ 36.0	nr	✓
CodeChain ( Le et al., 2024a )	2024	91	L	Modular chain of thought, then self-revisions conditioned on clustered representative sub-modules from earlier samples.	R/M	Modularity and reusability, GPT-4 Likert rating, 20 APPS tasks: shown only as a figure ( nr )	$\uparrow$	✓
SecCode ( Liu et al., 2024c )	2024	8	L	Score-based “encouragement” prompts drive repeated vulnerability detection and fixing, cross-checked with CodeQL.	SEC	Fix success rate after ten iterations, LEval, GPT-3.5: CoT 0.64 $\to$ 0.96	$\uparrow$	–
RefleXGen ( Wang et al., 2025a )	2025	7	L	Self-reflection on code security; reflections and secure snippets accumulate in a knowledge base retrieved for later tasks.	SEC	CodeQL security rate, SVEN scenarios, GPT-3.5 Turbo: base 75.5 $\to$ 89.1	mixed	–
GRASP ( Patir et al., 2025 )	2025	4	L	Revises a seed solution by traversing a dependency graph (DAG) of the relevant secure coding practices.	SEC	CodeQL security rate, 37 Python scenarios, GPT-4o: PromSec 0.52 $\to$ 0.79	mixed	–
Safety Sidecar ( Wang et al., 2026a )	2026	1	R	A lightweight self-check routes risky code to a reflective prompt fed by a memory of past secure repairs.	SEC	CodeQL security rate, SVEN scenarios, GPT-4o: base agent 85.7 $\to$ 95.0	mixed	–

Appendix

Table S33. Highlighted self-refinement techniques: landmarks (L, most cited) and recent leaders (R, 2025–2026, largest reported gain). Year = year of the cited version. Cit. = Semantic Scholar citations, October 2026. COR = direction of the effect on correctness as coded in our extraction. Values as reported by the original authors; settings differ across rows, so values are not directly comparable.

Technique	Year	Cit.	L/R	Core idea	Target	Reported result	COR	Code
SecAwareCoder ( Wang et al., 2026e )	2026	54	L	Task-adaptive threat modeling yields vulnerability hypotheses that guide constraint-aware generation and security-test synthesis; execution feedback drives repair.	SEC	S-Pass@1, CodeSecEval, GPT-5: CoT 27.45 $\to$ 34.90	$\uparrow$	✓
FlowGen ( Lin et al., 2025b )	2025	51	L	Agents in requirement-engineer, architect, developer, tester and scrum-master roles emulate Waterfall, TDD or Scrum, with self-refinement.	R/M	Pylint error-smell density, GPT-3.5: 81.8% to 96.0% lower than RawGPT	$\uparrow$	✓
INDICT ( Le et al., 2024b )	2024	4	L	A safety critic and a helpfulness critic, grounded in web search and code execution, give preemptive and post-hoc feedback to the coder.	SEC	Safety, CyberSecEval-1 split, GPT-4o-mini: direct 78.2 $\to$ 90.9	nr	✓
ContextEvolve ( Su et al., 2026 )	2026	2	R	Summarizer, Navigator and Sampler agents compress evolutionary-search context into semantic state, optimization direction and retrieved exemplars.	EFF	ADRS combined score vs. top baseline: 6.5% higher on average	$\leftrightarrow$	✓
Just-in-time remediation (P1) ( Surikov, 2026 )	2026	0	R	A validator LLM reviews generated code; its findings, enriched with ATT&CK techniques, CWE examples and best practices, drive remediation.	SEC	CodeQL+Bandit findings per sample, four Claude models pooled: initial 1.34 $\to$ 0.78	nr	✓
LessonL ( Liu et al., 2025c )	2025	0	R	Agents solicit lessons from each other’s successes and failures, bank them, and select relevant lessons for later rounds.	EFF	Geometric-mean speedup, ParEval serial: Reflexion (Qwen14B) 1.88 $\to$ 2.16	$\uparrow$	✓

Appendix

Table S34. Highlighted multi-agent techniques: landmarks (L, most cited) and recent leaders (R, 2025–2026, largest reported gain). Year = year of the cited version. Cit. = Semantic Scholar citations, October 2026. COR = direction of the effect on correctness as coded in our extraction. Values as reported by the original authors; settings differ across rows, so values are not directly comparable.

Method	Retrieved unit	Models; benchmark	Target-dimension effect	COR effect
SecCoder ( Zhang et al., 2024a )	Secure demonstrations	CodeGen, SantaCoder, InCoder; CodeQL	SEC $\uparrow$ 7.20% avg. on unseen cases	$\leftrightarrow$ “almost the same”
RESCUE ( Shi and Zhang, 2026 )	Distilled guidelines, sliced examples	6 LLMs; 4 benchmarks	SEC $\uparrow$ SecurePass@1 4.8 points	$\leftrightarrow$ comparable
SEEK ( Gadey et al., 2026 )	CWE-graph fix instances	5 open LLMs; CodeGuard+	SEC SecurePass@1 77.45%	in SecurePass@1
SOSecure ( Mukherjee and Hellendoorn, 2026 )	Stack Overflow critiques (post-draft)	GPT-4; LLMSecEval	SEC fix rate 91.3% vs. 56.5%	nr
A3-CodGen ( Liao et al., 2024 )	Local functions	GPT-3.5; 13,784 PyPI functions	R/M reuse F1 0.589 (vanilla 0)	nr

Appendix

Table S35. Reported effects of further retrieval-augmented methods on quality and correctness, as reported by the original authors; settings differ, so values are not directly comparable.

Technique	Year	Cit.	L/R	Core idea	Target	Reported result	COR	Code
PIE dynamic retrieval ( Shypula et al., 2024 )	2024	171	L	Retrieves the slow–fast program pairs most similar to the input and uses them as few-shot examples for optimization.	EFF	Speedup (Best@8), PIE, CodeLlama 34B: CoT 1.30 $\times$ $\to$ 2.28 $\times$	mixed	✓
LAIL ( Li et al., 2025c )	2025	92	L	Labels candidate examples by how much they raise the LLM’s likelihood of the reference program; trains a contrastive retriever on these labels.	R/M	Human code-quality score: best example selector 1.738 $\to$ 1.839	$\uparrow$	–
KPC ( Ren et al., 2023 )	2023	50	L	Builds an exception-handling knowledge base from Java API documentation; chained prompts check generated code for unhandled exceptions and rewrite it.	R/M	Good exception-handling practice (manual review), GPT-3.5: direct 14.58% $\to$ 98.86%	$\uparrow$	✓
RAPGen ( Garg et al., 2025 )	2025	36	L	Retrieves an instruction from a knowledge base of past performance-bug fixes and builds a zero-shot fix prompt from it.	EFF	Expert-judged closest-match top-100 accuracy, C# fixes, GPT-3.5: reasoning prompt 43.2 $\to$ 60.6	nr	–
CodeGuarder ( Lin et al., 2025a )	2025	30	L	Decomposes the query into sub-tasks, retrieves root causes and secure samples, re-ranks by model susceptibility, injects them into the prompt.	SEC	Security rate, CyberSecEval, average over LLMs: RACG without security knowledge 60.84 $\to$ 73.08	$\uparrow$	–
RAS ( Anupam et al., 2026 )	2026	6	R	Beam search over candidate optimizations; slow–fast exemplars are retrieved by an LLM-written natural-language description of the program.	EFF	Average speedup, PIE, Qwen3-Coder: dynamic retrieval 4.23 $\times$ $\to$ 8.70 $\times$	$\uparrow$	✓

Appendix

Table S36. Highlighted retrieval techniques: landmarks (L, most cited) and recent leaders (R, 2025–2026, largest reported gain). Year = year of the cited version. Cit. = Semantic Scholar citations, October 2026. COR = direction of the effect on correctness as coded in our extraction. Values as reported by the original authors; settings differ across rows, so values are not directly comparable.

Method	Dim.	Setting (model; benchmark; metric)	Reported
Type-constrained ( Mündler et al., 2025 )	COR	2B–34B; HumanEval (TypeScript); compiler-error reduction	74.8%
MRepE ( Gao et al., 2026 )	R/M	DeepSeek-R1 14B; MBPP; correctness loss	0.79%
CodeT ( Chen et al., 2023 )	COR	code-davinci-002; HumanEval; pass@1	65.8%
S* ( Li et al., 2025a )	COR	R1-Distill-Qwen-32B; LiveCodeBench; accuracy	85.7%

Appendix

Table S37. Further decoding-time results (highlighted techniques are in Table S38 ). Values as reported by the authors; settings differ, so values are not directly comparable.

Technique	Year	Cit.	L/R	Core idea	Target	Reported result	COR	Code
PG-TD, length reward ( Zhang et al., 2023b )	2023	272	L	Tree search during decoding; replacing the test reward with a length penalty or comment reward controls code style.	R/M	code length, APPS: default reward 248.42 $\to$ length penalty 190.73	$\leftrightarrow$	✓
Constrained beam sampling ( Fu et al., 2024 )	2024	61	L	Beam sampling with hand-written positive and negative key-phrase constraints per prompt, enforcing secure API patterns.	SEC	secure-pass@1, CodeGen-2.7B, CodeGuard+: beam 33.77 $\to$ 51.25	mixed	✓
RPG ( Dong et al., 2025b )	2025	15	L	A grammar-based pushdown automaton detects structural repetition during decoding and decays the likelihood of tokens causing it.	R/M	TR-S (lower is better), real-world repos: greedy 0.631, repetition penalty 0.358, RPG 0.335	$\uparrow$	✓
Surprisal-guided selection ( Barnes, 2026 )	2026	1	R	Among correct samples, keeps the one with the highest surprisal (lowest model confidence) instead of the most confident.	EFF	fast and correct, GPT-OSS-120B, KernelBench L1 subset: most confident 50% $\to$ 80%	nr	–
DuoSteer ( Yan and Yao, 2026 )	2026	0	R	Adds safety and correctness steering vectors at causally identified attention heads during inference.	SEC	vulnerability rate, Llama-3.1-8B-Instruct, five CWEs: average change $-$ 26.9% vs. unsteered	$\uparrow$	✓
FunPRM ( Zhang et al., 2026 )	2026	4	R	Prompts for code written function by function, treats each function as a step, and selects best-of- $N$ with a meta-corrected process reward model.	R/M	readability preference, Qwen2.5-7B-Coder, correct pairs: base 45.8 vs. FunPRM 54.2	$\uparrow$	–

Appendix

Table S38. Highlighted decoding and selection techniques: landmarks (L, most cited) and recent leaders (R, 2025–2026, largest reported gain). Year = year of the cited version. Cit. = Semantic Scholar citations, October 2026. COR = direction of the effect on correctness as coded in our extraction. Values as reported by the original authors; settings differ across rows, so values are not directly comparable.

Method	Signal	Base model	Target effect	COR effect
HexaCoder	SFT (CodeQL)	four LLMs	SEC $\uparrow$ up to 85% fewer vulnerable a (3 benchmarks)	$\leftrightarrow$ b
ProSec	PO (CWE tasks)	several code LLMs	SEC $\uparrow$ 25.2%–35.4% vs. SafeCoder data (PurpleLlama)	$\leftrightarrow$ b
PurpCode	SFT+RL	Qwen2.5-32B	SEC $\uparrow$ 12% over next best (CyberSecEval)	$\leftrightarrow$ b
SecCoderX	RL (learned RM)	CodeLlama-7B, Qwen2.5-Coder	SEC $\times$ COR $\uparrow$ ESR about 10%	in ESR
PIE	SFT (gem5)	GPT-3.5	EFF $\uparrow$ 6.86 speedup, best of 8 (PIE)	filtered
EffiCoder	SFT (profiler)	Qwen2.5-Coder-7B-Instr.	EFF $\uparrow$ time 48.4% lower (EffiBench)	44.8% $\to$ 57.7%

Appendix

Table S39. Reported effects of further training-time techniques; the highlighted ones are in Table S40 (signal source in parentheses; PO = preference optimization; a with two-step generation; b as stated by the authors). Values as reported by the original authors; settings differ across rows, so values are not directly comparable.

Technique	Year	Cit.	L/R	Core idea	Target	Reported result	COR	Code
SVEN ( He and Vechev, 2023 )	2023	337	L	Learns property-specific prefixes with region-specific losses on vulnerability-fix diffs; LM weights unchanged.	SEC	Secure rate, CodeGen-2.7B: original LM 59.1% $\to$ 92.3%	mixed	✓
SafeCoder ( He et al., 2024 )	2024	110	L	Joint standard and security instruction tuning; masked likelihood on secure code, masked unlikelihood on vulnerable code.	SEC	Code security, StarCoder-1B: standard instr. tuning 62.9 $\to$ 92.1	$\leftrightarrow$	✓
CodeUltraFeedback ( Weyssow et al., 2026 )	2026	81	L	GPT-3.5 judge rates responses of many LLMs on five coding preferences; SFT then DPO on the ratings.	R/M , EFF (judged)	Judge score, complexity/efficiency, CodeLlama-7B-Instr.: base 6.16 $\to$ 7.89	$\uparrow$	✓
CodeDPO ( Zhang et al., 2025a )	2025	63	L	Ranks self-generated code and tests by PageRank-style mutual validation, adds fastest-versus-slowest pairs, trains with DPO.	EFF	Mean time (s), HumanEval+, commonly solved: before 0.250 $\to$ 0.172	$\uparrow$	✓
Afterburner ( Du et al., 2025 )	2025	20	R	GRPO with execution-time rewards inside an iterative optimization loop that feeds back measured performance.	EFF	Beyond-I, Venus, Qwen2.5-3B: 31% $\to$ 45%	$\uparrow$	✓
PurpCode ( Liu et al., 2025a )	2025	23	R	Rule-learning SFT on safety rules, then GRPO with multi-objective rewards from code analyzers, tests and a learned verifier.	SEC	Secure code, CodeLMSec, Qwen2.5-32B: base 29.2 $\to$ 94.8	$\leftrightarrow$	✓

Appendix

Table S40. Highlighted training techniques: landmarks (L, most cited) and recent leaders (R, 2025–2026, largest reported gain). Year = year of the cited version. Cit. = Semantic Scholar citations, October 2026. COR = direction of the effect on correctness as coded in our extraction. Values as reported by the original authors; settings differ across rows, so values are not directly comparable.

Study	Scope	Dims	Technique coverage	Exp.	Years	Studies
Hou et al. (2024)	LLMs for software engineering	–	Model use, data, tuning, evaluation by task	no	2017–2024	395
Jiang et al. (2026)	LLM code generation	–	Data, training, prompting, retrieval, agents	no	nr	235
Dong et al. (2025a)	LLM-based code generation agents	–	Single- and multi-agent architectures	no	2022–2025	nr
Yang et al. (2024c)	Non-functional properties of code models	SEC ; efficiency of models, not code	Evaluation and enhancement methods per property	no	nr	146
Kamoi et al. (2024)	Self-correction of LLMs (all tasks)	–	Intrinsic and external feedback, fine-tuning	no	nr	nr
Chochlov et al. (2026)	Secondary studies of LLM code generation (tertiary)	–	Effects, scenarios, integration challenges	no	2017–2025	30

Appendix

Table S41. Existing secondary studies compared with this review. Dims: quality dimensions treated as a focus ( R/M , EFF , SEC ); “–” means none. Exp.: whether the study runs its own experiments. Years and study counts as stated by each survey ( nr : not stated). Values as reported by the original authors; settings differ across rows, so values are not directly comparable.

Figure S1. Left: included studies per year by technique family (2026 covers January to mid-October). Right: benchmarks used by included studies per year (top 12). Counts from LLM-assisted extraction reviewed by the author. Two stacked bar charts by year: included studies per technique family, and the twelve most used benchmarks.

LLM Code Generation

Large language model code generation studies how models translate instructions and context into executable programs and how to make them reliable.

See all 314 papers

May 22, 2026

Enhancing Reliability in LLM-Based Secure Code Generation

Large language models (LLMs) are widely used for code generation, but their security reliability remains inconsistent across languages and prompting strategies. Existing prompt engineering improves functional correctness but rarely ensures consistent security outcomes. We introduce the Mitigation-Aware Chain-of-Thought (MA-CoT) framework, which embeds task-specific CWE mitigation guidance and language-aware safeguards to reduce recurring vulnerabilities in generated code. We evaluate MA-CoT across three LLMs (gpt-5, claude-4.5, gemini-2.5), three programming languages (C, Java, Python), and four prompting strategies (Vanilla, Zero-shot, CoT, MA-CoT) on a 200-task primary dataset, with external validation on LLMSecEval. Using static analysis with expert validation, MA-CoT reduces total security findings from 92 to 39 (57.6%) on the primary dataset and from 73 to 4 (94.5%) on LLMSecEval. High-severity findings (Blocker + Critical) drop from 90 to 39 (56.7%) and from 45 to 2 (95.6%), respectively. Across both datasets, MA-CoT is the only strategy that consistently improves security reliability; Zero-shot and CoT are less reliable and may increase vulnerability, especially in C. We further introduce a strict layered attribution of vulnerability drivers (language-core vs. stack layers) and show that residual risk concentrates in hardening-oriented patterns (e.g., OS- and toolchain-dependent), motivating secure-by-construction primitives alongside prompting.

Mohammed F. Kharma, Mohammad Alkhanafseh, Ahmed Sabbah +1

the Department of Computer Science, Birzeit University, Ramallah, Palestine · the Department of Computer Science, University of Central Florida, Orlando, FL 32816 USA

LLM Evaluation CoT Reasoning LLM Security Secure Code Generation Prompt Engineering

Ask PDF

Oct 3, 2025

Investigating The Smells of LLM Generated Code

Context: Large Language Models (LLMs) are increasingly being used to generate program code. Much research has been reported on the functional correctness of generated code, but there is far less on code quality. Objectives: In this study, we propose a scenario-based method of evaluating the quality of LLM-generated code to identify the weakest scenarios, for which the quality of LLM-generated code should be improved. Methods: The method measures code smells, an important indicator of code quality, and compares them with a baseline formed from reference solutions of professionally written code. The test dataset is divided into various subsets according to the topics of the code and complexity of the coding tasks to represent different scenarios of using LLMs for code generation. We will also present an automated test system for this purpose and report experiments with the Java programs generated in response to prompts given to four state-of-the-art LLMs: Gemini Pro, ChatGPT, Codex, and Falcon. Results: We find that LLM-generated code has a higher incidence of code smells compared to reference solutions. Falcon performed the least badly, with a smell increase of 42.28%, followed by Gemini Pro (62.07%), ChatGPT (65.05%) and finally Codex (84.97%). The average smell increase across all LLMs was 63.34%, comprising 73.35% for implementation smells and 21.42% for design smells. We also found that the increase in code smells is greater for more complex coding tasks and for more advanced topics, such as those involving object-orientated concepts. Conclusion: In terms of code smells, LLMs' performances on various coding task complexities and topics are highly correlated to the quality of human-written code in the corresponding scenarios. However, the quality of LLM-generated code is noticeably poorer than human-written code.

Debalina Ghosh Paul, Hong Zhu, Ian Bayley

School of Engineering, Computing and Mathematics, Oxford Brookes University, Oxford, OX3 0BP, UK

Software Engineering Code Generation Evaluation LLM Code Generation

Ask PDF

May 6, 2026

Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code

Large language models (LLMs) frequently generate defective code, ranging from logical bugs to security vulnerabilities. Training-data quality is one contributor to these outcomes alongside model architecture, optimization and alignment, prompting and inference context, and decoding. However, the literature connecting specific training data quality issues to observable generated code quality issues has not been synthesized systematically. We review 114 included papers and establish taxonomies of eight generated-code quality dimensions and two broad groups of training-data issues. We then synthesize 12 recurring relationship families between training-data issues and generated-code outcomes, tracing each to its supporting papers, evidence basis, and limitations. We also synthesize detection and mitigation techniques across the data, model, and generation lifecycle. The findings position data curation as one layer of code-generation quality assurance coordinated with model- and inference-level controls, and identify provenance and controlled attribution as open research needs. Our repository is available at https://github.com/SYSUSELab/From-Data-to-Code.

Kaifeng He, Xiaojun Zhang, Peiliang Cai +7

Sun Yat-sen University, China · Nanyang Technological University, Singapore · Tongji University, China +1

Training Data Curation Code Generation Evaluation LLM Code Generation

Ask PDF

Abstract

LLM Code Generation

Explore similar work

Enhancing Reliability in LLM-Based Secure Code Generation

Investigating The Smells of LLM Generated Code

Bridging Generation and Training: A Systematic Review of Quality Issues in LLMs for Code