Secure and Functional Code Generation with Self-Evaluation

Md Afif Al Mamun; Gias Uddin

cs.CROct 24, 2026

Secure and Functional Code Generation with Self-Evaluation

Authors: Md Afif Al Mamun, Gias Uddin

Organizations: University of Calgary, Calgary, Canada · York University, Toronto, Canada

Abstract

Large Language Models (LLMs) can generate functionally correct code that remains insecure. Security-focused fine-tuning can be expensive and is not always available, motivating inference-time approaches. Security-oriented prompting and retrieval can reduce vulnerabilities, but may also remove required behavior or apply defenses incompatible with the task. The challenge is to identify relevant security knowledge and use it to improve security without sacrificing functionality. We present SecGent, an inference-time framework that retrieves weakness context from the Common Weakness Enumeration (CWE) database and applies it to secure code generation. By connecting task specifications to relevant textual CWE records, SecGent provides task-specific security guidance without relying on retrieved vulnerable or patched code examples. Complementary functional and security tests check the resulting implementation, and their execution feedback guides repairs toward satisfying both objectives. We evaluate SecGent with four open-source LLMs and Claude Sonnet 5.5 across five programming languages on the CWEval benchmark. It achieves the highest joint functional--security success against four inference-time baselines for all five models. Across the open-weight models, mean functional-security@1 (FS@1) improves by 15.5% over RAG, with gains in both functionality and security, and by 19.8% over the strongest prompt optimizer compiled for each model. On Sonnet, FS@1 reaches 92.1%, outperforming the strongest baseline AutoGen by 4.8%. In a separate comparison, SecGent outperforms Claude Code with security-focused prompting by over 6% in FS@1. Feedback-guided repair further improves FS@1 by up to 6.7% over generation without feedback, primarily by recovering functionality while checking security.

Figures & tables

Panel of Figure 1 — Figure 1. Excerpts for CWEval’s JavaScript CWE-113 header task under Qwen30B. Highlighted lines explain the security or functionality failure and SecGent ’s targeted check. Ticks and crosses show independent CWEval verdicts. Unrelated code are omitted to save space.

Figure 2. SecGent inference trace for CWEval’s CWE-117 task. Red marks the detected log-injection behavior; green marks the accepted neutralization. The same coding agent is used for code generation and repair through Pareto gate where $(1,1)\succ_{T}(1,0)$ authorizes replacement. A \system{} trace for CWEval's C CWE-117 task, organized into offline catalogue construction, task-conditioned weakness retrieval, test and code generation, test execution, and evidence-guided repair. Before inference, an LLM rewrites MITRE's CWE-117 record into a catalogue record with a code-level description and keywords. The initial candidate inserts an untrusted message directly into a log entry. A generated security test supplies newline and tab characters and fails. The same coding agent then proposes a repair that replaces those characters with spaces. A dashed arrow returns the proposal to the frozen test suite; both tests pass and the Pareto gate retains the repair.

Panel of Figure 3 — Figure 3. Offline catalogue construction. MITRE CWE records are automatically extracted and their names and descriptions are rewritten into code-level descriptions and keywords. The resulting catalogue retains the identifiers, names, and mitigations and is frozen and embedded for retrieval. Dashed arrows show possible additional sources, including NVD, and rebuilding after a source update.

Method	Gemma4				GPT-OSS				Qwen14B				Qwen30B				Sonnet-5.5
Method	F@1	S@1	FS@1	FS@5	F@1	S@1	FS@1	FS@5	F@1	S@1	FS@1	FS@5	F@1	S@1	FS@1	FS@5	F@1	S@1	FS@1	FS@5
Direct	70.9	52.3	49.9	63.9	74.1	54.8	53.6	75.6	70.4	51.6	48.6	65.5	76.5	58.0	54.6	64.7	84.7	81.2	81.2	87.4
RAG	71.3	75.6	67.6	79.8	73.9	78.0	70.1	84.0	67.4	59.0	52.9	75.6	65.4	69.1	57.1	73.9	89.6	89.1	86.6	89.9
CodeGuarder	61.0	60.0	53.1	69.7	63.2	66.6	59.3	79.8	54.3	52.1	43.0	67.2	62.0	59.5	52.1	69.7	72.8	73.3	69.6	76.5
AutoGen	78.0	68.6	64.2	75.6	83.9	65.0	62.5	77.3	76.0	56.8	53.3	68.9	78.7	59.2	56.5	63.0	89.9	88.6	87.9	89.9
SecGent	80.2	82.5	77.1	85.7	85.7	84.0	79.7	91.6	75.5	68.1	63.2	81.5	77.0	72.8	66.2	77.3	92.8	93.1	92.1	94.1

Table 1. RQ1: effectiveness against inference-time baselines on all 119 CWEval tasks (

n{=}5

samples per task; percentages). Func-Sec@

k

(FS@

k

) is the primary metric, and its best result per model and column is bold; all metrics at

k\in\{1,2,5\}

are in the replication package.

Method	Gemma4				GPT-OSS				Qwen14B				Qwen30B
Method	F@1	S@1	FS@1	FS@5	F@1	S@1	FS@1	FS@5	F@1	S@1	FS@1	FS@5	F@1	S@1	FS@1	FS@5
BSFewShot	62.5	67.1	61.7	75.0	70.8	76.2	68.8	87.5	42.1	45.8	34.6	64.6	72.9	66.2	61.2	77.1
MIPROv2	62.9	67.5	61.7	81.2	67.9	77.9	65.8	83.3	50.4	54.6	42.5	70.8	63.8	67.5	58.3	77.1
GEPA	73.8	75.4	71.7	83.3	69.6	75.8	67.9	85.4	30.0	37.1	23.3	45.8	63.3	57.1	52.5	72.9
SecGent	83.3	85.4	80.8	89.6	87.5	81.7	79.6	91.7	72.9	71.3	64.2	79.2	80.4	70.8	67.9	79.2

Table 2. RQ2: comparison with prompt optimizers for four open-weight models on the 48 held-out CWEval tasks.

Panel of Figure 4 — Figure 4. Preserving valid directory access (GPT-OSS, Python). A Python directory-listing example comparing BootstrapFewShot, which rejects valid absolute paths, with CWEgent, which passes the directory as a subprocess argument without shell evaluation. The five candidates yield one and five joint functional-security successes, respectively.

Figure 5. FS@1 (%) by language and model on the 48 held-out tasks. Grey band: range of the seven baselines; grey dot: highest baseline score in each row; blue dot: SecGent .

	Repair loop			Joint		Per objective
Model	Usable	Tried	Kept	Impr.	Regr.	F+	F $-$	S+	S $-$
Gemma4	485	270	45	30	1	26	2	28	0
GPT-OSS	500	259	130	13	7	17	2	14	5
Qwen14B	510	385	34	5	2	8	2	6	1
Qwen30B	470	305	30	7	0	7	0	5	0
Sonnet-5.5	590	73	52	16	1	16	1	5	1

Table 3. Generated-test repair outcomes (595 samples per model). Among kept repairs, Impr./Regr. count joint CWEval gains/losses in functional-security; F and S show functional and security changes.

Panel of Figure 6 — Figure 6. Repairing a functional failure with a passing security test. A C++ redirect-validation repair. The initial check rejects a valid subdomain while passing a generated security test. A functional test exposes the mistake, and the retained repair matches the complete domain suffix preceded by a dot while still rejecting the attacker-controlled host.

Method	F@1	S@1	FS@1	FS@5
Claude Code (func)	94.2	87.1	86.7	87.5
Claude Code (func-sec)	88.8	91.2	87.9	91.7
SecGent	93.3	93.8	93.3	95.8

Table 5. Claude Code and SecGent with Sonnet-5.5 on the 48 held-out tasks.

Panel of Figure 7 — Figure 7. A repair that satisfies an incorrect generated test (Gemma4, CWE-078). Candidate verdicts are CWEval’s; both generated tests pass after repair.

Configuration	R@5	R@10
Direct dense	47.1	64.7
Direct dense + rerank	67.2	72.3
Catalog + HyDE query	75.6	85.7
Catalog + HyDE query + rerank	89.1	93.3

Figure 8. Ground-truth CWE recall on all 119 tasks (%).

Secure Code Generation

Secure code generation studies how to produce code that meets functional requirements while avoiding security vulnerabilities.

See all 48 papers

Enhancing Reliability in LLM-Based Secure Code Generation
May 22, 2026Mohammed F. Kharma, Mohammad Alkhanafseh, Ahmed Sabbah +1LLM Evaluation CoT Reasoning
the Department of Computer Science, Birzeit University, Ramallah, Palestine · the Department of Computer Science, University of Central Florida, Orlando, FL 32816 USA
Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking
Jul 2, 2024Jiexin Wang, Liuwen Cao, Xitong Luo +6Execution-Guided Code Generation Agentic Code Generation
South China University of Technology, Guangzhou, China · University of Innsbruck, Innsbruck, Austria
SEEK: Agentic Knowledge Graph Retrieval for Secure Code Generation
Oct 10, 2026Varun Gadey, Markus Tohn, Lichao Wu +2Secure Code Generation RAG Security
University of Duisburg-Essen, Germany · University of Bristol, UK · Technical University of Darmstadt, Germany

May 22, 2026J/K move · Enter open · S save

Enhancing Reliability in LLM-Based Secure Code Generation

Mohammed F. Kharma, Mohammad Alkhanafseh, Ahmed Sabbah, David Mohaisen

the Department of Computer Science, Birzeit University, Ramallah, Palestine · the Department of Computer Science, University of Central Florida, Orlando, FL 32816 USA

LLM Evaluation CoT Reasoning LLM Security Secure Code Generation Prompt Engineering

Open paper AskPDF arXiv

Large language models (LLMs) are widely used for code generation, but their security reliability remains inconsistent across languages and prompting strategies. Existing prompt engineering improves functional correctness but rarely ensures consistent security outcomes. We introduce the Mitigation-Aware Chain-of-Thought (MA-CoT) framework, which embeds task-specific CWE mitigation guidance and language-aware safeguards to reduce recurring vulnerabilities in generated code. We evaluate MA-CoT across three LLMs (gpt-5, claude-4.5, gemini-2.5), three programming languages (C, Java, Python), and four prompting strategies (Vanilla, Zero-shot, CoT, MA-CoT) on a 200-task primary dataset, with external validation on LLMSecEval. Using static analysis with expert validation, MA-CoT reduces total security findings from 92 to 39 (57.6%) on the primary dataset and from 73 to 4 (94.5%) on LLMSecEval. High-severity findings (Blocker + Critical) drop from 90 to 39 (56.7%) and from 45 to 2 (95.6%), respectively. Across both datasets, MA-CoT is the only strategy that consistently improves security reliability; Zero-shot and CoT are less reliable and may increase vulnerability, especially in C. We further introduce a strict layered attribution of vulnerability drivers (language-core vs. stack layers) and show that residual risk concentrates in hardening-oriented patterns (e.g., OS- and toolchain-dependent), motivating secure-by-construction primitives alongside prompting.

Abstract

Secure Code Generation

Explore similar work

Enhancing Reliability in LLM-Based Secure Code Generation

Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking

SEEK: Agentic Knowledge Graph Retrieval for Secure Code Generation