Secure and Functional Code Generation with Self-Evaluation

Md Afif Al Mamun; Gias Uddin

cs.CROct 24, 2026

Secure and Functional Code Generation with Self-Evaluation

Authors: Md Afif Al Mamun, Gias Uddin

Organizations: University of Calgary, Calgary, Canada · York University, Toronto, Canada

Abstract

Large Language Models (LLMs) can generate functionally correct code that remains insecure. Security-focused fine-tuning can be expensive and is not always available, motivating inference-time approaches. Security-oriented prompting and retrieval can reduce vulnerabilities, but may also remove required behavior or apply defenses incompatible with the task. The challenge is to identify relevant security knowledge and use it to improve security without sacrificing functionality. We present SecGent, an inference-time framework that retrieves weakness context from the Common Weakness Enumeration (CWE) database and applies it to secure code generation. By connecting task specifications to relevant textual CWE records, SecGent provides task-specific security guidance without relying on retrieved vulnerable or patched code examples. Complementary functional and security tests check the resulting implementation, and their execution feedback guides repairs toward satisfying both objectives. We evaluate SecGent with four open-source LLMs and Claude Sonnet 5.5 across five programming languages on the CWEval benchmark. It achieves the highest joint functional--security success against four inference-time baselines for all five models. Across the open-weight models, mean functional-security@1 (FS@1) improves by 15.5% over RAG, with gains in both functionality and security, and by 19.8% over the strongest prompt optimizer compiled for each model. On Sonnet, FS@1 reaches 92.1%, outperforming the strongest baseline AutoGen by 4.8%. In a separate comparison, SecGent outperforms Claude Code with security-focused prompting by over 6% in FS@1. Feedback-guided repair further improves FS@1 by up to 6.7% over generation without feedback, primarily by recovering functionality while checking security.

Figures & tables

Panel of Figure 1 — Figure 1. Excerpts for CWEval’s JavaScript CWE-113 header task under Qwen30B. Highlighted lines explain the security or functionality failure and SecGent ’s targeted check. Ticks and crosses show independent CWEval verdicts. Unrelated code are omitted to save space.

Figure 2. SecGent inference trace for CWEval’s CWE-117 task. Red marks the detected log-injection behavior; green marks the accepted neutralization. The same coding agent is used for code generation and repair through Pareto gate where $(1,1)\succ_{T}(1,0)$ authorizes replacement. A \system{} trace for CWEval's C CWE-117 task, organized into offline catalogue construction, task-conditioned weakness retrieval, test and code generation, test execution, and evidence-guided repair. Before inference, an LLM rewrites MITRE's CWE-117 record into a catalogue record with a code-level description and keywords. The initial candidate inserts an untrusted message directly into a log entry. A generated security test supplies newline and tab characters and fails. The same coding agent then proposes a repair that replaces those characters with spaces. A dashed arrow returns the proposal to the frozen test suite; both tests pass and the Pareto gate retains the repair.

Panel of Figure 3 — Figure 3. Offline catalogue construction. MITRE CWE records are automatically extracted and their names and descriptions are rewritten into code-level descriptions and keywords. The resulting catalogue retains the identifiers, names, and mitigations and is frozen and embedded for retrieval. Dashed arrows show possible additional sources, including NVD, and rebuilding after a source update.

Method	Gemma4				GPT-OSS				Qwen14B				Qwen30B				Sonnet-5.5
Method	F@1	S@1	FS@1	FS@5	F@1	S@1	FS@1	FS@5	F@1	S@1	FS@1	FS@5	F@1	S@1	FS@1	FS@5	F@1	S@1	FS@1	FS@5
Direct	70.9	52.3	49.9	63.9	74.1	54.8	53.6	75.6	70.4	51.6	48.6	65.5	76.5	58.0	54.6	64.7	84.7	81.2	81.2	87.4
RAG	71.3	75.6	67.6	79.8	73.9	78.0	70.1	84.0	67.4	59.0	52.9	75.6	65.4	69.1	57.1	73.9	89.6	89.1	86.6	89.9
CodeGuarder	61.0	60.0	53.1	69.7	63.2	66.6	59.3	79.8	54.3	52.1	43.0	67.2	62.0	59.5	52.1	69.7	72.8	73.3	69.6	76.5
AutoGen	78.0	68.6	64.2	75.6	83.9	65.0	62.5	77.3	76.0	56.8	53.3	68.9	78.7	59.2	56.5	63.0	89.9	88.6	87.9	89.9
SecGent	80.2	82.5	77.1	85.7	85.7	84.0	79.7	91.6	75.5	68.1	63.2	81.5	77.0	72.8	66.2	77.3	92.8	93.1	92.1	94.1

Table 1. RQ1: effectiveness against inference-time baselines on all 119 CWEval tasks (

n{=}5

samples per task; percentages). Func-Sec@

k

(FS@

k

) is the primary metric, and its best result per model and column is bold; all metrics at

k\in\{1,2,5\}

are in the replication package.

Method	Gemma4				GPT-OSS				Qwen14B				Qwen30B
Method	F@1	S@1	FS@1	FS@5	F@1	S@1	FS@1	FS@5	F@1	S@1	FS@1	FS@5	F@1	S@1	FS@1	FS@5
BSFewShot	62.5	67.1	61.7	75.0	70.8	76.2	68.8	87.5	42.1	45.8	34.6	64.6	72.9	66.2	61.2	77.1
MIPROv2	62.9	67.5	61.7	81.2	67.9	77.9	65.8	83.3	50.4	54.6	42.5	70.8	63.8	67.5	58.3	77.1
GEPA	73.8	75.4	71.7	83.3	69.6	75.8	67.9	85.4	30.0	37.1	23.3	45.8	63.3	57.1	52.5	72.9
SecGent	83.3	85.4	80.8	89.6	87.5	81.7	79.6	91.7	72.9	71.3	64.2	79.2	80.4	70.8	67.9	79.2

Table 2. RQ2: comparison with prompt optimizers for four open-weight models on the 48 held-out CWEval tasks.

Panel of Figure 4 — Figure 4. Preserving valid directory access (GPT-OSS, Python). A Python directory-listing example comparing BootstrapFewShot, which rejects valid absolute paths, with CWEgent, which passes the directory as a subprocess argument without shell evaluation. The five candidates yield one and five joint functional-security successes, respectively.

Figure 5. FS@1 (%) by language and model on the 48 held-out tasks. Grey band: range of the seven baselines; grey dot: highest baseline score in each row; blue dot: SecGent .

	Repair loop			Joint		Per objective
Model	Usable	Tried	Kept	Impr.	Regr.	F+	F $-$	S+	S $-$
Gemma4	485	270	45	30	1	26	2	28	0
GPT-OSS	500	259	130	13	7	17	2	14	5
Qwen14B	510	385	34	5	2	8	2	6	1
Qwen30B	470	305	30	7	0	7	0	5	0
Sonnet-5.5	590	73	52	16	1	16	1	5	1

Table 3. Generated-test repair outcomes (595 samples per model). Among kept repairs, Impr./Regr. count joint CWEval gains/losses in functional-security; F and S show functional and security changes.

Panel of Figure 6 — Figure 6. Repairing a functional failure with a passing security test. A C++ redirect-validation repair. The initial check rejects a valid subdomain while passing a generated security test. A functional test exposes the mistake, and the retained repair matches the complete domain suffix preceded by a dot while still rejecting the attacker-controlled host.

Method	F@1	S@1	FS@1	FS@5
Claude Code (func)	94.2	87.1	86.7	87.5
Claude Code (func-sec)	88.8	91.2	87.9	91.7
SecGent	93.3	93.8	93.3	95.8

Table 5. Claude Code and SecGent with Sonnet-5.5 on the 48 held-out tasks.

Panel of Figure 7 — Figure 7. A repair that satisfies an incorrect generated test (Gemma4, CWE-078). Candidate verdicts are CWEval’s; both generated tests pass after repair.

Configuration	R@5	R@10
Direct dense	47.1	64.7
Direct dense + rerank	67.2	72.3
Catalog + HyDE query	75.6	85.7
Catalog + HyDE query + rerank	89.1	93.3

Figure 8. Ground-truth CWE recall on all 119 tasks (%).

Secure Code Generation

Secure code generation studies how to produce code that meets functional requirements while avoiding security vulnerabilities.

See all 48 papers

May 22, 2026

Enhancing Reliability in LLM-Based Secure Code Generation

Large language models (LLMs) are widely used for code generation, but their security reliability remains inconsistent across languages and prompting strategies. Existing prompt engineering improves functional correctness but rarely ensures consistent security outcomes. We introduce the Mitigation-Aware Chain-of-Thought (MA-CoT) framework, which embeds task-specific CWE mitigation guidance and language-aware safeguards to reduce recurring vulnerabilities in generated code. We evaluate MA-CoT across three LLMs (gpt-5, claude-4.5, gemini-2.5), three programming languages (C, Java, Python), and four prompting strategies (Vanilla, Zero-shot, CoT, MA-CoT) on a 200-task primary dataset, with external validation on LLMSecEval. Using static analysis with expert validation, MA-CoT reduces total security findings from 92 to 39 (57.6%) on the primary dataset and from 73 to 4 (94.5%) on LLMSecEval. High-severity findings (Blocker + Critical) drop from 90 to 39 (56.7%) and from 45 to 2 (95.6%), respectively. Across both datasets, MA-CoT is the only strategy that consistently improves security reliability; Zero-shot and CoT are less reliable and may increase vulnerability, especially in C. We further introduce a strict layered attribution of vulnerability drivers (language-core vs. stack layers) and show that residual risk concentrates in hardening-oriented patterns (e.g., OS- and toolchain-dependent), motivating secure-by-construction primitives alongside prompting.

Mohammed F. Kharma, Mohammad Alkhanafseh, Ahmed Sabbah +1

the Department of Computer Science, Birzeit University, Ramallah, Palestine · the Department of Computer Science, University of Central Florida, Orlando, FL 32816 USA

LLM Evaluation CoT Reasoning LLM Security Secure Code Generation Prompt Engineering

Ask PDF

Jul 2, 2024

Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking

Large language models (LLMs) are increasingly used for program synthesis, yet they often generate code that is functionally plausible but insecure. Progress in secure code generation has been hindered by benchmarks that are small, non-executable, leak mitigation details, or rely on noisy analyzers and subjective judgments, making it difficult to measure whether security improves without sacrificing correctness. We address these gaps with CodeSecEval, an execution-based benchmark for secure code generation, comprising 255 Python tasks spanning 77 CWE categories. Each task provides paired insecure and secure implementations together with executable functional and vulnerability-targeted security tests, enabling precise and reproducible evaluation of secure code generation and insecure-code repair. Building on CodeSecEval, we propose SecAwareCoder, an agent-based framework that shifts code generation toward secure-by-construction synthesis. SecAwareCoder performs task-adaptive threat modeling to identify security-sensitive regions and derive task-grounded vulnerability hypotheses, uses these hypotheses to guide both constraint-aware code generation and security-aware test synthesis, and leverages execution feedback for targeted refinement. Experiments across multiple LLM backbones show that SecAwareCoder consistently improves Pass@1 and security robustness over prompting and analyzer-driven baselines, narrowing the security--correctness gap in LLM code generation.

Jiexin Wang, Liuwen Cao, Xitong Luo +6

South China University of Technology, Guangzhou, China · University of Innsbruck, Innsbruck, Austria

Execution-Guided Code Generation Agentic Code Generation Secure Code Generation

Ask PDF

Oct 10, 2026

SEEK: Agentic Knowledge Graph Retrieval for Secure Code Generation

Large Language Models (LLMs) can generate functionally correct code while introducing exploitable security weaknesses, particularly when developers specify functionality without explicit security requirements. Recent state-of-the-art approaches mitigate this problem using external security knowledge, but largely rely on flat similarity-based retrieval over independent artifacts, making it difficult to infer the relevant vulnerability space and retrieve targeted knowledge when direct matches are unavailable. We propose SEEK, an agentic Graph RAG framework for secure code generation that constructs a security knowledge graph using the Common Weakness Enumeration (CWE) taxonomy as its structural backbone and injects it with curated real-world CVE vulnerability-fix instances. At inference time, a Planner Agent predicts potential CWE weaknesses associated with the developer query, which guide Graph Scoping to restrict retrieval to relevant regions of the CWE taxonomy before deterministic multi-faceted candidate ranking. A Selection Agent subsequently validates the predicted weaknesses and retrieved candidates prior to code generation. Across five open-source LLMs, we evaluate SEEK in an out-of-distribution setting on CodeGuard+ and CyberSecEval, spanning C, C++, and Python. SEEK achieves the highest SecurePass@1 of 77.45% on CodeGuard+ and the highest aggregate ICD pass rate of 79.45% on CyberSecEval, surpassing prior approaches. Moreover, SEEK achieves 85.2% SecurePass@1 across 13 unseen CWE types with no directly linked vulnerability-fix instances, demonstrating strong generalizability, while maintaining acceptable inference latency costs.

Varun Gadey, Markus Tohn, Lichao Wu +2

University of Duisburg-Essen, Germany · University of Bristol, UK · Technical University of Darmstadt, Germany

Secure Code Generation RAG Security Cybersecurity Graph Retrieval-Augmented Generation Agentic Code Generation

Ask PDF

Abstract

Secure Code Generation

Explore similar work

Enhancing Reliability in LLM-Based Secure Code Generation

Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking

SEEK: Agentic Knowledge Graph Retrieval for Secure Code Generation