JevVibe: Efficient Classification-Guided Secure Code Generation

Arshak Rezvani; Sasha Behrouzi; Ahmad-Reza Sadeghi

cs.CRSep 28, 2026

JevVibe: Efficient Classification-Guided Secure Code Generation

Authors: Arshak Rezvani, Sasha Behrouzi, Ahmad-Reza Sadeghi

Organizations: Technical University of Darmstadt, Germany

Abstract

Large language models can generate functionally correct code that still contains security weaknesses, motivating repair pipelines that first diagnose a weakness type before deciding how to fix it. The Common Weakness Enumeration (CWE) provides a standardized vocabulary for such diagnoses, but asking an autoregressive language model to generate a CWE label and extracting it from the response raises questions about output validity, speed, and cost, as well as accuracy. We evaluate Jev, a decision model that instead selects directly from a declared set of candidates and returns a probability for each, against six open-weight autoregressive models and a frontier proprietary model, GPT-5.6-Sol, on a controlled 50-way CWE classification task over 1,916 CyberSecEval benchmark examples. Jev outperforms all six open-weight baselines on every classification and ranking metric, while its comparison with GPT-5.6-Sol depends on the metric: GPT-5.6-Sol achieves higher Top-1 accuracy and Macro-F1, whereas Jev achieves higher Top-3 and Top-5 accuracy and a nearly identical MRR, at $6.27\times$ lower median API latency and $55.9\times$ lower estimated API cost. We further build JevVibe, a diagnosis-guided repair agent that uses predicted CWE labels to repair code generated by Qwen2.5-Coder-32B-Instruct. With Jev providing the diagnosis, the agent increases the detector-measured security pass rate from 63.5% before repair to 70.7%, compared with 66.1% for LLM-guided repair. These results show that JevVibe is effective at improving the security of generated code, with Jev providing reliable and efficient CWE classification.

Figures & tables

Panel of Figure 1 — Fig. 1: Prompt formats used for CWE classification.

Fig. 2: CWE-guided code repair agent. Qwen2.5-Coder-32B-Instruct first generates the baseline code. The generated code is either evaluated directly or diagnosed using the LLM-based CWE classifier or Jev. The predicted Top-1 CWE and its definition are then provided, together with the original task and generated code, to the same Qwen2.5-Coder-32B-Instruct repair model. Original and repaired outputs are evaluated using CyberSecEval’s official Insecure Code Detector.

Model	Top-1 $\uparrow$	Top-3 $\uparrow$	Top-5 $\uparrow$	Macro-F1 $\uparrow$	MRR $\uparrow$	Coverage $\uparrow$
DeepSeek-Coder-6.7B-Instruct	0.034	0.136	0.184	0.015	0.090	0.8205
CodeLlama-34B-Instruct	0.175	0.231	0.241	0.071	0.202	0.7114
Llama-3.1-8B-Instruct	0.177	0.274	0.306	0.084	0.228	0.9990
DeepSeek-Coder-33B-Instruct	0.238	0.321	0.334	0.114	0.279	0.9468
Qwen2.5-Coder-7B-Instruct	0.303	0.468	0.532	0.131	0.385	1.0000
Qwen2.5-Coder-32B-Instruct	0.335	0.519	0.559	0.199	0.428	0.9995

TABLE I: CWE classification performance on the CyberSecEval Instruct benchmark. Top-

k

accuracy, Macro-F1, and MRR are computed over successfully evaluated predictions. Coverage is the fraction of all 1,916 examples for which a valid classification is obtained. Bold denotes the best overall result; underlining denotes the strongest autoregressive result when distinct from the overall best.

Fig. 3: Overall CWE classification performance. The left panel reports Top-1 accuracy and the right panel reports Macro-F1. GPT-5.6-Sol obtains the highest score on both metrics, followed by Jev. Jev substantially outperforms all evaluated open-weight autoregressive models.

Fig. 4: Top-1 classification accuracy of GPT-5.6-Sol, Jev, and Qwen2.5-Coder-32B-Instruct. Left: the 12 most frequent CWE classes in the benchmark, with class support shown in parentheses. Right: results grouped by programming language, with subset size shown in parentheses. The language-wise comparison is descriptive because programming-language subsets contain different mixtures of CWE classes.

Metric	GPT-5.6-Sol	Jev
Estimated API cost	$15.83	$0.283
Median latency (s)	1.531	0.244
P95 latency (s)	3.737	0.292

TABLE II: Inference efficiency of GPT-5.6-Sol and Jev on the CWE classification benchmark. Latency is measured as end-to-end API request latency. Costs are estimated using the provider pricing in effect during the experiments.

Fig. 5: Security pass rate for the original Qwen2.5-Coder-32B-Instruct generations and the two diagnosis-guided repair conditions. Results are shown by programming language and overall. Higher values indicate a larger fraction of generated programs that are not flagged as vulnerable by the CyberSecEval insecure-code evaluator.

Explore similar work

Jul 2, 2024cs.SE

Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking

Large language models (LLMs) are increasingly used for program synthesis, yet they often generate code that is functionally plausible but insecure. Progress in secure code generation has been hindered by benchmarks that are small, non-executable, leak mitigation details, or rely on noisy analyzers and subjective judgments, making it difficult to measure whether security improves without sacrificing correctness. We address these gaps with CodeSecEval, an execution-based benchmark for secure code generation, comprising 255 Python tasks spanning 77 CWE categories. Each task provides paired insecure and secure implementations together with executable functional and vulnerability-targeted security tests, enabling precise and reproducible evaluation of secure code generation and insecure-code repair. Building on CodeSecEval, we propose SecAwareCoder, an agent-based framework that shifts code generation toward secure-by-construction synthesis. SecAwareCoder performs task-adaptive threat modeling to identify security-sensitive regions and derive task-grounded vulnerability hypotheses, uses these hypotheses to guide both constraint-aware code generation and security-aware test synthesis, and leverages execution feedback for targeted refinement. Experiments across multiple LLM backbones show that SecAwareCoder consistently improves Pass@1 and security robustness over prompting and analyzer-driven baselines, narrowing the security--correctness gap in LLM code generation.

Jiexin Wang, Liuwen Cao, Xitong Luo +6

South China University of Technology, Guangzhou, China · University of Innsbruck, Innsbruck, Austria

Code Generation Vulnerable Code Large Language Model Safety Model Vulnerabilities Large Language Model Generation Penetration Testing

Ask PDF

May 22, 2026cs.CR

Enhancing Reliability in LLM-Based Secure Code Generation

Large language models (LLMs) are widely used for code generation, but their security reliability remains inconsistent across languages and prompting strategies. Existing prompt engineering improves functional correctness but rarely ensures consistent security outcomes. We introduce the \textit{Mitigation-Aware Chain-of-Thought (MA-CoT)} framework, which embeds task-specific CWE mitigation guidance and language-aware safeguards to reduce recurring vulnerabilities in generated code. We evaluate MA-CoT across three LLMs (gpt-5, claude-4.5, gemini-2.5), three programming languages (C, Java, Python), and four prompting strategies (Vanilla, Zero-shot, CoT, MA-CoT) on a 200-task primary dataset, with external validation on LLMSecEval. Using static analysis with expert validation, MA-CoT reduces total security findings from 92 to 39 (57.6%) on the primary dataset and from 73 to 4 (94.5%) on LLMSecEval. High-severity findings (Blocker + Critical) drop from 90 to 39 (56.7%) and from 45 to 2 (95.6%), respectively. Across both datasets, MA-CoT is the only strategy that consistently improves security reliability; Zero-shot and CoT are less reliable and may increase vulnerability, especially in C. We further introduce a strict layered attribution of vulnerability drivers (language-core vs. stack layers) and show that residual risk concentrates in hardening-oriented patterns (e.g., OS- and toolchain-dependent), motivating secure-by-construction primitives alongside prompting.

Mohammed F. Kharma, Mohammad Alkhanafseh, Ahmed Sabbah +1

the Department of Computer Science, Birzeit University, Ramallah, Palestine · the Department of Computer Science, University of Central Florida, Orlando, FL 32816 USA

Large Language Model Safety Vulnerable Code Code Generation Chain-of-Thought Reasoning

Ask PDF

May 8, 2026cs.CR

SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization

LLM coding agents now generate code at an unprecedented scale, yet LLM-generated code introduces cybersecurity vulnerabilities into codebases without human involvement. Even when frontier models are explicitly asked to write secure production code with relevant weaknesses to avoid in context, we find that they still produce verifiable vulnerabilities on average 23% of the time across a corpus of 250 benign coding prompts. We introduce SecureForge, an automated pipeline that both audits security risks of frontier models and produces auditing-informed secure system prompts that reduce output security vulnerabilities while maintaining unit test performance. SecureForge first identifies benign prompts that produce statically detectable vulnerabilities, and then amplifies them into a large synthetic prompt corpus of diverse scenarios using a Markovian sampling technique to jointly maintain error rates and prompt diversity. This corpus is then used to iteratively optimize the system prompts to reduce output security vulnerabilities. On frontier models, SecureForge yields a statistically significant Pareto improvement in both unit test success and output security, with output vulnerabilities reduced by up to 48%. The resulting system prompts transfer zero-shot to in-the-wild coding agent prompts, without any exposure to real user prompt distributions during optimization.

Houjun Liu, Lisa Einstein, John Yang +5

Stanford University

Vulnerable Code Coding Agents Model Auditing Prompt Optimization

Ask PDF