cs.CRFeb 17, 2026

SecCodeBench-V2 Technical Report

Authors: Longfei Chen, Ji Zhao, Lanxiao Cui, Tong Su, Xingbo Pan, Ziyang Li, Yongxing Wu, Qijiang Cao, +19 more

Organizations: TSINGHUA UNIVERSITY, BEIJING, CHINA · ALIBABA GROUP, HANGZHOU, CHINA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We introduce SecCodeBench-V2, a publicly released benchmark for evaluating Large Language Model (LLM) copilots' capabilities of generating secure code. SecCodeBench-V2 comprises 98 generation and fix scenarios derived from Alibaba Group's industrial productions, where the underlying security issues span 22 common CWE (Common Weakness Enumeration) categories across five programming languages: Java, C, Python, Go, and JavaScript. SecCodeBench-V2 adopts a function-level task formulation: each scenario provides a complete project scaffold and requires the model to implement or patch a designated target function under fixed interfaces and dependencies. For each scenario, SecCodeBench-V2 provides executable proof-of-concept (PoC) test cases for both functional validation and security verification. All test cases are authored and double-reviewed by security experts, ensuring high fidelity, broad coverage, and reliable ground truth. Beyond the benchmark itself, we build a unified evaluation pipeline that assesses models primarily via dynamic execution. For most scenarios, we compile and run model-generated artifacts in isolated environments and execute PoC test cases to validate both functional correctness and security properties. For scenarios where security issues cannot be adjudicated with deterministic test cases, we additionally employ an LLM-as-a-judge oracle. To summarize performance across heterogeneous scenarios and difficulty levels, we design a Pass@K-based scoring protocol with principled aggregation over scenarios and severity, enabling holistic and comparable evaluation across models. Overall, SecCodeBench-V2 provides a rigorous and reproducible foundation for assessing the security posture of AI coding assistants, with results and artifacts released at https://alibaba.github.io/sec-code-bench. The benchmark is publicly available at https://github.com/alibaba/sec-code-bench.

Figures & tables

Figure 1. Overall pipeline of SecCodeBench-V2 .
ScenarioTaskEvaluation goal
genCode generationGenerating secure code.
gen-hintsGeneration w/ hintsGenerating secure code when given an explicit safety hint.
fixVulnerability repairIdentifying and fixing security vulnerabilities in existing code.
fix-hintsRepair w/ hintsFixing vulnerability when given an explicit safety hint.
Table 1. Four prompt scenarios supported by each test case.
ComponentDescriptionExample
Task descriptionDetailed business context and requirements.“Design a social-media content crawler…”
Method signatureThe required function interface.fetch_metadata(page_url: str) -> dict
Parameter spec.Detailed parameter definitions.Name/type/purpose of each argument.
Return spec.Return schema and semantics.“Return a dict with title and description.”
ContextExecution environment and relevant context.Use C functions such as strcpy and strncpy .
Dependency constraintsAllowed libraries and frameworks.Use only the listed dependencies.
Table 2. Core structure of each prompt.
SeverityJavaPythonC/C++GoJavaScriptTotal
Critical88115234
High33545249
Medium12003015
Table 3. Distribution of vulnerability severity levels across programming languages.
Vulnerability TypeCWE IDJavaPythonC/C++GoJavaScriptTotal
Path TraversalCWE-223–––25
Command InjectionCWE-78221229
XSSCWE-791––2–3
SQL InjectionCWE-8983–2–13
Code InjectionCWE-942113–7
Log InjectionCWE-1171––––1
Table 4. Distribution of security domains (CWE) across programming languages.

Explore similar work

CardsList
Jul 25, 2026

Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios

Large Language Models (LLMs) are widely used for code generation, yet their security behavior in realistic development workflows remains underexplored. Existing benchmarks often rely on explicitly specified security requirements, failing to capture real-world scenarios where prompts are frequently ambiguous or incomplete. In this paper, we adopt a developer-centric perspective and identify three representative risk scenarios that commonly lead to security vulnerabilities in LLM-generated code: Ambiguous Requirements, Under-Specified Operational Context, and Security--Functionality Conflict. Based on these scenarios, we construct a large-scale benchmark comprising 2,700 test cases, enabling fine-grained evaluation of LLM security under realistic conditions. Extensive evaluation of eight state-of-the-art LLMs reveals that all models exhibit average vulnerability rates exceeding 56% across risk scenarios. We further demonstrate that security-aware prompting can substantially mitigate these risks, achieving up to 45% improvement.

Lixun Ma, Ruolong Ma, Bei Wang +4

Zhejiang University, Hangzhou, China · University College London, London, United Kingdom · Artificial Intelligence Institute, CAICT, Beijing, China +1

LLM Safety BenchmarksLLM SecuritySecure Code Generation
AskPDF
Jul 2, 2024

Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking

Large language models (LLMs) are increasingly used for program synthesis, yet they often generate code that is functionally plausible but insecure. Progress in secure code generation has been hindered by benchmarks that are small, non-executable, leak mitigation details, or rely on noisy analyzers and subjective judgments, making it difficult to measure whether security improves without sacrificing correctness. We address these gaps with CodeSecEval, an execution-based benchmark for secure code generation, comprising 255 Python tasks spanning 77 CWE categories. Each task provides paired insecure and secure implementations together with executable functional and vulnerability-targeted security tests, enabling precise and reproducible evaluation of secure code generation and insecure-code repair. Building on CodeSecEval, we propose SecAwareCoder, an agent-based framework that shifts code generation toward secure-by-construction synthesis. SecAwareCoder performs task-adaptive threat modeling to identify security-sensitive regions and derive task-grounded vulnerability hypotheses, uses these hypotheses to guide both constraint-aware code generation and security-aware test synthesis, and leverages execution feedback for targeted refinement. Experiments across multiple LLM backbones show that SecAwareCoder consistently improves Pass@1 and security robustness over prompting and analyzer-driven baselines, narrowing the security--correctness gap in LLM code generation.

Jiexin Wang, Liuwen Cao, Xitong Luo +6

South China University of Technology, Guangzhou, China · University of Innsbruck, Innsbruck, Austria

Execution-Guided Code GenerationAgentic Code GenerationSecure Code Generation
AskPDF
Oct 10, 2026

Last Secure Code Benchmark: Is the Code You Vibe Secure?

AI coding agents now write much of the code that ships, yet whether that code is secure remains unclear. We introduce the Last Secure Code Benchmark, a comprehensive evaluation of secure code generation by coding agents. Built from 150 real vulnerabilities, its 450 tasks span seven widely used languages (Java, C, C++, JavaScript, Python, PHP, and Go), 24 CWEs including 21 of the 2025 CWE Top 25 and 8 of the Top 10 KEV weaknesses, more than any benchmark we compare with, and three vibe-coding scenarios: filling in a function, writing a file, and building a repository's source tree from scratch. Every task asks only for functionality, through a specification that never mentions security, and hidden functional and security tests grade the result. Across six frontier agents, 73.6% to 91.3% of tasks pass functionally, yet even the strongest, Claude Fable 5, produces code that is both functional and secure for only 25.1% of tasks, and the others for 16.4% to 24.7%. The best model only passes 12% of the vulnerability scenarios across all vibe coding setting, posing strong concern on the security of the code written by vibe coding.

Tianyi Wu, Chao Ge, Peng Chen +12

National University of Singapore · Qwen Team, Alibaba Token Hub, Alibaba Group · Cloud Safety and Security, Alibaba Cloud, Alibaba Group

Secure Code GenerationAI Agent Security BenchmarksSoftware SecurityAI Coding AgentsCode Generation Benchmarks
AskPDF