We introduce SecCodeBench-V2, a publicly released benchmark for evaluating Large Language Model (LLM) copilots' capabilities of generating secure code. SecCodeBench-V2 comprises 98 generation and fix scenarios derived from Alibaba Group's industrial productions, where the underlying security issues span 22 common CWE (Common Weakness Enumeration) categories across five programming languages: Java, C, Python, Go, and JavaScript. SecCodeBench-V2 adopts a function-level task formulation: each scenario provides a complete project scaffold and requires the model to implement or patch a designated target function under fixed interfaces and dependencies. For each scenario, SecCodeBench-V2 provides executable proof-of-concept (PoC) test cases for both functional validation and security verification. All test cases are authored and double-reviewed by security experts, ensuring high fidelity, broad coverage, and reliable ground truth. Beyond the benchmark itself, we build a unified evaluation pipeline that assesses models primarily via dynamic execution. For most scenarios, we compile and run model-generated artifacts in isolated environments and execute PoC test cases to validate both functional correctness and security properties. For scenarios where security issues cannot be adjudicated with deterministic test cases, we additionally employ an LLM-as-a-judge oracle. To summarize performance across heterogeneous scenarios and difficulty levels, we design a Pass@K-based scoring protocol with principled aggregation over scenarios and severity, enabling holistic and comparable evaluation across models. Overall, SecCodeBench-V2 provides a rigorous and reproducible foundation for assessing the security posture of AI coding assistants, with results and artifacts released at https://alibaba.github.io/sec-code-bench. The benchmark is publicly available at https://github.com/alibaba/sec-code-bench.
Figures & tables
Figure 1. Overall pipeline of SecCodeBench-V2 .
Scenario
Task
Evaluation goal
gen
Code generation
Generating secure code.
gen-hints
Generation w/ hints
Generating secure code when given an explicit safety hint.
fix
Vulnerability repair
Identifying and fixing security vulnerabilities in existing code.
fix-hints
Repair w/ hints
Fixing vulnerability when given an explicit safety hint.
Table 1. Four prompt scenarios supported by each test case.
Component
Description
Example
Task description
Detailed business context and requirements.
“Design a social-media content crawler…”
Method signature
The required function interface.
fetch_metadata(page_url: str) -> dict
Parameter spec.
Detailed parameter definitions.
Name/type/purpose of each argument.
Return spec.
Return schema and semantics.
“Return a dict with title and description.”
Context
Execution environment and relevant context.
Use C functions such as strcpy and strncpy .
Dependency constraints
Allowed libraries and frameworks.
Use only the listed dependencies.
Table 2. Core structure of each prompt.
Severity
Java
Python
C/C++
Go
JavaScript
Total
Critical
8
8
11
5
2
34
High
33
5
4
5
2
49
Medium
12
0
0
3
0
15
Table 3. Distribution of vulnerability severity levels across programming languages.
Vulnerability Type
CWE ID
Java
Python
C/C++
Go
JavaScript
Total
Path Traversal
CWE-22
3
–
–
–
2
5
Command Injection
CWE-78
2
2
1
2
2
9
XSS
CWE-79
1
–
–
2
–
3
SQL Injection
CWE-89
8
3
–
2
–
13
Code Injection
CWE-94
2
1
1
3
–
7
Log Injection
CWE-117
1
–
–
–
–
1
Table 4. Distribution of security domains (CWE) across programming languages.