WeSCE: A Benchmark for Measuring Security Drift in LLM-Driven Code Editing

Zhiyu Zhang; Tingyue Wen; Senke Sun; Dengxiang Liang; Enhao Huang

cs.CRAug 15, 2026

WeSCE: A Benchmark for Measuring Security Drift in LLM-Driven Code Editing

Authors: Zhiyu Zhang, Tingyue Wen, Senke Sun, Dengxiang Liang, Enhao Huang

Organizations: Zhejiang University, Hangzhou, China

Abstract

In this work, we introduce WeSCE, a benchmark for quantifying security drift in code editing under weak-security constraints, where tasks specify only functional objectives without explicit security requirements. WeSCE consists of 400 executable programs derived from real-world code, covering feature addition, feature removal, bug fixing, and refactoring. To quantify security drift, we propose a continuous risk representation that aggregates heterogeneous vulnerability signals through a unified formulation, and define drift measures capturing changes in overall risk, worst-case severity, and vulnerability distribution under code transformations, providing a multi-scale view of security spanning average-case behavior to worst-case emphasis.

Figures & tables

Figure 1: Overview of the WeSCE evaluation framework. An LLM edits a program under weak-security constraints (i.e., functional-only instructions). Both the original and edited code undergo static analysis (SAST), dynamic fuzzing, and complexity measurement. Vulnerability signals are then aggregated into continuous risk scores $E_{b}$ via LogSumExp, and security drift metrics ( $\overline{\Delta E_{0}}$ , $\overline{\Delta E_{\infty}}$ , $D_{\mathrm{TV}}$ , $R_{0}$ , $R_{\infty}$ , $R_{c}$ ) are computed by comparing pre- and post-edit risk representations.

Figure 2: The WeSCE dataset construction pipeline. Vulnerability seeds are collected from GitHub and Real-Vuln-Benchmark, expanded into fully executable programs via a code agent with fixed prompts, filtered by syntax and executability constraints, and deduplicated using CodeBERT embeddings. Four editing tasks (ADD, REMOVE, FIX, REFACTOR) are constructed with functional-only instructions. A final validation pipeline verifies program executability, vulnerability preservation, and task prompt correctness.

Normalization Method	CV of Density
No normalization ( $r_{i}$ )	0.82
$1/L$ scaling	0.74
$\log L$ scaling	0.63
$\sqrt{L}$ scaling (ours)	0.41

Table 1: Stability comparison of different normalization strategies for vulnerability density estimation. CV denotes the coefficient of variation across programs of varying sizes; lower values indicate more stable density estimates.

Model	ADD		REMOVE		FIX		REFACTOR		Total
	Mean	SD	Mean	SD	Mean	SD	Mean	SD	Mean	SD
Opus 4.6	0.517	0.317	0.409	0.263	0.725	0.241	0.866	0.341	0.629	0.343
GPT-5.4	0.449	0.391	0.334	0.359	0.743	0.209	0.834	0.219	0.590	0.368
Sonnet 4	0.613	0.305	0.384	0.228	0.688	0.190	0.815	0.267	0.625	0.297
Deepseek-v4-flash	0.501	0.289	0.420	0.432	0.713	0.128	0.772	0.201	0.602	0.321
Kimi k2.5	0.533	0.362	0.524	0.320	0.607	0.213	0.766	0.190	0.607	0.298

Table 2: Mean and standard deviation of total variation (TV) distance

D_{\mathrm{TV}}

for each model across four editing tasks. Higher

D_{\mathrm{TV}}

indicates larger distributional shifts in vulnerability profiles after editing.

Figure 3: Overall risk reduction magnitude $-\Delta E$ (plotted as positive values) for eight LLMs under average-sensitive ( $b{=}10^{-3}$ , dark blue) and worst-case-sensitive ( $b{=}10^{3}$ , light blue) aggregation regimes. Taller bars indicate larger security improvements after editing. Stronger models (left) consistently achieve greater reductions, with worst-case reductions exceeding average-case reductions for all models.

Model	ADD			REMOVE			FIX			REFACTOR			TOTAL
	$R_{\infty}$	$R_{0}$	$R_{c}$	$R_{\infty}$	$R_{0}$	$R_{c}$	$R_{\infty}$	$R_{0}$	$R_{c}$	$R_{\infty}$	$R_{0}$	$R_{c}$	$R_{\infty}$	$R_{0}$	$R_{c}$
Opus 4.6	71	71	34	70	70	29	90	91	62	98	98	80	82.25	82.50	51.25
GPT-5.4	66	66	29	67	67	32	89	89	59	98	97	82	80.00	79.75	50.50
Sonnet 4	67	67	31	68	68	37	87	87	55	95	95	79	79.25	79.25	50.50
Deepseek-v4-flash	65	65	24	59	59	22	83	83	66	91	91	85	74.50	74.50	49.25
Kimi k2.5	61	61	21	64	64	22	72	72	43	90	90	73	71.75	71.75	39.75

Table 3: Risk reduction rates (

R_{\infty}

: worst-case,

R_{0}

: average-sensitive) and complete clearance rate (

R_{c}

) in % for each model across four editing tasks. Best results per task are bolded .

Figure 4: Heatmap of per-task risk drift $\Delta E$ for eight models across four editing tasks under average-sensitive ( $b{=}10^{-3}$ ) and worst-case-sensitive ( $b{=}10^{3}$ ) regimes. Blue cells indicate risk reduction (negative drift); red cells indicate risk amplification (positive drift). REFACTOR and FIX consistently exhibit deeper blue across all models, while ADD occasionally produces near-zero or positive drift for weaker models.

Model	$\boldsymbol{\overline{\Delta E_{0}}}$ (Total)	$\boldsymbol{\overline{\Delta E_{0}^{(s)}}}$ (Static)	$\boldsymbol{\overline{\Delta E_{0}^{(d)}}}$ (Dynamic)
Opus 4.6	-0.5969	-0.4246	-0.1723
GPT-5.4	-0.5899	-0.4108	-0.1792
Sonnet 4	-0.5410	-0.3902	-0.1509
Deepseek-v4-flash	-0.5516	-0.4121	-0.1396
Kimi k2.5	-0.4464	-0.2611	-0.1854
Haiku 4.5	-0.3812	-0.2499	-0.1313

Table 4: Decomposition of average risk reduction

\overline{\Delta E_{0}}

into static (

\overline{\Delta E_{0}^{(s)}}

) and dynamic (

\overline{\Delta E_{0}^{(d)}}

) components for each model. Both components contribute to overall risk reduction, with static reductions consistently larger.

	Claude Gen	DeepSeek Gen
Claude Eval	-0.4410	-0.4013
DeepSeek Eval	-0.3933	-0.4127

Table 5: Fairness analysis: average risk drift

\overline{\Delta E_{0}}

under different generation–evaluation combinations.

Software Security

Software security studies how to find, prevent, validate, and repair vulnerabilities in code, systems, dependencies, and software supply chains.

See all 95 papers

Feb 17, 2026

SecCodeBench-V2 Technical Report

We introduce SecCodeBench-V2, a publicly released benchmark for evaluating Large Language Model (LLM) copilots' capabilities of generating secure code. SecCodeBench-V2 comprises 98 generation and fix scenarios derived from Alibaba Group's industrial productions, where the underlying security issues span 22 common CWE (Common Weakness Enumeration) categories across five programming languages: Java, C, Python, Go, and JavaScript. SecCodeBench-V2 adopts a function-level task formulation: each scenario provides a complete project scaffold and requires the model to implement or patch a designated target function under fixed interfaces and dependencies. For each scenario, SecCodeBench-V2 provides executable proof-of-concept (PoC) test cases for both functional validation and security verification. All test cases are authored and double-reviewed by security experts, ensuring high fidelity, broad coverage, and reliable ground truth. Beyond the benchmark itself, we build a unified evaluation pipeline that assesses models primarily via dynamic execution. For most scenarios, we compile and run model-generated artifacts in isolated environments and execute PoC test cases to validate both functional correctness and security properties. For scenarios where security issues cannot be adjudicated with deterministic test cases, we additionally employ an LLM-as-a-judge oracle. To summarize performance across heterogeneous scenarios and difficulty levels, we design a Pass@K-based scoring protocol with principled aggregation over scenarios and severity, enabling holistic and comparable evaluation across models. Overall, SecCodeBench-V2 provides a rigorous and reproducible foundation for assessing the security posture of AI coding assistants, with results and artifacts released at https://alibaba.github.io/sec-code-bench. The benchmark is publicly available at https://github.com/alibaba/sec-code-bench.

Longfei Chen, Ji Zhao, Lanxiao Cui +24

TSINGHUA UNIVERSITY, BEIJING, CHINA · ALIBABA GROUP, HANGZHOU, CHINA

Secure Code Generation Code Generation Benchmarks LLM Code Generation

Ask PDF

Jul 25, 2026

Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios

Large Language Models (LLMs) are widely used for code generation, yet their security behavior in realistic development workflows remains underexplored. Existing benchmarks often rely on explicitly specified security requirements, failing to capture real-world scenarios where prompts are frequently ambiguous or incomplete. In this paper, we adopt a developer-centric perspective and identify three representative risk scenarios that commonly lead to security vulnerabilities in LLM-generated code: Ambiguous Requirements, Under-Specified Operational Context, and Security--Functionality Conflict. Based on these scenarios, we construct a large-scale benchmark comprising 2,700 test cases, enabling fine-grained evaluation of LLM security under realistic conditions. Extensive evaluation of eight state-of-the-art LLMs reveals that all models exhibit average vulnerability rates exceeding 56% across risk scenarios. We further demonstrate that security-aware prompting can substantially mitigate these risks, achieving up to 45% improvement.

Lixun Ma, Ruolong Ma, Bei Wang +4

Zhejiang University, Hangzhou, China · University College London, London, United Kingdom · Artificial Intelligence Institute, CAICT, Beijing, China +1

LLM Safety Benchmarks LLM Security Secure Code Generation

Ask PDF

Jul 28, 2026

SecDrift: Measuring Sector-Conditioned Security Drift in AI-Generated Code

LLMs are increasingly used for code generation in critical infrastructure, yet the security effect of domain-specific prompting is understudied. We present SecDrift, a benchmark measuring sector-conditioned security drift: the change in static-analysis vulnerability rates when prompts are conditioned on industry contexts versus neutral baselines. We evaluate 7 LLMs (6 producing analyzable code) across 8 CISA critical infrastructure sectors and 9 CWE categories with 5 replicates (5,355 evaluations), using a 5-dimension transformation with a matched-baseline condition that holds the task fixed while substituting only domain terminology. Industry prompts naively appear more secure (14.0% vs. 11.4%, -2.7pp), but the gap is not statistically significant (Fisher's exact p = 0.24, Cohen's h = -0.08) and is a composition artifact of two CWE categories: excluding CWE-502 and CWE-22 eliminates and slightly reverses it (+0.4pp, p = 1.00). A mixed-effects logistic regression confirms sector identity is not a moderator and localizes the only detectable condition effect to those two vulnerability types. 0 of 8 sectors show drift distinguishable from baseline, corrected or uncorrected (|h| < 0.15). A placebo on two non-CISA sectors (e-commerce, online education) reproduces the CISA industry rate almost exactly (10.5% vs. 11.4%, p = 0.63): the small pooled pattern reflects generic industry-framing specificity, not critical-infrastructure identity. In contrast, model selection has a large and consistent effect: among full-output models vulnerability rates range from 11.6% to 16.1%, and these differences persist across conditions. Model choice, not prompt framing, is the more reliable security lever. We release the framework, prompts, generated code, findings, human-validation verdicts, and analysis scripts.

Narayanaswami Natraj Bharadwaj, Dhivya Chandramouleeswaran

Independent Researcher USA

Cybersecurity LLM Security Code Generation Evaluation Secure Code Generation LLM Code Generation

Ask PDF

Abstract

Software Security

Explore similar work

SecCodeBench-V2 Technical Report

Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios

SecDrift: Measuring Sector-Conditioned Security Drift in AI-Generated Code