WeSCE: A Benchmark for Measuring Security Drift in LLM-Driven Code Editing

Zhiyu Zhang; Tingyue Wen; Senke Sun; Dengxiang Liang; Enhao Huang

cs.CRAug 15, 2026

WeSCE: A Benchmark for Measuring Security Drift in LLM-Driven Code Editing

Authors: Zhiyu Zhang, Tingyue Wen, Senke Sun, Dengxiang Liang, Enhao Huang

Organizations: Zhejiang University, Hangzhou, China

Abstract

In this work, we introduce WeSCE, a benchmark for quantifying security drift in code editing under weak-security constraints, where tasks specify only functional objectives without explicit security requirements. WeSCE consists of 400 executable programs derived from real-world code, covering feature addition, feature removal, bug fixing, and refactoring. To quantify security drift, we propose a continuous risk representation that aggregates heterogeneous vulnerability signals through a unified formulation, and define drift measures capturing changes in overall risk, worst-case severity, and vulnerability distribution under code transformations, providing a multi-scale view of security spanning average-case behavior to worst-case emphasis.

Figures & tables

Figure 1: Overview of the WeSCE evaluation framework. An LLM edits a program under weak-security constraints (i.e., functional-only instructions). Both the original and edited code undergo static analysis (SAST), dynamic fuzzing, and complexity measurement. Vulnerability signals are then aggregated into continuous risk scores $E_{b}$ via LogSumExp, and security drift metrics ( $\overline{\Delta E_{0}}$ , $\overline{\Delta E_{\infty}}$ , $D_{\mathrm{TV}}$ , $R_{0}$ , $R_{\infty}$ , $R_{c}$ ) are computed by comparing pre- and post-edit risk representations.

Figure 2: The WeSCE dataset construction pipeline. Vulnerability seeds are collected from GitHub and Real-Vuln-Benchmark, expanded into fully executable programs via a code agent with fixed prompts, filtered by syntax and executability constraints, and deduplicated using CodeBERT embeddings. Four editing tasks (ADD, REMOVE, FIX, REFACTOR) are constructed with functional-only instructions. A final validation pipeline verifies program executability, vulnerability preservation, and task prompt correctness.

Normalization Method	CV of Density
No normalization ( $r_{i}$ )	0.82
$1/L$ scaling	0.74
$\log L$ scaling	0.63
$\sqrt{L}$ scaling (ours)	0.41

Table 1: Stability comparison of different normalization strategies for vulnerability density estimation. CV denotes the coefficient of variation across programs of varying sizes; lower values indicate more stable density estimates.

Model	ADD		REMOVE		FIX		REFACTOR		Total
	Mean	SD	Mean	SD	Mean	SD	Mean	SD	Mean	SD
Opus 4.6	0.517	0.317	0.409	0.263	0.725	0.241	0.866	0.341	0.629	0.343
GPT-5.4	0.449	0.391	0.334	0.359	0.743	0.209	0.834	0.219	0.590	0.368
Sonnet 4	0.613	0.305	0.384	0.228	0.688	0.190	0.815	0.267	0.625	0.297
Deepseek-v4-flash	0.501	0.289	0.420	0.432	0.713	0.128	0.772	0.201	0.602	0.321
Kimi k2.5	0.533	0.362	0.524	0.320	0.607	0.213	0.766	0.190	0.607	0.298

Table 2: Mean and standard deviation of total variation (TV) distance

D_{\mathrm{TV}}

for each model across four editing tasks. Higher

D_{\mathrm{TV}}

indicates larger distributional shifts in vulnerability profiles after editing.

Figure 3: Overall risk reduction magnitude $-\Delta E$ (plotted as positive values) for eight LLMs under average-sensitive ( $b{=}10^{-3}$ , dark blue) and worst-case-sensitive ( $b{=}10^{3}$ , light blue) aggregation regimes. Taller bars indicate larger security improvements after editing. Stronger models (left) consistently achieve greater reductions, with worst-case reductions exceeding average-case reductions for all models.

Model	ADD			REMOVE			FIX			REFACTOR			TOTAL
	$R_{\infty}$	$R_{0}$	$R_{c}$	$R_{\infty}$	$R_{0}$	$R_{c}$	$R_{\infty}$	$R_{0}$	$R_{c}$	$R_{\infty}$	$R_{0}$	$R_{c}$	$R_{\infty}$	$R_{0}$	$R_{c}$
Opus 4.6	71	71	34	70	70	29	90	91	62	98	98	80	82.25	82.50	51.25
GPT-5.4	66	66	29	67	67	32	89	89	59	98	97	82	80.00	79.75	50.50
Sonnet 4	67	67	31	68	68	37	87	87	55	95	95	79	79.25	79.25	50.50
Deepseek-v4-flash	65	65	24	59	59	22	83	83	66	91	91	85	74.50	74.50	49.25
Kimi k2.5	61	61	21	64	64	22	72	72	43	90	90	73	71.75	71.75	39.75

Table 3: Risk reduction rates (

R_{\infty}

: worst-case,

R_{0}

: average-sensitive) and complete clearance rate (

R_{c}

) in % for each model across four editing tasks. Best results per task are bolded .

Figure 4: Heatmap of per-task risk drift $\Delta E$ for eight models across four editing tasks under average-sensitive ( $b{=}10^{-3}$ ) and worst-case-sensitive ( $b{=}10^{3}$ ) regimes. Blue cells indicate risk reduction (negative drift); red cells indicate risk amplification (positive drift). REFACTOR and FIX consistently exhibit deeper blue across all models, while ADD occasionally produces near-zero or positive drift for weaker models.

Model	$\boldsymbol{\overline{\Delta E_{0}}}$ (Total)	$\boldsymbol{\overline{\Delta E_{0}^{(s)}}}$ (Static)	$\boldsymbol{\overline{\Delta E_{0}^{(d)}}}$ (Dynamic)
Opus 4.6	-0.5969	-0.4246	-0.1723
GPT-5.4	-0.5899	-0.4108	-0.1792
Sonnet 4	-0.5410	-0.3902	-0.1509
Deepseek-v4-flash	-0.5516	-0.4121	-0.1396
Kimi k2.5	-0.4464	-0.2611	-0.1854
Haiku 4.5	-0.3812	-0.2499	-0.1313

Table 4: Decomposition of average risk reduction

\overline{\Delta E_{0}}

into static (

\overline{\Delta E_{0}^{(s)}}

) and dynamic (

\overline{\Delta E_{0}^{(d)}}

) components for each model. Both components contribute to overall risk reduction, with static reductions consistently larger.

	Claude Gen	DeepSeek Gen
Claude Eval	-0.4410	-0.4013
DeepSeek Eval	-0.3933	-0.4127

Table 5: Fairness analysis: average risk drift

\overline{\Delta E_{0}}

under different generation–evaluation combinations.

Software Security

Software security studies how to find, prevent, validate, and repair vulnerabilities in code, systems, dependencies, and software supply chains.

See all 95 papers

SecCodeBench-V2 Technical Report
Feb 17, 2026Longfei Chen, Ji Zhao, Lanxiao Cui +24Secure Code Generation Code Generation Benchmarks
TSINGHUA UNIVERSITY, BEIJING, CHINA · ALIBABA GROUP, HANGZHOU, CHINA
Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios
Jul 25, 2026Lixun Ma, Ruolong Ma, Bei Wang +4LLM Safety Benchmarks LLM Security
Zhejiang University, Hangzhou, China · University College London, London, United Kingdom · Artificial Intelligence Institute, CAICT, Beijing, China +1
SecDrift: Measuring Sector-Conditioned Security Drift in AI-Generated Code
Jul 28, 2026Narayanaswami Natraj Bharadwaj, Dhivya ChandramouleeswaranCybersecurity LLM Security
Independent Researcher USA

Feb 17, 2026J/K move · Enter open · S save

SecCodeBench-V2 Technical Report

Longfei Chen, Ji Zhao, Lanxiao Cui, Tong Su+23

TSINGHUA UNIVERSITY, BEIJING, CHINA · ALIBABA GROUP, HANGZHOU, CHINA

Secure Code Generation Code Generation Benchmarks LLM Code Generation

Open paper AskPDF arXiv

We introduce SecCodeBench-V2, a publicly released benchmark for evaluating Large Language Model (LLM) copilots' capabilities of generating secure code. SecCodeBench-V2 comprises 98 generation and fix scenarios derived from Alibaba Group's industrial productions, where the underlying security issues span 22 common CWE (Common Weakness Enumeration) categories across five programming languages: Java, C, Python, Go, and JavaScript. SecCodeBench-V2 adopts a function-level task formulation: each scenario provides a complete project scaffold and requires the model to implement or patch a designated target function under fixed interfaces and dependencies. For each scenario, SecCodeBench-V2 provides executable proof-of-concept (PoC) test cases for both functional validation and security verification. All test cases are authored and double-reviewed by security experts, ensuring high fidelity, broad coverage, and reliable ground truth. Beyond the benchmark itself, we build a unified evaluation pipeline that assesses models primarily via dynamic execution. For most scenarios, we compile and run model-generated artifacts in isolated environments and execute PoC test cases to validate both functional correctness and security properties. For scenarios where security issues cannot be adjudicated with deterministic test cases, we additionally employ an LLM-as-a-judge oracle. To summarize performance across heterogeneous scenarios and difficulty levels, we design a Pass@K-based scoring protocol with principled aggregation over scenarios and severity, enabling holistic and comparable evaluation across models. Overall, SecCodeBench-V2 provides a rigorous and reproducible foundation for assessing the security posture of AI coding assistants, with results and artifacts released at https://alibaba.github.io/sec-code-bench. The benchmark is publicly available at https://github.com/alibaba/sec-code-bench.

Abstract

Software Security

Explore similar work

SecCodeBench-V2 Technical Report

Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios

SecDrift: Measuring Sector-Conditioned Security Drift in AI-Generated Code