Detecting Data Poisoning in Code Generation LLMs via Black-Box, Vulnerability-Oriented Scanning

Shenao Yan; Shan Jin; Shimaa Ahmed; Sunpreet Singh Arora; Yiwei Cai; Yizhen Wang; Yuan Hong

cs.CRMar 17, 2026

Detecting Data Poisoning in Code Generation LLMs via Black-Box, Vulnerability-Oriented Scanning

Authors: Shenao Yan, Shan Jin, Shimaa Ahmed, Sunpreet Singh Arora, Yiwei Cai, Yizhen Wang, Yuan Hong

Organizations: University of Connecticut · Visa Research

Abstract

Code generation large language models (LLMs) are increasingly integrated into modern software development workflows. Recent work has shown that these models are vulnerable to backdoor and poisoning attacks that induce the generation of insecure code, yet effective defenses remain limited. Existing scanning approaches rely on token-level generation consistency to invert attack targets, which is ineffective for source code where identical semantics can appear in diverse syntactic forms. We present CodeScan, the first black-box, vulnerability-specific scanning framework for auditing code generation LLMs, assuming that the defender specifies the target vulnerability classes and provides corresponding task-relevant prompts. CodeScan identifies attack targets by analyzing structural similarities across multiple generations conditioned on different clean prompts. It combines iterative divergence analysis with abstract syntax tree (AST)-based normalization to abstract away surface-level variation and unify semantically equivalent code, isolating structures that recur consistently across generations. CodeScan then applies LLM-based vulnerability analysis to determine whether the extracted structures contain security vulnerabilities and flags the model as compromised when such a structure is found. We evaluate CodeScan against four representative attacks under both backdoor and poisoning settings across three real-world vulnerability classes. Experiments on 117 models spanning three architectures and multiple model sizes demonstrate 97%+ detection accuracy with substantially lower false positives than prior methods.

Figures & tables

Figure 1 . Example of Attacks to Code Generation LLMs

Figure 2 . Overview of CodeScan. Given a vulnerability of interest, CodeScan iterates over candidate tokens by appending each token to a set of clean prompts and collecting the resulting generations. It first performs structural divergence analysis to identify structurally consistent code clusters, and then applies vulnerability analysis only to those clusters. If a vulnerable cluster is found, the model is flagged as poisoned; otherwise, CodeScan continues scanning the next token.

Inversion		No	Wrong			Correct
Measurement		Ct.(Pct.)	Ct.(Pct.)	AST_D	BLEU	Ct.(Pct.)
V1	BAIT	15(50%)	5(16.7%)	0.751	0.172	10(33.3%)
V1	CodeScan	0(0%)	2(6.7%)	1.000	0.223	28(93.3%)
V2	BAIT	9(30%)	7(23.3%)	0.713	0.266	14(46.7%)
V2	CodeScan	0(0%)	0(0%)	–	–	30(100%)
V3	BAIT	0(0%)	12(50%)	0.344	0.761	12(50%)

Table 1 . Inversion Results Given the First Token

Figure 4 . Q-Score of Inversions for BAIT

Attack			Backdoor Attack			Poisoning Attack			Overall	Clean
Model (7B)			CodeLlama	QwenCoder	StarCoder	CodeLlama	QwenCoder	StarCoder	Overall	Clean
		Precision	0.0000	0.0000	0.4000	0.2000	0.0000	0.2000	0.1333
		Recall	0.0000	0.0000	1.0000	1.0000	0.0000	1.0000	1.0000
		F1-score	0.0000	0.0000	0.5714	0.3333	0.0000	0.3333	0.2353	FPR 100%
	BAIT	Runtime (s)	10979.5	14673.0	5461.4	14854.0	19421.6	10179.4	12594.8
		Precision	1.0000	0.0000	1.0000	1.0000	0.0000	1.0000	0.6667

Table 2 . CodeScan vs. BAIT vs. BAIT+VA performance and efficiency on attacked models (left), and FPR on clean models (right)

	Cost	Runtime (s)	# of Calls	# Tokens	API Cost ($)
V1	BAIT+VA	5,502.5	19.3	5,162.0	0.0014
	VA	142.4	5.7	18,696.0	0.0276
	CodeScan	16.6	1.0	209.0	0.00006
V2	BAIT+VA	12,708.9	42.0	17,892.0	0.0048
	VA	11,078.8	601.7	1,818,447.3	2.6830
	CodeScan	1,574.1	201.0	69,613.0	0.0186

Table 3. Costs of CodeScan, BAIT+VA and VA

Figure 5 . Breakdown of Time Cost Per Scan

Model (7B)			CodeLlama	QwenCoder	StarCoder
V1	BAIT	AST_D	0.8199	0.9185	0.9643
	BAIT	BLEU	0.0978	0.0078	0.1745
	BAIT+ VA	AST_D	0.7532	0.9025	0.5845
	BAIT+ VA	BLEU	0.1981	0.0065	0.4285
	CodeScan	AST_D	0.0553	0.1023	0.1171
	CodeScan	BLEU	0.9119	0.8184	0.8051

Table 4 . AST Distance and BLEU of Inverted Codes

Dataset	SA		GPT-4
Prompting	Zero-shot	One-Shot	Zero-shot	One-Shot
GPT-4	36.00%	67.33%	12.00%	22.67%
GPT-5 mini	94.67%	100.00%	94.67%	98.00%
GPT-5.4	91.33%	99.33%	88.67%	100.00%

Table 5 . Vulnerability Analysis Across Models

Figure 6 . Adaptive Attacks

Target		T1	T2	T3
CodeScan	Detect (?)	✔	✔	✘
CodeScan	Overhead (s)	158.2	583.3	–
Backdoor Attack	Attack@1/3/5	3/7/8	0/1/2	1/1/2

Table 6. Multi-target Adaptive Attack

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8 . Insufficiency of BAIT

Panel of Figure 9 — Figure 9 . Attack Targets In the Evaluation

Figure 10 . Visualization of Algorithm 2

Figure 11 . Triggers for Backdoor Attacks

Panel of Figure 12 — Figure 12 . Inversion examples under different variants

Dataset	SA						GPT
Prompting Method	Zero-Shot			One-Shot			Zero-Shot			One-Shot
LLM (GPT)	4	5 mini	5.4	4	5 mini	5.4	4	5 mini	5.4	4	5 mini	5.4
direct-use-of-jinja2	1/10	4/10	10/10	2/10	10/10	10/10	0/10	10/10	10/10	0/10	10/10	10/10
user-exec-format-string	10/10	10/10	10/10	9/10	10/10	10/10	1/10	10/10	10/10	1/10	10/10	10/10
avoid-pickle	10/10	10/10	10/10	10/10	10/10	10/10	2/10	10/10	10/10	4/10	10/10	10/10
unsanitized-input-in-response	3/10	10/10	10/10	1/10	10/10	10/10	3/10	10/10	8/10	0/10	10/10	10/10

Appendix

Table 7 . Vulnerability Analysis Results

Panel of Figure 13 — Figure 13 . Hyper-parameter Sensitivity

Figure 14 . BAIT Failure Cases

Figure 15 . CodeScan Successful Cases

Figure 16 . CodeScan Failure Cases

Algorithms	CodeScan	BAIT+VA	GCG	PEZ	DBS
Precision	0.9500	1.0000	0.0000	0.0000	0.0000
Recall	1.0000	0.5000	0.0000	0.0000	0.0000
F1-Score	0.9744	0.6667	0.0000	0.0000	0.0000
AST_D	0.0621	0.5685	0.9647	0.8823	0.9329
BLEU	0.8540	0.4623	0.0345	0.0698	0.0398

Appendix

Table 8 . CodeScan, BAIT+VA vs. Baselines

Figure 17 . Obfuscated Payloads againt GPT-5

Model		CodeLlama-34B	QwenCoder-14B	StarCoder-15B
CodeScan	Precision	1.0000	1.0000	1.0000
	Recall	1.0000	1.0000	1.0000
	F1-score	1.0000	1.0000	1.0000
	Runtime (s)	3218.9	448.5	3139.2
	AST_D	0.1039	0.0243	0.0276
	BLEU	0.7852	0.8866	0.9295

Appendix

Table 9 . CodeScan Performance on Larger LLMs

Vocabulary			CodeLlama	QwenCoder	StarCoder
Model	CodeLlama	Detect (?)	✔	✔	✔
		Runtime (s)	3013.3	2036.4	2526.2
		AST_D	0.0158	0.0198	0.0198
		BLEU	0.9184	0.9184	0.9184
	QwenCoder	Detect (?)	✔	✔	✔
		Runtime (s)	34.4	128.5	31.7

Appendix

Table 10. Cross Vocabulary Validation

Explore similar work

Jul 25, 2026cs.CR

Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios

Large Language Models (LLMs) are widely used for code generation, yet their security behavior in realistic development workflows remains underexplored. Existing benchmarks often rely on explicitly specified security requirements, failing to capture real-world scenarios where prompts are frequently ambiguous or incomplete. In this paper, we adopt a developer-centric perspective and identify three representative risk scenarios that commonly lead to security vulnerabilities in LLM-generated code: Ambiguous Requirements, Under-Specified Operational Context, and Security--Functionality Conflict. Based on these scenarios, we construct a large-scale benchmark comprising 2,700 test cases, enabling fine-grained evaluation of LLM security under realistic conditions. Extensive evaluation of eight state-of-the-art LLMs reveals that all models exhibit average vulnerability rates exceeding 56% across risk scenarios. We further demonstrate that security-aware prompting can substantially mitigate these risks, achieving up to 45% improvement.

Lixun Ma, Ruolong Ma, Bei Wang +4

Zhejiang University, Hangzhou, China · University College London, London, United Kingdom · Artificial Intelligence Institute, CAICT, Beijing, China +1

Large Language Model Safety Vulnerable Code Code Generation Security Large-Scale Benchmark Agentic Workflow Design

Ask PDF

May 8, 2026cs.CR

SecureForge: Finding and Preventing Vulnerabilities in LLM-Generated Code via Prompt Optimization

LLM coding agents now generate code at an unprecedented scale, yet LLM-generated code introduces cybersecurity vulnerabilities into codebases without human involvement. Even when frontier models are explicitly asked to write secure production code with relevant weaknesses to avoid in context, we find that they still produce verifiable vulnerabilities on average 23% of the time across a corpus of 250 benign coding prompts. We introduce SecureForge, an automated pipeline that both audits security risks of frontier models and produces auditing-informed secure system prompts that reduce output security vulnerabilities while maintaining unit test performance. SecureForge first identifies benign prompts that produce statically detectable vulnerabilities, and then amplifies them into a large synthetic prompt corpus of diverse scenarios using a Markovian sampling technique to jointly maintain error rates and prompt diversity. This corpus is then used to iteratively optimize the system prompts to reduce output security vulnerabilities. On frontier models, SecureForge yields a statistically significant Pareto improvement in both unit test success and output security, with output vulnerabilities reduced by up to 48%. The resulting system prompts transfer zero-shot to in-the-wild coding agent prompts, without any exposure to real user prompt distributions during optimization.

Houjun Liu, Lisa Einstein, John Yang +5

Stanford University

Vulnerable Code Coding Agents Model Auditing Prompt Optimization

Ask PDF

May 22, 2026cs.CR

Enhancing Reliability in LLM-Based Secure Code Generation

Large language models (LLMs) are widely used for code generation, but their security reliability remains inconsistent across languages and prompting strategies. Existing prompt engineering improves functional correctness but rarely ensures consistent security outcomes. We introduce the \textit{Mitigation-Aware Chain-of-Thought (MA-CoT)} framework, which embeds task-specific CWE mitigation guidance and language-aware safeguards to reduce recurring vulnerabilities in generated code. We evaluate MA-CoT across three LLMs (gpt-5, claude-4.5, gemini-2.5), three programming languages (C, Java, Python), and four prompting strategies (Vanilla, Zero-shot, CoT, MA-CoT) on a 200-task primary dataset, with external validation on LLMSecEval. Using static analysis with expert validation, MA-CoT reduces total security findings from 92 to 39 (57.6%) on the primary dataset and from 73 to 4 (94.5%) on LLMSecEval. High-severity findings (Blocker + Critical) drop from 90 to 39 (56.7%) and from 45 to 2 (95.6%), respectively. Across both datasets, MA-CoT is the only strategy that consistently improves security reliability; Zero-shot and CoT are less reliable and may increase vulnerability, especially in C. We further introduce a strict layered attribution of vulnerability drivers (language-core vs. stack layers) and show that residual risk concentrates in hardening-oriented patterns (e.g., OS- and toolchain-dependent), motivating secure-by-construction primitives alongside prompting.

Mohammed F. Kharma, Mohammad Alkhanafseh, Ahmed Sabbah +1

the Department of Computer Science, Birzeit University, Ramallah, Palestine · the Department of Computer Science, University of Central Florida, Orlando, FL 32816 USA

Large Language Model Safety Vulnerable Code Code Generation Chain-of-Thought Reasoning

Ask PDF