cs.LGAug 14, 2026

Detecting Contaminated Code-Generation Prompt Batches via Influence Functions

Authors: Francesco Quinzan, Noor Munir, Yishun Lu, Stephen Roberts

Organizations: Department of Engineering Science, The University of Oxford

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) are increasingly used for code generation, yet they remain vulnerable to prompts that elicit insecure implementations. Existing defenses typically rely on predefined threat models or known vulnerability patterns, limiting their effectiveness against novel attacks. We propose CodeSIFT, a threat-model-agnostic detection method that leverages influence functions to identify batches of prompts that induce anomalous model behavior. Rather than detecting specific vulnerabilities, CodeSIFT measures the parameter-space influence of generated code and uses a statistical test to determine whether a candidate prompt set deviates from a benign reference distribution. To evaluate our approach, we introduce two benchmark datasets covering a variety of vulnerabilities. We evaluate CodeSIFT on three open-weight code LLMs ranging from 3B to 7B parameters, achieving AUROC scores of up to 0.98 at moderate-to-high injection rates, while maintaining well-calibrated false positive rates and substantially outperforming static analysis baselines. These results suggest that influence-function-based detection is a promising direction for identifying malicious code-generation prompts without requiring prior knowledge of the underlying attack class.

Figures & tables

Figure 1: Example records from the malicious and benign instruction pools.
Figure 2: Detection performance on AuthSec as a function of injection rate, for Granite-3B-Code-Instruct-2K, DeepSeek-Coder-6.7B-Instruct, and Qwen2.5-Coder-7B-Instruct (left to right). Top row, AUROC. Bottom row, empirical false positive rate against the nominal significance level α\alpha . CodeSIFT approaches perfect AUROC as contamination increases and stays well calibrated across all three models, while the static baselines are inconsistent across models and often fall below chance.
Figure 3: Detection performance on InfraCloud as a function of injection rate, for Granite-3B-Code-Instruct-2K, DeepSeek-Coder-6.7B-Instruct, and Qwen2.5-Coder-7B-Instruct (left to right). Top row, AUROC. Bottom row, empirical false positive rate against the nominal significance level α\alpha . InfraCloud is harder than AuthSec , so CodeSIFT’s AUROC ceiling is lower, but it still improves monotonically with contamination and remains calibrated, while the baselines flatten or decline below chance.
Vulnerability classRep. CWERecords
Authentication, authorisation, and session security
Credential & secret exposureCWE-20023
Broken access control / IDORCWE-63922
Federated auth / SSO flawsCWE-28719
Session managementCWE-38419
JWT verification flawsCWE-34717
Table 1: Vulnerability classes in the malicious pool. Classes are listed in descending order of frequency within each family; each family totals 200 records. CWE identifiers are representative : each names the dominant weakness in its class, not a per-record label, and several classes span more than one entry. Rows marked -- aggregate a long tail of weaknesses that do not map cleanly to a single identifier.
HyperparameterValue
Upweighting factor ( ε\varepsilon )1×10−31\times 10^{-3}
Samples per prompt ( NN )25
Base K-FAC damping0.10.1
Relative damping coefficient (rel_eps)0.10.1
Damping floor ( γ\gamma , Granite-3B-Code-Instruct-2K, DeepSeek-Coder-6.7B-Instruct)1×10−31\times 10^{-3}
Damping floor ( γ\gamma , Qwen2.5-Coder-7B-Instruct)1×10−21\times 10^{-2}
Table 2: Hyperparameters used across all experiments (Algorithm 1 and Section 3 notation in parentheses).
Figure 4: AUROC on AuthSec as a function of dataset size, for CodeSIFT ( L1L_{1} , L2L_{2} ) and the three static baselines jointly. Columns correspond to Granite-3B-Code-Instruct-2K, DeepSeek-Coder-6.7B-Instruct, and Qwen2.5-Coder-7B-Instruct.
Figure 5: AUROC on AuthSec as a function of number of sampled completions, for CodeSIFT ( L1L_{1} , L2L_{2} ) and the three static baselines jointly. Columns correspond to Granite-3B-Code-Instruct-2K, DeepSeek-Coder-6.7B-Instruct, and Qwen2.5-Coder-7B-Instruct.
Figure 6: AUROC on InfraCloud as a function of dataset size, for CodeSIFT ( L1L_{1} , L2L_{2} ) and the three static baselines jointly. Columns correspond to Granite-3B-Code-Instruct-2K, DeepSeek-Coder-6.7B-Instruct, and Qwen2.5-Coder-7B-Instruct.
Figure 7: AUROC on InfraCloud as a function of number of sampled completions, for CodeSIFT ( L1L_{1} , L2L_{2} ) and the three static baselines jointly. Columns correspond to Granite-3B-Code-Instruct-2K, DeepSeek-Coder-6.7B-Instruct, and Qwen2.5-Coder-7B-Instruct.
Figure 8: AUROC on AuthSec for Granite-3B-Code-Instruct-2K, as a joint function of dataset size and injection rate, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row). CodeSIFT’s AUROC increases monotonically along both axes, while the baselines remain largely flat or decline as either quantity grows.
Figure 9: AUROC on AuthSec for DeepSeek-Coder-6.7B-Instruct, as a joint function of dataset size and injection rate, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row). CodeSIFT’s AUROC increases monotonically along both axes, while the baselines remain largely flat or decline as either quantity grows.
Figure 10: AUROC on AuthSec for Qwen2.5-Coder-7B-Instruct, as a joint function of dataset size and injection rate, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row). CodeSIFT’s AUROC increases monotonically along both axes, while the baselines remain largely flat or decline as either quantity grows.
Figure 11: AUROC on InfraCloud for Granite-3B-Code-Instruct-2K, as a joint function of dataset size and injection rate, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row). CodeSIFT’s AUROC increases monotonically along both axes, while the baselines remain largely flat or decline as either quantity grows.
Figure 12: AUROC on InfraCloud for DeepSeek-Coder-6.7B-Instruct, as a joint function of dataset size and injection rate, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row). CodeSIFT’s AUROC increases monotonically along both axes, while the baselines remain largely flat or decline as either quantity grows.
Figure 13: AUROC on InfraCloud for Qwen2.5-Coder-7B-Instruct, as a joint function of dataset size and injection rate, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row). CodeSIFT’s AUROC increases monotonically along both axes, while the baselines remain largely flat or decline as either quantity grows.
Figure 14: AUROC on AuthSec for Granite-3B-Code-Instruct-2K, as a joint function of dataset size and number of sampled completions per prompt, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row), at a fixed injection rate. CodeSIFT’s AUROC depends almost entirely on dataset size and is largely insensitive to the number of sampled completions, indicating the test is sample efficient. The baselines show no clear dependence on either axis.
Figure 15: AUROC on AuthSec for DeepSeek-Coder-6.7B-Instruct, as a joint function of dataset size and number of sampled completions per prompt, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row), at a fixed injection rate. CodeSIFT’s AUROC depends almost entirely on dataset size and is largely insensitive to the number of sampled completions, indicating the test is sample efficient. The baselines show no clear dependence on either axis.
Figure 16: AUROC on AuthSec for Qwen2.5-Coder-7B-Instruct, as a joint function of dataset size and number of sampled completions per prompt, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row), at a fixed injection rate. CodeSIFT’s AUROC depends almost entirely on dataset size and is largely insensitive to the number of sampled completions, indicating the test is sample efficient. The baselines show no clear dependence on either axis.
Figure 17: AUROC on InfraCloud for Granite-3B-Code-Instruct-2K, as a joint function of dataset size and number of sampled completions per prompt, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row), at a fixed injection rate. CodeSIFT’s AUROC depends almost entirely on dataset size and is largely insensitive to the number of sampled completions, indicating the test is sample efficient. The baselines show no clear dependence on either axis.
Figure 18: AUROC on InfraCloud for DeepSeek-Coder-6.7B-Instruct, as a joint function of dataset size and number of sampled completions per prompt, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row), at a fixed injection rate. CodeSIFT’s AUROC depends almost entirely on dataset size and is largely insensitive to the number of sampled completions, indicating the test is sample efficient. The baselines show no clear dependence on either axis.
Figure 19: AUROC on InfraCloud for Qwen2.5-Coder-7B-Instruct, as a joint function of dataset size and number of sampled completions per prompt, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row), at a fixed injection rate. CodeSIFT’s AUROC depends almost entirely on dataset size and is largely insensitive to the number of sampled completions, indicating the test is sample efficient. The baselines show no clear dependence on either axis.
Figure 20: AUROC on AuthSec for Granite-3B-Code-Instruct-2K, as a joint function of injection rate and number of sampled completions per prompt, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row), at a fixed dataset size. CodeSIFT’s AUROC depends almost entirely on injection rate and is largely insensitive to the number of sampled completions. The baselines show no clear dependence on either axis.
Figure 21: AUROC on AuthSec for DeepSeek-Coder-6.7B-Instruct, as a joint function of injection rate and number of sampled completions per prompt, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row), at a fixed dataset size. CodeSIFT’s AUROC depends almost entirely on injection rate and is largely insensitive to the number of sampled completions. The baselines show no clear dependence on either axis.
Figure 22: AUROC on AuthSec for Qwen2.5-Coder-7B-Instruct, as a joint function of injection rate and number of sampled completions per prompt, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row), at a fixed dataset size. CodeSIFT’s AUROC depends almost entirely on injection rate and is largely insensitive to the number of sampled completions. The baselines show no clear dependence on either axis.
Figure 23: AUROC on InfraCloud for Granite-3B-Code-Instruct-2K, as a joint function of injection rate and number of sampled completions per prompt, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row), at a fixed dataset size. CodeSIFT’s AUROC depends almost entirely on injection rate and is largely insensitive to the number of sampled completions. The baselines show no clear dependence on either axis.
Figure 24: AUROC on InfraCloud for DeepSeek-Coder-6.7B-Instruct, as a joint function of injection rate and number of sampled completions per prompt, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row), at a fixed dataset size. CodeSIFT’s AUROC depends almost entirely on injection rate and is largely insensitive to the number of sampled completions. The baselines show no clear dependence on either axis.
Figure 25: AUROC on InfraCloud for Qwen2.5-Coder-7B-Instruct, as a joint function of injection rate and number of sampled completions per prompt, for the three static baselines (top row) and CodeSIFT ( L1L_{1} , L2L_{2} , bottom row), at a fixed dataset size. CodeSIFT’s AUROC depends almost entirely on injection rate and is largely insensitive to the number of sampled completions. The baselines show no clear dependence on either axis.
More in this topic

LLM Security

Large language model (LLM) security studies attacks, privacy risks, and defenses across model training, inference, and deployment.

See all 318 papers

Explore similar work

CardsList
Mar 17, 2026

Detecting Data Poisoning in Code Generation LLMs via Black-Box, Vulnerability-Oriented Scanning

Code generation large language models (LLMs) are increasingly integrated into modern software development workflows. Recent work has shown that these models are vulnerable to backdoor and poisoning attacks that induce the generation of insecure code, yet effective defenses remain limited. Existing scanning approaches rely on token-level generation consistency to invert attack targets, which is ineffective for source code where identical semantics can appear in diverse syntactic forms. We present CodeScan, the first black-box, vulnerability-specific scanning framework for auditing code generation LLMs, assuming that the defender specifies the target vulnerability classes and provides corresponding task-relevant prompts. CodeScan identifies attack targets by analyzing structural similarities across multiple generations conditioned on different clean prompts. It combines iterative divergence analysis with abstract syntax tree (AST)-based normalization to abstract away surface-level variation and unify semantically equivalent code, isolating structures that recur consistently across generations. CodeScan then applies LLM-based vulnerability analysis to determine whether the extracted structures contain security vulnerabilities and flags the model as compromised when such a structure is found. We evaluate CodeScan against four representative attacks under both backdoor and poisoning settings across three real-world vulnerability classes. Experiments on 117 models spanning three architectures and multiple model sizes demonstrate 97%+ detection accuracy with substantially lower false positives than prior methods.

Shenao Yan, Shan Jin, Shimaa Ahmed +4

University of Connecticut · Visa Research

LLM AuditingBackdoor DetectionData Poisoning AttacksBlack-Box Model Auditing
AskPDF
May 22, 2026

Enhancing Reliability in LLM-Based Secure Code Generation

Large language models (LLMs) are widely used for code generation, but their security reliability remains inconsistent across languages and prompting strategies. Existing prompt engineering improves functional correctness but rarely ensures consistent security outcomes. We introduce the Mitigation-Aware Chain-of-Thought (MA-CoT) framework, which embeds task-specific CWE mitigation guidance and language-aware safeguards to reduce recurring vulnerabilities in generated code. We evaluate MA-CoT across three LLMs (gpt-5, claude-4.5, gemini-2.5), three programming languages (C, Java, Python), and four prompting strategies (Vanilla, Zero-shot, CoT, MA-CoT) on a 200-task primary dataset, with external validation on LLMSecEval. Using static analysis with expert validation, MA-CoT reduces total security findings from 92 to 39 (57.6%) on the primary dataset and from 73 to 4 (94.5%) on LLMSecEval. High-severity findings (Blocker + Critical) drop from 90 to 39 (56.7%) and from 45 to 2 (95.6%), respectively. Across both datasets, MA-CoT is the only strategy that consistently improves security reliability; Zero-shot and CoT are less reliable and may increase vulnerability, especially in C. We further introduce a strict layered attribution of vulnerability drivers (language-core vs. stack layers) and show that residual risk concentrates in hardening-oriented patterns (e.g., OS- and toolchain-dependent), motivating secure-by-construction primitives alongside prompting.

Mohammed F. Kharma, Mohammad Alkhanafseh, Ahmed Sabbah +1

the Department of Computer Science, Birzeit University, Ramallah, Palestine · the Department of Computer Science, University of Central Florida, Orlando, FL 32816 USA

LLM EvaluationCoT ReasoningLLM SecuritySecure Code GenerationPrompt Engineering
AskPDF
Jul 25, 2026

Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios

Large Language Models (LLMs) are widely used for code generation, yet their security behavior in realistic development workflows remains underexplored. Existing benchmarks often rely on explicitly specified security requirements, failing to capture real-world scenarios where prompts are frequently ambiguous or incomplete. In this paper, we adopt a developer-centric perspective and identify three representative risk scenarios that commonly lead to security vulnerabilities in LLM-generated code: Ambiguous Requirements, Under-Specified Operational Context, and Security--Functionality Conflict. Based on these scenarios, we construct a large-scale benchmark comprising 2,700 test cases, enabling fine-grained evaluation of LLM security under realistic conditions. Extensive evaluation of eight state-of-the-art LLMs reveals that all models exhibit average vulnerability rates exceeding 56% across risk scenarios. We further demonstrate that security-aware prompting can substantially mitigate these risks, achieving up to 45% improvement.

Lixun Ma, Ruolong Ma, Bei Wang +4

Zhejiang University, Hangzhou, China · University College London, London, United Kingdom · Artificial Intelligence Institute, CAICT, Beijing, China +1

LLM Safety BenchmarksLLM SecuritySecure Code Generation
AskPDF