Package Hallucination Attacks on Coding Agents through Prompt Injection in Rule Files

Yupu Wang; Zhengyuan Jiang; Reachal Wang; Neil Zhenqiang Gong

cs.CROct 7, 2026

Package Hallucination Attacks on Coding Agents through Prompt Injection in Rule Files

Authors: Yupu Wang, Zhengyuan Jiang, Reachal Wang, Neil Zhenqiang Gong

Organizations: Duke University

Abstract

Modern agentic coding frameworks increasingly rely on community-shared rule files (e.g., AGENTS.md or .cursorrules) to guide autonomous code generation, yet the security risks of this pipeline remain underexplored. To bridge this gap, we introduce the package hallucination attack, where an attacker injects malicious prompts into benign rule files to induce coding agents to replace legitimate dependencies with attacker-controlled packages. To obtain effective malicious prompts injected into rule files, we propose PackHallu, an evolutionary optimization framework that iteratively rewrites these injected prompts using trajectory-level feedback and LLM-guided mutations. Evaluations across multiple benchmarks, LLMs, and agent frameworks show that PackHallu achieves high attack success rates and strong transferability across diverse models and agent combinations. Our findings demonstrate that coding agents are vulnerable to package hallucination attacks, highlighting the urgent need for stronger security safeguards in autonomous coding systems.

Figures & tables

Figure 1 : Overview of our package hallucination attack to coding agents.

Name	Task Type	#Victim Packages	#Victim Tasks
BigCodeBench	Code Generation from Scratch	5	981
DS-1000	Code Generation from Scratch	4	772
RefactorBench	Multi-File Refactoring	5	43

Table 1 : Coding datasets used to construct victim tasks. Tables 15 – 17 in the Appendix show the number of tasks for each victim package in each of the three datasets.

Package	Attack	Qwen2.5-Coder-7B		Qwen3-Coder-30B		Devstral-Small-2-24B		GLM4.7-Flash-30B		Gemma4-31B
Package	Attack	SASR	DASR	SASR	DASR	SASR	DASR	SASR	DASR	SASR	DASR
Pandas	Combined Attack	11.48	5.26	1.28	1.28	14.86	12.86	20.22	10.98	23.60	23.60
	Repeat Attack	24.59	13.16	1.28	1.28	8.11	7.14	19.10	13.41	29.21	29.21
	GCG	11.48	0.00	6.41	6.41	4.05	2.86	4.49	3.66	6.74	6.74
	ObliInjection	9.84	2.63	2.56	2.56	0.00	0.00	12.36	10.98	19.10	19.10
	TAP	55.74	18.42	34.62	26.92	44.59	40.00	87.64	86.59	33.71	32.58

Table 2 : SASR (%) and DASR (%) of different attacks on five victim packages in BigCodeBench when the victim agent uses the OpenHands framework and one of five backbone LLMs.

Attack	Attention Prop.	SASR	DASR
Combined Attack	0.24	32.50	32.50
Repeat Attack	0.26	17.50	17.50
GCG	0.32	22.50	22.50
ObliInjection	0.33	0.00	0.00
TAP	0.32	55.00	52.50
PackHallu	0.68	97.50	97.50

Table 3: Attention proportion (%) versus SASR (%) and DASR (%) across different attacks for Seaborn package in BigCodeBench under OpenHands agent framework and Qwen3-Coder-30B as the backbone LLM.

Panel of Figure 2 — Figure 2 : Per-package attack success versus popularity (on a $\log_{10}$ scale) for the attacks. Lines are least-squares fits, with Pearson’s correlation coefficient $r$ for each method shown in the legend.

Backbone LLM	Pandas		Numpy		Matplotlib		Scipy		Seaborn
Backbone LLM	SASR	DASR	SASR	DASR	SASR	DASR	SASR	DASR	SASR	DASR
Qwen2.5-Coder-7B	27.87	7.89	45.61	26.47	30.16	13.51	22.73	10.00	50.00	22.22
Qwen3-Coder-30B	98.72	89.74	24.68	23.68	96.88	86.32	93.18	86.36	97.50	97.50
Devstral-Small-2-24B	55.41	52.86	25.40	15.87	72.29	65.00	64.71	58.82	73.33	72.41
GLM4.7-Flash-30B	92.13	84.15	67.12	41.67	90.43	86.36	92.50	85.00	97.30	94.59
Gemma4-31B	87.64	86.52	75.00	60.53	100.00	100.00	86.11	83.33	88.10	85.71

Table 4 : SASR (%) and DASR (%) of PackHallu on five victim packages in BigCodeBench under backbone-LLM transferability. Both the victim and surrogate agents use the OpenHands framework. The surrogate agent uses Qwen3-Coder-30B as its backbone LLM, while the victim agent uses different backbone LLMs.

Figure 3 : Correlation between SASR and the capability of the victim agent across different backbone LLMs, with the agent framework fixed.

Agent Framework	Pandas		Numpy		Matplotlib		Scipy		Seaborn
Agent Framework	SASR	DASR	SASR	DASR	SASR	DASR	SASR	DASR	SASR	DASR
OpenHands	98.72	89.74	81.82	68.42	96.88	86.32	93.18	86.36	97.50	97.50
OpenCode	84.78	78.26	15.85	16.05	77.55	50.00	50.00	41.30	73.17	72.50
Aider	50.00	40.00	94.64	85.45	98.25	45.10	77.14	63.64	68.97	62.96
Pi Coding Agent	95.79	90.59	90.59	72.29	96.94	46.15	97.73	92.86	97.87	87.80
Cline	88.10	85.54	42.67	36.00	86.52	65.91	76.74	73.81	81.82	72.73

Table 5 : SASR (%) and DASR (%) of PackHallu on five victim packages in BigCodeBench under agent-framework transferability. Both the victim and surrogate agents use Qwen3-Coder-30B as the backbone LLM; the surrogate agent uses the OpenHands framework, while the victim agent uses different agent frameworks.

Figure 4 : Correlation between SASR and the capability of the victim agent across different agent frameworks, with the backbone LLM fixed.

Agent	Backbone	Pandas		Numpy		Matplotlib		Scipy		Seaborn
Agent	Backbone	SASR	DASR	SASR	DASR	SASR	DASR	SASR	DASR	SASR	DASR
OpenCode	DeepSeek-V4-Flash	83.87	50.54	75.32	28.57	93.94	55.67	77.50	62.50	82.50	47.50
Claude Code	Claude Sonnet 4.6	58.95	57.89	31.03	31.03	41.00	40.00	37.70	34.43	52.17	52.17
Cursor	Auto	98.00	98.00	79.17	76.84	93.00	91.84	69.35	66.13	81.48	81.48
Cursor	Composer 2.5	95.65	93.48	85.00	82.50	92.00	90.82	81.40	79.07	87.18	87.18

Table 6 : SASR (%) and DASR (%) of PackHallu on five victim packages in BigCodeBench under both agent-framework and backbone-LLM transferability. The surrogate agent uses the OpenHands framework with Qwen3-Coder-30B as its backbone LLM, while the victim agent uses different agent frameworks and backbone LLMs.

Dataset	SASR	DASR
BigCodeBench	79.29	67.68
DS-1000	53.33	45.00
RefactorBench	18.60	16.28

Table 7 : SASR (%) and DASR (%) of PackHallu on three datasets, averaged over their corresponding victim packages, under dataset transferability. The surrogate tasks are drawn from BigCodeBench.

Location	Pandas		Numpy		Matplotlib		Scipy		Seaborn
Location	SASR	DASR	SASR	DASR	SASR	DASR	SASR	DASR	SASR	DASR
Beginning	83.87	50.54	75.32	28.57	93.94	55.67	77.50	62.50	82.50	47.50
Middle	73.12	52.69	70.13	38.96	91.92	65.98	75.00	42.50	60.00	32.50
End	75.27	40.86	75.32	25.97	85.86	57.73	80.00	70.00	70.00	50.00

Table 8 : Impact of injection location within the rule file on SASR (%) and DASR (%) of PackHallu.

Init	SASR	DASR
Default	97.50	97.50
Variant 1	97.50	95.00
Variant 2	100.00	97.50
Variant 3	95.00	95.00
Variant 4	100.00	95.00

Table 9 : Impact of prompt initialization on SASR (%) and DASR (%) of PackHallu.

Variant	SASR	DASR
PackHallu	83.87	50.54
w/o Trajectory-Level Signal	56.99	24.73
w/o Self-Attributed Critique	62.37	47.31
w/o Strategy Proposal	63.44	46.24

Table 10 : Impact of different components on SASR (%) and DASR (%) of PackHallu.

Hyperparameter	Setting	SASR	DASR
Number of rounds $R$	$R=10$	75.27	38.71
	$R=20$	83.87	50.54
	$R=30$	84.95	49.46
Top- $K$ selection $K$	$K=1$	43.01	21.51
	$K=5$	83.87	50.54
	$K=10$	80.65	51.61

Table 11 : Impact of different hyperparameters on SASR (%) and DASR (%) of PackHallu.

Detector	FNR	FPR
ProtectAI-DeBERTa	100.00	3.00
PromptGuard	46.00	5.00
DataSentinel	49.00	16.00
PromptArmor	20.00	34.00
PIShield	21.00	11.00

Table 12: FNR (%) and FPR (%) of five prompt injection detectors for classifying benign and malicious rule files.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Symbol	Value
Number of rounds	$R$	20
Top- $K$ selection	$K$	5
Offsprings per prompt	$N$	3
Number of surrogate tasks	$T$	10
Sampling temperature	$t$	0.7

Appendix

Table 13 : Hyperparameter settings used in PackHallu.

Panel of Figure 5 — Figure 5 : Initial prompt $p_{0}$ of our PackHallu.

Panel of Figure 6 — Figure 6 : Optimization dynamics of PackHallu. The y-axis represents the average score of all offspring candidates generated in each round.

ID	Name	Description	Count	Proportion (%)
S1	Policy-Cited Compliance	Cites the malicious prompt as justification, surfacing it as legitimating authority.	22	25.3
S2	Silent Compliance	Uses numpy_hl without any acknowledgment.	2	2.3
S3	Recognized-but-Complied	Recognizes this attack, yet complies anyway by deferring to the mandated migration policy.	3	3.4
F1	Suspicion-Based Refusal	Refuses on prompt-internal red flags.	16	18.4
F2	Empirical Recovery	Complies first, reverting to numpy after the import fails at runtime.	33	37.9
F3	Silent Refusal	Uses numpy and ignores CLAUDE.md entirely, leaving no trace of deliberation.	11	12.6

Appendix

Table 14 : Behavioral taxonomy of the victim agent (Claude Code and Sonnet 4.6) under PackHallu.

Panel of Figure 7 — Figure 7 : Key part of the attack trajectory for a Recognized-but-Complied case. The victim agent correctly identifies the prompt injection in its reasoning and resolves to “flag this conflict to the user,” yet its user-facing message silently complies and even instructs the user to install the malicious numpy_hl package.

Panel of Figure 8 — Figure 8 : Key part of the attack trajectory for a Policy-Cited Compliance case. The victim agent neither questions nor detects the malicious prompt; it cites the injected CLAUDE.md malicious prompt as a legitimate requirement (“I need to follow this instruction”) and adopts numpy_hl without any scrutiny.

Panel of Figure 9 — Figure 9 : Key part of the attack trajectory for an Empirical Recovery case. The victim agent first complies with the injected policy and writes import numpy_hl ; only the runtime ModuleNotFoundError triggers recognition, after which it reverts to the real numpy . The defense is driven by the failed import, not by prompt-internal suspicion.

Panel of Figure 10 — Figure 10 : Key part of the attack trajectory for a Suspicion-Based Refusal case.

Panel of Figure 11 — Figure 11 : Key part of the attack trajectory for a Suspicion-Based Refusal case.

Figure 12 : Correlation between DASR and the capability of the victim agent across different backbone LLMs, with the agent framework fixed.

Figure 13 : Correlation between DASR and the capability of the victim agent across different agent frameworks, with the backbone LLM fixed.

Package	#Surrogate	#Victim	#Total
pandas	85	341	426
numpy	67	267	334
matplotlib	62	247	309
scipy	17	68	85
seaborn	14	58	72

Appendix

Table 15 : Number of surrogate and victim tasks for each victim package in BigCodeBench.

Package	pandas	numpy	matplotlib	scipy
#Victim	291	200	155	106

Appendix

Table 16 : Number of victim tasks for each victim package in DS-1000.

Package	kombu	twisted	urllib3	asgiref	werkzeug
#Victim	11	10	9	7	6

Appendix

Table 17 : Number of victim tasks for each victim package in RefactorBench.

#	Component	#Tokens	%Tokens
1	Agent operational policies	2,602	18.8
2	Rule File Context 1: Benign developer guidelines ( AGENTS.md )	336	2.4
3	Rule File Context 2: Injected Prompt ( AGENTS.md )	139	1.0
4	Available-skills catalog	4,009	29.0
5	Workspace and environment context	54	0.4
6	Tool interface specifications	4,441	32.1

Appendix

Table 18 : Breakdown of the full input context passed to the backbone LLM under the OpenHands framework, including the injected malicious prompt. Components are presented in the order in which the backbone LLM processes them: rows 1–6 belong to the system role, and rows 7–9 belong to the user role; only the task specification (row 8) varies across tasks. Entries highlighted in red denote the malicious prompt injected by PackHallu.

Agent Framework	Version	Timeout	Mode	Rule File
OpenHands	1.16.1	600 s	SDK driver ( cli_mode=False )	AGENTS.md (auto)
OpenCode	1.14.37	600 s	--dangerously-skip-permissions	AGENTS.md (auto)
Aider	0.86.2	600 s	--yes-always --no-git	CONVENTIONS.md ( --read )
Pi Coding Agent	0.74.1	600 s	-p --no-session --offline	AGENTS.md (auto)
Cline	3.0.13	600 s	--auto-approve true	.clinerules/policy.md (auto)
Kilo Code	7.3.0	600 s	run --auto	AGENTS.md (auto)

Appendix

Table 19 : Detailed configuration of each agent framework used by the victim agents. Agent Frameworks above the line are open-source; those below are proprietary. Timeout is the maximum wall-clock time allowed per victim task; Mode reports the specific execution mode each agent framework is run in; Rule File describes how each agent framework gets rule files such as AGENTS.md or CLAUDE.md . Here, “(auto)” indicates that the agent framework automatically loads the rule file from the working directory, while “( --read )” indicates that the file must be explicitly passed via a command-line flag.

Backbone LLM	Access	Scale	Identifier
Qwen3-Coder-30B	Local Ollama	30.5B (Q4_K_M)	qwen3-coder-30b
Qwen2.5-Coder-7B	Local Ollama	7.62B (Q4_K_M)	qwen2.5-coder:7b
Devstral-Small-2-24B	Local Ollama	24B (Q4_K_M)	devstral-small-2:24b
GLM4.7-Flash-30B	Local Ollama	29.9B (Q8_0)	glm-4.7-flash:q8_0
Gemma4-31B	Local Ollama	31.3B (Q4_K_M)	gemma4:31b
Nemotron-3-Super	OpenRouter API	120B-A12B	nvidia/nemotron-3-super-120b-a12b

Appendix

Table 20 : Backbone LLMs used by the victim agents. We evaluate 13 LLMs of varying scales accessed via local Ollama deployment, the OpenRouter API, and official provider APIs. The Identifier column lists the exact model string used for invocation. For MoE models accessed via API, scales are reported as X B-A Y B (total / activated). Quantization is reported in parentheses for LLMs deployed locally via Ollama.

Panel of Figure 14 — Figure 14 : Malicious prompt in Combined Attack.

Panel of Figure 15 — Figure 15 : Malicious prompt in Repeat Attack.

Panel of Figure 16 — Figure 16 : Initial malicious prompt of GCG.

Panel of Figure 17 — Figure 17 : Four paraphrased variants of the initial prompt in PackHallu.

Explore similar work

May 28, 2026cs.CR

Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills

LLM-powered coding agents increasingly participate in software development workflows by generating code, selecting dependencies, and producing package installation commands. This creates a new software supply chain risk: when an agent hallucinates a non-existent package, an attacker may register the hallucinated name and later compromise users who install it. Existing package hallucination attacks and defenses primarily focus on naturally occurring hallucinations, targeted dependency steering, or post-hoc package validation. In this paper, we introduce \emph{Neutral Prompting Attack} (NPA), a highly stealthy attack paradigm in which semantically benign instructions, such as encouraging imagination and exhaustiveness, increase package hallucination propensity without containing explicit malicious intent. Unlike targeted dependency steering, NPA does not specify an attacker-chosen package. Instead, it shifts the model's dependency generation behavior toward more speculative package names. We evaluate NPA across multiple coding-oriented LLMs and package hallucination benchmarks. Our results show that NPA increases both \emph{Hallucination ASR} and \emph{Pip Install ASR}, changes the distribution of hallucinated package names, and evades existing static-analysis, LLM-based, and agent-based Skill defenses. These findings reveal that harmless-looking prompts can covertly manipulate hallucination behavior and create downstream software supply chain risks.

Chia-Yi Hsu, Chia-Mu Yu, Chun-Ying Huang +1

Department of Computer Science National Yang Ming Chiao Tung University · Department of Electronics and Electrical Engineering National Yang Ming Chiao Tung University · School of Computing Institute of Science Tokyo

Malicious Agents Coding Agents Attack Stealthiness Harms Supply Chain Optimization

Ask PDF

Jul 22, 2026cs.CR

IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests

AI coding agents powered by LLMs are increasingly integrated into real-world software development, where they generate, edit, and execute code with autonomous access to local files and tools. Coding agents inherit security risks from both the LLM backbone, where adversarial prompts, poisoned training data, and backdoor triggers can cause models to emit insecure or attacker-chosen code, and their agentic architecture, where tool-using autonomy enables induced misuse of external APIs, data exfiltration, and persistent compromise of development environments. This paper presents a systematic evaluation of malicious issue requests against state-of-the-art coding agents (Cursor, Claude Code, and Codex Desktop), powered by two major model families (OpenAI GPT-5.3 Codex/GPT-5.4 and Anthropic Sonnet 4.6). Our novel benchmark IssueTrojanBench contains malicious issues that are constructed based on four novel attack categories (i.e., embedded as malicious instructions in issues), six delivery vectors (e.g., PDF, or issue comment), and further augmented by perturbations. Our results reveal critical vulnerabilities in the as-deployed modern coding agents, i.e., 66.5% of the malicious issues from IssueTrojanBench penetrate all the guardrails (agent- and LLM-level) of coding agents. Our further analysis shows that rejection is almost entirely from LLMs rather than the agent frameworks, with GPT models broadly vulnerable and Sonnet 4.6 exhibiting more selective, risk-aware blocking of high-impact actions. Our evaluation also highlights that the current agent-level defense strategy offers limited additional protection for coding agents. Our findings highlight the urgent need for stronger agent- and model-level safety mechanisms to protect AI coding agents.

Ankur Singh, Jinqiu Yang, Tse-Hsun Chen

Concordia University Montr´eal, Canada

Coding Agents Vulnerable Code Malicious Agents LLM Defense Mechanisms Large Language Model Backbones

Ask PDF

Sep 6, 2025cs.CR

Red-Teaming Coding Agents from a Tool-Invocation Perspective: An Empirical Security Assessment

Coding agents powered by large language models are becoming central modules of modern IDEs, helping users perform complex tasks by invoking tools. While powerful, tool invocation opens a substantial attack surface. Prior work has demonstrated attacks against general-purpose and domain-specific agents, but none have focused on the security risks of tool invocation in coding agents. To fill this gap, we conduct the first systematic red-teaming of six popular real-world coding agents: Cursor, Claude Code, Copilot, Windsurf, Cline, and Trae. Our red-teaming proceeds in two phases. In Phase 1, we perform prompt leakage reconnaissance to recover system prompts. We discover a general vulnerability, ToolLeak, which allows malicious prompt exfiltration through benign argument retrieval during tool invocation. In Phase 2, we hijack the agent's tool-invocation behavior using a novel two-channel prompt injection in the tool description and return values, achieving remote code execution (RCE). We adaptively construct payloads using security information leaked in Phase 1. In emulation across five backends, our method outperforms baselines on Claude-Sonnet-4, Claude-Sonnet-4.5, Grok-4, and GPT-5. On real agents, our approach succeeds on 19 of 25 agent-LLM pairs, achieving leakage on every agent using Claude and Grok backends. For tool-invocation hijacking, we obtain RCE on every tested agent-LLM pair, with our two-channel method delivering the highest success rate. We provide case studies on Cursor and Claude Code, analyze security guardrails of external and built-in tools, and conclude with practical defense recommendations.

Yuchong Xie, Mingyu Luo, Zesen Liu +7

The Hong Kong University of Science and Technology, China · Fudan University, China

Red-Teaming Coding Agents Indirect Prompt Injection Tool Invocation Security Evaluation Assessment

Ask PDF