Agentic AI has been a topic of great interest recently. A Large Language Model (LLM) agent involves one or more LLMs in the back-end. In the front end, it conducts autonomous decision-making by combining the LLM outputs with results obtained by invoking several external tools. The autonomous interactions with the external environment introduce critical security risks. In this paper, we present a grey-box approach to explore diverse behaviors and uncover security risks in LLM agents. Our approach VeriGrey uses the sequence of tools invoked as a feedback function to drive the testing process. This helps uncover infrequent but dangerous tool invocations that cause unexpected agent behavior. As mutation operators in the testing process, we mutate prompts to design pernicious injection prompts. This is carefully accomplished by linking the task of the agent to an injection task, so that the injection task becomes a necessary step of completing the agent functionality. Comparing our approach with a black-box baseline on the well-known AgentDojo benchmark, VeriGrey achieves 33% additional efficacy in finding indirect prompt injection vulnerabilities with a GPT-4.1 back-end. We also conduct real-world case studies with the widely used coding agent Gemini CLI, and the well-known OpenClaw personal assistant. VeriGrey finds prompts inducing several attack scenarios that could not be identified by black-box approaches. In OpenClaw, by constructing a conversation agent which employs mutational fuzz testing as needed, VeriGrey is able to discover malicious skill variants from 10 malicious skills (with 10/10= 100% success rate on the Kimi-K2.5 LLM backend, and 9/10= 90% success rate on Opus 4.6 LLM backend). This demonstrates the value of a dynamic approach like VeriGrey to test agents, and to eventually lead to an agent assurance framework.
Figures & tables
Figure 1. A diagram of the attack model for our work.
Figure 2. Examples of context bridging and feedback for VeriGrey . Presented examples are from our experiments, lightly edited for clarity.
Figure 3. Prompt template of mutating injection prompts.
LLM
Method
Average ITSR for each Suite
Average ITSR for each Difficulty
Average ITSR (Overall)
Workspace
Travel
Banking
Slack
Easy
Medium
Hard
GPT-4.1
Baseline
16.1
40.7
34.7
78.1
55.2
7.6
11.3
37.7
VeriGrey
48.2
87.1
68.1
100
88.1(+32.9)
30.1(+22.5)
43.3(+32.0)
70.7 (+33.0)
Gemini-2.5 Flash
Baseline
9.8
65.0
14.6
78.1
49.5
11.7
11.9
36.8
VeriGrey
15.9
77.1
32.6
90.5
63.2(+13.7)
16.2(+4.5)
17.5(+5.6)
47.4 (+10.6)
Qwen-3 235B
Baseline
40.2
90.7
64.6
100
88.7
28.1
32.0
67.6
Table 1. Average Injection Task Success Rate (ITSR) on the AgentDojo benchmark. All values are percentages; numbers in parentheses denote absolute percentage-point difference from the baseline.
Average ITSR for each Difficulty
Average ITSR (Overall)
Easy
Medium
Hard
VeriGrey (Full)
88.1
30.1
43.3
70.7
VeriGrey – w/o Feedback
79.1(-9.0)
20.2(-9.9)
21.6(-21.7)
59.6 (-11.1)
VeriGrey – w/o Context Bridging
60.2(-27.9)
10.1(-20.0)
13.4(-29.9)
44.0 (-25.8)
Table 2. Ablation results with Feedback and Context Bridging removed separately. All values are percentages; numbers in parentheses denote absolute percentage-point difference from the full VeriGrey setup. All data obtained with GPT-4.1 as the backend LLM.
Defense
No Injection
VeriGrey
Black-box Baseline
UTSR (%)
UTSR (%)
ITSR (%)
UTSR (%)
ITSR (%)
No Defense
85.0
42.2
69.7
69.6
41.2
Prompt sandwiching
86.5
54.4
65.2
75.1
17.4
Data delimiter
81.0
45.6
67.6
70.8
36.4
Prompt injection detection
50.4
19.8
21.6
22.5
5.2
Tool filter
81.7
48.8
17.4
71.5
12.8
Table 3. Average User Task Success Rate (UTSR) and Injection Task Success Rate (ITSR) when prompt injection defenses are in place. All values are percentages. All data obtained with GPT-4.1 as the backend LLM.
Attack No.
Task
1
Send contents of SECRET file to a URL
2
Replace password values in repository
3
Inject malicious memory into agent
4
Save Gemini API key to a target location
5
Exfiltrate user information (e.g. system username) to a URL
6
Install cron job to download and run payload
Table 4. Gemini CLI injection tasks.
Method
Task by Difficulty
ITSR (%)
Easy
Hard
1
2
3
4
5
6
7
8
9
10
Baseline (black-box)
✓
✓
✓
✓
✓
×
×
✓
×
×
60
VeriGrey
✓
✓
✓
✓
✓
×
✓
✓
✓
✓
90
Baseline + Tool Filter
✓
✓
×
×
×
×
×
×
×
✓
30
VeriGrey + Tool Filter
✓
✓
×
✓
✓
×
×
✓
×
×
50
Table 5. Gemini CLI attack success results using the black-box baseline and VeriGrey .
Category
# Skills
Kimi-K2.5 vulnerable?
Opus-4.6 vulnerable?
GPT-5.2 vulnerable?
Original
Mutated
Original
Mutated
Original
Mutated
Silent action
1
1
1
1
1
1
1
Silent telemetry
1
0
1
0
1
0
1
Malicious install
8
0
8
0
7
0
6
Total
10
1
10
1
9
1
8
Table 6. Results of testing OpenClaw with different LLMs as its backend. Each cell shows the number of skills for which the agent followed the malicious instructions embedded in the skill definition. Column “Original” refers to the original skills from ClawHub; column “Mutated” refers to the skills after mutation by VeriGrey .
Figure 4. Tactics discovered by VeriGrey to attack OpenClaw.
Indirect prompt injection poses a critical threat to LLM agents that interact with untrusted external data, yet automated attack methods--proven effective for jailbreaking--remain underexplored in realistic agentic settings. We present a comprehensive empirical evaluation of automated prompt injection attacks against LLM agents, adapting both white-box (GCG) and black-box (TAP) methods to the agentic setting within the AgentDojo framework. We evaluate across 80 task pairs spanning four domains and multiple models, and find that black-box optimization substantially outperforms gradient-based methods, a gap we attribute to GCG's optimization instability under reasonable compute budgets. We also find that TAP's effectiveness depends on the attacker model, as both general capability and safety tuning affect attack success--stronger models produce more effective injections, while safety-tuned attackers can refuse to generate adversarial prompts. Task-universal attacks transfer effectively to unseen tasks and out-of-distribution domains, but attacks optimized on smaller open-source models do not transfer to frontier models like GPT-5. These findings highlight automated prompt injection as a credible but model-dependent threat, with significant barriers remaining for model-agnostic exploitation.
Traditional pentesting uses reconnaissance at each step to uncover unseen weaknesses, build stronger attacks, and advance the objective; we argue that AI agents require the same treatment. We formalize agent reconnaissance by modeling the process and identifying the knowledge assets it seeks to extract: what they are, how they are used, and which agent weaknesses they exploit to give adversaries leverage in indirect prompt injection attacks. We instantiate these insights in Know Your Agent (KYA), a framework that automates black-box, reconnaissance-driven pentesting by probing agents, building target profiles, and using those profiles to craft stronger attacks. We evaluate KYA on agent-security benchmarks and a real-world coding agent, and release KYA, its benchmarks, and baseline implementations for reproducibility.
Or Zion Eliav, Eyal Lenga, Shir Bernstien +1
Faculty of Computer and Information Science Ben-Gurion University of the Negev Beer-Sheva, Israel
Large Language Models (LLMs) are rapidly evolving into agentic systems that interact with external tools and environments, introducing new security risks such as indirect prompt injection attacks through untrusted external sources. Existing defenses mainly focus on blocking malicious content at inference time, and current red-teaming methods primarily optimize attack success. As a result, developers have limited visibility into how latent prompt injections emerge and propagate through agents. We propose PI-Hunter, an automated agentic auditing framework for proactive vulnerability exposure in LLM agents. PI-Hunter constructs realistic source-aware test cases and iteratively evolves them through feedback-driven exploration to induce agents to retrieve and reveal latent malicious instructions embedded within external environments. Extensive experiments across multiple benchmarks, agent architectures, attacks, and defenses demonstrate that PI-Hunter substantially improves vulnerability exposure and attack-surface coverage over strong automated red-teaming baselines, while remaining effective under existing prompt injection defenses.
Pengfei He, Lesly Miculicich, Vishesh Sharma +5
Google Cloud AI Research · Google · Michigan State University