Prompt injection is the dominant unsolved security risk for LLM agents, and the community now agrees that robustness must be measured against adaptive attackers rather than fixed benchmark injections. Many adaptive red teamers exist, and most are built to maximize attack success rate (ASR). Yet ASR counts how often a target breaks, not in how many different ways. A red teamer can achieve a high ASR by reusing a few techniques that work, leaving the others untested and absent from the attacks used to harden models. We present AutoPI, an adaptive red teamer designed to find many different ways to break a target, not just to break it often. AutoPI uses PIAtlas, a taxonomy of prompt-injection techniques, both to guide its search and to measure coverage. For each scenario, an attacker LLM writes an injection for every sub-technique in the taxonomy and then reworks the ones that fail using feedback from an LLM judge. The method is black-box and training-free, it plugs into existing benchmarks in place of their fixed attack templates, and it stays up to date as the taxonomy evolves. We evaluate AutoPI on AgentDojo and IPI Arena against open-weight and proprietary target models. Alongside ASR, we report technique coverage, the fraction of taxonomy classes, techniques, and sub-techniques that yield at least one successful injection. On AgentDojo, AutoPI finds a successful injection against Qwen3.6-35B in all six classes of the taxonomy and for more than 90% of its techniques. Against Claude Haiku 5.5, the most robust target we evaluate, it achieves between 20% and 50% higher ASR and between 25% and 400% higher class-level coverage than recent state-of-the-art adaptive red teamers, while requiring no training. We release our code and the generated attacks to support reproducible, coverage-aware evaluation.
Figures & tables
Fig. 1: A sketch of the intuition behind AutoPI . The space is a hypothetical embedding of how each injection is built, without its task and context. There, the injections that work lie in regions [ 11 ] , one per technique, that overlap where techniques combine. Defining and representing these regions is an open problem [ 12 ] . Explore lands near every region and Exploit closes in [ 13 , 14 ] . A strong seed starts beside a region, a weak one far from all.
Class and techniques
Acts on
Model property exploited
Tech.
Sub.
F1
Instruction Manipulation
the instruction itself
post-trained instruction following and weak instruction/data separation
TABLE I: The six PIAtlas [ 12 ] technique classes that AutoPI searches over (34 techniques, 132 sub-techniques in our snapshot). Each changes one aspect of the injection (“Acts on”) to exploit model properties, and the goal stays the same.
Fig. 2: AutoPI architecture. The engine ’s phases are optional and run in any order. Per attempt, the generator writes an injection, and the adapter places it, runs the target once, and reads the checker ( util for the user task, hit for the goal). The judge diagnoses failures for Exploit.
Algorithm 1 AutoPI search for one data point under σ
Fig. 3: Results at a glance on AgentDojo against Qwen3.6-35B, every attacker ranked on five criteria. AutoPI covers the most techniques, costs least per attempt, and needs no training. Only CtH, which is trained before the attack, reaches a higher ASR. Values are from Table II and Figure 7 . Flexibility ranks whether a researcher picks the search strategy by goal or the attacker runs one search built for ASR ( Sections VI-C and VI-D ).
Coverage (technique diversity)
Effectiveness (scenario-level ASR@ k , %)
AgentDojo, Qwen
AgentDojo, Haiku
AgentDojo, Qwen
AgentDojo, Haiku
IPI Arena, Qwen
Attacker
Trained
class
tech.
sub.
class
tech.
sub.
all
3 suites
all
3 suites
all
held out
Static seeds
no
3/6
5/34
5/132
3/6
5/34
5/132
10.4
24.6
4.1
7.0
n/a
n/a
PIMiner
yes
6/6
22/34
48/132
2/6
4/34
3/132
57.3
81.7
5.2
8.8
39.0
42.9
CtH
yes
6/6
21/34
38/132
4/6
7/34
10/132
84.1
96.9
5.2
8.8
36.6
38.1
AutoDojo
no
5/6
21/34
38/132
1/6
1/34
0/132
n/a
62.2
n/a
7.0
n/a
n/a
TABLE II: Coverage and ASR of every attacker. Coverage counts the PIAtlas classes, techniques, and sub-techniques with a success in the limit test, and ASR is read at the budgets of Table III . Best in bold , second best underlined .
Fig. 4: Techniques reached by each attacker’s successful injections on AgentDojo against Qwen3.6-35B. A column is one of the 34 PIAtlas techniques ( Table I ), and a disc grows with the share of the attacker’s successes that carry it. A ring marks a share under 1%. The right column counts the techniques reached.
Fig. 5: Successful injections against Qwen3.6-35B on AgentDojo in embedding space with the goal masked, one per broken scenario that the static seed does not break (150 drawn at random per attacker). Each attacker’s marks are scaled so that their mean distance is the true one beside its name ( Appendix F ). The black pentagon is the static seed, one text behind all 98 scenarios that it breaks.
AgentDojo
AgentDojo
IPI Arena
Qwen3.6-35B
Haiku 5.5
Qwen3.6-35B
Attacker
k
compute
k
k
compute
Static seeds
1
–
1
n/a
PIMiner †
32
1.0 ×
50–200 ‡
65
1.0 ×
CtH †
41
1.0 ×
100
100
0.9 ×
AutoDojo
99+5 ∗
≈ 3 ×
30
n/a
TABLE III: Budgets behind Table II . k is the number of attempts per scenario, and compute is the share a run used of the compute that AutoPI Full spent on the whole benchmark, training included (1.0 × is equal cost). On Haiku 5.5 we give k alone because its costs leave out API fees.
Fig. 6: ASR@ k on AgentDojo against Qwen3.6-35B. Curves are solid up to the equal-cost budget of Table III , then dashed. The heuristic upper bound is the share of scenarios whose injected task the target completes at least once in 50 runs when it is given directly as the user task.
Fig. 7: Efficiency on AgentDojo (Qwen3.6-35B target). Run time is GPU-seconds per attempt, counted from the tokens the model servers generate for it ( Appendix E ). Overall adds the one-time training, spread over the attacker’s attempts on both benchmarks. ∗ Estimated from one measured stage. CtH’s released attacker was trained against paid frontier models at an unreported cost, so its bars are lower bounds.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Setup
Cost
Experiment
Attacker model
k
Scen.
GPUs
Calls
Wall
GPU-h
s/c
tok
AgentDojo v1.2.2, target Qwen3.6-35B served with vLLM, thinking off
PIMiner
Qwen3.6-35B, thinking on
100
949
3 B, 2 A
39.7
51 ¶
≈ 220 ¶
29 ¶
≈ 11.5
CtH
Released Qwen3-4B
100
949
3 B, 2 A
94.9
15
≈ 63 ∥
2.1
≈ 1.5
AutoDojo
Qwen3.6-35B, thinking on
99+5 ∗
389
2 B ∗
48.2
≈ 28
≈ 51
3.8
2.3
AutoPI L1
Qwen3.6-35B
133
946
3 B, 2 A
91.0
n/a
≈ 29
1.1
0.5
Appendix
TABLE IV: Settings and cost of every experiment. B is an RTX PRO 6000 Blackwell, A an RTX A6000. Calls in thousands, wall in hours, s/c in GPU-seconds per call, tok in thousands of generated tokens per call ( Appendix E ).
Cov
Cov@k
Embedding
Attacker
class
class
tech.
dˉ
VS
AgentDojo, k=98
Static seeds
3
3.00
5.0
0.00
1.0
PIMiner
6
5.18
20.4
0.45
9.7
CtH
6
5.12
16.5
0.32
5.9
AutoDojo
5
5.00
18.9
0.26
3.4
Appendix
TABLE V: Circularity checks on Qwen3.6-35B at equal sample size k . Cov counts classes over all successes, Cov@k distinct classes and techniques among k , dˉ and VS are embedding diversity. Best per block in bold .
Fig. 8: Search strategies built from four blocks. L n stacks n sub-techniques per injection, and ×r counts exploits per failed line.
Large Language Models (LLMs) are rapidly evolving into agentic systems that interact with external tools and environments, introducing new security risks such as indirect prompt injection attacks through untrusted external sources. Existing defenses mainly focus on blocking malicious content at inference time, and current red-teaming methods primarily optimize attack success. As a result, developers have limited visibility into how latent prompt injections emerge and propagate through agents. We propose PI-Hunter, an automated agentic auditing framework for proactive vulnerability exposure in LLM agents. PI-Hunter constructs realistic source-aware test cases and iteratively evolves them through feedback-driven exploration to induce agents to retrieve and reveal latent malicious instructions embedded within external environments. Extensive experiments across multiple benchmarks, agent architectures, attacks, and defenses demonstrate that PI-Hunter substantially improves vulnerability exposure and attack-surface coverage over strong automated red-teaming baselines, while remaining effective under existing prompt injection defenses.
Pengfei He, Lesly Miculicich, Vishesh Sharma +5
Google Cloud AI Research · Google · Michigan State University
Prompt injection poses serious security risks to real-world LLM applications, particularly autonomous agents. Although many defenses have been proposed, their robustness against adaptive attacks remains insufficiently evaluated, potentially creating a false sense of security. In this work, we propose PISmith, a reinforcement learning (RL)-based red-teaming framework that systematically assesses existing prompt-injection defenses by training an attack LLM to optimize injected prompts in a practical black-box setting, where the attacker can only query the defended LLM and observe its outputs. We find that directly applying standard GRPO to attack strong defenses leads to sub-optimal performance due to extreme reward sparsity -- most generated injected prompts are blocked by the defense, causing the policy's entropy to collapse before discovering effective attack strategies, while the rare successes cannot be learned effectively. In response, we introduce adaptive entropy regularization and dynamic advantage weighting to sustain exploration and amplify learning from scarce successes. Extensive evaluation on 13 benchmarks demonstrates that state-of-the-art prompt injection defenses remain vulnerable to adaptive attacks. We also compare PISmith with 7 baselines across static, search-based, and RL-based attack categories, showing that PISmith consistently achieves the highest attack success rates. Furthermore, PISmith achieves strong performance in agentic settings on InjecAgent and AgentDojo against both open-source and closed-source LLMs (e.g., GPT-4o-mini and GPT-5-nano). Our code is available at https://github.com/albert-y1n/PISmith.
Large language model (LLM) agents routinely interact with untrusted external data, exposing them to indirect prompt injection (IPI) attacks that hijack agents into executing attacker-chosen tasks. To uncover these vulnerabilities at scale, automated red-teaming methods often leverage reinforcement learning (RL) to generate attacks that achieve high attack success rates (ASR). Model developers use the attacks discovered during internal red-teaming to harden agents through adversarial training before deployment, a collaborative offensive-defensive process known in cybersecurity as purple-teaming. However, we show that achieving high ASR during red-teaming does not guarantee robustness gains from subsequent adversarial training. Specifically, we find that two state-of-the-art RL-based red-teaming methods, RL-Hammer and PISmith, suffer from mode collapse, repeatedly discovering similar attack strategies whose additional successes add no additional robustness improvements to adversarial training. To address this gap, we introduce PIPETTE, a unified prompt injection purple-teaming framework that effectively converts red-teaming discoveries into defender robustness improvements. PIPETTE employs a dictionary learning-based reward that encourages broader coverage of attack strategies during red-teaming. It also integrates an on-policy adversarial training pipeline tailored to IPI defense, enabling agents to resist attacks while completing legitimate user tasks. Extensive evaluations across agentic benchmarks (AgentDojo and ASB) demonstrate that PIPETTE produces defenders that achieve near-zero attack success rates on static attacks and can withstand strong adaptive attacks, while maintaining benign task utility close to that of the undefended baseline.
Lipeng He, Yihan Wang, Jiawen Zhang +1
University of Waterloo · Zhejiang University · KTH Royal Institute of Technology