PIPETTE: Prompt Injection Purple-Teaming for Robust LLM Agents
Organizations: University of Waterloo · Zhejiang University · KTH Royal Institute of Technology
Abstract
Large language model (LLM) agents routinely interact with untrusted external data, exposing them to indirect prompt injection (IPI) attacks that hijack agents into executing attacker-chosen tasks. To uncover these vulnerabilities at scale, automated red-teaming methods often leverage reinforcement learning (RL) to generate attacks that achieve high attack success rates (ASR). Model developers use the attacks discovered during internal red-teaming to harden agents through adversarial training before deployment, a collaborative offensive-defensive process known in cybersecurity as purple-teaming. However, we show that achieving high ASR during red-teaming does not guarantee robustness gains from subsequent adversarial training. Specifically, we find that two state-of-the-art RL-based red-teaming methods, RL-Hammer and PISmith, suffer from mode collapse, repeatedly discovering similar attack strategies whose additional successes add no additional robustness improvements to adversarial training. To address this gap, we introduce PIPETTE, a unified prompt injection purple-teaming framework that effectively converts red-teaming discoveries into defender robustness improvements. PIPETTE employs a dictionary learning-based reward that encourages broader coverage of attack strategies during red-teaming. It also integrates an on-policy adversarial training pipeline tailored to IPI defense, enabling agents to resist attacks while completing legitimate user tasks. Extensive evaluations across agentic benchmarks (AgentDojo and ASB) demonstrate that PIPETTE produces defenders that achieve near-zero attack success rates on static attacks and can withstand strong adaptive attacks, while maintaining benign task utility close to that of the undefended baseline.
Figures & tables
| Use | Attack | Type | # Steps | # Pairs |
| Red-Teaming | RL-Hammer [ 11 ] | Offline | 250 | 560 |
| PISmith [ 12 ] | Offline | 250 | 560 | |
| Evaluation | ChatInject [ 33 ] | Static | 1 | 40 |
| TAP [ 34 ] | Iterative-adaptive | 250 | 40 | |
| Genetic [ 10 ] | Iterative-adaptive | 248 | 40 |
| Red-teamer | Model | ASR | # Strings |
| RL-Hammer [ 11 ] | Qwen3-4B | 33.39% | 4,282 |
| Llama-8B | 25.18% | 1,671 | |
| PISmith [ 12 ] | Qwen3-4B | 68.75% | 7,132 |
| Llama-8B | 27.32% | 1,783 |
| Model | Dataset | Base | RL | SFT |
| Qwen3-4B | RL-Hammer | 61.86% | 54.64% | 49.48% |
| PISmith | 61.86% | 60.82% | 53.61% | |
| Llama-8B | RL-Hammer | 40.21% | 29.90% | 15.46% |
| PISmith | 40.21% | 25.77% | 16.49% |
| Model | Base | RL | SFT | Pipette -Blue |
| Qwen3-4B | 61.86% | 59.79% | 58.76% | 64.95% |
| Llama-8B | 40.21% | 35.05% | 29.90% | 41.24% |
| Attack | TAP | Genetic | ||
| Defense | ASR | UA | ASR | UA |
| Llama-3.1-8B-Instruct | ||||
| None | 7.50% | 12.50% | 17.50% | 32.50% |
| Meta-SecAlign | 2.50% | 15.00% | 12.50% | 17.50% |
| Pipette | 0.00% | 25.00% | 2.50% | 30.00% |
| Qwen3.6-27B | ||||
| Attack | IgnorePrev | InjecAgent | ImportantMsgs | ToolKnow | ChatInject | Average | None | ||||||
| Defense | ASR | UA | ASR | UA | ASR | UA | ASR | UA | ASR | UA | ASR | UA | BU |
| Qwen3-4B-Instruct-2507 | |||||||||||||
| None | 25.24% | 45.95% | 25.71% | 45.95% | 27.38% | 50.48% | 41.90% | 48.10% | 46.19% | 31.90% | 33.29% | 44.48% | 61.86% |
| Spotlighting | 17.38% | 46.43% | 18.10% | 47.86% | 29.05% | 49.29% | 35.48% | 44.76% | 45.71% | 28.33% | 29.14% | 43.33% | 57.73% |
| Sandwiching | 10.95% | 48.33% | 13.10% | 52.14% | 18.57% | 50.71% | 25.48% | 47.14% | 17.38% | 47.38% | 17.10% | 49.14% | 60.82% |
| PromptGuard | 0.00% | 19.52% | 0.00% | 17.86% | 20.00% | 37.62% | 31.19% | 36.43% | 0.24% | 16.19% | 10.29% | 25.52% | 63.92% |
| Configuration | ASR | UA | BU |
| Base (None) | 46.19% | 31.90% | 61.86% |
| Pipette -Blue : | |||
| w/o reasoning, w/o | 3.57% | 45.48% | 59.79% |
| w/ reasoning, w/o | 2.38% | 56.90% | 60.82% |
| w/ reasoning, w/ ( Pipette ) | 1.43% | 58.10% | 64.95% |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Task Suite | Slack | Banking | Workspace | Travel | ||||
| Defense | TK | TT | TK | TT | TK | TT | TK | TT |
| None | 12955 | 3.50 | 14042 | 4.00 | 18378 | 4.40 | 41728 | 7.70 |
| Spotlighting | 13476 | 5.00 | 13199 | 4.50 | 17874 | 4.70 | 37503 | 8.50 |
| Sandwiching | 11934 | 3.90 | 11523 | 4.40 | 18400 | 5.20 | 44337 | 9.10 |
| PromptGuard | 11887 | 4.20 | 12399 | 4.80 | 18082 | 5.10 | 41370 | 8.80 |
| PIGuard | 11708 | 4.80 | 12367 | 5.00 | 22010 | 5.60 | 42144 | 9.20 |
| Attack | IgnorePrev | InjecAgent | ImportantMsgs | ToolKnow | ChatInject | Average | None | ||||||
| Defense | ASR | UA | ASR | UA | ASR | UA | ASR | UA | ASR | UA | ASR | UA | BU |
| Llama-3.1-8B-Instruct | |||||||||||||
| None | 7.62% | 37.14% | 7.14% | 37.14% | 9.29% | 37.86% | 12.86% | 36.43% | 30.48% | 16.67% | 13.48% | 33.05% | 40.21% |
| Meta-SecAlign | 0.71% | 36.67% | 0.00% | 33.33% | 0.71% | 29.76% | 1.19% | 32.62% | 1.43% | 30.48% | 0.81% | 32.57% | 29.90% |
| Pipette | 0.00% | 39.29% | 0.00% | 39.05% | 0.00% | 39.76% | 0.00% | 38.81% | 2.14% | 38.33% | 0.43% | 39.05% | 41.24% |
| Qwen3.6-27B | |||||||||||||
| Attack | Naive | Escape | Ignore | Completion | Combined | Average | ||||||
| Defense | ASR | UA | ASR | UA | ASR | UA | ASR | UA | ASR | UA | ASR | UA |
| Llama-3.1-8B-Instruct | ||||||||||||
| None | 4.25% | 61.75% | 3.75% | 58.75% | 6.75% | 42.25% | 9.50% | 46.75% | 7.00% | 31.75% | 6.25% | 48.25% |
| Meta-SecAlign | 0.50% | 52.25% | 0.75% | 49.50% | 1.00% | 49.00% | 0.75% | 45.00% | 0.75% | 45.75% | 0.75% | 48.30% |
| Pipette | 0.25% | 65.25% | 0.50% | 61.50% | 0.75% | 45.25% | 0.25% | 52.50% | 0.75% | 34.25% | 0.50% | 51.75% |
| Qwen3.6-27B | ||||||||||||
| Suite | Task pairs |
| Banking | 7/0, 5/1, 11/1, 13/1, 7/2, 13/2, 13/3, 8/4, 13/4, 13/5 |
| Slack | 0/1, 1/1, 2/1, 3/1, 4/1, 5/1, 8/1, 11/1, 13/1, 15/1 |
| Travel | 6/0, 6/2, 6/5, 0/6, 2/6, 4/6, 6/6, 8/6, 9/6, 12/6 |
| Workspace | 9/0, 1/1, 2/2, 35/2, 10/3, 20/4, 9/10, 21/10, 26/10, 39/10 |
| Symbol | Description | Value |
| Pipette -Red | ||
| RL steps | ||
| group size | ||
| task pairs per step | ||
| learning rate | ||
| reward weights (Eq. 12 ) | , , | |
Prompt Injection Attacks on AI Agents
Prompt injection attacks on AI agents exploit untrusted content to override instructions and misuse tools, data, memory, or delegated workflows.
See all 221 papers