cs.CRJun 11, 2025
SaveLLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge
Organizations: Microsoft · ISTA · Trend Micro · RainaResearch · University of Coimbra · Vietnamese German University · SK Shieldus · HiddenLayer
Abstract
Indirect prompt injection attacks exploit a fundamental weakness of LLMs: executing unauthorized tasks from untrusted sources. Despite the practical significance of this attack vector, existing benchmarks rely on small sets of template-based attacks (100-1,000 examples) that fail to capture the diversity and adaptiveness of real-world adversaries. Consequently, benchmarks are saturated and overfit to static attacks. We introduce LLMail-Inject, a large-scale benchmark of 208,095 distinct human-written attack prompts collected through a public red-teaming challenge with 839 participants. LLMail-Inject evaluates the attack's complete chain of execution: prompts must be retrieved from an email database, evade production-level defenses, and trigger unauthorized tool calls with correct arguments. Evaluation reveals that models that achieve 5% attack success rate (ASR) on existing benchmarks fail more on LLMail-Inject: Meta SecAlign--specifically trained against prompt injection--still yields 32% ASR, GPT-4o exhibits 58% ASR, and GPT-5 mini 47% ASR. We further demonstrate that attacks transfer across domains, achieving 37% ASR on proof-of-concept coding tasks and 28-45% ASR on file management tasks. We release our challenge platform and curated evaluation subsets at multiple difficulty tiers and the full dataset with rich annotations.
Figures & tables
Figure 1: The challenge simulated an LLM-based email assistant. Participants submitted prompt injection in an email. To succeed, the attacker’s email had to be retrieved by the assistant, evade the prompt injection defenses, and correctly call a tool. Note: The email shown is an actual successful submission.
Figure 2: Over all submissions, “Tool Call” denotes submissions where the send_email tool was correctly called while “E2E Attack Success” denotes end-to-end successful attacks (i.e., the send_email tool was correctly called and the defense was evaded). The figure compares Prompt Shield (the text-based classifier), TaskTracker, and LLM Judge.
Figure 3: Spotlight vs. others with Figure 2 metrics.
Figure 4: Figure 2 metrics per level and LLM
Figure 5: Left: out of the 371k submissions for phase 1, the attack email was retrieved in 362k cases. The attacks were undetected in 180k cases, and the send_email tool was called in 7.2k cases (denoted by the exfil flag). Among these, the destination was correct in 6.1k cases, and the content was correct in 3k cases, representing less than 1% of total submissions. Right: out of all the 183k detected attacks, the send_email tool was called in 34k cases. Of these, the destination was correct in 28.6k cases, and the content was correct in 21.4k cases. See Appendix N for multi-stage diagrams for each LLM and level.
| Defense | TSR |
|---|---|
| Text-based classifier | 0.600 |
| TaskTracker | 0.440 |
| Spotlight | 0.529 |
| LLM Judge | 0.318 |
| All | 0.166 |
Table 1: TSR for level 4 GPT-4o-mini .
Figure 6: (a) Team Success Rate (TSR) per LLM, averaged over sub-levels. (b) TSR per retrieval level, averaged over all defenses. (c) TSR for different detection defenses, averaged over all GPT-4 sub-levels. (c) also shows post-competition analysis of recall ratio of the different defenses over a subset of the collected phase 1 dataset.
| Level | Classifier | TaskTracker | LLM Judge |
|---|---|---|---|
| Level 1 | 156.6 668.6 | 224.4 790.4 | 677.1 1140.2 |
| Level 2 | 48.89 213.6 | 129.1 339.2 | 224.9 405.09 |
| Level 3 | 46.40 98.51 | 86.33 185.3 | 517.5 957.24 |
| Level 4 | 55.37 74.08 | 165.2 310.0 | 535.4 959.63 |
| Average | 88.90 426.2 | 159.3 518.5 | 506.2 948.46 |
Table 2: Average number of trials before first success per team for the three detection defenses in phase 1.
| Defense | Phase 1 | Phase 2 |
|---|---|---|
| Text-based classifier v1 | 0.604 | 0.425 |
| Text-based classifier v2 | 0.678 | 0.487 |
| TaskTracker Phi-3 14B | 0.761 | 0.896 |
| TaskTracker Phi-3.5 MoE | 0.949 | 0.991 |
| TaskTracker Phi-4 14B | 0.881 | 0.998 |
| LLM Judge | 0.994 | 0.965 |
Table 3: Detection rates (i.e., recall) on all attack submissions that called the send_email tool in phase 1 (25,323 attacks) and phase 2 (3,688 attacks).
| Text-based classifier v1 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ||||
| TaskTracker Phi-3 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ||||
| TaskTracker Phi-4 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ||||
| LLM Judge | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ||||
| Recall | 0.890 | 0.924 | 0.927 | 0.956 | 0.996 | 0.996 | 0.997 | 0.997 | 0.997 | 0.998 | 0.998 |
Table 4: Detection rates on all attack submissions that called the send_email tool in phase 1 for combinations of the text-based classifier, TaskTracker Phi-3, TaskTracker Phi-4, and LLM Judge.
| Model | ASR (%) | Notes |
|---|---|---|
| GPT-4o | 58.0 | Larger than GPT-4o-mini |
| GPT-5 mini | 47.0 | More recent |
| GPT-5 | 22.8 | High-reasoning |
| Meta SecAlign 70B | 32.0 | DPO-trained defense |
Table 5: Attack success rates on models not included in the original competition.
| Transfer Method | Target Domain | ASR (%) |
| Baseline (original task) | ||
| None | Email assistant | 58.0 |
| Scaffolding change | ||
| New system prompt | Coding agent | 37.0 |
| Attack paraphrasing | ||
| LLM paraphrase | File upload task | 28.0 |
Table 6: Attack success rates when transferring LLMail-Inject attacks to new domains on GPT-4o .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Indirect | End-to-end | Adaptive | Tests defenses | Public release | # Attacks |
|---|---|---|---|---|---|---|
| HackAPrompt ( Schulhoff et al., 2023 ) | ✗ | ✗ | ✓ | ✗ | ✓ | 600K+ |
| SaTML LLM CTF ( Debenedetti et al., 2024a ) | ✗ | ✗ | ✓ | ✓ | ✓ | 137K |
| Gandalf ( Pfister et al., 2025 ) | ✗ | ✗ | ✓ | ✓ | ✓ | 279K |
| InjecAgent ( Zhan et al., 2024 ) | ✓ | ✗ | ✗ | ✗ | ✓ | 1,054 |
| BIPIA ( Yi et al., 2025 ) | ✓ | ✗ | ✗ | ✓ | ✓ | 35K |
| SEP ( Zverev et al., 2025 ) | ✓ | ✗ | ✗ | ✗ | ✓ | 9,160 |
Table 7: Comparison to related prompt-injection benchmarks and competitions. LLMail-Inject is the only fully open-sourced indirect prompt injection benchmark that evaluates end-to-end attacks (retrieval defense evasion tool call with correct arguments), at large scale, under an adaptive two-phase setup where defenses are hardened between phases in response to observed failure modes. Template-based benchmarks (InjecAgent, BIPIA, SEP, AgentDojo, WASP) report total test cases obtained by combining a small set of attack templates with multiple scenarios, contexts, and injection positions; the number of unique underlying attack templates is typically one to two orders of magnitude smaller (e.g., 50 for BIPIA, 2 for WASP).
| Defense | FPR (%) |
|---|---|
| Text-based classifier | 0.0 |
| TaskTracker (Phi-3 medium) | 4.0 |
| TaskTracker (Phi-4) | 5.7 |
| LLM judge | 5.0 |
| All (any flags) | 10.0 |
Table 8: False positive rates on a 1,000-email subset of the Enron Email Dataset, using the same thresholds and prompts as in the challenge.
| Axis | Released fields or mapping | What can be recovered |
|---|---|---|
| Strategy | attack_attempt , judge_category , judge_answer | Attack-attempt status, a coarse category (direct instruction, obfuscation, or social engineering), and the annotator’s free-text rationale. |
| Target objective and outcome | email.retrieved , defense.undetected , exfil.sent , exfil.destination , exfil.content | Which stage of the end-to-end objective was reached, including tool invocation and argument correctness. |
| Defense and model | scenario joined with levels_descriptions.json | Target LLM, retrieval level, and individual or stacked defense configuration. |
| Difficulty | Retrieval level, phase, objective flags, and curated subset membership | Designed difficulty (Levels 1–4), whether the prompt targeted hardened Phase 2 defenses, and empirical tiers such as tool-triggering or defense-evasive. |
| Provenance | team_id , timestamps, and phase | Team-level or temporal splits that reduce near-duplicate and adaptation leakage. |
Table 9: Released metadata and annotation axes.
Figure 7: Participants were able to send jobs via the website by selecting which sub-level to submit to and including the subject and body of their attack email.
Figure 8: Participants were able to receive feedback about which objectives were achieved.
Figure 9: There was a leaderboard that shows the number of solves per sub-levels and was updated online each time a team solves a sub-level.
Figure 10: To engage participants, the leaderboard also showed the order of teams and was automatically updated according to solves (team names and their Github usernames are anonymized).
Figure 11: Multi-stage outcome diagram for Level 1 (all submissions). Left is Phi-3, right is GPT-4o-mini.
Figure 12: Multi-stage outcome diagram for Level 2 (all submissions). Left is Phi-3, right is GPT-4o-mini.
Figure 13: Multi-stage outcome diagram for Level 3 (all submissions). Left is Phi-3, right is GPT-4o-mini.
Figure 14: Multi-stage outcome diagram for Level 4 (all submissions). Left is Phi-3, right is GPT-4o-mini.
Figure 15: Multi-stage outcome diagram for Level 1 (detected submissions). Left is Phi-3, right is GPT-4o-mini.
Figure 16: Multi-stage outcome diagram for Level 2 (detected submissions). Left is Phi-3, right is GPT-4o-mini.
Figure 17: Multi-stage outcome diagram for Level 3 (detected submissions). Left is Phi-3, right is GPT-4o-mini.
Figure 18: Multi-stage outcome diagram for Level 4 (detected submissions). Left is Phi-3, right is GPT-4o-mini.