Large language model email assistants are particularly vulnerable to indirect prompt injection because untrusted email content can be retrieved into the model context and influence subsequent tool use. Existing prompt injection detectors mainly formulate this problem as binary malicious text classification, which overlooks the important factor that harmful agent behavior often arises through a sequence of stages. We propose a detection framework that models this attack chain by combining a text detector, verifiers specific to each stage, explicit rule-based risk signals, user intent and action consistency analysis, and a logistic decision policy. To support this framework, we derive attack chain labels from prompt injection datasets, evaluate the proposed framework under random splits, temporal phase transfer, conditional stage transfer, cross-dataset transfer, and conduct ablation studies on multiple benchmarks. Results show that random train test splits substantially overestimate robustness under distribution shift, while later tool argument stages are more predictable than earlier stages in the framework. We also show that training on harmless emails that resemble attacks helps reduce false alarms while preserving the ability to detect real attacks. Across five binary benchmarks, our framework achieves a mean F1 score of 0.406 under the strict threshold setting policy, compared with 0.216 for the strongest of five pretrained detectors evaluated without additional training. These results highlight the value of combining attack stage predictions with checks for conflicts between the user's request and instructions in retrieved emails. Our experiments also demonstrate the importance of training with challenging benign examples to balance attack detection and false alarms.
Figures & tables
Phase
Rows
Retrieved
Bypassed
Tool
Full success
Phase 1
370,724
97.74%
50.62%
11.11%
0.814%
Phase 2
90,916
99.88%
48.50%
6.31%
0.337%
TABLE I: LLMail-Inject Phase Summary
Dataset
Rows
Attack
Benign
BIPIA generated binary
176,200
175,000
1,200
NotInject
339
0
339
NVIDIA Agentic IPI
1,272
1,272
0
PromptShield
43,425
16,441
26,984
ShieldLM
54,162
18,965
35,197
Neuralchemy
6,274
3,736
2,538
TABLE II: BIPIA, NotInject, NVIDIA Agentic IPI, PromptShield, ShieldLM, and Neuralchemy Data
Fig. 1: Proposed framework workflow. The detector first converts email agent inputs into broad text, stage verifier, rule, and action and context signals, then uses a meta policy to select a review or strict decision.
Fig. 2: Outcomes across the full LLMail-Inject attack chain. Almost half of submissions are detected after retrieval, and full success is below 1% of all submissions.
Target
Random F1
Random AUC
Phase F1
Oracle F1
Full success
0.167
0.825
0.000
0.024
Defense bypass
0.821
0.906
0.457
0.654
Tool called
0.527
0.892
0.077
0.137
TABLE III: Random Split Versus Phase 1 to Phase 2 Transfer
Transition
Random F1
Phase F1
Phase AUC
Retrieved → bypassed
0.814
0.413
0.455
Bypassed → tool call
0.394
0.053
0.704
Tool call → destination
0.969
0.939
0.625
Tool call → content
0.874
0.814
0.668
Tool call → full success
0.475
0.122
0.593
TABLE IV: Conditional Stage-Transition Results
Train
Test
F1 / FPR
AUC
LLMail
BIPIA
0.453
0.572
BIPIA
BIPIA
0.738
0.904
BIPIA
LLMail
0.747
0.780
LLMail+BIPIA
BIPIA
0.770
0.899
LLMail+BIPIA
NotInject
65.2% FPR
–
LLMail+BIPIA+NotInject
NotInject
0.6% FPR
–
TABLE V: Cross-Dataset and Hard Negative Results
Variant
Policy
Mean F1
NotInject FPR
NVIDIA Rec.
BIPIA F1
Selective stages
review
0.643±0.002
27.73%
100.0%
0.745
All stages
review
0.643±0.003
27.14%
100.0%
0.744
Text/rules only
strict
0.281±0.149
0.88%
4.01%
0.182
All stages
strict
0.348±0.096
1.77%
12.40%
0.183
Action/context only
strict
0.386±0.024
0.88%
2.33%
0.735
Selective stages + action/context
strict
0.400±0.057
2.36%
2.33%
0.733
TABLE VI: Framework Component Ablation
Method
Mean F1
NotInject FP / 339
NVIDIA Rec.
BIPIA F1
ProtectAI v2
0.216
13 (3.83%)
0.00%
0.000
TestSavant tiny
0.130
8 (2.36%)
0.00%
0.000
ProtectAI v1
0.114
3 (0.88%)
0.00%
0.000
Arkaean DistilBERT
0.094
5 (1.47%)
0.00%
0.000
deepset DeBERTa
0.014
6 (1.77%)
0.00%
0.013
Proposed, all-stage action/context
0.406
7 (2.06%)
2.33%
0.732
TABLE VII: Matched Strict Transfer Comparison
Method
NotInject FPR
NVIDIA Rec.
BIPIA F1
LLMail F1
Framework, review a
27.73%
100.00%
0.745
0.665
Framework, strict b
2.06%
2.33%
0.732
0.665
OpenAI judge, default
1.47%
75.16%
0.630
0.919
OpenAI judge, strict
1.77%
75.16%
0.625
0.891
OpenAI judge, ultra-strict
1.18%
73.45%
0.565
0.810
TABLE VIII: Matched Comparison with a Zero-Shot OpenAI LLM Judge
Linear probes on a model's hidden states are used as cheap monitors for indirect prompt injection: they screen the tool outputs an agent reads and send suspicious ones to a slower second check, so every false alarm has a cost. But a probe learns whatever separates its training examples. The standard evaluation of injection probes can be passed without the model: conversation position and text lengths alone reach 0.91-0.98 AUROC. When each attack is compared with the same conversation without the injection, or with harmless text formatted like an injection, standard probes flag most of these harmless cases. We add such cases to the training data as matched negatives. Across six open models, in applications seen during training and at the default threshold, changing only the training data cuts false alarms on injection-free conversations from 41-80% to 4-27%, and on harmless look-alikes from 92-98% to 3-16%. With a threshold set on normal traffic, our probe still catches a median 95% of the attacks that fooled the agent (91% without our data). In new applications, false alarms stay low, but detection depends on the model and on how the threshold is set. The same data also helps a text-only filter.
Multiple prompt injection attacks have been proposed against web agents. At the same time, various methods have been developed to detect general prompt injection attacks, but none have been systematically evaluated for web agents. In this work, we bridge this gap by presenting the first comprehensive benchmark study on detecting prompt injection attacks targeting web agents. We begin by introducing a fine-grained categorization of such attacks based on the threat model. We then construct datasets containing both malicious and benign samples: malicious text segments generated by different attacks, benign text segments from four categories, malicious images produced by attacks, and benign images from two categories. Next, we systematize both text-based and image-based detection methods. Finally, we evaluate their performance across multiple scenarios. Our key findings show that while some detectors can identify attacks that rely on explicit textual instructions or visible image perturbations with moderate to high accuracy, they largely fail against attacks that omit explicit instructions or employ imperceptible perturbations. Our datasets and code are released at: https://github.com/Norrrrrrr-lyn/WAInjectBench.
Indirect prompt injection attacks exploit a fundamental weakness of LLMs: executing unauthorized tasks from untrusted sources. Despite the practical significance of this attack vector, existing benchmarks rely on small sets of template-based attacks (100-1,000 examples) that fail to capture the diversity and adaptiveness of real-world adversaries. Consequently, benchmarks are saturated and overfit to static attacks. We introduce LLMail-Inject, a large-scale benchmark of 208,095 distinct human-written attack prompts collected through a public red-teaming challenge with 839 participants. LLMail-Inject evaluates the attack's complete chain of execution: prompts must be retrieved from an email database, evade production-level defenses, and trigger unauthorized tool calls with correct arguments. Evaluation reveals that models that achieve 5% attack success rate (ASR) on existing benchmarks fail more on LLMail-Inject: Meta SecAlign--specifically trained against prompt injection--still yields 32% ASR, GPT-4o exhibits 58% ASR, and GPT-5 mini 47% ASR. We further demonstrate that attacks transfer across domains, achieving 37% ASR on proof-of-concept coding tasks and 28-45% ASR on file management tasks. We release our challenge platform and curated evaluation subsets at multiple difficulty tiers and the full dataset with rich annotations.