Indirect prompt injection (IPI) attacks, in which adversarial instructions are embedded in external data processed by a language model, pose a significant threat to the integrity of deployed LLM agents. Attention Tracker (Hung et al., 2025) is a recently proposed white-box detection method that identifies prompt injection attacks by monitoring how a model's attention is distributed between user task tokens and data tokens, claiming dataset-agnostic detection via a scalar focus score threshold. In this paper, we provide the first evaluation of Attention Tracker in a realistic agentic IPI setting using the AgentDojo benchmark, finding that the dataset agnosticism claim does not hold. We find that in agentic environments, long structured tool outputs subvert the fundamental assumptions Attention Tracker makes about attention that the method depends on and produces near-chance detection on AgentDojo across most model families. We propose AttnDA, an extension replacing the scalar threshold with a Fisher Linear Discriminant Analysis classifier over the per-head focus score vector, and evaluate both methods in a cross-dataset setting spanning four benchmarks. AttnDA improves detection performance across agentic and tool-integrated benchmarks, though transfer to AgentDojo remains challenging regardless of classifier choice. A covariance ablation further shows that removing inter-head correlation structure substantially contributes to performance, with complex covariance assumptions causing significant degradation on cross-domain data. These results indicate that the attentional distraction signal underlying focus-score-based detection does not transfer reliably to agentic environments, and that learned linear classifiers over head-level attention features offer improved detection performance, provided training data from the target deployment domain is available.
Figures & tables
Figure 1: Prompt length (characters) vs Focus score FS from Equation 3 for Llama3-8B-Instruct for the deepset and AgentDojo datasets.
Synthetic
Model
Dataset
Method
Llama
Qwen
Gemma
Phi
Mistral
Avg.
deepset
AT
0.92
0.98
0.99
0.97
0.99
0.97
AttnDA
0.91
0.97
0.97
0.92
0.96
0.95
AgentDojo †
AT
0.53
0.39
0.83
0.50
0.59
0.57
AttnDA
0.57
0.42
0.69
0.77
0.66
0.62
InjecAgent
AT
0.73
0.63
0.71
0.76
0.76
0.72
Table 2
In-Domain
Model
Dataset
Llama
Qwen
Gemma
Phi
Mistral
Avg.
deepset
AT
0.98
0.99
0.99
0.97
0.99
0.98
AttnDA
0.88
0.93
0.95
0.89
0.94
0.92
AgentDojo †
AT
0.62
0.44
0.53
0.78
0.71
0.62
AttnDA
0.97
0.88
0.95
0.95
0.95
0.94
InjecAgent
AT
0.99
0.98
0.98
0.90
0.90
0.95
Table 3
Cross-Domain
Eval.
Train
deepset
AgentDojo †
InjecAgent
LLMail-Inject
deepset
AT
0.98
0.55
0.82
0.87
FC
0.99
0.56
0.77
0.70
AttnDA
0.92
0.65
0.86
0.83
AgentDojo †
AT
0.85
0.62
0.73
0.82
FC
0.56
0.91
0.55
0.54
Table 4
Evaluation Dataset
Covariance Structure
AgentDojo †
deepset
LLMail-Inject
InjecAgent
Avg.
Full
0.9170
0.9400
0.5130
0.4110
0.6952
Block-layer
0.9830
0.5740
0.3100
0.2620
0.5323
Diagonal (AttnDA)
0.8830
0.8110
0.8580
0.8360
0.8470
Identity
0.8330
0.7560
0.3100
0.7380
0.6593
Table 5
deepset
AgentDojo †
InjecAgent
LLMail-Inject
Synthetic
deepset
0.98
0.17
0.53
0.65
0.53
AgentDojo †
0.80
0.09
0.15
0.06
InjecAgent
0.77
0.31
0.26
LLMail-Inject
0.99
0.58
Synthetic
0.96
Table 6
Figure 2: AttnDA weight vector w for Qwen2-1.5B fit on each dataset’s training split. The weight of a head has the sign of μmalicious−μbenign for its focus score: blue (negative) marks heads whose focus on the instruction decreases under the presences of an attack, as Attention Tracker assumes, and red (positive) marks heads whose focus score increases. Each panel is scaled by its own largest ∣w∣ . AgentDojo † mixes positive and negative weights, while deepset and LLMail are almost entirely negative.
Bootstrap of Test Set for Agent-Dojo Trained Models
Prompt injection detection identifies malicious instructions in prompts, retrieved content, webpages, images, or tool outputs before they hijack language models and agents.
Indirect prompt injection poses a critical threat to LLM agents that interact with untrusted external data, yet automated attack methods--proven effective for jailbreaking--remain underexplored in realistic agentic settings. We present a comprehensive empirical evaluation of automated prompt injection attacks against LLM agents, adapting both white-box (GCG) and black-box (TAP) methods to the agentic setting within the AgentDojo framework. We evaluate across 80 task pairs spanning four domains and multiple models, and find that black-box optimization substantially outperforms gradient-based methods, a gap we attribute to GCG's optimization instability under reasonable compute budgets. We also find that TAP's effectiveness depends on the attacker model, as both general capability and safety tuning affect attack success--stronger models produce more effective injections, while safety-tuned attackers can refuse to generate adversarial prompts. Task-universal attacks transfer effectively to unseen tasks and out-of-distribution domains, but attacks optimized on smaller open-source models do not transfer to frontier models like GPT-5. These findings highlight automated prompt injection as a credible but model-dependent threat, with significant barriers remaining for model-agnostic exploitation.
Large Language Model (LLM) agents are susceptible to Indirect Prompt Injection (IPI) attacks, where malicious instructions in retrieved content hijack the agent's execution. Existing defenses typically rely on strict filtering or refusal mechanisms, which suffer from a critical limitation: over-refusal, prematurely terminating valid agentic workflows. We propose ICON, a probing-to-mitigation framework that neutralizes attacks while preserving task continuity. Our key insight is that IPI attacks leave distinct over-focusing signatures in the latent space. We introduce a Latent Space Trace Prober to detect attacks based on high intensity scores. Subsequently, a Mitigating Rectifier performs surgical attention steering that selectively manipulate adversarial query key dependencies while amplifying task relevant elements to restore the LLM's functional trajectory. Extensive evaluations on multiple backbones show that ICON achieves a competitive 0.4% ASR, matching commercial grade detectors, while yielding a over 50% task utility gain. Furthermore, ICON demonstrates robust Out of Distribution(OOD) generalization and extends effectively to multi-modal agents, establishing a superior balance between security and efficiency.
Che Wang, Fuyao Zhang, Jiaming Zhang +6
School of Computer Science, Peking University, China · College of Computing and Data Science, Nanyang Technological University, Singapore · Ant Group +1
LLM agents increasingly screen tool outputs with small prompt-injection detectors, and teams choose among detectors by their scores on public benchmarks. We ask whether those scores predict how a detector behaves inside an agent. We replay the ground-truth tool calls of two agent benchmarks, AgentDojo and tau-bench, without an LLM to obtain tool outputs that are benign by construction, label injected outputs by differential replay, and evaluate fifteen detectors, including Meta's Prompt Guard 2, and two task-aware LLM judges on these outputs and on the BIPIA benchmark. Detection rankings transfer poorly between benchmarks: the best detector on BIPIA catches 2% of AgentDojo injections at a 1% false-positive rate, and a detector that catches 72% of AgentDojo injections catches 15% on tau-bench. False-positive rates on tool outputs, which range from none to over 90%, do transfer between the two agent benchmarks. Where training data is public, the form of the training inputs explains the results. The BIPIA leader was trained on full BIPIA inputs, but having seen InjecAgent's attack strings as short prompts does not help it find them inside tool outputs; the best detector on both agent benchmarks shares no data with any benchmark and was trained on agent-style inputs. Evaluations meant to inform deployment should use the agent's own tool outputs, report detection at a low false-positive rate, and audit what the detector was trained on.
Zhuowen Liu
Cybersecurity Lab Japan Advanced Institute of Science and Technology (JAIST) Nomi, Ishikawa, Japan