Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents

Zhuowen Liu

cs.CROct 2, 2026

Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents

Authors: Zhuowen Liu

Organizations: Cybersecurity Lab Japan Advanced Institute of Science and Technology (JAIST) Nomi, Ishikawa, Japan

Abstract

LLM agents increasingly screen tool outputs with small prompt-injection detectors, and teams choose among detectors by their scores on public benchmarks. We ask whether those scores predict how a detector behaves inside an agent. We replay the ground-truth tool calls of two agent benchmarks, AgentDojo and tau-bench, without an LLM to obtain tool outputs that are benign by construction, label injected outputs by differential replay, and evaluate fifteen detectors, including Meta's Prompt Guard 2, and two task-aware LLM judges on these outputs and on the BIPIA benchmark. Detection rankings transfer poorly between benchmarks: the best detector on BIPIA catches 2% of AgentDojo injections at a 1% false-positive rate, and a detector that catches 72% of AgentDojo injections catches 15% on tau-bench. False-positive rates on tool outputs, which range from none to over 90%, do transfer between the two agent benchmarks. Where training data is public, the form of the training inputs explains the results. The BIPIA leader was trained on full BIPIA inputs, but having seen InjecAgent's attack strings as short prompts does not help it find them inside tool outputs; the best detector on both agent benchmarks shares no data with any benchmark and was trained on agent-style inputs. Evaluations meant to inform deployment should use the agent's own tool outputs, report detection at a low false-positive rate, and audit what the detector was trained on.

Figures & tables

Panel of Figure 1 — Figure 1: Running example. A detector between the tools and the agent blocks a harmless confirmation message (top) and, once its threshold is set to keep false alarms rare, passes almost every injected output (bottom). Both outputs come from AgentDojo; the scores are measured (§ 5.1 , § 5.2 ).

Set	Benign	Injected	Content
General text	7,504	–	README, email, web
AD-Clean	339	–	tool outputs, no injection
AD-Matched	585	432	tool outputs, 3 attacks
AD-Shift	975	720	tool outputs, 5 other attacks
TB-Matched	1,533	584	$\tau$ -bench tool outputs (JSON)
BIPIA	1,200	2,435	email, table, code

Table 1: Evaluation sets. Benign / injected counts. AD = AgentDojo v1.2; TB =

\tau

-bench. The benign side of TB-Matched is TB-Clean.

				General	AgentDojo				$\tau$ -bench			BIPIA
			Agent	FPR	FPR	Tasks	TPR @ 1% FPR		FPR	Tasks	TPR	TPR
Detector	Year	Size	data	(%)	(%)	(%)	Matched	Shift	(%)	(%)	@ 1%	@ 1%
JasperLS	2023	184M	no	99.9	78.5	97.9	23.4	22.1	93.1	99.6	18.7	4.9
ProtectAI v1	2023	184M	no	0.3	3.5	12.4	0.7	7.5	5.1	6.8	23.3	0.4
ProtectAI v2	2024	184M	no	3.8	30.1	69.1	0.5	0.8	66.9	80.3	10.1	0.7
ProtectAI v2-small	2024	142M	no	1.6	12.1	34.0	7.6	3.1	1.4	2.8	30.1	1.4

Table 2: All detectors on all sets. General: README, email, and web text. FPR at each detector’s default threshold on benign tool outputs (AD-Clean, TB-Clean); “tasks” is the share of tasks with at least one flagged benign output. TPR at the threshold that gives 1% FPR on the same set. BIPIA uses test-half attacks. “Agent data”: the model card states training on agent or tool-output inputs; “eval.” means the card reports evaluation on AgentDojo-derived data. † Undocumented use of Sheltron’s user-goal segment. Bootstrap intervals for four methods are in Appendix Table 4 .

Figure 2: Each point is a detector or judge (diamonds). (a) Detection on BIPIA does not predict detection on AgentDojo. (b) Detection on one agent benchmark predicts detection on the other only partly: Prismor (PR) and Sheltron (SH) trade places. Dashed line: equal performance. Key: HL Horizon-Labs, WD Wolf Defender, PR Prismor, P2/P2s Prompt Guard 2 86M/22M, P1 Prompt Guard 1, PG PIGuard, SH Sheltron, QS Qualifire Sentinel, S2 Sentinel v2, TS TestSavant, JS JasperLS, A1/A2/A2s ProtectAI v1/v2/v2-small, JQ/JL Qwen/Llama judge.

BIPIA component	Found in training split
Text attacks, training half	65 of 75
Code attacks, training half	6 of 50
Text attacks, test half	0 of 75
Code attacks, test half	0 of 50
Clean email contexts	31%
Clean code contexts	22%

Table 3: Verbatim overlap between BIPIA and PIGuard’s published training split (whitespace and case normalized). A lower bound: paraphrases are not counted.

Figure 3: Detection by BIPIA attack group (test half), each method at its own 1%-FPR threshold on BIPIA. Only PIGuard, which was trained on BIPIA inputs, covers all groups; every other approach shown misses most task-irrelevant injections.

Figure 4: Format alone moves false positives. Left: removing markup from benign READMEs. Right: removing YAML keys from AgentDojo tool outputs (circle: as-is; square: keys removed) lowers PIGuard’s FPR threefold at unchanged TPR.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Detector	Hugging Face model	Ctx	Input
ProtectAI v1	protectai/…-injection	512	text
ProtectAI v2	protectai/…-injection-v2	512	text
JasperLS	JasperLS/deberta-v3-base-injection	512	text
TestSavant-L	testsavantai/…-large-v0	512	text
PIGuard	leolee99/PIGuard	512	text
Sheltron	sheltron-ai/prompt-guard-68m	1024	role/source tokens

Appendix

Table 8

Set	Method	AUROC	TPR @ 1% FPR (%)	FPR @ 0.5 (%)	TPR @ 0.5 (%)
AD-Matched	ProtectAI v2	0.82	0.5 [0.0–1.2]	20.0	58.6
	PIGuard	0.83	2.1 [0.0–4.4]	20.0	73.1
	Judge (Qwen2.5-7B)	0.90	57.4 [52.5–62.7]	6.7	77.5
	Judge (Llama-3.1-8B)	0.93	55.3 [26.6–75.2]	11.8	87.0
AD-Shift	ProtectAI v2	0.84	0.8 [0.3–1.5]	20.0	65.7
	PIGuard	0.84	7.9 [3.1–12.4]	20.0	80.7

Appendix

Table 4: Detection on the three matched sets for four methods. Brackets: 95% bootstrap interval. BIPIA uses only test-half attacks.

Figure 5: ROC curves with logarithmic FPR axis for the four methods of Table 4 ; the dashed line marks 1% FPR. The Llama judge’s wide interval on AD-Matched comes from a few benign outputs with high scores that make the 1% threshold unstable.

Explore similar work

AgentShield: Deception-based Compromise Detection for Tool-using LLM Agents
May 10, 2026Yassin H. Rassul, Tarik A. RashidLLM Agent Security AI Agent Monitoring
Computer Science and Engineering Department, School of Science and Engineering, University of Kurdistan Hewlêr, Erbil, Iraq · Artificial Intelligence and Innovation Centre, University of Kurdistan Hewlêr, Erbil, Iraq
Defusing Explosive Prompts: Understanding and Preventing Trigger-Based Prompt Injections in LLM Agents
Sep 18, 2026Justin Szczepaniak, Elad Feldman, Naum Viner +1Prompt Injection Attacks Prompt Injection Attacks on AI Agents
Tel Aviv University
AgentRedBench: Dynamic Redteaming and Integration-Aware Defense for LLM Agents over SaaS Integrations
Jun 1, 2026Hiskias Dingeto, William LeeneyLLM Red Teaming LLM Agent Security
StackOne Technologies

May 10, 2026J/K move · Enter open · S save

AgentShield: Deception-based Compromise Detection for Tool-using LLM Agents

Yassin H. Rassul, Tarik A. Rashid

Computer Science and Engineering Department, School of Science and Engineering, University of Kurdistan Hewlêr, Erbil, Iraq · Artificial Intelligence and Innovation Centre, University of Kurdistan Hewlêr, Erbil, Iraq

LLM Agent Security AI Agent Monitoring Prompt Injection Attacks on LLMs Prompt Injection Attacks on AI Agents Prompt Injection Detection

Open paper AskPDF arXiv

Defenses against indirect prompt injection (IPI) in tool-using LLM agents share two structural weaknesses. First, they all attempt to prevent attacks rather than detect the compromises that slip through. Second, they have only been evaluated in English, leaving users of low-resource languages such as Kurdish and Arabic without tested protection. This paper addresses both gaps with AgentShield, a deception-based detection framework that places three layers of traps inside the agent's tool interface: fake tools, fake credentials, and allowlisted parameters. The same trap triggers serve as high-precision labels for a self-supervised classifier. An LLM agent that follows an attacker's hidden instruction almost always touches one of these traps, which gives both a real-time compromise signal and a zero-FP label for training a downstream detector without manual annotation. Across 176 cross-lingual attack prompts and four LLMs from three providers, and because modern LLMs already refuse most IPI attempts on their own (attack success rate <= 10%), AgentShield's job is to catch the attacks that do slip through. On commercial models, it catches 90.7%-100% of such successful attacks, with zero false alarms on 485 normal-use tests. It survives a systematic adaptive-attack evaluation with zero evasion on commercial models, and the self-supervised classifier transfers across models and languages without retraining.

Abstract

Explore similar work

AgentShield: Deception-based Compromise Detection for Tool-using LLM Agents

Defusing Explosive Prompts: Understanding and Preventing Trigger-Based Prompt Injections in LLM Agents

AgentRedBench: Dynamic Redteaming and Integration-Aware Defense for LLM Agents over SaaS Integrations