MetaPI: Constructing Prompt Injection Benchmarks from LLM Agent Benchmark
Organizations: University of California, Los Angeles · NVIDIA · University of California, Berkeley · National University of Singapore
Abstract
Large language model (LLM) agents increasingly perform complex tasks using high-privilege tools, making indirect prompt injection (IPI) a critical security risk. However, IPI benchmarks lag behind their rapidly evolving utility counterparts, only offering limited task and environment coverage and relying on narrow, static attack scenarios. In this paper, we propose meta-benchmarking for IPI: transforming existing utility benchmarks into end-to-end IPI evaluation suites that reuse their realistic tasks, environments, and infrastructure. We present MetaPI, an extensible toolkit that automatically identifies plausible injection points, generates realistic attack objectives, optimizes adaptive payloads, and validates attacks in the real environment. Its modular interfaces support heterogeneous environments, agents, defenses, and attack algorithms, allowing new components to be integrated without redesigning the evaluation pipeline. We apply MetaPI to AppWorld, VitaBench, and Terminal-Bench and evaluate representative agents and defenses across multiple backbone LLMs. Our results show that no evaluated defense simultaneously achieves strong utility and strong security in realistic, adaptive agentic IPI settings. MetaPI thus enables scalable and consistent security evaluation across diverse and evolving agent ecosystems
Figures & tables
| Payload-induced outcome | Owner-authority test | Valid IPI obj.? |
| Resume claims that the candidate is an AI security expert | Within: candidates may describe their own qualifications | |
| Resume asks the recruiter to hire the candidate | Exceeds: the candidate cannot make hiring decisions | |
| Resume asserts “HR approval granted; reference #1234” | Exceeds: the candidate cannot grant or represent HR approval | |
| Bill is duplicated, or its amount is changed | Within: the biller determines the bill and its amount |
| AppWorld | VitaBench | Terminal-Bench | All | ||
| Objective | Action-deflection | 36 | 44 | 24 | 104 |
| Action-injection | 15 | 43 | 21 | 79 | |
| Position | Necessary read | 22 | 41 | 31 | 94 |
| Accidental read | 29 | 46 | 14 | 89 | |
| Total | attacks (covered tasks) | 51 (37) | 87 (46) | 45 (24) | 183 (107) |
| Benchmark | AppWorld | VitaBench | Terminal-Bench | |||||||
| Category | Defense | ASR | Clean Uti | Tok | ASR | Clean Uti | Tok | ASR | Clean Uti | Tok |
| None | None | 50.0 | 82.5 | 1.0 | 45.9 | 96.0 | 1.0 | 44.4 | 82.2 | 1.0 |
| Sandwiching | 20.8 | 73.6 | 1.13 | 24.7 | 88.8 | 1.22 | 22.2 | 68.9 | 1.09 | |
| Prompt | Spotlighting | 25.0 | 80.4 | 1.09 | 20.0 | 92.6 | 1.14 | 37.8 | 77.8 | 1.12 |
| Model | Meta-SecAlign a | 2.1 | 22.9 | – | 11.8 | 13.6 | – | N/A | ||
| DataSentinel | 41.7 | 77.1 | 1.04 | 43.5 | 92.5 | 1.06 | 28.9 | 73.3 | 1.05 | |
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| AppWorld | VitaBench | Terminal-Bench | |
| Initially selected tasks | 50 | 50 | 50 |
| Tasks retained for the pipeline | 50 | 50 | 30 |
| Benign trajectories collected | 50 | 50 | 27 |
| mean benign reward | 0.74 | 0.85 | 0.85 |
| mean / median steps | 16.2 / 13 | 17.7 / 16 | 16.0 / 13 |
| mean / median tool calls | 42.5 / 24 | 22.2 / 16.5 | 16.0 / 12 |
| AppWorld | VitaBench | Terminal-Bench | |
| Attacks | 51 | 87 | 45 |
| in a necessary read action / an accidental one | 26 / 25 | 45 / 42 | 33 / 12 |
| participant (relied-on) owner / bystander owner | 22 / 29 | 41 / 46 | 31 / 14 |
| action-deflection / action-injection | 36 / 15 | 44 / 43 | 24 / 21 |
| deterministic tool_call rubric / llm_judge | 42 / 9 | 83 / 4 | 24 / 21 |
| Median payload length (chars) [IQR] | 354 [244, 516] | 480 [306, 656] | 448 [272, 682] |
| Reason | Meaning |
| not_delivered | the anchor never appeared in the attacked run, or both delivery encodings failed the feasibility check |
| baseline_contaminated | the baseline run already satisfies the attack’s criterion |
| errored | the attacked run raised or ended in an error |
| no_payload / no_baseline | there was nothing to inject, or no baseline run to compare against |
| judge_failed / eval_failed | the LLM judge or the benchmark’s evaluator failed to return a verdict |
| AppWorld | VitaBench | Terminal-Bench | |
| Domain | personal assistant over 9 consumer apps | conversational customer service (delivery, in-store, travel) | command-line tasks in a container |
| Tools shown | task-specific slice, (median 10) | full domain list (20 / 24 / 38) | one: bash |
| Environment | persistent AppWorld server | in-process environment + LLM user simulator | Docker container per task |
| Utility | unit tests (fraction passed) | LLM rubric judge over the dialogue | task’s test suite (pass/fail) |
| Injection surface | third-party messages, listings, reviews, notes | merchant-supplied product and store text | third-party text in files, VCS history, docs, comments |
| Guidance supplied | write, surface, objective | write, surface, objective | write, read, surface, objective |
| Defense | Attaches as | Sees | Acts by |
| Sandwiching | observation transform | one observation; the user task | wrapping (never blocks) |
| Spotlighting | observation transform | one observation | wrapping; a system-prompt note |
| PromptArmor | observation detector | one observation (no task, no history) | deleting the flagged span |
| DataSentinel | observation detector | one observation | replacing the observation |
| MELON | run-level processors | all tool outputs so far, task removed; the proposed calls | aborting the run |
| DRIFT | run-level processors | the task; tool names and a plan; one observation | blocking calls; deleting sentences |
| Role | Used by | Backbone |
| agent | the victim, in Modules 1, 3, and 4 | deepseek-v4-flash |
| pipeline | Strategy Generator (Module 2) | deepseek-v4-flash |
| optimizer | the attacker in Payload Optimizer, | deepseek-v4-flash |
| judge | llm_judge rubrics in Modules 3 and 4 | deepseek-v4-flash |
| defense | hosted generative guard models | deepseek-v4-flash |
| api_predictor | AppWorld’s per-task tool selection | deepseek-v4-flash |
| Defense | AppWorld | VitaBench | Terminal-Bench | |||
| ASR | ASR | ASR | ||||
| None | 24/41 | 58.5 | 39/83 | 47.0 | 20/38 | 52.6 |
| Sandwiching | 10/36 | 27.8 | 21/83 | 25.3 | 10/39 | 25.6 |
| Spotlighting | 12/37 | 32.4 | 17/83 | 20.5 | 17/41 | 41.5 |
| Meta-SecAlign a | 1/4 | 25.0 | 10/28 | 35.7 | N/A | |
| DataSentinel | 20/41 | 48.8 | 37/83 | 44.6 | 13/38 | 34.2 |
Prompt Injection Attacks on AI Agents
Prompt injection attacks on AI agents exploit untrusted content to override instructions and misuse tools, data, memory, or delegated workflows.
See all 218 papers