cs.CRNov 5, 2026

MetaPI: Constructing Prompt Injection Benchmarks from LLM Agent Benchmark

Authors: Peiran Wang, Chong Xiang, Wenjie Qu, Ying Li, Jiaheng Zhang, G. Edward Suh, Yuan Tian

Organizations: University of California, Los Angeles · NVIDIA · University of California, Berkeley · National University of Singapore

Abstract

Large language model (LLM) agents increasingly perform complex tasks using high-privilege tools, making indirect prompt injection (IPI) a critical security risk. However, IPI benchmarks lag behind their rapidly evolving utility counterparts, only offering limited task and environment coverage and relying on narrow, static attack scenarios. In this paper, we propose meta-benchmarking for IPI: transforming existing utility benchmarks into end-to-end IPI evaluation suites that reuse their realistic tasks, environments, and infrastructure. We present MetaPI, an extensible toolkit that automatically identifies plausible injection points, generates realistic attack objectives, optimizes adaptive payloads, and validates attacks in the real environment. Its modular interfaces support heterogeneous environments, agents, defenses, and attack algorithms, allowing new components to be integrated without redesigning the evaluation pipeline. We apply MetaPI to AppWorld, VitaBench, and Terminal-Bench and evaluate representative agents and defenses across multiple backbone LLMs. Our results show that no evaluated defense simultaneously achieves strong utility and strong security in realistic, adaptive agentic IPI settings. MetaPI thus enables scalable and consistent security evaluation across diverse and evolving agent ecosystems

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LivePI: More Realistic Benchmarking of Agents Against Indirect Prompt Injection

    May 18, 2026Lei Zhao, Abhay Bhaskar, Edgar DobribanAI Agent SecurityAI Agent Safety

  2. AutoDojo: A Generative Benchmark for Evaluating Prompt Injection Defenses in LLM Agents

    Jun 13, 2026Xinhang Ma, Taoran Li, Chaowei Xiao +3AI Agent Security BenchmarksIndirect Prompt Injection