Prompt Injection Attacks on AI Agents

Follow this topic Ask about this topic

Prompt injection occurs when an agent treats instructions hidden in retrieved pages, tool outputs, files, skills, images, or memory as authoritative, potentially redirecting actions or exposing data. Researchers study these risks with agent benchmarks, realistic tool-use environments, adaptive red-teaming, and tests that track how attacks move through multi-step or multi-agent workflows. Defenses include instruction-hierarchy training, content screening, provenance and information-flow controls, isolated agents, and runtime checks that authorize each tool action against the user’s task. A central tension is that fixed benchmarks and pattern-based defenses can miss adaptive attacks, while stricter safeguards may block legitimate work; persistent memory, varied input surfaces, and long-horizon behavior remain difficult to secure and evaluate.

Latest papers

191 papers46 papers in the last four weeks, up 84% on the four weeks before. 0.3% of all new papers.

Owner-Harm: A Missing Threat Model for AI Agent Safety
Apr 20, 2026Dongcheng Zhang, Yiqing JiangAI Agent Security AI Agent Safety
BlueFocus Communication Group · Tongji University
Conjunctive Prompt Attacks in Multi-Agent LLM Systems
Apr 17, 2026Nokimul Hasan Arif, Qian Lou, Mengxin ZhengMulti-Agent LLM Systems Adversarial Attacks on LLMs
University of Central Florida, USA
ClawGuard: A Runtime Security Framework for Tool-Augmented LLM Agents Against Indirect Prompt Injection
Apr 13, 2026Wei Zhao, Zhe Li, Peixin Zhang +1AI Agent Security LLM Agent Security
Singapore Management University
STARS: Skill-Triggered Audit for Request-Conditioned Invocation Safety in Agent Systems
Apr 11, 2026Guijia Zhang, Shu Yang, Xilin Gong +1Agent Skill Security Auditing AI Agent Security Benchmarks
Shenzhen University · King Abdullah University of Science & Technology · PRADA Lab +1
ACIArena: Toward Unified Evaluation for Agent Cascading Injection
Apr 9, 2026Hengyu An, Minxi Li, Jinghuai Zhang +6Multi-Agent System Security AI Agent Security Benchmarks
Zhejiang University · State Key Laboratory of Internet Architecture, Tsinghua University · University of California, Los Angeles +2
ClawSafety: "Safe" LLMs, Unsafe Agents
Apr 1, 2026Bowen Wei, Yunbei Zhang, Jinhao Pan +5AI Agent Security AI Agent Safety
George Mason University · Tulane University · Oak Ridge National Laboratory +1
Evaluating Privilege Usage of Agents with Real-World Tools
Mar 30, 2026Quan Zhang, Lianhang Fu, Lvsi Lian +5AI Agent Security Benchmarks Tool Access Control for LLM Agents
ECNU Shanghai, China · Xinjiang University Urumqi, Xinjiang, China · Tsinghua University Beijing, China
Kill-Chain Canaries: Stage-Level Tracking of Prompt Injection Across Attack Surfaces and Five Production LLMs
Mar 30, 2026Haochuan Kevin Wang, Zechen ZhangMulti-Agent LLM Systems Language Model Safety Evaluation
Massachusetts Institute of Technology · University of Chicago
Invisible Threats from Model Context Protocol: Generating Stealthy Injection Payload via Tree-based Adaptive Search
Mar 25, 2026Yulin Shen, Xudong Pan, Geng Hong +1Prompt Injection Attacks on AI Agents MCP Security
School of Computer Science, Fudan University · Shanghai Innovation Institute
The Cognitive Firewall:Securing Browser Based AI Agents Against Indirect Prompt Injection Via Hybrid Edge Cloud Defense
Mar 24, 2026Qianlong Lan, Anuj KaulAI Agent Security Prompt Injection Defense
eBay Inc
Agent-Sentry: Bounding LLM Agents via Execution Provenance
Mar 24, 2026Rohan Sequeira, Stavros Damianakis, Umar Iqbal +1Computer-Use Agent Safety AI Agent Security
University of Southern California, Los Angeles, California, USA · Washington University in St. Louis, St. Louis, Missouri, USA
Trojan's Whisper: Stealthy Manipulation of OpenClaw through Injected Bootstrapped Guidance
Mar 20, 2026Fazhong Liu, Zhuoyan Chen, Tu Lan +6AI Agent Security Agent Skill Security Auditing
Shanghai Jiao Tong University, Shanghai, China
A Framework for Formalizing LLM Agent Security
Mar 19, 2026Vincent Siu, Jingxuan He, Kyle Montgomery +4LLM Agent Security Prompt Injection Attacks on AI Agents
UC Santa Cruz · UC Berkeley · Duke University
The Autonomy Tax: Defense Training Breaks LLM Agents
Mar 19, 2026Shawn Li, Yue ZhaoAI Agent Security LLM Agents
University of Southern California
VeriGrey: Greybox Agent Validation
Mar 18, 2026Yuntong Zhang, Sungmin Kang, Ruijie Meng +2LLM Agent Security Prompt Injection Attacks on AI Agents
National University of Singapore · Max-Planck Insitute of Security and Privacy
Caging the Agents: A Zero Trust Security Architecture for Autonomous AI in Healthcare
Mar 18, 2026Saikat MaitiAI Agent Security LLM Agent Security
VP of Trust, Commure · Founder & CEO, nFactor Technologies
How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition
Mar 16, 2026Mateusz Dziemian, Maxwell Lin, Xiaohan Fu +28Indirect Prompt Injection Prompt Injection Attacks
Gray Swan AI · Carnegie Mellon University · Center for AI Safety +5
PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defenses
Mar 13, 2026Chenlong Yin, Runpeng Geng, Yanting Wang +1Prompt Injection Attacks on LLMs LLM Red Teaming
The Pennsylvania State University
Agent Privilege Separation in OpenClaw: A Structural Defense Against Prompt Injection
Mar 13, 2026Darren Cheng, Wen-Kwang TsaoPrompt Injection Defense AI Agent Security
TrendAI Lab
Security Considerations for Artificial Intelligence Agents
Mar 12, 2026Ninghui Li, Kaiyuan Zhang, Kyle Polley +1AI Agent Security LLM Agent Security
Perplexity · Purdue University
You Told Me to Do It: Measuring Instructional Text-induced Private Data Leakage in LLM Agents
Mar 12, 2026Ching-Yu Kao, Xinfeng Li, Shenyu Dai +4LLM Agent Security Prompt Injection Attacks on AI Agents
Fraunhofer AISEC · NTU · KTH +2
Taming OpenClaw: Security Analysis and Mitigation of Autonomous LLM Agent Threats
Mar 12, 2026Xinhao Deng, Yixiang Zhang, Jiaqing Wu +15AI Agent Security LLM Agent Security
Ant Group & Tsinghua University, China · Tsinghua University, China · Ant Group, China
IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs
Mar 11, 2026Chuan Guo, Juan Felipe Ceron Uribe, Sicheng Zhu +10Adversarial Robustness of Language Models Prompt Injection Attacks on AI Agents
Michael Pokorny · OpenAI
Compatibility at a Cost: Systematic Discovery and Exploitation of MCP Clause-Compliance Vulnerabilities
Mar 10, 2026Nanzi Yang, Weiheng Bai, Kangjie LuMCP Security AI Agent Security
University of Minnesota Minneapolis, USA
Governance Architecture for Autonomous Agent Systems: Threats, Framework, and Engineering Practice
Mar 7, 2026Yuxu GeLLM Agent Security AI Agent Governance
University of York, Department of Computer Science, York, United Kingdom
AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purification
Feb 26, 2026Tian Zhang, Yiwei Xu, Juan Wang +8Prompt Injection Defense Prompt Injection Attacks on AI Agents
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, Wuhan 430072, China · Department of Computer Science and Engineering, University at Buffalo, SUNY · Department of Computer Science, College of Information Science and Technology, Jinan University, Guangzhou 510632, China +1
Silent Egress: When Implicit Prompt Injection Makes LLM Agents Leak Without a Trace
Feb 25, 2026Qianlong Lan, Anuj Kaul, Shaun Jones +1LLM Agent Security Prompt Injection Attacks on AI Agents
eBay
AdapTools: Adaptive Tool-based Indirect Prompt Injection Attacks on Agentic LLMs
Feb 24, 2026Che Wang, Jiaming Zhang, Ziqi Zhang +6Indirect Prompt Injection Prompt Injection Attacks
School of Computer Science, Peking University, China · College of Computing and Data Science, Nanyang Technological University, Singapore · Ant Group
Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks
Feb 23, 2026David Schmotz, Luca Beurer-Kellner, Sahar Abdelnabi +1LLM Agent Security Prompt Injection Attacks on AI Agents
Max Planck Institute for Intelligent Systems, ELLIS Institute Tübingen, Tübingen AI Center · Snyk, Switzerland
The LLMbda Calculus: AI Agents, Conversations, and Information Flow
Feb 23, 2026Zac Garby, Andrew D. Gordon, David SandsPrompt Injection Attacks on AI Agents LLM Agent Harnesses
University of Nottingham, Nottingham, UK · University of Edinburgh, Edinburgh, UK · Chalmers University of Technology and University of Gothenburg, Gothenburg, Sweden

Apr 20, 2026J/K move · Enter open · S save

Owner-Harm: A Missing Threat Model for AI Agent Safety

Dongcheng Zhang, Yiqing Jiang

BlueFocus Communication Group · Tongji University

AI Agent Security AI Agent Safety Prompt Injection Attacks on AI Agents AI Agent Safety Benchmarks

Open paper AskPDF arXiv

Existing AI agent safety benchmarks focus on generic criminal harm (cybercrime, harassment, weapon synthesis), leaving a systematic blind spot for a distinct and commercially consequential threat category: agents harming their own deployers. Real-world incidents illustrate the gap: Slack AI credential exfiltration (Aug 2024), Microsoft 365 Copilot calendar-injection leaks (Jan 2024), and a Meta agent unauthorized forum post exposing operational data (Mar 2026). We propose Owner-Harm, a formal threat model with eight categories of agent behavior damaging the deployer. We quantify the defense gap on two benchmarks: a compositional safety system achieves 100% TPR / 0% FPR on AgentHarm (generic criminal harm) yet only 14.8% (4/27; 95% CI: 5.9%-32.5%) on AgentDojo injection tasks (prompt-injection-mediated owner harm). A controlled generic-LLM baseline shows the gap is not inherent to owner-harm (62.7% vs. 59.3%, delta 3.4 pp) but arises from environment-bound symbolic rules that fail to generalize across tool vocabularies. On a post-hoc 300-scenario owner-harm benchmark, the gate alone achieves 75.3% TPR / 3.3% FPR; adding a deterministic post-audit verifier raises overall TPR to 85.3% (+10.0 pp) and Hijacking detection from 43.3% to 93.3%, demonstrating strong layer complementarity. We introduce the Symbolic-Semantic Defense Generalization (SSDG) framework relating information coverage to detection rate. Two SSDG experiments partially validate it: context deprivation amplifies the detection gap 3.4x (R = 3.60 vs. R = 1.06); context injection reveals structured goal-action alignment, not text concatenation, is required for effective owner-harm detection.