Prompt Injection Attacks on AI Agents

Follow this topic Ask about this topic

Prompt injection occurs when an agent treats instructions hidden in retrieved pages, tool outputs, files, skills, images, or memory as authoritative, potentially redirecting actions or exposing data. Researchers study these risks with agent benchmarks, realistic tool-use environments, adaptive red-teaming, and tests that track how attacks move through multi-step or multi-agent workflows. Defenses include instruction-hierarchy training, content screening, provenance and information-flow controls, isolated agents, and runtime checks that authorize each tool action against the user’s task. A central tension is that fixed benchmarks and pattern-based defenses can miss adaptive attacks, while stricter safeguards may block legitimate work; persistent memory, varied input surfaces, and long-horizon behavior remain difficult to secure and evaluate.

Latest papers

191 papers46 papers in the last four weeks, up 84% on the four weeks before. 0.3% of all new papers.

Heads of Security: Learned Attention Gates Improve Prompt Injection Robustness
Oct 27, 2026Igor Shilov, Xiaoxue Yang, Alexander Panfilov +6LLM Security Prompt Injection Defense
Imperial College London · Tübingen AI Center · ELLIS Institute Tübingen +2
Log-To-Leak: Prompt Injection Attacks on Tool-Using LLM Agents via the Model Context Protocol
Oct 26, 2026Yuepeng Hu, Chongyu Fan, Sirat Samyoun +1Prompt Injection Attacks on AI Agents LLM Agent Safety
Michigan State University East Lansing, MI, USA · TikTok San Jose, CA, USA
Scope-Gap Injection: Exploiting Evaluator Channel Blind Spots in the OpenClaw Agent Security Competition
Oct 26, 2026Weihang Niu, Mengxing Ren, Che QuPrompt Injection Attacks on AI Agents Indirect Prompt Injection
University of Science and Technology of China · University of Electronic Science and Technology of China
Agentic AI Defense: Decode-Time Provenance Binding for MCP
Oct 22, 2026Eric Yocam, Varghese Vaidyan, Paris Kalathas +3AI Agent Security Prompt Injection Attacks on AI Agents
Beacom College of Computer and Cyber Sciences, Dakota State University, Madison, SD, USA. · Information Networking Institute, Carnegie Mellon University, Pittsburgh, PA, USA.
Certified Execution Harness for Agentic AI: Safe and Least-Intrusive Petri-Net Supervisory Control Between Reasoning and Action
Oct 22, 2026Chao Gu, Quanyan ZhuTool Access Control for LLM Agents Runtime Enforcement for AI Agents
C. Gu is with the School of Electronics, Electrical Engineering and Computer Science, Queen’s University Belfast, Belfast, UK · Q. Zhu is with the Department of Electrical and Computer Engineering, New York University, Brooklyn, NY 11201, USA
Correlated Failures of AI Approval Monitors
Oct 21, 2026Luca ZanoliniAI Agent Monitoring AI Agent Reliability
Independent researcher
Attention-based Classification for Indirect Prompt Injection
Oct 20, 2026Amelia Kawasaki, Stefan LeePrompt Injection Detection Prompt Injection Attacks on AI Agents
Corvallis, Oregon, USA
After the Attacker Leaves: Benchmarking Carryover Attacks on LLM Agents
Oct 20, 2026Zhiyue Xu, Xukai Wang, Zhiyuan Huang +3Adversarial Attacks on LLMs Prompt Injection Attacks on AI Agents
MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artiﬁcial Intelligence, UCAS · Zhongguancun Academy +2
ClawMMU: Protection-Coherent Virtual Memory for Stateful LLM Agents
Oct 20, 2026Jinhao Hu, Bardia Mohammadi, Mofasshara Binte Rafique +4AI Agent Security Memory Management for LLM Agents
MPI-SWS · Unaffiliated · EPFL +2
SoK: PIAtlas: Decomposing the Unbounded Prompt Injection Attack Space
Oct 18, 2026Stavros Damianakis, Konstantinos PsounisPrompt Injection Attacks Prompt Injection Attacks on LLMs
University of Southern California
The Window Is in the Reasoning: When LLM Agents Decide to Follow Prompt Injections
Oct 16, 2026Haoxuan Jia, Yang Liu, Bin Chong +13Prompt Injection Attacks on AI Agents Prompt Injection Detection
Fullive-AI · Nanyang Technological University · Supply Chain Tech Team Y, JD.com +6
MobileInject: Evaluating Reasoning-to-Action Security of MLLM-Based Mobile Agents under Cross-Modal Consistency Injection
Oct 16, 2026Songping Wang, Peiyang Li, Yi Huang +5LLM Agent Security AI Agent Security Benchmarks
From Approval to Abuse: Privilege Escalation Attacks on Harness-Protected Coding Agents
Oct 16, 2026Xianglin Yang, Yechao Zhang, Jiahao Liu +4AI Agent Security LLM Agent Harnesses
National University of Singapore · Nanyang Technological University · Ant Group
The Model Knows Better: Defending Medical Multi-Agent Systems Against Prompt Injection
Oct 15, 2026Fardin Ahsan Sakib, Ziwei Zhu, Ozlem UzunerPrompt Injection Defense Prompt Injection Attacks on AI Agents
Department of Computer Science, George Mason University · Department of Information Sciences and Technology, George Mason University
INFUSE: A Unified Surface Benchmark for Indirect Prompt Injection in Agentic Systems
Oct 14, 2026Thuy Dung Nguyen, Jingyi Guo, Rana Seda Dubauskas +3Indirect Prompt Injection Prompt Injection Attacks on AI Agents
Vanderbilt University
Policy-Verified Self-Training: Learning Agent Compliance from Verifiable Policy Rewards
Oct 14, 2026Min-Young YuAI Agent Safety Reinforcement Learning with Verifiable Rewards
Corners Co., Ltd.
Towards Safe Autonomy for Computer Use
Oct 13, 2026Dominik Seip, Bartol Markovinović, Matthias HeinComputer-Use Agent Safety AI Agent Safety
Tübingen AI Center, University of Tübingen
Kairos: Transactional Security Enforcement at the System Boundary for LLM Agents
Oct 13, 2026Kristoffer Hempel, Nils Loose, Lorenzo Cavallaro +1LLM Agent Security Runtime Enforcement for AI Agents
Universität zu Lübeck · University College London · Mohamed bin Zayed University of Artificial Intelligence
Encrypted Skills Can Become New Security Exploits If AI Agents Know How to Decipher
Oct 13, 2026Yan-Lun Chen, Pin-Yu Chen, Chia-Mu YuPrompt Injection Attacks on AI Agents Prompt Injection Attacks
National Yang Ming Chiao Tung University · IBM Research
BrowseSafe++: Benchmarking Prompt Injection Detection Across Web Contexts, Languages, and Browser Views
Oct 12, 2026Yuetian Chen, Kaiyuan Zhang, Yuntao Du +6Prompt Injection Detection AI Agent Security Benchmarks
Perplexity · Purdue University · Rutgers University
Only What You've Signed up For: Secure Workflows with Constrained Diffusion Agents
Oct 10, 2026Michael Cardei, Anjun Hu, Chenrui Ma +4AI Agent Security Prompt Injection Defense
University of Virginia · University of Oxford · Google DeepMind +1
Can agents whose sole duty is to keep humans alive be hijacked to kill? Delayed multi-turn derived-action trap attacks in simulated high-stakes environments
Oct 10, 2026Josef A. HabdankPrompt Injection Attacks on AI Agents Prompt Injection Attacks
DXC Technology
APEX-Harm: Evaluating Agent Safety in Shared Professional Workspaces
Oct 9, 2026Ziyi Yang, Adel Bibi, Guohao Li +2AI Agent Security LLM Agent Security
King Abdullah University of Science and Technology · University of Oxford
Forgetting is Not Always Bad: A Neuro-Inspired Memory Repair Mechanism for Poisoned LLM Agents
Oct 9, 2026Lei Liu, Yunji Liang, Xiaowen Zhang +4Agent Memory Poisoning Backdoor Defense in LLMs
Northwestern Polytechnical University · National Univerisity of Singapore
One Word Opens the Gate: The Option-Channel Attack on Typed Decision Models as Agent Guardrails
Oct 8, 2026Seyedarmin Azizi, Erfan Baghaei Potraghloo, Massoud PedramLLM Guardrails LLM Agent Security
University of Southern California (USC), USA
Constrained-Action AI Remediation for SIEM/XDR via a NeMo-Guardrails Proxy
Oct 7, 2026Georgios Koutidis, Nikolaos Kekatos, Tom Nianios +1LLMs for Cybersecurity Tool Access Control for LLM Agents
Clone Systems, Larnaca, Cyprus · University of Thessaly, Department of Intelligent Energy Systems, Larissa, Greece
Secure-CUA: Controlling Untrusted Influence in Computer-Use Agents
Oct 7, 2026Sarthak Choudhary, Mihai Christodorescu, Ashish Hooda +3Computer-Use Agents AI Agent Security
University of Wisconsin–Madison · Google · Google DeepMind
Package Hallucination Attacks on Coding Agents through Prompt Injection in Rule Files
Oct 7, 2026Yupu Wang, Zhengyuan Jiang, Reachal Wang +1AI Agent Security Adversarial Attacks
Duke University
AdvSim2Real : Training Web Agents Against Adaptive Prompt Injection in a Web World Model
Oct 6, 2026Sarim Hashmi, Mukul Ranjan, Kshitij Mishra +3Adversarial Training Sim-to-Real Transfer
Mohamed bin Zayed University of Artificial Intelligence · Amazon · Massachusetts Institute of Technology
Surviving the Router: Optimizing Skill Injections for Retrieval and Execution
Oct 6, 2026Haneen Najjar, Luca Scionis, Haritz Puerto +1AI Agent Security Benchmarks LLM Agent Skill Retrieval
Max Planck Institute for Intelligent Systems; Tel Aviv University · Sapienza University of Rome; University of Cagliari · ELLIS Institute Tübingen

Oct 27, 2026J/K move · Enter open · S save

Heads of Security: Learned Attention Gates Improve Prompt Injection Robustness

Igor Shilov, Xiaoxue Yang, Alexander Panfilov, Evgenii Kortukov+5

Imperial College London · Tübingen AI Center · ELLIS Institute Tübingen+2

LLM Security Prompt Injection Defense Prompt Injection Attacks on LLMs Prompt Injection Attacks on AI Agents

Open paper AskPDF arXiv

We introduce Heads of Security (HoS), a prompt injection defense that enforces instruction-data separation via a small set of learned per-head attention gates. HoS learns how each head should redistribute attention between trusted and untrusted tokens, enabling models to reason over untrusted content without following injected instructions. We evaluate HoS on Llama-3.3-70B-Instruct and GPT-OSS-20B. We compare it against SecAlign++, a state-of-the-art finetuning-based defense, and ASIDE, a defense that, like ours, builds instruction-data separation into the model architecture. We show that HoS offers greater robustness to adaptive attacks, better performance on standard utility benchmarks, and matches top performance on standard agentic prompt injection benchmarks. More specifically, the black-box RL-Hammer attack reaches 97% ASR against GPT-OSS-20B and 85% against SecAlign, but only 10% against HoS. Claudini, a state-of-the-art white-box attack, strengthened against HoS through autoresearch, requires 3 times more FLOPs to reach the same ASR on HoS as it does on GPT-OSS-20B or SecAlign. HoS preserves base-model utility across standard multiple-choice benchmarks and prompting setups, whereas SecAlign and ASIDE degrade in few-shot settings. On agentic benchmarks with static attacks, HoS and SecAlign perform similarly and substantially improve the security-utility trade-off over the base model. ASIDE struggles with tool use, achieving only 10% utility. These results show that prompt injection robustness can be achieved and improved through targeted attention interventions.