AI Agent Security

Follow this topic Ask about this topic

AI agent security studies how tool use, memory, delegation, and interaction with people or environments create risks beyond those of a standalone language model, including prompt injection, privilege misuse, data leakage, and unsafe actions. Researchers test these risks with red-teaming and execution-grounded benchmarks for coding, browsing, GUI, embodied, and multi-agent tasks, measuring whether harmful effects occur across trajectories rather than judging text alone. Defenses range from input and memory provenance checks to sandboxing, least-privilege permissions, stateful runtime authorization, action verification, and cryptographic identity or audit mechanisms. An open challenge is making controls robust to adaptive, long-horizon attacks and interactions between tools or agents while preserving useful task completion; benchmark realism and comparable measures remain uneven.

Latest papers

517 papers74 papers in the last four weeks, up 32% on the four weeks before. 0.4% of all new papers.

LivePI: More Realistic Benchmarking of Agents Against Indirect Prompt Injection
May 18, 2026Lei Zhao, Abhay Bhaskar, Edgar DobribanAI Agent Security AI Agent Safety
University of Pennsylvania
ESLD (External Surrogate Latent Defense): A Latent-Space Architecture for Faster, Stronger Prompt-Injection Defense
May 18, 2026Yash NarendraAI Agent Security Safety Filtering
Microsoft
AI Agents May Always Fall for Prompt Injections
May 17, 2026Sahar Abdelnabi, Eugene BagdasarianContextual Integrity AI Agent Security
ELLIS Institute Tübingen & MPI-IS & Tübingen AI Center · University of Massachusetts Amherst
ADR: An Agentic Detection System for Enterprise Agentic AI Security
May 17, 2026Chenning Li, Pan Hu, Justin Xu +9MCP Security AI Agent Security
Uber Technologies, Inc., USA · Massachusetts Institute of Technology, USA · University of Oxford, United Kingdom
Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security
May 17, 2026Jinhu Qi, Muzhi Li, Jiahong Liu +9AI Agent Reliability AI Agent Evaluation
Faculty of Engineering, Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China. · Shanghai Academy of AI for Science, Shanghai, China.
PrivScope: Task-scoped Disclosure Control for Hybrid Agentic Systems
May 15, 2026Shafizur Rahman Seeam, Zhengxiong Li, Zhiyuan Yu +3Data Leakage Privacy-Preserving Language Models
Ian · Rochester Institute of Technology, Rochester, NY, USA · University of Colorado Denver, Denver, CO, USA +2
SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors
May 15, 2026Elle Najt, Colin Toft, Tyler Tracy +2Software Engineering Agents Adversarial Attacks
1Anthropic Fellows Program · University of Waterloo · 3Redwood Research +1
The End of Trust: How Agentic AI Breaks Security Assumptions
May 14, 2026Osama Zafar, Alexander Nemecek, Erman AydayAI Agent Security AI Agent Governance
Dept. of Computer and Data Sciences Case Western Reserve University Cleveland, USA
Do Coding Agents Understand Least-Privilege Authorization?
May 14, 2026Zheng Yan, Jingxiang Weng, Charles Chen +9Software Engineering Agents AI Agent Security
Evolvent AI Research Team
Verifiable Agentic Infrastructure: Proof-Derived Authorization for Sovereign AI Systems
May 13, 2026Jun He, Deying YuAI Agent Security AI Agent Governance
AgentTrap: Measuring Runtime Trust Failures in Third-Party Agent Skills
May 13, 2026Haomin Zhuang, Hanwen Xing, Yujun Zhou +5AI Agent Security LLM Agent Security
University of Notre Dame · University of Southern California · LMU Munich +1
Language-Based Agent Control
May 13, 2026Timothy Zhou, Loris D'Antoni, Nadia PolikarpovaAI Agent Security Information Flow Control
Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety
May 12, 2026Muhammad Bilal, Jon Crowcroft, Ruizhi Wang +2LLM Agent Evaluation AI Agent Security
School of Computing and Communications Lancaster University, United Kingdom · Department of Computer Science and Technology, University of Cambridge, United Kingdom · School of Software, Nanjing University of Information Science and Technology, China +1
Attacks and Mitigations for Distributed Governance of Agentic AI under Byzantine Adversaries
May 12, 2026Matthew D. Laws, Alina Oprea, Cristina Nita-RotaruAI Agent Security AI Agent Monitoring
Northeastern University
No More, No Less: Task Alignment in Terminal Agents
May 12, 2026Sina Mavali, David Pape, Jonathan Evertz +5Computer-Use Agent Benchmarks AI Alignment
CISPA Helmholtz Center for Information Security · ELLIS Institute Tübingen and MPI for Intelligent Systems · Tübingen AI Center
IPI-proxy: An Intercepting Proxy for Red-Teaming Web-Browsing AI Agents Against Indirect Prompt Injection
May 12, 2026Chia-Pei, Chen, Kentaroh Toyoda +2AI Agent Security AI Agent Security Benchmarks
Janet · Vulcan Research, AIFT
Behavioral Integrity Verification for AI Agent Skills
May 12, 2026Yuhao Wu, Tung-Ling Li, Hongliang LiuAgent Reliability AI Agent Auditing
Palo Alto Networks
Digital Identity for Agentic Systems: Toward a Portable Authorization Standard for Autonomous Agents
May 12, 2026Partha MadhiraAI Agent Security AI Agent Governance
Mobius Injection: From One-Shot Prompt Injection to Sustained Resource Exhaustion in AI Agent Infrastructure
May 12, 2026Zi Liang, Ronghua Li, Yanyun Wang +2AI Agent Security LLM Agent Security
The Hong Kong Polytechnic University, Hong Kong SAR, China · HKUST (GZ), Guangzhou, China
Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry
May 12, 2026Shoumik Saha, Kazem Faghih, Soheil FeiziAI Agent Security LLM Agent Skill Retrieval
Department of Computer Science, University of Maryland - College Park
ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
May 11, 2026Zhun Wang, Nico Schiller, Hongwei Li +13AI Agent Security AI Agent Security Benchmarks
1UC Berkeley · 2Max Planck Institute for Security and Privacy · 3UC Santa Barbara +4
MATRA: Modeling the Attack Surface of Agentic AI Systems -- OpenClaw Case Study
May 11, 2026Tim Van hamme, Thomas Vissers, Javier Carnerero-Cano +4AI Risk Management AI Agent Security
DistriNet, KU Leuven, 3001 Leuven, Belgium · IBM Research · CISPA Helmholtz Center for Information Security +2
Red-Teaming Agent Execution Contexts: Open-World Security Evaluation on OpenClaw
May 11, 2026Hongwei Yao, Yiming Liu, Yiling He +1AI Agent Security LLM Agent Security
Department of Computer Science, City University of Hong Kong, Hong Kong · School of Computer Science and Technology, Beijing Institute of Technology, Beijing, China · University College London, London, United Kingdom +1
Position: Academic Conferences are Potentially Facing Denominator Gaming Caused by Fully Automated Scientific Agents
May 11, 2026Rong Shan, Te Gao, Hang Zheng +6Scholarly Peer Review AI Agent Security
Shanghai Jiao Tong University · Central South University · Carnegie Mellon University
Skill Description Deception Attack against Task Routing in Internet of Agents
May 11, 2026Jiayi He, Xiaofeng Luo, Jiawen Kang +3Adversarial Attacks AI Agent Security
School of Automation Guangdong University of Technology Guangzhou, China · College of Computing and Data Science Nanyang Technological University Singapore, Singapore · The State Key Laboratory of Public Big Data Guizhou University Guiyang, China +1
Oracle Poisoning: Corrupting Knowledge Graphs to Weaponise AI Agent Reasoning
May 10, 2026Ben Kereopa-Yorke, Guillermo Diaz, Holly Wright +3AI Agent Security LLM Agent Security
1Microsoft · 2UNSW Canberra · 3SAP +1
Portable Agent Memory: A Protocol for Cryptographically-Verified Memory Transfer Across Heterogeneous AI Agents
May 10, 2026Santhosh Kumar RavindranAgent Memory Persistent Memory for Language Models
Microsoft Corporation, Redmond, WA, USA
Security Risks in Tool-Enabled AI Agents: A Systematic Analysis of Privileged Execution Environments
May 10, 2026Hardik GoelAI Agent Security Tool-Using Agents
Microsoft Corporation, USA
Restricting the Model, Missing the System: Measurement and Accountability in Offensive AI Governance
May 10, 2026Michael Alexander Riegler, Finn Schwall, Annika Willoch Olstad +4AI Agent Security AI Safety Evaluation
AI Safety Department SimulaMet and OsloMet Oslo, Norway · Department of Computer Science NTNU Trondheim, Norway
Don't Click That: Teaching Web Agents to Resist Deceptive Interfaces
May 10, 2026Yilin Zhang, Yingkai Hua, Chunyu Wei +2Adversarial Robustness AI Agent Security
Renmin University of China · Ant Digital Technologies, Ant Group

May 18, 2026J/K move · Enter open · S save

LivePI: More Realistic Benchmarking of Agents Against Indirect Prompt Injection

Lei Zhao, Abhay Bhaskar, Edgar Dobriban

University of Pennsylvania

AI Agent Security AI Agent Safety AI Agent Security Benchmarks Indirect Prompt Injection Prompt Injection Defense Prompt Injection Attacks on LLMs+2

Open paper AskPDF arXiv

AI agents such as OpenClaw are increasingly deployed in local workflows with access to external tools. This creates indirect prompt-injection (IPI) risk: an agent may execute harmful instructions embedded in untrusted inputs such as email, downloaded files, webpages, repositories, or group-chat messages. Existing evaluations are often small, purely simulated, or focused on a narrow set of channels. We introduce LivePI (Live Prompt Injection), a structured benchmark for IPI risk in a production-like but test-controlled environment. LivePI covers seven input surfaces, twelve attack/rendering families, and five malicious goals, including protected-information exfiltration, unauthorized security-control changes, unsafe code retrieval or execution, inbox-summary exfiltration, and cryptocurrency transfer. We run LivePI on a real virtual machine with live but test-controlled email, chat, web, local-file, repository, and wallet interfaces. Across GPT-5.3-Codex, Claude Opus 4.6, Gemini 3.1 Pro, Kimi K2.5, and GLM-5, total attack success rates range from 10.7% to 29.6%. Group-chat injection is uniformly successful across the evaluated backbones in our deployment, and repository-link attacks produce high-severity failures despite a small denominator. We also evaluate a two-layer defense consisting of prompt-level filtering and pre-execution tool-call authorization. In the GPT-5.3-Codex setting, the defense intercepts all tested malicious-goal completions in LivePI before execution while preserving benign utility on PinchBench-derived workloads.