AI Agent Security

Follow this topic Ask about this topic

AI agent security studies how tool use, memory, delegation, and interaction with people or environments create risks beyond those of a standalone language model, including prompt injection, privilege misuse, data leakage, and unsafe actions. Researchers test these risks with red-teaming and execution-grounded benchmarks for coding, browsing, GUI, embodied, and multi-agent tasks, measuring whether harmful effects occur across trajectories rather than judging text alone. Defenses range from input and memory provenance checks to sandboxing, least-privilege permissions, stateful runtime authorization, action verification, and cryptographic identity or audit mechanisms. An open challenge is making controls robust to adaptive, long-horizon attacks and interactions between tools or agents while preserving useful task completion; benchmark realism and comparable measures remain uneven.

Latest papers

517 papers74 papers in the last four weeks, up 32% on the four weeks before. 0.4% of all new papers.

StepShield: When, Not Whether to Intervene on Rogue Agents
Jan 29, 2026Gloria Felicia, Zitha Sasindran, Jinfeng He +3LLM Guardrails AI Agent Security
University of Virginia · Indian Institute of Science Bangalore · Indian Institute of Science, Bangalore +5
Mind the Gap: Action Rebinding Attacks against Android GUI Agents
Jan 18, 2026Yi Qian, Kunwei Qian, Xingbang He +7Mobile GUI Automation Adversarial Attacks
State Key Laboratory for Novel Software Technology, Nanjing University · Hornor Device Co., Ltd · Institute of Dataspace, Hefei Comprehensive National Science Center China
MiniScope: Authorizing Agents with Least-Privilege Permissions
Dec 11, 2025Jinhao Zhu, Xiao Huang, Kevin Tseng +4Tool Access Control for LLM Agents AI Agent Security
UC Berkeley · IBM Research
Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks
Oct 1, 2025Shoumik Saha, Jifan Chen, Sam Mayers +3AI Coding Agents AI Agent Security
University of Maryland – College Park · AWS AI Labs · Meta Superintelligence Labs
Enabling Regulatory Multi-Agent Collaboration: Architecture, Challenges, and Solutions
Sep 11, 2025Qinnan Hu, Yuntao Wang, Yuan Gao +3Agent Reliability AI Agent Security
School of Cyber Science and Engineering, Xi’an Jiaotong University, China · School of Mechatronic Engineering and Automation, Shanghai University, China
Web3 x AI Agents: Landscape, Integrations, and Foundational Challenges
Aug 4, 2025Yiming Shen, Jiashuo Zhang, Zhenzhe Shao +5AI Agent Reliability AI Agent Governance
School of Software Engineering, Sun Yat-sen University, Zhuhai, Guangdong, China · School of Computer Science, Peking University, Beijing, China · School of Computer Science and Engineering (School of Cyber Security), University of Electronic Science and Technology of China, Chengdu 611731, China +1
Easier Said Than Done: Unpacking Intent-Behavior Gap in Jailbreaking LLM-Based Robots
Dec 21, 2024Xuancun Lu, Zhengxian Huang, Xinfeng Li +3Large Language Model-Based Robot Planning Adversarial Attacks on LLMs
Zhejiang University

Jan 29, 2026J/K move · Enter open · S save

StepShield: When, Not Whether to Intervene on Rogue Agents

Gloria Felicia, Zitha Sasindran, Jinfeng He, Michael Eniolade+2

University of Virginia · Indian Institute of Science Bangalore · Indian Institute of Science, Bangalore+5

LLM Guardrails AI Agent Security AI Agent Security Benchmarks AI Agent Monitoring

Open paper AskPDF arXiv

Agent safety benchmarks measure whether a monitor detects harm, not when. Yet timing is the difference between intervention and autopsy. We introduce StepShield, the first benchmark that treats detection timeliness as a first-class metric. On 9,429 incident-grounded code-agent trajectories, we define the Early Intervention Rate (EIR): the fraction of detected rogue trajectories where the alert fires within a k-step window after the divergence point, isolating timing quality from coverage. This metric exposes what we call the Forensics Trap: a pattern-based guardrail with 847 rules achieves 86% recall yet is statistically indistinguishable from random timing on EIR (0.23 vs. 0.24; p = 0.66, one-sided binomial; difference within CI), because over three-quarters of its alerts trigger on benign prefix code before any violation occurs. The 4x EIR gap between rule-based and semantic detectors is completely invisible to accuracy, recall, or F1. Our finding is structural: regex guardrails detect syntax, not intent, and therefore cannot distinguish the moment an agent turns rogue, rendering the entire deployed class of pattern-based monitors unsuited for real-time oversight. No existing method simultaneously achieves high recall, low false-positive rate, and timely intervention, establishing step-level rogue detection as genuinely unsolved.