AI Agent Safety Benchmarks

Follow this topic Ask about this topic

AI agent safety benchmarks test whether systems using tools, software, or physical devices follow instructions, respect permissions, avoid harmful side effects, and know when to abstain. Common approaches use executable sandboxes, simulated workplaces and embodied environments, multi-turn red teaming, and analysis of full action trajectories rather than final answers alone. Evaluations increasingly measure safety alongside task success, usability, authorization, and recovery, while runtime guards and deterministic checks are tested as possible safeguards. Results depend heavily on the model, harness, environment, and scoring method, and fragmented benchmarks make safety rankings difficult to compare; a central challenge is capturing long-horizon and real-world risks without confusing safe behavior with inability to act.

Latest papers

220 papers46 papers in the last four weeks, up 53% on the four weeks before. 0.3% of all new papers.

Don't Make Models Guess Security and Safety: Symbolic Guardrails for Domain-Specific AI Agents
Apr 16, 2026Yining Hong, Yining She, Eunsuk Kang +2LLM Guardrails AI Agent Security
Carnegie Mellon University
HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?
Apr 16, 2026Yukun Jiang, Yage Zhang, Michael Backes +2LLM Agent Security AI Agent Security Benchmarks
CISPA Helmholtz Center for Information Security
Benchmarks for Trajectory Safety Evaluation and Diagnosis in OpenClaw and Codex: ATBench-Claw and ATBench-Codex
Apr 16, 2026Zhonghao Yang, Yu Li, Yanxu Zhu +6Benchmark Design Software Engineering Benchmarks
Shanghai Artificial Intelligence Laboratory
HINTBench: Horizon-agent Intrinsic Non-attack Trajectory Benchmark
Apr 15, 2026Jiacheng Wang, Jinchang Hou, Fabian Wang +3AI Agent Auditing Long-Horizon Agent Evaluation
Baidu Inc., Beijing, China · Beijing Institute of Technology, Beijing, China
Policy-Invisible Violations in LLM-Based Agents
Apr 14, 2026Jie Wu, Ming GongLLM Agent Safety Runtime Enforcement for AI Agents
Atlassian
EmbodiedGovBench: A Benchmark for Governance, Recovery, and Upgrade Safety in Embodied Agent Systems
Apr 13, 2026Xue Qin, Simin Luan, John See +2AI Agent Governance AI Agent Safety Benchmarks
School of Software, Harbin Institute of Technology, Harbin, China · School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China · School of Mathematical and Computer Sciences, Heriot-Watt University, Malaysia Campus, Putrajaya, Malaysia +1
The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agents
Apr 12, 2026Xuwei Ding, Skylar Zhai, Linxin Song +6Computer-Use Agent Safety AI Agent Safety
University of Wisconsin–Madison · University of Minnesota · University of Southern California +3
Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents
Apr 11, 2026Miles Q. Li, Benjamin C. M. Fung, Boyang Li +2AI Agent Safety Benchmarks AI Agent Evaluation
School of Information Studies, McGill University, Canada · Department of Computer Science, Kean University, USA · College of Technological Innovation, Zayed University, UAE
Many-Tier Instruction Hierarchy in LLM Agents
Apr 10, 2026Jingyu Zhang, Tianjian Li, William Jurayj +3LLM Agents AI Agent Safety Benchmarks
Johns Hopkins University
CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI Automation
Apr 10, 2026Yushi Feng, Junye Du, Qifan Wang +5AI Agent Safety Mobile GUI Agents
The University of Hong Kong · The Chinese University of Hong Kong · The University of Tokyo
PilotBench: A Benchmark for General Aviation Agents with Safety Constraints
Apr 10, 2026Yalun Wu, Haotian Liu, Zhoujun Li +1AI Agent Safety Benchmarks Trajectory Prediction
School of Computing, National University of Singapore, China · School of Informatics, Xiamen University, China · CESS, Beihang University, China
SysAdmin: Measuring Instrumental Power-Seeking in Frontier AI
Apr 10, 2026Mana Azarm, Qiyao Wei, Rahul NambiarLanguage Model Safety Evaluation AI Alignment
Analytics & Information Systems University of San Francisco · Department of Applied Mathematics and Theoretical Physics University of Cambridge · Propensity Labs
TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectories
Apr 8, 2026Yen-Shan Chen, Sian-Yao Huang, Cheng-Lin Yang +1Language Model Safety Evaluation LLM Guardrails
CyCraft AI Lab, Taiwan · National Taiwan University
Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents
Apr 7, 2026Bowen Ye, Rang Li, Qibin Yang +10AI Agent Evaluation AI Agent Benchmarks
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The University of Hong Kong
ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces
Apr 6, 2026Xiangyi Li, Kyoung Whan Choe, Yimin Liu +12LLM Agent Evaluation LLM Agents
BenchFlow · RLWRLD · Ohio State University +9
Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw
Apr 6, 2026Zijun Wang, Haoqin Tu, Letian Zhang +11AI Agent Security AI Agent Safety
UC Santa Cruz · NUS · Tencent +3
AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agents
Apr 3, 2026Yunhao Feng, Yifan Ding, Yingshui Tan +6LLM Safety Benchmarks Computer-Use Agent Benchmarks
Alibaba Group · Fudan University · Hunan Institute of Advanced Technology +2
ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis
Apr 2, 2026Yu Li, Haoyu Luo, Yuejin Xie +10AI Agent Safety Benchmarks LLM Agent Safety
Shanghai AI Lab · Fudan Univ. · Shanghai Jiao Tong Univ. +2
AgentSocialBench: Evaluating Privacy Risks in Human-Centered Agentic Social Networks
Apr 1, 2026Prince Zizhuang Wang, Shuli JiangPrivacy Protection for LLM Agents LLM Agent Security
Carnegie Mellon University
ClawSafety: "Safe" LLMs, Unsafe Agents
Apr 1, 2026Bowen Wei, Yunbei Zhang, Jinhao Pan +5AI Agent Security AI Agent Safety
George Mason University · Tulane University · Oak Ridge National Laboratory +1
Doctorina MedBench: A Dialogue-Based Benchmark and Evaluation Framework for Agent-Based Medical AI
Mar 26, 2026Anna Kozlova, Stanislau Salavei, Pavel Satalkin +3Medical AI Agents AI Agent Benchmarks
A.I. Doctor Medical Assist LTD, 13 Myrtiotissis Street, Aqua Mansions, Office 1, Germasogeia, 4041 Limassol, Cyprus
EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings
Mar 13, 2026Shiva Krishna Reddy Malay, Shravan Nayak, Jishnu Sethumadhavan Nair +6AI Agent Benchmarks LLM Agent Planning
ServiceNow Research · Mila - Quebec AI Institute · Université de Montréal
LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratories
Mar 12, 2026Qianpu Sun, Xiaowei Chi, Yuhan Rui +5AI Agent Safety Benchmarks Multimodal Large Language Models
Tsinghua University, Beijing, China · The Hong Kong University of Science and Technology, Hong Kong, China · Southern University of Science and Technology, Shenzhen, China +2
HomeSafe-Bench: Evaluating Vision-Language Models on Unsafe Action Detection for Embodied Agents in Household Scenarios
Mar 12, 2026Jiayue Pu, Zhongxiang Sun, Zilu Zhang +2Cultural Safety Evaluation of VLMs AI Agent Safety
Gaoling School of Artificial Intelligence, Renmin University of China · University of Chinese Academy of Sciences · Beijing University of Posts and Telecommunications
Real-Time Trust Verification for Safe Agentic Actions using TrustBench
Mar 10, 2026Tavishi Sharma, Vinayak Sharma, Pragya SharmaAI Agent Safety AI Agent Reliability
School of Computing and Augmented Intelligence, Arizona State University, Tempe, USA · Dept. of Electrical and Computer Engineering, University of California Los Angeles, Los Angeles, USA
Tool Receipts, Not Zero-Knowledge Proofs: Practical Hallucination Detection for AI Agents
Mar 9, 2026Abhinaba BasuHallucination Detection LLM Agent Verification
ConflictBench: Evaluating Human-AI Conflict via Interactive and Visually Grounded Environments
Mar 9, 2026Weixiang Zhao, Haozhen Li, Yanyan Zhao +5Multimodal Agent Evaluation AI Agent Safety Benchmarks
Harbin Institute of Technology · SERES
AutoControl Arena: Synthesizing Executable Test Environments for Frontier AI Risk Evaluation
Mar 8, 2026Changyi Li, Pengfei Lu, Xudong Pan +2Language Model Safety Evaluation AI Agent Safety Benchmarks
Fudan University, Shanghai, China · Shanghai Innovation Institute, Shanghai, China · University of Oxford, Oxford, United Kingdom +2
Safety Under Scaffolding: How Evaluation Conditions Shape Measured Safety
Mar 8, 2026David GringrasLLM Safety Benchmarks LLM Agent Evaluation
Harvard University
Quantifying Frontier LLM Capabilities for Container Sandbox Escape
Mar 1, 2026Rahul Marchand, Art O Cathain, Jerome Wynne +8LLM Safety Benchmarks LLM Agent Security
University of Oxford · UK AI Security Institute

Apr 16, 2026J/K move · Enter open · S save

Don't Make Models Guess Security and Safety: Symbolic Guardrails for Domain-Specific AI Agents

Yining Hong, Yining She, Eunsuk Kang, Christopher S. Timperley+1

Carnegie Mellon University

LLM Guardrails AI Agent Security AI Agent Safety AI Agent Security Benchmarks AI Agent Safety Benchmarks

Open paper AskPDF arXiv

There is increasing interest in integrating AI agents that invoke tools into domain-specific commercial software, where unintended tool calls can cause serious security and safety incidents. This has drawn growing research attention, and many agent security and safety benchmarks have emerged. They implicitly shape how the community approaches security and safety. Yet existing work exhibits a blind spot: it emphasizes training-based methods and neural guardrails, which reduce the likelihood of insecure or unsafe actions but cannot guarantee their prevention. It generally overlooks opportunities for deductive, symbolic guardrails grounded in standard software engineering practices, which can provide guarantees for some security and safety requirements. Our study has three parts: (1) a systematic review of 80 agent security and safety benchmarks finding that that 85% of benchmarks do not state verifiable requirements (61% provide none, and 24% give only high-level goals); (2) an applicability analysis of which security and safety requirements symbolic guardrails can and cannot enforce on

τ^2

-Bench, CAR-bench, and MedAgentBench, finding that 74% of requirements are symbolically enforceable and 95% of these need only simple, low-cost checks; and (3) an empirical evaluation of symbolic guardrails on the same three benchmarks, finding that symbolic guardrails improve security and safety without sacrificing utility, and often improve it. Our work draws attention to the potential for symbolic guardrails for AI agents, suggesting them as an overlooked but practical path toward deploying domain-specific AI agents in risk-averse commercial software. We release all codes and artifacts at https://github.com/hyn0027/agent-symbolic-guardrails.