AI Agent Safety Benchmarks

Ask about this topic

Overview

AI agent safety benchmarks test whether systems using tools, software, or physical devices follow instructions, respect permissions, avoid harmful side effects, and know when to abstain. Common approaches use executable sandboxes, simulated workplaces and embodied environments, multi-turn red teaming, and analysis of full action trajectories rather than final answers alone. Evaluations increasingly measure safety alongside task success, usability, authorization, and recovery, while runtime guards and deterministic checks are tested as possible safeguards. Results depend heavily on the model, harness, environment, and scoring method, and fragmented benchmarks make safety rankings difficult to compare; a central challenge is capturing long-horizon and real-world risks without confusing safe behavior with inability to act.

Latest papers

CardsList

220 papers46 papers in the last four weeks, up 53% on the four weeks before. 0.3% of all new papers.

  1. An Alignment Benchmark for Enterprise Agents on Era by Eon

    Oct 28, 2026Hai Sadon, Benjamin Gruenbaum, Doron Porat +4Enterprise AI AgentsAI Agent Safety

  2. ShutdownMod: Turning Any Trajectory into a Shutdown Evaluation

    Oct 26, 2026Param Biyani, Sebastien Luyasu, Cedric Potvliege +3AI Agent SafetyAI Agent Evaluation

    MATS · SEAR · University of Washington +2

  3. V3^3AST-Bench: Verifiable, Valid, and Versatile Safety Evaluation for Coding Agents

    Oct 26, 2026Weitao Feng, Hengheng Xiong, Zihao Zeng +6AI Agent SafetyAI Agent Safety Benchmarks

    Nanyang Technological University, Singapore · Alibaba Group, Hangzhou, China

  4. SentryBench: A Cost-Sensitive Benchmark for Adversarial Robustness of Tool-Using LLM Agents in Public-Safety Deployments

    Oct 25, 2026Md Abu Yusuf, Oyshi Tabassum Aditi, Md Faysal Hosain MazedLLM Agent SafetyAI Agent Safety Benchmarks

    Department of Computer Science, Maharishi International University, IA, USA · Department of Computer Science and Engineering, Daffodil International University, Dhaka, Bangladesh · Department of Information Technology, University of the Cumberlands, Williamsburg, KY, USA

  5. Task Success Is Not Enough: Side-effect-Aware Evaluation of Tool-Using Language Model Agents

    Oct 25, 2026Jiaju Huang, Shaobin Chen, Xinglong Liang +5LLM Agent EvaluationLLM Agent Safety

    Macao Polytechnic University · Southwest Medical University · Netherlands Cancer Institute +1

  6. BARC-Bench: Benign Asks, Risky Consequences

    Oct 23, 2026Kangkang Li, Kelvin Luu, Lily Ma +3AI Agent Safety BenchmarksLLM Agent Safety

    Scale AI

  7. The Helpful Agent Problem: When Emergent Cooperation Among LLM Agents Bypasses Intended Workflows

    Oct 22, 2026Moumita Choudhury, Amit Sarker, Shlomo ZilbersteinLLM Agent SafetyEmergent Behavior in Multi-Agent Systems

    University of Massachusetts Amherst

  8. EpiSEM-Bench: Evaluating Safety and Evidence Recovery in LLM Agents under Missing Evidence

    Oct 22, 2026Qijin Zhang, Hanzhen Wang, Yuehan Zhao +4AI Agent Safety BenchmarksLLM Agent Safety

    City University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong, Shenzhen

  9. Post-Audit Mirage: Identifiability Limits of Offline Agent Verification

    Oct 22, 2026Aman Behera, Payal PayalLLM Agent VerificationAI Agent Monitoring

    IIT Roorkee

  10. CAPSafe: A Benchmark for Contextual Action and Plan Safety Reasoning in Embodied Agents

    Oct 20, 2026Youmna Farag, Svetlana Stoyanchev, Simon Keizer +2AI Agent SafetyAI Agent Safety Benchmarks

    Cambridge Research Laboratory, Toshiba Europe Ltd, Cambridge, UK

  11. RSI Control: Can AI Models Perform Covert Sabotage in the Self-Improvement Process?

    Oct 20, 2026Kaiwen Zhou, Udari Madhushani SehwagRecursive Self-ImprovementAI Control

    Preparedness and Policy, Scale AI

  12. Act on Your Behalf, Beyond Your Authorization: Benchmarking LLM Agent Authorization Alignment

    Oct 20, 2026Zhiping Zhang, Franklin Mingzhe Li, Zihao Huang +1LLM Agent EvaluationAI Agent Safety Benchmarks

    Northeastern University · Carnegie Mellon University · UNC-Chapel Hill

  13. AM-Bench: A Unified Taxonomy and Evaluation Suite for Agentic Misalignment

    Oct 16, 2026Eric Zhang, Terry Jingchen Zhang, Chijioke Ugwuanyi +3AI Agent Safety BenchmarksLLM Agent Safety

    Jinesis Lab, University of Toronto & Vector Institute · EuroSafeAI · Carnegie Mellon University +1

  14. ToolRosetta: Surface-Form Invariance of Tool-Call Guards

    Oct 16, 2026Hasan Demirkiran, Jens ErnstbergerLLM GuardrailsRuntime Enforcement for AI Agents

    Kontext

  15. SIT-Eval: Situation-Based Interactive Turn-by-Turn Evaluation of Conversational Agents

    Oct 16, 2026Ravil Mussabayev, Saveliy Zavertyaev, Dmitriy Parpulov +1LLM Agent EvaluationMulti-Turn Dialogue Evaluation

    AI Research Lab, Satbayev University, Almaty, Kazakhstan · Sber AI Lab, Moscow, Russia

  16. Ignored, Not Refused: What Low Prompt-Injection Success Rates Hide in Small Tool-Using Agents

    Oct 16, 2026Farhan Zarif, Kazi Shahid Ahmed Galib, Synthia Akter Promi +4Prompt InjectionIndirect Prompt Injection

    Department of Computer Science and Engineering, BRAC University

  17. Language Is a Security Surface for Agent Guards

    Oct 15, 2026Hasan Demirkiran, Jens ErnstbergerLLM GuardrailsLLM Agent Safety

    Kontext

  18. Shortcut Sensitivity in Agent-Safety Evaluations

    Oct 15, 2026Hasan Demirkiran, Jens ErnstbergerAI Safety EvaluationAI Agent Auditing

    Kontext

  19. Auditing Escalation Reliability in Healthcare LLM Agents: A Perturbation-Controlled Benchmark of Workflow-State Failures

    Oct 15, 2026Zihao Zheng, Junyi Yao, Baichuan Li +1LLM Agent ReliabilityAI Agent Safety Benchmarks

    Washington University in St. Louis · Southern Methodist University

  20. AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety

    Oct 14, 2026Chenxu Wang, Xiaoyu Wen, Han Qi +4LLM Agent SafetyAdversarial Training

    Shanghai Artificial Intelligence Laboratory · Fudan University · Shanghai Jiaotong University +1

  21. SWE-MISBEHAVIOR: Benchmarking and Mitigating Boundary Violations in Software Coding Agents

    Oct 13, 2026Do Xuan Trong, Duc Anh Nguyen, Do Xuan Long +12AI Coding AgentsAI Agent Safety

    National University of Singapore · Vietnam National University, Hanoi · University of Virginia, USA +4

  22. LatentRiskBench: Measuring State-Dependent Risks in Enterprise Agents

    Oct 13, 2026Jingyue Cong, Estrid He, Zhuang Li +5AI Agent SafetyAI Safety Evaluation

    RMIT University, Melbourne, Australia · Independent · YepAI, Melbourne, Australia +1

  23. Same Operation, Different Permission: Evaluating Whether Guardrails Honor Authorization

    Oct 12, 2026Huihao Jing, Shaojin Chen, Wenbin Hu +5LLM GuardrailsTool-Use Authorization

    HKUST

  24. AuthGateBench: Benchmarking LLMs as Authorization Gate

    Oct 12, 2026Zhixiong Zhuang, Jin Jia, Zhiwei Shang +1Tool Access Control for LLM AgentsAI Agent Safety Benchmarks

    Huawei Technologies Ltd. · Sun Yat-sen University

  25. RetryBench: Separating Retry Permission from Recovery Outcomes in GPT Tool Use

    Oct 11, 2026You Liu, Yue Liu, Quanchao LuTool UseTool Access Control for LLM Agents

  26. Can agents whose sole duty is to keep humans alive be hijacked to kill? Delayed multi-turn derived-action trap attacks in simulated high-stakes environments

    Oct 10, 2026Josef A. HabdankPrompt Injection Attacks on AI AgentsPrompt Injection Attacks

    DXC Technology

  27. EAIR: Interaction Risks in Enterprise LLM Multi-Agent Systems

    Oct 10, 2026Yuchen Ma, Swadheen Shukla, Paulo Salem +5Enterprise AI AgentsMulti-Agent LLM Systems

    LMU Munich · Microsoft · University of Notre Dame

  28. APEX-Harm: Evaluating Agent Safety in Shared Professional Workspaces

    Oct 9, 2026Ziyi Yang, Adel Bibi, Guohao Li +2AI Agent SecurityLLM Agent Security

    King Abdullah University of Science and Technology · University of Oxford

  29. DRYLAB-Bench: Benchmarking the Biosecurity Risk of Agentic LLM Mutation Design

    Oct 9, 2026Yuzhen Wang, Chunxiao Li, Baichen Le +6LLM SecurityAI Agent Safety Benchmarks

    Shanghai Artificial Intelligence Laboratory · Fudan University

  30. Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents

    Oct 8, 2026Jiaming Zhang, Xuan Wang, Fuyao Zhang +3AI Agent MonitoringLatent World Models

    Nanyang Technological University · Institute of Science Tokyo · Sony AI