AI Agent Safety Benchmarks

Follow this topic Ask about this topic

AI agent safety benchmarks test whether systems using tools, software, or physical devices follow instructions, respect permissions, avoid harmful side effects, and know when to abstain. Common approaches use executable sandboxes, simulated workplaces and embodied environments, multi-turn red teaming, and analysis of full action trajectories rather than final answers alone. Evaluations increasingly measure safety alongside task success, usability, authorization, and recovery, while runtime guards and deterministic checks are tested as possible safeguards. Results depend heavily on the model, harness, environment, and scoring method, and fragmented benchmarks make safety rankings difficult to compare; a central challenge is capturing long-horizon and real-world risks without confusing safe behavior with inability to act.

Latest papers

220 papers46 papers in the last four weeks, up 53% on the four weeks before. 0.3% of all new papers.

When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Models
May 27, 2026Dasol Choi, Alex KwonLLM Safety Benchmarks Counterfactual Evaluation
AIM Intelligence · Independent Researcher
Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems
May 26, 2026Aman Priyanshu, Supriti Vijay, Esha PahwaAI Agent Safety Benchmarks Multi-Agent LLM Systems
Foundation AI · USA · Corvic AI
Position: AI Safety Requires Effective Controllability
May 26, 2026Yige Li, Yunhao Feng, Jun SunLLM Safety Alignment AI Agent Safety
Singapore Management University · Ant Group
Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety
May 21, 2026Piercosma Bisconti, Matteo Prandi, Federico Pierucci +11LLM Safety Benchmarks Computer-Use Agent Benchmarks
Icaro Foundation · Sapienza University of Rome · Sant’Anna School of Advanced Studies +6
Measuring Security Without Fooling Ourselves: Why Benchmarking Agents Is Hard
May 21, 2026Sahar Abdelnabi, Chris Hicks, Konrad Rieck +1AI Agent Evaluation AI Agent Security
ELLIS Institute Tübingen & MPI-IS & Tübingen AI Center, Germany · The Alan Turing Institute, London, UK · BIFOLD & Technische Universität Berlin, Germany +1
Measuring Safety Alignment Effects in Autonomous Security Agents
May 19, 2026Isaac David, Arthur GervaisLanguage Model Safety Evaluation LLM Safety Alignment
University College London
Overeager Coding Agents: Measuring Out-of-Scope Actions on Benign Tasks
May 18, 2026Yubin Qu, Ying Zhang, Yanjun Zhang +4Tool Access Control for LLM Agents Computer-Use Agent Benchmarks
Griffith University · Wake Forest University · Nanyang Technological University +2
LivePI: More Realistic Benchmarking of Agents Against Indirect Prompt Injection
May 18, 2026Lei Zhao, Abhay Bhaskar, Edgar DobribanAI Agent Security AI Agent Safety
University of Pennsylvania
Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback
May 17, 2026Lecheng Yan, Ruizhe Li, Xicheng Han +5AI Agent Reliability LLM Agent Evaluation
Southern University of Science and Technology · University of Science and Technology of China · Zhejiang University +2
ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agents
May 17, 2026Udari Madhushani Sehwag, Zhengyang Shan, Heming Liu +3LLM Agent Security AI Agent Security Benchmarks
Scale AI · 2Boston University · 4Human Frontier Collective +3
Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks
May 14, 2026Karthik Raghu Iyer, Yazdan Jamshidi, Nicholas Bray +1LLM Safety Benchmarks Benchmark Auditing
Palo Alto Networks
History Anchors: How Prior Behavior Steers LLM Decisions Toward Unsafe Actions
May 13, 2026Alberto G. Rodríguez SalgadoLLM Safety Evaluation AI Agent Safety Benchmarks
Independent Researcher
AgentTrap: Measuring Runtime Trust Failures in Third-Party Agent Skills
May 13, 2026Haomin Zhuang, Hanwen Xing, Yujun Zhou +5AI Agent Security LLM Agent Security
University of Notre Dame · University of Southern California · LMU Munich +1
Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
May 12, 2026Hao Wang, Hanchen Li, Qiuyang Mang +3Computer-Use Agent Benchmarks Reward Hacking
UC Berkeley
SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces
May 12, 2026Chang Jin, An Wang, Zeming Wei +7LLM Agent Security AI Agent Security Benchmarks
1Shanghai AI Laboratory · 2Peking University · 3East China Normal University
LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments
May 11, 2026Chiyu Zhang, Huiqin Yang, Bendong Jiang +8AI Agent Security Benchmarks LLM Jailbreak Attacks
Nanjing University of Aeronautics and Astronautics · Collaborative lnnovation Center of Novel Software Technology and Industrialization · Zhejiang University
Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values
May 11, 2026Haonan Dong, Qiguan Feng, Kehan Jiang +3AI Alignment AI Agent Evaluation
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University · Key Laboratory of Machine Perception (Ministry of Education), Peking University · School of Software and Microelectronics, Peking University +1
FORTIS: Benchmarking Over-Privilege in Agent Skills
May 9, 2026Shawn Li, Chenxiao Yu, Han Wang +8LLM Agent Security AI Agent Security Benchmarks
University of Southern California · University of Illinois at Urbana-Champaign · 3Adobe Research +3
Safe, or Simply Incapable? Rethinking Safety Evaluation for Phone-Use Agents
May 8, 2026Zhengyang Tang, Yi Zhang, Chenxin Li +18Mobile GUI Agents Computer-Use Agents
Tencent Hunyuan · The Chinese University of Hong Kong, Shenzhen · Tsinghua University
Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors
May 7, 2026Jonas Wiedermann-Möller, Leonard Dung, Maksym AndriushchenkoAI Agent Evaluation AI Agent Safety
Universität Bielefeld · Ruhr-Universität Bochum · ELLIS Institute Tübingen +2
MANTRA: Synthesizing SMT-Validated Compliance Benchmarks for Tool-Using LLM Agents
May 7, 2026Ashwani Anand, Ivi Chatzi, Ritam Raha +1Computer-Use Agent Benchmarks AI Agent Reliability
Max Planck Institute for Software Systems, Kaiserslautern, Germany
Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges
May 7, 2026Shihao Weng, Yang Feng, Xiaofei XieLLM-as-a-Judge AI Agent Safety
Nanjing University Nanjing, China · Singapore Management University Singapore
AgentTrust: Runtime Safety Evaluation and Interception for AI Agent Tool Use
May 6, 2026Chenglin YangAI Agent Security AI Agent Security Benchmarks
Independent Researcher
MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents
May 5, 2026Jonathan Steinberg, Oren GalLanguage Model Safety Evaluation AI Coding Agents
Swarms & AI Lab (SAIL), University of Haifa
Enhancing Agent Safety Judgment: Controlled Benchmark Rewriting and Analogical Reasoning for Deceptive Out-of-Distribution Scenarios
May 5, 2026Zuoyu Zhang, Yancheng ZhuAnalogical Reasoning for Language Models AI Agent Safety
Shenzhen University, Shenzhen, China.
Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
May 3, 2026Kunvar ThamanAI Agent Reliability Reward Hacking
Independent Researcher
A Comparative Evaluation of AI Agent Security Guardrails
Apr 27, 2026Qi Li, Jiu Li, Pingtao Wei +8LLM Guardrails AI Agent Security
Beijing Caizhi Tech, Beijing, China
SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Models
Apr 21, 2026Josue Torres-Fonseca, Naihao Deng, Yinpei Dai +5Safety Evaluation of VLA Models LLM Safety Benchmarks
University of Michigan · Boise State University
Owner-Harm: A Missing Threat Model for AI Agent Safety
Apr 20, 2026Dongcheng Zhang, Yiqing JiangAI Agent Security AI Agent Safety
BlueFocus Communication Group · Tongji University
HealthCraft: A Reinforcement Learning Safety Environment for Emergency Medicine
Apr 18, 2026Brandon DentHealthcare AI Agent Benchmarks
GOATnote Inc.

May 27, 2026J/K move · Enter open · S save

When Context Flips, Safety Breaks: Diagnosing Brittle Safety in Aligned Language Models

Dasol Choi, Alex Kwon

AIM Intelligence · Independent Researcher

LLM Safety Benchmarks Counterfactual Evaluation Language Model Safety Evaluation LLM Guardrails AI Agent Safety Benchmarks

Open paper AskPDF arXiv

Safety benchmark scores provide incomplete evidence of deployment readiness: aligned language models often adhere to rigid rules even when a situational update flips which action is safe. We term this failure brittle safety. To diagnose it, we introduce context-flip evaluation, testing 12 models across a safety benchmark (PacifAIst) and two commonsense controls using paired variants where the nominally safe action produces harm. Three findings emerge. First, brittle safety is safety-specific: all 12 models exhibit a safety-commonsense gap (mean +17.4 pp). Baseline accuracy fails to predict brittleness: among models above 90% baseline accuracy, brittleness rates range from 13.7% to 90.0%. Second, failures stem from policy override rather than miscomprehension: despite acknowledging the context change in every case, models persist via three distinct mechanisms that vary by update type and model family. Third, on a hand-audited probe of catastrophic consequence-flip scenarios, standard action-level guardrails catch none, while a state-aware validator catches all without false alarms on correct interventions. This indicates that action-level content moderation is systematically blind to consequence-flips, motivating state-aware architectural alternatives. We release our protocol, perturbed benchmarks, and deployment probe.