AI Agent Safety Benchmarks

Follow this topic Ask about this topic

Momentum

22 papers in the last four weeks, up 83% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 143

All topics

Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents
Oct 8, 2026Jiaming Zhang, Xuan Wang, Fuyao Zhang +3AI Agent Monitoring Latent World Models
Nanyang Technological University · Institute of Science Tokyo · Sony AI
Safe Actions Alone Do Not Ensure Safe Agents: Identifying Unfulfilled Obligations with Guard Models
Oct 8, 2026Youwei Feng, Yitong Zhang, Yuetong Liu +1LLM Agent Safety LLM Guardrails
Tsinghua University, China · College of AI, Tsinghua University, China
Workerville: Towards an Organizational Behavior Account of Agent Safety
Oct 8, 2026Hanjun Luo, Junting Mao, Yuhan Lu +5LLM Agent Safety Agentic AI
New York University · New York University Abu Dhabi · McGill University +1
RT-Safe: Benchmarking Agent Safety in Real-Time Embodied Environment
Oct 7, 2026Tianruo Rose Xu, Jiawei Ren, Yichi Yang +2AI Agent Safety Safe Robot Navigation
Cornell University · University of California, San Diego
DecepEval: A Benchmark for Evaluating Deception in LLM Agents
Oct 6, 2026Yiming Xu, Hongyue Yu, Beihua Yang +8Deception in Language Models LLM Agent Evaluation
Xi’an Jiaotong University · University of Virginia · Griffith University +2
From Evidence to Action: How Tool-Using Agents Fail
Oct 6, 2026Hongzhan Lin, Shidong Cao, Ziyang Luo +3Agent Failure Analysis Evidence-Grounded Reasoning
National University of Singapore · Hong Kong Baptist University · Amazon Web Services +1
BazaarBench: Delegation Safety in Decentralized C2C Marketplaces Run by LLM Agents
Oct 5, 2026Ziyan Wang, Shuqing Shi, James Oldfield +5AI Agent Safety Benchmarks LLM Agent Safety
King’s College London · Institute for Decentralized AI · University of Oxford +1
Inspect Robots: Evaluating the Capabilities and Safety of Embodied AI
Oct 5, 2026Christopher Leet, Achu Menon, Sravanthi Machcha +12AI Agent Evaluation Robotic Policy Evaluation
Robocurve · University of Southern California · San Jose State University +7
Towards a Unified Misuse Monitoring Benchmark
Oct 5, 2026Aniruddh Pramod, James Oldfield, Adel BibiAI Agent Security AI Agent Monitoring
University of Oxford
Benchmarking Jailbreak Guardrails for Embodied Agents
Oct 5, 2026Xunguang Wang, Qingyue Wang, Yuguang Zhou +3LLM Guardrails AI Agent Safety
Department of Computer Science and Engineering The Hong Kong University of Science and Technology Clear Water Bay, Kowloon, Hong Kong · School of Information Renmin University of China Beijing, China
The Backdrop Exposes What the World Around an Agent Costs It
Sep 29, 2026Nusrat Jahan Lia, Shubhashis Roy DiptaAI Agent Reliability Prompt Injection Attacks on AI Agents
University of Dhaka · University of Maryland, Baltimore County
NAQD Env: A benchmark for selective withdrawal in language agents
Sep 29, 2026Mohamed AbouzahraLLM Agent Evaluation AI Agent Safety Benchmarks
California State University, Monterey Bay
CheatBench: Measuring Reward Gaming in AI Agents
Sep 28, 2026Long Phan, Stephen K. Yang, Jason J. Lim +10Reward Hacking AI Agent Benchmarks
Center for AI Safety · Work done while at the Center for AI Safety.
SCOUT: Synergizing Reasoning and Tool-Use for Computer-Use Safety
Sep 28, 2026Jianxing Chen, Xiao Yu, Shipra Agrawal +1AI Agent Safety AI Agent Safety Benchmarks
Columbia University
SEABench: Benchmarking Endogenous Misalignment In Self-Evolving Agents
Sep 28, 2026Saswat Das, Parvati Viswanathan, Daniel Donnelly +3AI Agent Safety AI Agent Safety Benchmarks
University of Virginia · ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems +1
PROACT-Agent: Progressive Runtime Oversight and Active Circuit-breaking for Real-Time Safety
Sep 28, 2026Ding Jia, Wei Liu, Xianglong Du +5LLM Guardrails Runtime Enforcement for AI Agents
State Key Laboratory of Intelligent Vehicle Safety Technology, Changan Automobile · Independent Researcher
AdaGuard: An Adaptive Guard Model with User-defined Policies
Sep 28, 2026Yunhao Feng, Yifan Ding, Yuxiang Xie +4LLM Guardrails AI Agent Safety
National University of Defense Technology · Fudan University
AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents
Sep 27, 2026Tianzhuo Yang, Zirui Mi, Yantao Huang +4LLM Agent Evaluation LLM Agents
Peking University · Beijing Academy of Artificial Intelligence
Instrumental Monitor Evasion Emerges Under Ordinary Task Pressure
Sep 24, 2026David Schmotz, Derck Prinzhorn, Luca Beurer-Kellner +3Runtime Enforcement for AI Agents AI Agent Safety Benchmarks
ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems · Tübingen AI Center +3
PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety
Sep 23, 2026Jiapeng Sun, Yujin Zhou, Han Zhu +4AI Agent Safety Benchmarks Inference-Time Intervention
The Hong Kong University of Science and Technology, Hong Kong, China · Peking University, Beijing, China
ClashBench: Conflicts Leading Agents to Seize and Harm
Sep 17, 2026Yuejin Xie, Yu Li, Dadi Guo +6AI Agent Security AI Agent Safety
Tsinghua University · Shanghai AI Lab · Fudan University +2
PACT: Can Enterprise AI Assistants Be Trusted Under Pressure?
Sep 16, 2026Mika Okamoto, Ansel Kaplan ErolAI Agent Auditing LLM Agent Evaluation
Georgia Institute of Technology · Decagon AI · Baseten
Collective Loss of Control in LLM Agent Systems: An Epidemic Account of Mutation, Contagion, and Recovery
Sep 16, 2026Xiangfan Wu, Zonghao Ying, Huiyu Wu +4Multi-Agent LLM Systems AI Agent Safety
Tencent Zhuque Lab
Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems
Sep 15, 2026Deepak Akkil, Tamer Abuelsaad, Karthik Vikram +5Multi-Agent LLM Systems Long-Horizon Agent Evaluation
Emergence AI
RiskChainBench: A Benchmark for Obfuscated Platform Message Restoration and Evidence-Grounded Web Investigation
Sep 15, 2026ZhuoXin Liu, Zhiming Ma, Ying Zhang +10Web Agent Benchmarks AI Agent Safety Benchmarks
Baidu · SmartFlowAI · JD Technology +3
BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agents
Sep 14, 2026Sadia Asif, Mohammad Mohammadi Amiri, Momin Abbas +2Long-Horizon Agent Evaluation AI Agent Safety Benchmarks
Rensselaer Polytechnic Institute · IBM Research
HazardAuditor: From Executable Threats to Safer Computer-Use Agents
Sep 14, 2026Yunhao Feng, Ruixiao Lin, Ming Wen +5Language Model Safety Evaluation AI Agent Auditing
Ant Group · Hunan Institute of Advanced Technology · Zhejiang University +3
Guardrailed Meta-Agent Loops: Stress-Testing Policy Pinning, Budget Bounds, and Crash Recovery
Sep 14, 2026Qinzhen Ma, Jialin WuAgent Failure Analysis AI Agent Reliability
Rice University · University of California, San Diego
MemRiskBench: Trace-Aware Risk-Preserving Evaluation for Long-Horizon LLM Agents
Sep 14, 2026Jianhua Jiang, Dongbo Yuan, Weihua LiLong-Horizon Agent Evaluation AI Agent Safety Benchmarks
School of Artificial Intelligence and Computer Science, Jilin University of Finance and Economics, 130117 Changchun, China · Jilin Province Key Laboratory of Fintech, Jilin University of Finance and Economics, 130117 Changchun, China · School of Engineering, Computer and Mathematical Sciences, Auckland University of Technology, Auckland, New Zealand
ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs
Sep 12, 2026Yizhan Li, Jianxin You, Mengyang Xiong +5Physical Reasoning Robotics Simulation
Université de Montréal · Mila – Quebec AI Institute · McGill University +2

Oct 8, 2026, cs.CVJ/K move · Enter open · S save

Right Screen, Wrong Transition: World Models as Verifiers for GUI Agents

Jiaming Zhang, Xuan Wang, Fuyao Zhang, Yang Cao+2

Nanyang Technological University · Institute of Science Tokyo · Sony AI

AI Agent Monitoring Latent World Models Action-Conditioned World Models AI Agent Safety Benchmarks

Open paper AskPDF arXiv

A login screen that appears after a tap on Sign in is expected; the same screen after a tap on View order is an attack. For GUI agents, safety is therefore a property of the transition rather than of the screen, and a monitor that inspects only screens can be defeated by reusing a legitimate one. Judging a transition requires an expectation of what should have followed the action. Existing GUI world models provide one, but they output it as text, code, or images, so checking it against the observed screen requires a second model to judge the two. We argue that a world model meant for verification should instead predict in the space in which observations are encoded, and present LGWM, a decoder-free, action-conditioned world model that predicts the representation of the next screen directly, trained without semantic annotation on 1.85M real GUI transitions. Verification reduces to a vector comparison, and the same signal reveals whether a mismatch is harmful. We evaluate on RSWT-BENCH, a diagnostic where each credential screen appears under both a legitimate and a hijacked transition, so detectors that see only the screen are at chance by construction. The training-free score reaches 0.987 AUC at 17 ms per decision, on par with the strongest closed-source VLMs and about ten AUC points above generative GUI world models at over three orders of magnitude lower latency. The residual direction reaches 0.953 AUC at separating harmful from benign violations, where prompted VLMs are near chance. Further analyses show that the prediction is a usable future state rather than an anomaly score. World models have mostly served as simulators or planners; our results point to a third role, verification, for which predicting in representation space is the natural design.

Momentum

Related topics

Latest papers 143