AI Agent Safety

Follow this topic Ask about this topic

Momentum

22 papers in the last four weeks, up 144% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 207

All topics

What Benchmarks Don't Measure: The Case for Evaluating Abstention Competence in Autonomous Agents
Jun 1, 2026Victor Ojewale, Suresh VenkatasubramanianAgent Evaluation AI Agent Safety
Brown University Providence, Rhode Island, USA
Toward a Modular Architecture for Embedded AI Agent Systems at the Edge
Jun 1, 2026Marcus Rüb, Michael GerhardsAI Agent Safety AI Agent Governance
Foresthub.Ai Villingen, Germany · Deloitte Consulting DÜsseldorf, Germany
SPADE-Bench: Evaluating Spontaneous Strategic Deception in Agents via Plan-Action Divergence
Jun 1, 2026Yuyan Bu, Haowei Li, Qirui Zheng +7LLM Agent Evaluation AI Agent Safety
Beijing Academy of Artificial Intelligence · University of Science and Technology of China · Peking University +2
BraveGuard: From Open-World Threats to Safer Computer-Use Agents
May 31, 2026Yunhao Feng, Xiaohu Du, Xinhao Deng +13LLM Guardrails AI Agent Safety
1Fudan University · Ant Group · 8ShanghaiUniversity +5
TRACE: Trajectory Risk-Aware Compression for Long-Horizon Agent Safety
May 30, 2026Zhepei Hong, Lin Wang, Liting Li +5AI Agent Safety LLM Safety Evaluation
University of Science and Technology of China · National University of Singapore · South China Normal University
ROGUE: Evaluating Corrigibility Failures in Frontier Computer-Use Agents
May 29, 2026Jeremy Tien, Abishek Anand, Yu-Rou Tuan +3Agent Reliability Computer-Use Agents
Carnegie Mellon University
EMBGuard: Constructing Hazard-Aware Guardrails for Safe Planning in Embodied Agents
May 29, 2026Dongwook Choi, Taeyoon Kwon, Bogyung Jeong +6LLM Guardrails AI Agent Safety
Department of Artificial Intelligence, Yonsei University · Department of Biomedical Engineering and the Department of Intelligent Precision Healthcare Convergence, Sungkyunkwan University · Independent Researcher
Gram: Assessing sabotage propensities via automated alignment auditing
May 28, 2026David Lindner, Victoria Krakovna, Sebastian FarquharAI Coding Agents AI Agent Auditing
1Google DeepMind
AI Loss of Control Incident Management: Response & Resilience
May 28, 2026Ross GruetzemacherAI Risk Management AI Agent Safety
Wichita State University · Transformative Futures Institute
AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security
May 28, 2026Dongrui Liu, Yu Li, Zhonghao Yang +47LLM Safety Alignment AI Agent Security
Realistic honeypot evaluations for scheming propensity
May 28, 2026Victoria Krakovna, David Lindner, Lewis Ho +2LLM Agent Evaluation AI Agent Safety
1Google DeepMind
Training Deliberative Monitors for Black-Box Scheming Detection
May 28, 2026Aditya Sinha, Akshat Naik, Victor Gillioz +5AI Agent Safety AI Agent Monitoring
1Independent · 2MATS Research† · 4Apollo Research +1
The Importance of Out-of-Band Metadata for Safe Autonomous Agents: The Redpanda Agentic Data Plane
May 27, 2026Tyler Akidau, Tyler Rockwood, Johannes Brüderl +1AI Agent Security AI Agent Safety
Redpanda · USA / Germany
Calibrating Conservatism for Scalable Oversight
May 27, 2026William Overman, Mohsen BayatiScalable Oversight AI Agent Safety
Stanford Graduate School of Business, Stanford University, Stanford, CA, USA.
Intelligence as Managed Autonomy: Failure, Escalation, and Governance for Agentic AI Systems
May 26, 2026Srini RamaswamyAgent Failure Analysis AI Agent Reliability
DNRS.ai USA
Reasoning and Planning with Dynamically Changing Norms
May 26, 2026Taylor Olson, Roberto Salas-Damian, Kenneth D. ForbusSymbolic Planning AI Agent Safety
University of Iowa · Northwestern University
Position: AI Safety Requires Effective Controllability
May 26, 2026Yige Li, Yunhao Feng, Jun SunLLM Safety Alignment AI Agent Safety
Singapore Management University · Ant Group
Foundations of a Time-Consistent Counterfactual Actuarial Runtime for Autonomous AI Agents
May 26, 2026Hao-Hsuan ChenCounterfactual Evaluation AI Risk Management
Department of Risk Management and Insurance National Chengchi University
Retrying vs Resampling in AI Control
May 25, 2026James Lucassen, Adam KaufmanAI Agent Safety AI Agent Monitoring
Redwood Research
ProofAgent Harness: Open Infrastructure for Adversarial Evaluation of AI Agents
May 22, 2026Fouad BousetouaneAI Agent Reliability AI Agent Auditing
The University of Chicago, USA
Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety
May 21, 2026Piercosma Bisconti, Matteo Prandi, Federico Pierucci +11LLM Safety Benchmarks Computer-Use Agent Benchmarks
Icaro Foundation · Sapienza University of Rome · Sant’Anna School of Advanced Studies +6
LLM Agents Make Collective Belief Dynamics Programmable: Challenges and Research Directions
May 19, 2026Xin He, Junxi Shen, Yuchen Mou +4Opinion Dynamics AI Agent Safety
Centre for Frontier AI Research, Agency for Science, Technology and Research (A*STAR), Singapore · Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore · College of Design and Engineering, National University of Singapore, Singapore +1
Agent Meltdowns: The Road to Hell Is Paved with Helpful Agents
May 18, 2026Rishi Jha, Harold Triedman, Arkaprabha Bhattacharya +1Agent Failure Analysis AI Agent Reliability
Department of Computer Science Cornell University
Surviving the Unseen: Predictive Defense for Novel Multi-Turn Multimodal Attacks
May 18, 2026Doohee YouAdversarial Attacks on VLMs Multimodal Robustness
Trust and Safety, Google
LivePI: More Realistic Benchmarking of Agents Against Indirect Prompt Injection
May 18, 2026Lei Zhao, Abhay Bhaskar, Edgar DobribanAI Agent Security AI Agent Safety
University of Pennsylvania
Ethical Hyper-Velocity (EHV): A Hardware-Rooted Zero-Trust Runtime Enforcement Architecture for Agentic AI Systems
May 18, 2026Riddhi Mohan SharmaAI Agent Safety Runtime Enforcement for AI Agents
IAM Architect & Non-Human Identity Governance Enterprise Identity, M&A Integration, Regulated Environments
Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security
May 17, 2026Jinhu Qi, Muzhi Li, Jiahong Liu +9AI Agent Reliability AI Agent Evaluation
Faculty of Engineering, Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China. · Shanghai Academy of AI for Science, Shanghai, China.
TClone: Low-Latency Forking of Live GUI Environments for Computer-Use Agents
May 17, 2026Yutong Huang, Vikranth Srivatsa, Alex Asch +2Computer-Use Agents AI Agent Safety
University of California, San Diego · University of California, San Diego and GenseeAI
LiSA: Lifelong Safety Adaptation via Conservative Policy Induction
May 14, 2026Minbeom Kim, Lesly Miculicich, Bhavana Dalvi Mishra +6Continual Learning for LLM Agents LLM Guardrails
Google Cloud AI Research · Google · Seoul National University
Conformity Generates Collective Misalignment in AI Agents Societies
May 11, 2026Giordano De Marzo, Alessandro Bellina, Claudio Castellano +2Opinion Dynamics AI Agent Safety
University of Konstanz, Konstanz, Germany · Centro Ricerche Enrico Fermi, Rome, Italy · Complexity Science Hub, Vienna, Austria +5

Jun 1, 2026, cs.AIJ/K move · Enter open · S save

What Benchmarks Don't Measure: The Case for Evaluating Abstention Competence in Autonomous Agents

Victor Ojewale, Suresh Venkatasubramanian

Brown University Providence, Rhode Island, USA

Agent Evaluation AI Agent Safety AI Agent Benchmarks AI Agent Safety Benchmarks LLM Abstention

Open paper AskPDF arXiv

Benchmarks for autonomous agents measure whether agents complete tasks, yet this framing is systematically blind to whether an agent should have proceeded at all. Agents trained under human-feedback objectives develop a structural tendency to proceed even when they lack the inputs, evidence, or authorization to act safely, a disposition we term compliance bias, because both the reward signal and the benchmark scoring regime treat proceeding as the correct default regardless of whether the preconditions for safe action are present. We make three contributions. We first show that compliance bias originates in reward hacking within human-feedback pipelines and is entrenched by prominent agent benchmarks, which either penalize agents for pausing or are architecturally unable to distinguish a principled pause from a silent failure. We then introduce a three-gap taxonomy of abstention-warranted scenarios, covering specification gaps where required information is absent, verification gaps where world state cannot be confirmed, and authority gaps where explicit authorization has not been given, which together provide a principled basis for constructing abstention-aware agent benchmarks. Finally, we propose abstention evaluation protocols (Safety Rate, Usability Rate, and Informed Refusal Rate) and report preliminary results across 144 enterprise agent scenarios and five model families, in which a runtime-enforced abstention mechanism achieves up to 89.2% hazardous-action blocking and 87.5% usability on authorized scenarios, demonstrating that the safety--usability tradeoff is tunable rather than inherent and that its shape varies substantially across model families. We treat this as preliminary work and offer the taxonomy and composite metrics as a starting point for further conversations.

Momentum

Related topics

Latest papers 207