AI Agent Safety

Ask about this topic

Overview

AI agent safety asks how to keep systems that plan over multiple steps and use tools, memory, or other agents within authorized and harmless bounds. Researchers test computer-use, coding, web, and embodied agents with prompt-injection, misuse, and long-horizon trajectory benchmarks, measuring executed effects as well as final answers. Approaches include red-teaming and monitoring, safety fine-tuning and refusal training, and runtime controls such as least-privilege access, pre-execution policy gates, sandboxing, human approval, and formal verification. A central challenge is that safety can fail through ambiguous goals, changing context, tool or agent interactions, and pressure over time, while stronger restrictions can block legitimate tasks and assurances may not transfer to deployment.

Latest papers

CardsList

330 papers47 papers in the last four weeks, up 68% on the four weeks before. 0.3% of all new papers.

  1. AI Organizations are More Effective but Less Aligned than Individual Agents

    Apr 11, 2026Judy Hanwen Shen, Daniel Zhu, Siddarth Srinivasan +5Multi-Agent SystemsAI Alignment

    Constellation Institute · MATS Program · Anthropic

  2. CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI Automation

    Apr 10, 2026Yushi Feng, Junye Du, Qifan Wang +5AI Agent SafetyMobile GUI Agents

    The University of Hong Kong · The Chinese University of Hong Kong · The University of Tokyo

  3. Harnessing Embodied Agents: Runtime Governance for Policy-Constrained Execution

    Apr 9, 2026Xue Qin, Simin Luan, John See +3Runtime Enforcement for AI AgentsAI Agent Governance

    School of Software, Harbin Institute of Technology, Harbin, China. · School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China. · School of Mathematical and Computer Sciences, Heriot-Watt University, Malaysia Campus, Putrajaya, Malaysia. +2

  4. Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw

    Apr 6, 2026Zijun Wang, Haoqin Tu, Letian Zhang +11AI Agent SecurityAI Agent Safety

    UC Santa Cruz · NUS · Tencent +3

  5. Implementing surrogate goals for safer bargaining in LLM-based agents

    Apr 6, 2026Caspar Oesterheld, Maxime Riché, Filip Sondej +2AI Agent SafetyLLM Agent Safety

  6. AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agents

    Apr 3, 2026Yunhao Feng, Yifan Ding, Yingshui Tan +6LLM Safety BenchmarksComputer-Use Agent Benchmarks

    Alibaba Group · Fudan University · Hunan Institute of Advanced Technology +2

  7. I must delete the evidence: AI Agents Explicitly Cover up Fraud and Violent Crime

    Apr 2, 2026Thomas RivasseauAI Agent SafetyAI Ethics

    Data Mining and Security Lab, School of Information Studies, McGill University.

  8. ClawSafety: "Safe" LLMs, Unsafe Agents

    Apr 1, 2026Bowen Wei, Yunbei Zhang, Jinhao Pan +5AI Agent SecurityAI Agent Safety

    George Mason University · Tulane University · Oak Ridge National Laboratory +1

  9. UK AISI Alignment Evaluation Case-Study

    Apr 1, 2026Alexandra Souly, Robert Kirk, Jacob Merizian +2AI Agent SafetyAI Safety Evaluation

  10. The Persistent Vulnerability of Aligned AI Systems

    Mar 31, 2026Aengus LynchAI Agent SafetyAI Alignment

    Department of Computer Science, University College London

  11. Empirical Validation of the Classification-Verification Dichotomy for AI Safety Gates

    Mar 31, 2026Arsenios ScrivensAI Agent SafetyAI Safety Evaluation

  12. Peer-Preservation in Frontier Models

    Mar 30, 2026Yujin Potter, Nicholas Crispino, Vincent Siu +2AI Agent SafetyLLM Safety Evaluation

    University of California, Berkeley · University of California, Santa Cruz

  13. SafetyDrift: Predicting When AI Agents Cross the Line Before They Actually Do

    Mar 28, 2026Aditya Dhodapkar, Farhaan PishoriAI Agent SafetyLLM Agent Safety

    Rensselaer Polytechnic Institute Department of Computer Science · Santa Clara University Department of Engineering

  14. Evaluating a Multi-Agent Voice-Enabled Smart Speaker for Care Homes: A Safety-Focused Framework

    Mar 24, 2026Zeinab Dehghani, Rameez Raja Kureshi, Koorosh Aslansefat +6Voice Agent EvaluationConversational Agents

    University of Hull, UK · University of Southampton, UK · Connexin, Hull, UK +1

  15. Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models

    Mar 24, 2026Massimiliano Pappa, Luca Romani, Valentino Sacco +5Efficient Inference for World Action ModelsAI Agent Safety

    Sapienza University of Rome, Italy · Inria, Univ. Grenoble Alpes, CNRS, LJK

  16. AgentWall: A Runtime Safety Layer for Local AI Agents

    Mar 24, 2026Ashwin AravindAI Agent SafetyRuntime Enforcement for AI Agents

  17. NuHF Claw: A Risk Constrained Cognitive Agent Framework for Human Centered Procedure Support in Digital Nuclear Control Rooms

    Mar 23, 2026Xingyu Xiao, Jiejuan Tong, Jun Sun +4AI Agent SafetyHuman-in-the-Loop AI

    Institute of Nuclear and New Energy Technology, Tsinghua University, Beijing, 100084, China · National Key Laboratory of Human Factors Engineering, Beijing, 100094, China · Institute of Software, Chinese Academy of Sciences, Beijing, 100045, China

  18. Active Inference Agency Formalization, Metrics, and Convergence Assessments

    Mar 22, 2026Eduard KapelkoAI Agent SafetyActive Inference

  19. Emergent Formal Verification: How an Autonomous AI Ecosystem Independently Discovered SMT-Based Safety Across Six Domains

    Mar 22, 2026Octavian UntilaFormal VerificationAI Agent Safety

    Aisophical SRL, Bucharest, Romania

  20. Session Risk Memory (SRM): Temporal Authorization for Deterministic Pre-Execution Safety Gates

    Mar 22, 2026Florin Adrian ChitanAI Agent SafetyRuntime Enforcement for AI Agents

    Independent Researcher, ILION Project

  21. The Intelligent Disobedience Game: Formulating Disobedience in Stackelberg Games and Markov Decision Processes

    Mar 22, 2026Benedikt Hornig, Reuth MirskyAI Agent SafetyShared Autonomy

    Independent Researcher, Colorado Springs, United States · Tufts University, Medford, United States

  22. Toward Reliable, Safe, and Secure LLMs for Scientific Applications

    Mar 18, 2026Saket Sanjeev Chaturvedi, Joshua Bergerson, Tanwi MallickLLM SafetyAI Agent Safety

    Argonne National Laboratory, Lemont, 60439, USA

  23. What if Pinocchio Were a Reinforcement Learning Agent: A Normative End-to-End Pipeline

    Mar 17, 2026Benoît AlcarazAI Agent SafetyAgentic RL

    Faculty of Sciences, Technology, and Medicine · Doctoral School in Science and Engineering

  24. Proactive Rejection and Grounded Execution: A Dual-Stage Intent Analysis Paradigm for Safe and Efficient AIoT Smart Homes

    Mar 17, 2026Xinxin Jin, Zhengwei Ni, Zhengguo Sheng +1Language Model-Based ControlAI Agent Reliability

    School of Information and Electronic Engineering (Sussex Artificial Intelligence Institute), Zhejiang Gongshang University, Hangzhou 310018, China. · Sussex Artificial Intelligent Institute, Zhejiang Gongshang University, Hangzhou 310018, China · Department of Engineering and Design, University of Sussex, BN1 9RH, Brighton, U.K. +3

  25. From Workflow Automation to Capability Closure: A Formal Framework for Safe and Revenue-Aware Customer Service AI

    Mar 16, 2026Cosimo SperaAI Agent SafetyCustomer Support Automation

    Minerva CQ, 114 Lester Ln, Los Gatos, CA 95032

  26. Safety is Non-Compositional: A Formal Framework for Capability-Based AI Systems

    Mar 16, 2026Cosimo SperaAI Agent SafetyCapability-Based Security

    Minerva CQ, Los Gatos, CA 95032

  27. Evasive Intelligence: Lessons from Malware Analysis for Evaluating AI Agents

    Mar 16, 2026Simone Aonzo, Merve Sahin, Aurélien Francillon +1AI Agent EvaluationAI Agent Safety

    Eurecom, Sophia Antipolis, France · SAP Labs, Sophia Antipolis, France · Depthfirst, San Francisco, United States

  28. Beyond Benchmark Islands: Toward Representative Trustworthiness Evaluation for Agentic AI

    Mar 16, 2026Jinhu Qi, Yifan Li, Minghao Zhao +4AI Agent EvaluationAI Agent Safety

    The Chinese University of Hong Kong, Hong Kong SAR, China · Macao Polytechnic University, Macao SAR, China · Jilin University, Changchun, China

  29. Why Agents Compromise Safety Under Pressure

    Mar 16, 2026Hengle Jiang, Ke TangSafety-Utility Trade-Off in Language ModelsAI Agent Safety

    Guangdong Provincial Key Laboratory of Brain-inspired Intelligent Computation, Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China

  30. Emotional Cost Functions for AI Safety: Teaching Agents to Feel the Weight of Irreversible Consequences

    Mar 15, 2026Pandurang MopgarAI Agent SafetyLLM Agent Safety

    Independent Researcher