AI Agent Safety

Ask about this topic

Overview

AI agent safety asks how to keep systems that plan over multiple steps and use tools, memory, or other agents within authorized and harmless bounds. Researchers test computer-use, coding, web, and embodied agents with prompt-injection, misuse, and long-horizon trajectory benchmarks, measuring executed effects as well as final answers. Approaches include red-teaming and monitoring, safety fine-tuning and refusal training, and runtime controls such as least-privilege access, pre-execution policy gates, sandboxing, human approval, and formal verification. A central challenge is that safety can fail through ambiguous goals, changing context, tool or agent interactions, and pressure over time, while stronger restrictions can block legitimate tasks and assurances may not transfer to deployment.

Latest papers

CardsList

330 papers47 papers in the last four weeks, up 68% on the four weeks before. 0.3% of all new papers.

  1. Governing What You Cannot Observe: Adaptive Runtime Governance for Autonomous AI Agents

    Apr 27, 2026German Marin, Jatin ChaudharyAI Agent SafetyAI Agent Monitoring

    Department of Computing, University of Turku, Turku, Finland

  2. Evaluating whether AI models would sabotage AI safety research

    Apr 27, 2026Robert Kirk, Alexandra Souly, Kai Fronsdal +2Language Model Safety EvaluationLLM Auditing

    UK AI Security Institute · UK AISI Research Affiliate

  3. OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agents

    Apr 27, 2026Zheng Wu, Yi Hua, Zhaoyuan Huang +8Computer-Use AgentsAI Agent Evaluation

    School of Computer Science, Shanghai Jiao Tong University

  4. Benchmarking Web Agent Safety under E-commerce Deceptive Interfaces

    Apr 26, 2026Zijing Shi, Meng Fang, Ling ChenWeb Agent BenchmarksAI Agent Safety

    AAII, University of Technology Sydney, NSW, Australia · University of Liverpool, Liverpool, UK

  5. Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanisms

    Apr 26, 2026Qi Li, Bo Yin, Weiqi Huang +6AI Agent SafetyAdversarial Attacks on VLA Models

    National University of Singapore · Peking University · Monash University

  6. Structural Enforcement of Goal Integrity in AI Agents via Separation-of-Powers Architecture

    Apr 26, 2026Rong XiangAI Agent SecurityAI Agent Safety

  7. Operationalizing Reconstructive Authority: Runtime Construction, Dependency Resolution, and Execution Gating in Autonomous Agent Systems

    Apr 24, 2026Marcelo Fernandez - TraslaIAAI Agent SecurityAI Agent Safety

    TraslaIA

  8. Sovereign Agentic Loops: Decoupling AI Reasoning from Execution in Real-World Systems

    Apr 24, 2026Jun He, Deying YuLLM Agent SecurityAI Agent Safety

  9. Interval POMDP Shielding for Imperfect-Perception Agents

    Apr 22, 2026William Scarbro, Ravi MangalSafety-Critical ControlAI Agent Safety

    Department of Computer Science Colorado State University Fort Collins, USA

  10. AgentSOC: A Multi-Layer Agentic AI Framework for Security Operations Automation

    Apr 22, 2026Joyjit Roy, Samaresh Kumar SinghAutonomous Cyber DefenseCybersecurity

    Austin, Texas, USA · Leander, Texas, USA

  11. SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Models

    Apr 21, 2026Josue Torres-Fonseca, Naihao Deng, Yinpei Dai +5Safety Evaluation of VLA ModelsLLM Safety Benchmarks

    University of Michigan · Boise State University

  12. AI-Native Network Controller: A Modular Framework for Safe Agentic Control of Multi-Domain Network Infrastructure

    Apr 20, 2026Merim DzaferagicAI Agent Safety

    School of Computer Science and Statistics, Trinity College Dublin, Ireland

  13. SDOF: Taming the Alignment Tax in Multi-Agent Orchestration with State-Constrained Dispatch

    Apr 20, 2026Zhantao WangMulti-Agent OrchestrationAI Agent Safety

    Digital China

  14. Owner-Harm: A Missing Threat Model for AI Agent Safety

    Apr 20, 2026Dongcheng Zhang, Yiqing JiangAI Agent SecurityAI Agent Safety

    BlueFocus Communication Group · Tongji University

  15. From Craft to Kernel: A Governance-First Execution Architecture and Semantic ISA for Agentic Computers

    Apr 20, 2026Xiangyu Wen, Yuang Zhao, Xiaoyu Xu +9AI Agent SecurityAI Agent Safety

    CUHK · Hong Kong SAR, China · Shanghai Jiao Tong University +6

  16. Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMs

    Apr 19, 2026Carissa Cullen, Harry Garland, Alexander Roman +3Agent ReliabilityAI Agent Safety

    University of Oxford · University College London · New College of Florida +3

  17. Beyond Task Success: An Evidence-Synthesis Framework for Evaluating, Governing, and Orchestrating Agentic AI

    Apr 18, 2026Christopher Koch, Joshua Andreas WellbrockLLM Agent OrchestrationAI Agent Evaluation

    Independent Researcher

  18. Governance by Design: Architecting Agentic AI for Organizational Learning and Scalable Autonomy

    Apr 17, 2026Nelly Dux, Cristina Alaimo, Philippe Roussiere +1AI AccountabilityAI Agent Safety

    ESSEC Business School · Accenture Research

  19. Don't Make Models Guess Security and Safety: Symbolic Guardrails for Domain-Specific AI Agents

    Apr 16, 2026Yining Hong, Yining She, Eunsuk Kang +2LLM GuardrailsAI Agent Security

    Carnegie Mellon University

  20. Agentic Microphysics: A Manifesto for Generative AI Safety

    Apr 16, 2026Federico Pierucci, Matteo Prandi, Marcantonio Bracale Syrnikov +2AI Agent SafetyAI Safety

    DEXAI – Icaro Lab · Sant’Anna School of Advanced Studies · Sapienza University of Rome +1

  21. Benchmarks for Trajectory Safety Evaluation and Diagnosis in OpenClaw and Codex: ATBench-Claw and ATBench-Codex

    Apr 16, 2026Zhonghao Yang, Yu Li, Yanxu Zhu +6Benchmark DesignSoftware Engineering Benchmarks

    Shanghai Artificial Intelligence Laboratory

  22. Bounded Autonomy for Enterprise AI: Typed Action Contracts and Consumer-Side Execution

    Apr 16, 2026Sarmad Sohail, Ghufran HaiderLLM GuardrailsAI Agent Security

  23. CBCL: Safe Self-Extending Agent Communication

    Apr 16, 2026Hugo O'ConnorAgent Communication ProtocolsAI Agent Safety

    Anuna Research

  24. HINTBench: Horizon-agent Intrinsic Non-attack Trajectory Benchmark

    Apr 15, 2026Jiacheng Wang, Jinchang Hou, Fabian Wang +3AI Agent AuditingLong-Horizon Agent Evaluation

    Baidu Inc., Beijing, China · Beijing Institute of Technology, Beijing, China

  25. Golden Handcuffs make safer AI agents

    Apr 15, 2026Aram Ebtekar, Michael K. CohenAI Agent SafetySafe RL

    University of California, Berkeley

  26. SciFi: A Safe, Lightweight, User-Friendly, and Fully Autonomous Agentic AI Workflow for Scientific Applications

    Apr 14, 2026Qibin Liu, Julia GonskiAgentic WorkflowsScientific Workflow Automation

    SLAC National Accelerator Laboratory, 2575 Sand Hill Rd, Menlo Park CA 94025 USA

  27. Playing Along: Learning a Double-Agent Defender for Belief Steering via Theory of Mind

    Apr 13, 2026Hanqi Xiao, Vaidehi Patil, Zaid Khan +3Theory of MindAI Agent Safety

    UNC Chapel Hill · University of Texas at Austin

  28. SemaClaw: A Step Towards General-Purpose Personal AI Agents through Harness Engineering

    Apr 13, 2026Ningyan Zhu, Huacan Wang, Jie Zhou +8LLM Agent HarnessesPersonal AI Agents

    Midea AIRC

  29. Speaking to No One: Ontological Dissonance and the Double Bind of Conversational AI

    Apr 12, 2026Hugh Brosnahan, Izabela LipinskaHuman-AI InteractionAI Agent Safety

    Bioethics Centre, University of Otago · Independent Researcher

  30. The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agents

    Apr 12, 2026Xuwei Ding, Skylar Zhai, Linxin Song +6Computer-Use Agent SafetyAI Agent Safety

    University of Wisconsin–Madison · University of Minnesota · University of Southern California +3