AI Agent Safety

Ask about this topic

Momentum

22 papers in the last four weeks, up 144% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 207

All topics
CardsList

  1. AI Safety Considerations for Agents With Limited Time to Act

    Oct 7, 2026Leo Zeitler, Jack Richings, Victoria NocklesAI Agent SafetyAI Safety

    The Alan Turing Institute 96 Euston Road London, NW1 2DB, United Kingdom

  2. RT-Safe: Benchmarking Agent Safety in Real-Time Embodied Environment

    Oct 7, 2026Tianruo Rose Xu, Jiawei Ren, Yichi Yang +2AI Agent SafetySafe Robot Navigation

    Cornell University · University of California, San Diego

  3. SpecGuard: Proving a Task Is Broken Before the Agent Cheats

    Oct 6, 2026Param Biyani, Krishnamurthy DvijothamAI Agent SafetySoftware Engineering Agents

    MATS · Google DeepMind

  4. Learning to Report Unsafe Tasks in a Multi-Agent Game

    Oct 6, 2026Avyay M. Casheekar, Hariganesh TangiralaAI Agent SafetyMulti-Agent Systems

    University of Michigan Law School Ann Arbor, USA · School of Computing National University of Singapore Singapore, Singapore

  5. Toward Evidence-Driven Human-Agent-Robot Teaming for Earth-Independent Anomaly Triage

    Oct 6, 2026Ignacio G Lopez-Francos, Alexis Gallagher, Samira ShalalHuman-Robot CollaborationSpace Robotics

    NASA Ames Research Center, Moffett Field, CA, USA · SETI Institute, Mountain View, CA, USA · The University of Texas at Austin, Austin, TX, USA +2

  6. Benchmarking Jailbreak Guardrails for Embodied Agents

    Oct 5, 2026Xunguang Wang, Qingyue Wang, Yuguang Zhou +3LLM GuardrailsAI Agent Safety

    Department of Computer Science and Engineering The Hong Kong University of Science and Technology Clear Water Bay, Kowloon, Hong Kong · School of Information Renmin University of China Beijing, China

  7. G-CARB: Graph-Localized Conformal Agent Risk Budget for Compositional Harm

    Oct 4, 2026Zijun Yu, Yu Gu, Vahid Partovi Nia +1AI Agent SafetyConformal Risk Control

    Department of Mathematics and Statistics McGill University Montréal, QC, Canada · Department of Mathematics and Industrial Engineering Polytechnique de Montréal Montréal, QC, Canada

  8. Chaining Skills to Hijack LLM Agents

    Oct 1, 2026Tian Dong, Zixuan Ma, Haodong Zhao +3LLM Agent SecurityAI Agent Safety

    The University of Hong Kong · Shandong University · Shanghai Jiao Tong University +1

  9. Mean field games as a tool for AI safety: a worked example from the July 2026 Hugging Face incident

    Oct 1, 2026P. Jameson GraberMulti-Agent CoordinationMean Field Games

  10. Safety of Latent Communication in Multi-Agent Systems

    Sep 30, 2026Muhammad Huzaifa, Sina Mavali, Thorsten EisenhoferMulti-Agent LLM SystemsAI Agent Safety

    CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

  11. Trust Is Not a Score: Runtime Assurance Contracts for High-Risk AI Agents

    Sep 30, 2026Serhii ZabolotniiAI Agent SafetyRuntime Enforcement for AI Agents

    Cherkasy State Business College, Cherkasy, Ukraine · State Scientific Research Institute of Armament and Military Equipment Testing and Certification, Cherkasy, Ukraine · healthPrecision, New York, NY, USA

  12. A Competing-Hazards Systematization of Loss of Control in Autonomous Agents

    Sep 29, 2026Mohamed Aly BoukeAgent Failure AnalysisAI Agent Auditing

    Centre for Intelligent Cloud Computing, CoE for Advanced Cloud, Faculty of Information Science and Technology, Multimedia University, Jalan Ayer Keroh Lama, Bukit Beruang, 75450, Melaka, Malaysia

  13. Character Training for Risk-Averse Agents

    Sep 29, 2026Arav Dhoot, Punya Syon Pandey, Jamie Johnson +3Risk-Sensitive RLAI Agent Safety

    UK AI Security Institute · Resolution

  14. SCOUT: Synergizing Reasoning and Tool-Use for Computer-Use Safety

    Sep 28, 2026Jianxing Chen, Xiao Yu, Shipra Agrawal +1AI Agent SafetyAI Agent Safety Benchmarks

    Columbia University

  15. SEABench: Benchmarking Endogenous Misalignment In Self-Evolving Agents

    Sep 28, 2026Saswat Das, Parvati Viswanathan, Daniel Donnelly +3AI Agent SafetyAI Agent Safety Benchmarks

    University of Virginia · ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems +1

  16. Before Acting, Change the State: Prospective State Intervention for Web Agents under Deceptive Interfaces

    Sep 28, 2026Ruozhao Yang, Mingfei Cheng, Xiaofei XieAI Agent SafetyRuntime Enforcement for AI Agents

    School of Computing and Information Systems Singapore Management University Singapore 188065

  17. AdaGuard: An Adaptive Guard Model with User-defined Policies

    Sep 28, 2026Yunhao Feng, Yifan Ding, Yuxiang Xie +4LLM GuardrailsAI Agent Safety

    National University of Defense Technology · Fudan University

  18. Hard Stop: Kernel-Level Preemption and Containment for Rogue Agentic Execution

    Sep 24, 2026José Luis PinoAutonomous Cyber DefenseAI Agent Security

    Independent Researcher Westlake Village, CA, USA

  19. Shutdown Sabotage Propensities in Multi-Agent Systems

    Sep 23, 2026Amelie Knecht, Ulysse Schaller, Christopher Summerfield +1AI Agent SafetyMulti-Agent System Security

    AI Safety Research Group, University of Stuttgart, Stuttgart, Germany · University of Oxford, Oxford, United Kingdom

  20. Finite-Sample Probabilistic Safety Certification for AI-Based Grid-Edge Coordination

    Sep 23, 2026Yihong Zhou, Hanbin Yang, Thomas MorstynAI Agent Safety

    Department of Engineering Science, University of Oxford, U.K.

  21. Safety Nudges: User-Facing Interventions for Real-Time AI Risk Awareness

    Sep 22, 2026Varshini Elangovan, James Wedgwood, Chhavi Yadav +3AI Agent SafetyHuman-AI Interaction

    Carnegie Mellon University

  22. ClashBench: Conflicts Leading Agents to Seize and Harm

    Sep 17, 2026Yuejin Xie, Yu Li, Dadi Guo +6AI Agent SecurityAI Agent Safety

    Tsinghua University · Shanghai AI Lab · Fudan University +2

  23. Collective Loss of Control in LLM Agent Systems: An Epidemic Account of Mutation, Contagion, and Recovery

    Sep 16, 2026Xiangfan Wu, Zonghao Ying, Huiyu Wu +4Multi-Agent LLM SystemsAI Agent Safety

    Tencent Zhuque Lab

  24. Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems

    Sep 15, 2026Deepak Akkil, Tamer Abuelsaad, Karthik Vikram +5Multi-Agent LLM SystemsLong-Horizon Agent Evaluation

    Emergence AI

  25. Delegating Authorization to Misaligned Agents: Coalitional Alignment and Safe Control

    Sep 14, 2026Natalie Collina, Surbhi Goel, Aaron Roth +1AI Agent SafetyRuntime Enforcement for AI Agents

  26. The Troy Moment: How LLM Agents Adjudicate the Decision Point Under Impossible Tasks, Claimed Authority, and Peer Information

    Sep 14, 2026Ivy ZhangAgent Failure AnalysisSoftware Engineering Agents

    Apart Research

  27. HazardAuditor: From Executable Threats to Safer Computer-Use Agents

    Sep 14, 2026Yunhao Feng, Ruixiao Lin, Ming Wen +5Language Model Safety EvaluationAI Agent Auditing

    Ant Group · Hunan Institute of Advanced Technology · Zhejiang University +3

  28. Safety Signals to Verify NetOps Agents with Action-Level Granularity

    Sep 13, 2026Tobias Labarta, Frederik Pahde, Novak Boškov +5Long-Horizon Agent EvaluationAI Agent Safety

    Fraunhofer Heinrich-Hertz-Institut, Berlin, Germany · Nokia Bell Labs, Stuttgart, Germany · Nokia Bell Labs, Murray Hill, NJ 07974, USA +2

  29. Do GUI Agents Know When Not to Act? Enabling Conflict-Aware Termination for Multimodal GUI Agents

    Sep 3, 2026Zhaoyuan Huang, Tianjie Ju, Pengzhou Cheng +7GUI AgentsAI Agent Safety

    School of Computer Science, Shanghai Jiao Tong University · Ant Group

  30. PACE: Towards Surfacing Hidden Conflicts in User Requests

    Sep 3, 2026Yoojin Kim, Jihyoung Jang, Hyounghun KimAI Agent SafetyLLM Refusal Behavior

    Department of Computer Science and Engineering, POSTECH · Graduate School of Artificial Intelligence, POSTECH · Department of Computer Science

  31. Beyond Task Completion: Training Capable and Safe Computer-Use Agents

    Aug 27, 2026Zeyu Kang, Zhenyun Yin, Yang Zhang +5Computer-Use AgentsAI Agent Safety

    Shanghai Artificial Intelligence Laboratory · Harbin Institute of Technology · Fudan University +2

  32. AI Agents Push Humans Out of the Loop

    Aug 24, 2026Margaret Mitchell, Avijit Ghosh, Samir PassiAI Agent SafetyHuman-in-the-Loop AI

    Hugging Face · Data & Society

  33. OpenAgentFlow: Enabling System-Wide Safety Boundaries for Heterogeneous AI Agent Fleets

    Aug 14, 2026Dongsheng Chen, Xiangyu Zhao, Xin Yao +1AI Agent SafetyAI Agent Governance

    Southern University of Science and Technology Shenzhen, China · City University of Hong Kong Hong Kong, China · Lingnan University Hong Kong, China

  34. Agent Safety Should Be a Runtime Contract

    Aug 11, 2026Albus W. Ng, Yi Han, Jusheng Zhang +1AI Agent SafetyAI Agent Monitoring

    1Vast Intelligence Lab · 2Southwest University · 3Sun Yat-sen University

  35. Multi-Agent AI Safety as an Institutional Design Problem

    Aug 10, 2026Abdullah XAI Agent SafetyMulti-Agent Systems

    POLIS Research Programme, Project AWARE

  36. ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents

    Aug 10, 2026Hongwei Yao, Yiming Liu, Meihui Chen +6AI Agent SecurityAI Agent Safety

    City University of Hong Kong, Hong Kong · Zhejiang University, China · Xiaomi Corporation, China +1

  37. Software Engineering for and with GUI Agent

    Aug 10, 2026Shengcheng Yu, Yuchen Ling, Junyang Xing +3AI Agent ReliabilityGUI Agents

    State Key Laboratory for Novel Software Technology, Nanjing University, China · Technical University of Munich, Germany

  38. Unaccountable Delegation, Fading Skills: Mapping the Risks of Workplace AI Agents

    Aug 9, 2026Gabriele La Malfa, Lakmal Meegahapola, Edyta Bogucka +4AI Agent SafetyHuman-AI Collaboration

    1King’s College London, UK · 2Nokia Bell Labs, Cambridge, UK · University of Sussex, UK +1

  39. Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution

    Aug 8, 2026Anton Razzhigaev, Andrei Gritsaev, Andrei Kaznacheev +3AI Coding AgentsAgent Harness Optimization

    1Lomonosov Moscow State University · 2Skolkovo Institute of Science and Technology · 4FusionBrain Lab at AIRI +2

  40. StepJack: Benchmarking Computer-Use Agent Safety Against Multi-Step Indirect Prompt Injection

    Aug 6, 2026Zhuoxin Zhan, Akbar Rafiey, Avery Ma +2Computer-Use AgentsAI Agent Safety

    Simon Fraser University · RBC Borealis · New York University

  41. Resourced Authority A Mechanism-Design Model for Participatory Governance of Deployed AI Agents

    Aug 6, 2026Praphul Chandra, Sujit Gujar, Ganesh GhalmeAI Agent SafetyAI Agent Governance

    Atria University · IIIT Hyderabad · IIT Hyderabad

  42. SafeCommit: Certifying When Memory-Grounded Agents May Safely Act

    Aug 4, 2026Mayur Akewar, Ravi RanjanAgent MemoryAI Agent Reliability

    Florida International University

  43. Securing Agentic AI: From Per-Action Checks to Trajectory Assurance

    Aug 3, 2026Alireza Lotfi, Subangkar Karmaker Shanto, Imtiaz Karim +1AI Agent SecurityLLM Agent Security

    Purdue University · Department of Computer Science · West Lafayette, IN, USA +2

  44. Humans Are More Diverse: Frontier LLMs Show Extreme Policies in Idealised AI Development Races

    Aug 2, 2026Phu Hoa Pham, Duy Minh Dao Sy, Trung Kiet Huynh +9LLM Agent EvaluationAI Agent Safety

    Ho Chi Minh City University of Science Vietnam National University Ho Chi Minh City (VNU-HCM) Ho Chi Minh City, Vietnam · Ho Chi Minh City University of Technology (HCMUT) Vietnam National University Ho Chi Minh City (VNU-HCM) Ho Chi Minh City, Vietnam · Vrije Universiteit Brussel Université Libre de Bruxelles Brussels, Belgium +1

  45. When Prompts Control Robots: Prompt Injection Attacks in Multi-Agent Robotic Systems

    Aug 1, 2026Neha Nagaraja, Amisha Bagari, Hayretdin BahsiMulti-Agent LLM SystemsLarge Language Model-Based Robot Planning

    School of Informatics, Computing, and Cyber Systems, Northern Arizona University, Flagstaff, USA · Department of Software Science, Tallinn University of Technology, Tallinn, Estonia

  46. Beyond Component Testing: Validating Agentic AI Systems

    Jul 31, 2026Fabio Orazio Mirto, Luca D'Agati, Giuseppe Tricomi +4AI Agent EvaluationAI Agent Safety

    Department of Biomedical, Dental, and Morphological and Functional Imaging Sciences, University of Messina, A.O.U. Policlinico “G.Martino” - Via Consolare Valeria, Messina, 98125, Italy. · Department of Engineering, University of Messina, Contrada di Dio, Sant’Agata, Messina, 98158, Italy. · Institute for High Performance Computing and Networking of National Research Council of Italy, ICAR-CNR, Via Pietro Castellino 111, Naples, 80131, Italy. +1

  47. LabEvolver: Training-Free Experience Evolution for Safe and Grounded Wet-Lab Agents

    Jul 30, 2026Jingya Wang, Yuyang Gao, Liuzhenghao Lv +2Agent MemoryAI Agents for Scientific Discovery

    School of Computer Science, Peking University · School of AI for Science, Peking University · School of Electronic and Computer Engineering, Peking University

  48. HALO: Heterogeneous Admission through Localized Obligations for Safe Agentic Execution

    Jul 30, 2026Taewoo Park, Kyeonghyun Yoo, Kiseok Kim +2AI Agent SafetyRuntime Enforcement for AI Agents

    Korea University

  49. Towards a Systems Foundation for Agentic Cloud Management

    Jul 28, 2026Minghao Li, Ziqian Liu, Ziyu Mao +5Multi-Agent OrchestrationAI Agent Safety

    The University of Hong Kong · Microsoft · University of Illinois Urbana-Champaign

  50. PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents

    Jul 28, 2026Korosh Vatanparvar, Ashutosh Joshi, Maria Xenochristou +11AI Agent EvaluationAI Agent Safety

    Amazon Health AI

  51. Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response

    Jul 28, 2026Abu Bakar SiddikAI Agent EvaluationCybersecurity

    Department of Computer Science and Engineering, Rajshahi University of Engineering & Technology, Rajshahi 6204, Bangladesh

  52. SafeFlow: Semantic Information-Flow Control for Blocking Malicious Propagation in Multi-Agent Systems

    Jul 28, 2026Haowen Dai, Zonghao Ying, Wenfeng Li +10AI Agent SafetyInformation Flow Control

    University of Nottingham Ningbo China · Beihang University · Shandong University +6

  53. GuardianAgentBench: Where Agents Fail and How to Guard Them

    Jul 23, 2026Vishal Ishwar Naik, Chenyu Xu, Donna Dong +5AI Agent ReliabilityLLM Guardrails

    Vectara, Inc., Palo Alto, CA, USA · Iowa State University, Ames, IA, USA

  54. OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills

    Jul 22, 2026Qiyuan Liu, Tingfeng Hui, Kun Zhan +2LLM Agent EvaluationAI Agent Safety

    Department of Data Science, City University of Hong Kong · Hong Kong Institute of AI for Science, City University of Hong Kong · Beijing University of Posts and Telecommunications +1

  55. JANUS: Foreseeing Latent Risk for Long-Horizon Agent Safety

    Jul 22, 2026Yuan Xiong, Linji Hao, Shizhu He +2AI Agent SafetyLLM Agent Safety

    The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Peking University, Beijing, China +2

  56. Engineering Trustworthy Agentic AI for Critical Systems

    Jul 20, 2026Omar Al-Refai, Ibrahim Shahbaz, Adam Ali Husseinat +3AI AssuranceAgent Reliability

    Innovations in Systems Trust and Resilience (iSTAR) Laboratory, Texas A&M University, TX, USA · Texas A&M Institute of Data Science - Security, Privacy and Resilience for Trusted AI (SPARTA) Thematic Lab., Texas A&M University, TX, USA · Texas A&M Global Cyber Research Institute (GCRI), Texas A&M University, TX, USA

  57. Operational Hallucination and Safety Drift in AI Agents

    Jul 20, 2026Shasha Yu, Fiona Carroll, Barry L. BentleyLLM Agent EvaluationAI Agent Safety

    Cardiff School of Technologies, Cardiff Metropolitan University Cardiff, UK · Harvard Medical School, Harvard University Boston, USA

  58. Specifying the Delegated-Autonomy Boundary: Requirements Engineering for Agentic AI

    Jul 19, 2026Chetan Arora, Andreas Vogelsang, Abbi SharmaAI Agent SafetyAI Agent Governance

    Faculty of IT, Monash University Melbourne, Victoria, Australia · University of Duisburg-Essen Essen, Germany

  59. PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution

    Jul 18, 2026Yang Liu, Weixing Chen, Xinshuai Song +8LLM Agent OrchestrationAI Agent Safety

    1X-Era Lab · 2HCP Lab, Sun Yat-sen University · 3Peng Cheng Laboratory

  60. SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction

    Jul 17, 2026Xue Yu, Bo Yuan, Kailin Zhao +3GUI AgentsAI Agent Safety

    JIUTIAN Research