AI Agent Safety

Ask about this topic

Momentum

22 papers in the last four weeks, up 175% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 201

All topics
CardsList

  1. Benchmarking Jailbreak Guardrails for Embodied Agents

    Oct 5, 2026Xunguang Wang, Qingyue Wang, Yuguang Zhou +3LLM GuardrailsAI Agent Safety

    Department of Computer Science and Engineering The Hong Kong University of Science and Technology Clear Water Bay, Kowloon, Hong Kong · School of Information Renmin University of China Beijing, China

  2. G-CARB: Graph-Localized Conformal Agent Risk Budget for Compositional Harm

    Oct 4, 2026Zijun Yu, Yu Gu, Vahid Partovi Nia +1AI Agent SafetyConformal Risk Control

    Department of Mathematics and Statistics McGill University Montréal, QC, Canada · Department of Mathematics and Industrial Engineering Polytechnique de Montréal Montréal, QC, Canada

  3. Chaining Skills to Hijack LLM Agents

    Oct 1, 2026Tian Dong, Zixuan Ma, Haodong Zhao +3LLM Agent SecurityAI Agent Safety

    The University of Hong Kong · Shandong University · Shanghai Jiao Tong University +1

  4. Mean field games as a tool for AI safety: a worked example from the July 2026 Hugging Face incident

    Oct 1, 2026P. Jameson GraberMulti-Agent CoordinationMean Field Games

  5. Safety of Latent Communication in Multi-Agent Systems

    Sep 30, 2026Muhammad Huzaifa, Sina Mavali, Thorsten EisenhoferMulti-Agent LLM SystemsAI Agent Safety

    CISPA Helmholtz Center for Information Security, Saarbrücken, Germany

  6. Trust Is Not a Score: Runtime Assurance Contracts for High-Risk AI Agents

    Sep 30, 2026Serhii ZabolotniiAI Agent SafetyRuntime Enforcement for AI Agents

    Cherkasy State Business College, Cherkasy, Ukraine · State Scientific Research Institute of Armament and Military Equipment Testing and Certification, Cherkasy, Ukraine · healthPrecision, New York, NY, USA

  7. A Competing-Hazards Systematization of Loss of Control in Autonomous Agents

    Sep 29, 2026Mohamed Aly BoukeAgent Failure AnalysisAI Agent Auditing

    Centre for Intelligent Cloud Computing, CoE for Advanced Cloud, Faculty of Information Science and Technology, Multimedia University, Jalan Ayer Keroh Lama, Bukit Beruang, 75450, Melaka, Malaysia

  8. Character Training for Risk-Averse Agents

    Sep 29, 2026Arav Dhoot, Punya Syon Pandey, Jamie Johnson +3Risk-Sensitive RLAI Agent Safety

    UK AI Security Institute · Resolution

  9. SCOUT: Synergizing Reasoning and Tool-Use for Computer-Use Safety

    Sep 28, 2026Jianxing Chen, Xiao Yu, Shipra Agrawal +1AI Agent SafetyAI Agent Safety Benchmarks

    Columbia University

  10. SEABench: Benchmarking Endogenous Misalignment In Self-Evolving Agents

    Sep 28, 2026Saswat Das, Parvati Viswanathan, Daniel Donnelly +3AI Agent SafetyAI Agent Safety Benchmarks

    University of Virginia · ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems +1

  11. Before Acting, Change the State: Prospective State Intervention for Web Agents under Deceptive Interfaces

    Sep 28, 2026Ruozhao Yang, Mingfei Cheng, Xiaofei XieAI Agent SafetyRuntime Enforcement for AI Agents

    School of Computing and Information Systems Singapore Management University Singapore 188065

  12. AdaGuard: An Adaptive Guard Model with User-defined Policies

    Sep 28, 2026Yunhao Feng, Yifan Ding, Yuxiang Xie +4LLM GuardrailsAI Agent Safety

    National University of Defense Technology · Fudan University

  13. Hard Stop: Kernel-Level Preemption and Containment for Rogue Agentic Execution

    Sep 24, 2026José Luis PinoAutonomous Cyber DefenseAI Agent Security

    Independent Researcher Westlake Village, CA, USA

  14. Shutdown Sabotage Propensities in Multi-Agent Systems

    Sep 23, 2026Amelie Knecht, Ulysse Schaller, Christopher Summerfield +1AI Agent SafetyMulti-Agent System Security

    AI Safety Research Group, University of Stuttgart, Stuttgart, Germany · University of Oxford, Oxford, United Kingdom

  15. Finite-Sample Probabilistic Safety Certification for AI-Based Grid-Edge Coordination

    Sep 23, 2026Yihong Zhou, Hanbin Yang, Thomas MorstynAI Agent Safety

    Department of Engineering Science, University of Oxford, U.K.

  16. Safety Nudges: User-Facing Interventions for Real-Time AI Risk Awareness

    Sep 22, 2026Varshini Elangovan, James Wedgwood, Chhavi Yadav +3AI Agent SafetyHuman-AI Interaction

    Carnegie Mellon University

  17. ClashBench: Conflicts Leading Agents to Seize and Harm

    Sep 17, 2026Yuejin Xie, Yu Li, Dadi Guo +6AI Agent SecurityAI Agent Safety

    Tsinghua University · Shanghai AI Lab · Fudan University +2

  18. Collective Loss of Control in LLM Agent Systems: An Epidemic Account of Mutation, Contagion, and Recovery

    Sep 16, 2026Xiangfan Wu, Zonghao Ying, Huiyu Wu +4Multi-Agent LLM SystemsAI Agent Safety

    Tencent Zhuque Lab

  19. Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems

    Sep 15, 2026Deepak Akkil, Tamer Abuelsaad, Karthik Vikram +5Multi-Agent LLM SystemsLong-Horizon Agent Evaluation

    Emergence AI

  20. Delegating Authorization to Misaligned Agents: Coalitional Alignment and Safe Control

    Sep 14, 2026Natalie Collina, Surbhi Goel, Aaron Roth +1AI Agent SafetyRuntime Enforcement for AI Agents

  21. The Troy Moment: How LLM Agents Adjudicate the Decision Point Under Impossible Tasks, Claimed Authority, and Peer Information

    Sep 14, 2026Ivy ZhangAgent Failure AnalysisSoftware Engineering Agents

    Apart Research

  22. HazardAuditor: From Executable Threats to Safer Computer-Use Agents

    Sep 14, 2026Yunhao Feng, Ruixiao Lin, Ming Wen +5Language Model Safety EvaluationAI Agent Auditing

    Ant Group · Hunan Institute of Advanced Technology · Zhejiang University +3

  23. Safety Signals to Verify NetOps Agents with Action-Level Granularity

    Sep 13, 2026Tobias Labarta, Frederik Pahde, Novak Boškov +5Long-Horizon Agent EvaluationAI Agent Safety

    Fraunhofer Heinrich-Hertz-Institut, Berlin, Germany · Nokia Bell Labs, Stuttgart, Germany · Nokia Bell Labs, Murray Hill, NJ 07974, USA +2

  24. Do GUI Agents Know When Not to Act? Enabling Conflict-Aware Termination for Multimodal GUI Agents

    Sep 3, 2026Zhaoyuan Huang, Tianjie Ju, Pengzhou Cheng +7GUI AgentsAI Agent Safety

    School of Computer Science, Shanghai Jiao Tong University · Ant Group

  25. PACE: Towards Surfacing Hidden Conflicts in User Requests

    Sep 3, 2026Yoojin Kim, Jihyoung Jang, Hyounghun KimAI Agent SafetyLLM Refusal Behavior

    Department of Computer Science and Engineering, POSTECH · Graduate School of Artificial Intelligence, POSTECH · Department of Computer Science

  26. Beyond Task Completion: Training Capable and Safe Computer-Use Agents

    Aug 27, 2026Zeyu Kang, Zhenyun Yin, Yang Zhang +5Computer-Use AgentsAI Agent Safety

    Shanghai Artificial Intelligence Laboratory · Harbin Institute of Technology · Fudan University +2

  27. AI Agents Push Humans Out of the Loop

    Aug 24, 2026Margaret Mitchell, Avijit Ghosh, Samir PassiAI Agent SafetyHuman-in-the-Loop AI

    Hugging Face · Data & Society

  28. Agent Safety Should Be a Runtime Contract

    Aug 11, 2026Albus W. Ng, Yi Han, Jusheng Zhang +1AI Agent SafetyAI Agent Monitoring

    1Vast Intelligence Lab · 2Southwest University · 3Sun Yat-sen University

  29. Multi-Agent AI Safety as an Institutional Design Problem

    Aug 10, 2026Abdullah XAI Agent SafetyMulti-Agent Systems

    POLIS Research Programme, Project AWARE

  30. ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents

    Aug 10, 2026Hongwei Yao, Yiming Liu, Meihui Chen +6AI Agent SecurityAI Agent Safety

    City University of Hong Kong, Hong Kong · Zhejiang University, China · Xiaomi Corporation, China +1

  31. Software Engineering for and with GUI Agent

    Aug 10, 2026Shengcheng Yu, Yuchen Ling, Junyang Xing +3AI Agent ReliabilityGUI Agents

    State Key Laboratory for Novel Software Technology, Nanjing University, China · Technical University of Munich, Germany

  32. Unaccountable Delegation, Fading Skills: Mapping the Risks of Workplace AI Agents

    Aug 9, 2026Gabriele La Malfa, Lakmal Meegahapola, Edyta Bogucka +4AI Agent SafetyHuman-AI Collaboration

    1King’s College London, UK · 2Nokia Bell Labs, Cambridge, UK · University of Sussex, UK +1

  33. Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution

    Aug 8, 2026Anton Razzhigaev, Andrei Gritsaev, Andrei Kaznacheev +3AI Coding AgentsAgent Harness Optimization

    1Lomonosov Moscow State University · 2Skolkovo Institute of Science and Technology · 4FusionBrain Lab at AIRI +2

  34. StepJack: Benchmarking Computer-Use Agent Safety Against Multi-Step Indirect Prompt Injection

    Aug 6, 2026Zhuoxin Zhan, Akbar Rafiey, Avery Ma +2Computer-Use AgentsAI Agent Safety

    Simon Fraser University · RBC Borealis · New York University

  35. Resourced Authority A Mechanism-Design Model for Participatory Governance of Deployed AI Agents

    Aug 6, 2026Praphul Chandra, Sujit Gujar, Ganesh GhalmeAI Agent SafetyAI Agent Governance

    Atria University · IIIT Hyderabad · IIT Hyderabad

  36. SafeCommit: Certifying When Memory-Grounded Agents May Safely Act

    Aug 4, 2026Mayur Akewar, Ravi RanjanAgent MemoryAI Agent Reliability

    Florida International University

  37. Securing Agentic AI: From Per-Action Checks to Trajectory Assurance

    Aug 3, 2026Alireza Lotfi, Subangkar Karmaker Shanto, Imtiaz Karim +1AI Agent SecurityLLM Agent Security

    Purdue University · Department of Computer Science · West Lafayette, IN, USA +2

  38. Humans Are More Diverse: Frontier LLMs Show Extreme Policies in Idealised AI Development Races

    Aug 2, 2026Phu Hoa Pham, Duy Minh Dao Sy, Trung Kiet Huynh +9LLM Agent EvaluationAI Agent Safety

    Ho Chi Minh City University of Science Vietnam National University Ho Chi Minh City (VNU-HCM) Ho Chi Minh City, Vietnam · Ho Chi Minh City University of Technology (HCMUT) Vietnam National University Ho Chi Minh City (VNU-HCM) Ho Chi Minh City, Vietnam · Vrije Universiteit Brussel Université Libre de Bruxelles Brussels, Belgium +1

  39. When Prompts Control Robots: Prompt Injection Attacks in Multi-Agent Robotic Systems

    Aug 1, 2026Neha Nagaraja, Amisha Bagari, Hayretdin BahsiMulti-Agent LLM SystemsLarge Language Model-Based Robot Planning

    School of Informatics, Computing, and Cyber Systems, Northern Arizona University, Flagstaff, USA · Department of Software Science, Tallinn University of Technology, Tallinn, Estonia

  40. Beyond Component Testing: Validating Agentic AI Systems

    Jul 31, 2026Fabio Orazio Mirto, Luca D'Agati, Giuseppe Tricomi +4AI Agent EvaluationAI Agent Safety

    Department of Biomedical, Dental, and Morphological and Functional Imaging Sciences, University of Messina, A.O.U. Policlinico “G.Martino” - Via Consolare Valeria, Messina, 98125, Italy. · Department of Engineering, University of Messina, Contrada di Dio, Sant’Agata, Messina, 98158, Italy. · Institute for High Performance Computing and Networking of National Research Council of Italy, ICAR-CNR, Via Pietro Castellino 111, Naples, 80131, Italy. +1

  41. LabEvolver: Training-Free Experience Evolution for Safe and Grounded Wet-Lab Agents

    Jul 30, 2026Jingya Wang, Yuyang Gao, Liuzhenghao Lv +2Agent MemoryAI Agents for Scientific Discovery

    School of Computer Science, Peking University · School of AI for Science, Peking University · School of Electronic and Computer Engineering, Peking University

  42. HALO: Heterogeneous Admission through Localized Obligations for Safe Agentic Execution

    Jul 30, 2026Taewoo Park, Kyeonghyun Yoo, Kiseok Kim +2AI Agent SafetyRuntime Enforcement for AI Agents

    Korea University

  43. Towards a Systems Foundation for Agentic Cloud Management

    Jul 28, 2026Minghao Li, Ziqian Liu, Ziyu Mao +5Multi-Agent OrchestrationAI Agent Safety

    The University of Hong Kong · Microsoft · University of Illinois Urbana-Champaign

  44. PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents

    Jul 28, 2026Korosh Vatanparvar, Ashutosh Joshi, Maria Xenochristou +11AI Agent EvaluationAI Agent Safety

    Amazon Health AI

  45. Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response

    Jul 28, 2026Abu Bakar SiddikAI Agent EvaluationCybersecurity

    Department of Computer Science and Engineering, Rajshahi University of Engineering & Technology, Rajshahi 6204, Bangladesh

  46. SafeFlow: Semantic Information-Flow Control for Blocking Malicious Propagation in Multi-Agent Systems

    Jul 28, 2026Haowen Dai, Zonghao Ying, Wenfeng Li +10AI Agent SafetyInformation Flow Control

    University of Nottingham Ningbo China · Beihang University · Shandong University +6

  47. GuardianAgentBench: Where Agents Fail and How to Guard Them

    Jul 23, 2026Vishal Ishwar Naik, Chenyu Xu, Donna Dong +5AI Agent ReliabilityLLM Guardrails

    Vectara, Inc., Palo Alto, CA, USA · Iowa State University, Ames, IA, USA

  48. OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills

    Jul 22, 2026Qiyuan Liu, Tingfeng Hui, Kun Zhan +2LLM Agent EvaluationAI Agent Safety

    Department of Data Science, City University of Hong Kong · Hong Kong Institute of AI for Science, City University of Hong Kong · Beijing University of Posts and Telecommunications +1

  49. JANUS: Foreseeing Latent Risk for Long-Horizon Agent Safety

    Jul 22, 2026Yuan Xiong, Linji Hao, Shizhu He +2AI Agent SafetyLLM Agent Safety

    The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Peking University, Beijing, China +2

  50. Engineering Trustworthy Agentic AI for Critical Systems

    Jul 20, 2026Omar Al-Refai, Ibrahim Shahbaz, Adam Ali Husseinat +3AI AssuranceAgent Reliability

    Innovations in Systems Trust and Resilience (iSTAR) Laboratory, Texas A&M University, TX, USA · Texas A&M Institute of Data Science - Security, Privacy and Resilience for Trusted AI (SPARTA) Thematic Lab., Texas A&M University, TX, USA · Texas A&M Global Cyber Research Institute (GCRI), Texas A&M University, TX, USA

  51. Operational Hallucination and Safety Drift in AI Agents

    Jul 20, 2026Shasha Yu, Fiona Carroll, Barry L. BentleyLLM Agent EvaluationAI Agent Safety

    Cardiff School of Technologies, Cardiff Metropolitan University Cardiff, UK · Harvard Medical School, Harvard University Boston, USA

  52. Specifying the Delegated-Autonomy Boundary: Requirements Engineering for Agentic AI

    Jul 19, 2026Chetan Arora, Andreas Vogelsang, Abbi SharmaAI Agent SafetyAI Agent Governance

    Faculty of IT, Monash University Melbourne, Victoria, Australia · University of Duisburg-Essen Essen, Germany

  53. PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution

    Jul 18, 2026Yang Liu, Weixing Chen, Xinshuai Song +8LLM Agent OrchestrationAI Agent Safety

    1X-Era Lab · 2HCP Lab, Sun Yat-sen University · 3Peng Cheng Laboratory

  54. SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction

    Jul 17, 2026Xue Yu, Bo Yuan, Kailin Zhao +3GUI AgentsAI Agent Safety

    JIUTIAN Research

  55. A Self-Evolving Agent for Longitudinal Personal Health Management

    Jul 15, 2026Haoran Li, Jiebi Deng, Tong Jin +10Agent MemoryHealthcare

    School of Data Science, Fudan University, Shanghai, China · School of Life Sciences, Beijing University of Chinese Medicine, Beijing, China · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China +4

  56. Agent-Safety Evaluations as Load-Bearing Evidence: A Vendor-Neutral, Cross-Harness Reconstructability Metric

    Jul 14, 2026Oleg SolozobovAI Agent SafetyAI Safety Evaluation

    Independent Researcher (Global)

  57. ANCHOR: Automated Alignment Auditing for CLI Agents on Real-World Harm

    Jul 11, 2026Kefan Song, Yanjun QiAI Agent AuditingLLM Auditing

    University of Virginia, Charlottesville, VA, USA

  58. Neuro-Agentic Control: A Deep Learning-based LLM-Powered Agentic AI Framework for Controlling Security Controls

    Jul 10, 2026Saroj Gopali, Bipin Chhetri, Deepika Giri +2Language Model-Based ControlSafety-Critical Control

    Department of Computer Science1, Texas Tech University1 · Cumberland University2 · Advanced Academic Programs Science3, Johns Hopkins University3

  59. Institutional Red-Teaming: Deployment Rules, Not Just Models, Causally Shape Multi-Agent AI Safety

    Jul 8, 2026Yujiao ChenAlgorithmic FairnessAI Agent Safety

    Massachusetts Institute of Technology · Cambridge, MA 02139

  60. Beyond Attack-Success Rate: Action-Graded Severity Scale for Tool-Using AI Agents

    Jul 8, 2026Harry Owiredu-AshleyLLM Red TeamingAI Agent Safety

    Independent Researcher New Jersey, USA