AI Agent Safety

Follow this topic Ask about this topic

Momentum

22 papers in the last four weeks, up 144% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 207

All topics

Beyond Task Completion: Training Capable and Safe Computer-Use Agents
Aug 27, 2026Zeyu Kang, Zhenyun Yin, Yang Zhang +5Computer-Use Agents AI Agent Safety
Shanghai Artificial Intelligence Laboratory · Harbin Institute of Technology · Fudan University +2
AI Agents Push Humans Out of the Loop
Aug 24, 2026Margaret Mitchell, Avijit Ghosh, Samir PassiAI Agent Safety Human-in-the-Loop AI
Hugging Face · Data & Society
OpenAgentFlow: Enabling System-Wide Safety Boundaries for Heterogeneous AI Agent Fleets
Aug 14, 2026Dongsheng Chen, Xiangyu Zhao, Xin Yao +1AI Agent Safety AI Agent Governance
Southern University of Science and Technology Shenzhen, China · City University of Hong Kong Hong Kong, China · Lingnan University Hong Kong, China
Agent Safety Should Be a Runtime Contract
Aug 11, 2026Albus W. Ng, Yi Han, Jusheng Zhang +1AI Agent Safety AI Agent Monitoring
1Vast Intelligence Lab · 2Southwest University · 3Sun Yat-sen University
Multi-Agent AI Safety as an Institutional Design Problem
Aug 10, 2026Abdullah XAI Agent Safety Multi-Agent Systems
POLIS Research Programme, Project AWARE
ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents
Aug 10, 2026Hongwei Yao, Yiming Liu, Meihui Chen +6AI Agent Security AI Agent Safety
City University of Hong Kong, Hong Kong · Zhejiang University, China · Xiaomi Corporation, China +1
Software Engineering for and with GUI Agent
Aug 10, 2026Shengcheng Yu, Yuchen Ling, Junyang Xing +3AI Agent Reliability GUI Agents
State Key Laboratory for Novel Software Technology, Nanjing University, China · Technical University of Munich, Germany
Unaccountable Delegation, Fading Skills: Mapping the Risks of Workplace AI Agents
Aug 9, 2026Gabriele La Malfa, Lakmal Meegahapola, Edyta Bogucka +4AI Agent Safety Human-AI Collaboration
1King’s College London, UK · 2Nokia Bell Labs, Cambridge, UK · University of Sussex, UK +1
Ouroboros: A Self-Developing Frontier Coding Agent with Reviewed Core Evolution
Aug 8, 2026Anton Razzhigaev, Andrei Gritsaev, Andrei Kaznacheev +3AI Coding Agents Agent Harness Optimization
1Lomonosov Moscow State University · 2Skolkovo Institute of Science and Technology · 4FusionBrain Lab at AIRI +2
StepJack: Benchmarking Computer-Use Agent Safety Against Multi-Step Indirect Prompt Injection
Aug 6, 2026Zhuoxin Zhan, Akbar Rafiey, Avery Ma +2Computer-Use Agents AI Agent Safety
Simon Fraser University · RBC Borealis · New York University
Resourced Authority A Mechanism-Design Model for Participatory Governance of Deployed AI Agents
Aug 6, 2026Praphul Chandra, Sujit Gujar, Ganesh GhalmeAI Agent Safety AI Agent Governance
Atria University · IIIT Hyderabad · IIT Hyderabad
SafeCommit: Certifying When Memory-Grounded Agents May Safely Act
Aug 4, 2026Mayur Akewar, Ravi RanjanAgent Memory AI Agent Reliability
Florida International University
Securing Agentic AI: From Per-Action Checks to Trajectory Assurance
Aug 3, 2026Alireza Lotfi, Subangkar Karmaker Shanto, Imtiaz Karim +1AI Agent Security LLM Agent Security
Purdue University · Department of Computer Science · West Lafayette, IN, USA +2
Humans Are More Diverse: Frontier LLMs Show Extreme Policies in Idealised AI Development Races
Aug 2, 2026Phu Hoa Pham, Duy Minh Dao Sy, Trung Kiet Huynh +9LLM Agent Evaluation AI Agent Safety
Ho Chi Minh City University of Science Vietnam National University Ho Chi Minh City (VNU-HCM) Ho Chi Minh City, Vietnam · Ho Chi Minh City University of Technology (HCMUT) Vietnam National University Ho Chi Minh City (VNU-HCM) Ho Chi Minh City, Vietnam · Vrije Universiteit Brussel Université Libre de Bruxelles Brussels, Belgium +1
When Prompts Control Robots: Prompt Injection Attacks in Multi-Agent Robotic Systems
Aug 1, 2026Neha Nagaraja, Amisha Bagari, Hayretdin BahsiMulti-Agent LLM Systems Large Language Model-Based Robot Planning
School of Informatics, Computing, and Cyber Systems, Northern Arizona University, Flagstaff, USA · Department of Software Science, Tallinn University of Technology, Tallinn, Estonia
Beyond Component Testing: Validating Agentic AI Systems
Jul 31, 2026Fabio Orazio Mirto, Luca D'Agati, Giuseppe Tricomi +4AI Agent Evaluation AI Agent Safety
Department of Biomedical, Dental, and Morphological and Functional Imaging Sciences, University of Messina, A.O.U. Policlinico “G.Martino” - Via Consolare Valeria, Messina, 98125, Italy. · Department of Engineering, University of Messina, Contrada di Dio, Sant’Agata, Messina, 98158, Italy. · Institute for High Performance Computing and Networking of National Research Council of Italy, ICAR-CNR, Via Pietro Castellino 111, Naples, 80131, Italy. +1
LabEvolver: Training-Free Experience Evolution for Safe and Grounded Wet-Lab Agents
Jul 30, 2026Jingya Wang, Yuyang Gao, Liuzhenghao Lv +2Agent Memory AI Agents for Scientific Discovery
School of Computer Science, Peking University · School of AI for Science, Peking University · School of Electronic and Computer Engineering, Peking University
HALO: Heterogeneous Admission through Localized Obligations for Safe Agentic Execution
Jul 30, 2026Taewoo Park, Kyeonghyun Yoo, Kiseok Kim +2AI Agent Safety Runtime Enforcement for AI Agents
Korea University
Towards a Systems Foundation for Agentic Cloud Management
Jul 28, 2026Minghao Li, Ziqian Liu, Ziyu Mao +5Multi-Agent Orchestration AI Agent Safety
The University of Hong Kong · Microsoft · University of Illinois Urbana-Champaign
PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents
Jul 28, 2026Korosh Vatanparvar, Ashutosh Joshi, Maria Xenochristou +11AI Agent Evaluation AI Agent Safety
Amazon Health AI
Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response
Jul 28, 2026Abu Bakar SiddikAI Agent Evaluation Cybersecurity
Department of Computer Science and Engineering, Rajshahi University of Engineering & Technology, Rajshahi 6204, Bangladesh
SafeFlow: Semantic Information-Flow Control for Blocking Malicious Propagation in Multi-Agent Systems
Jul 28, 2026Haowen Dai, Zonghao Ying, Wenfeng Li +10AI Agent Safety Information Flow Control
University of Nottingham Ningbo China · Beihang University · Shandong University +6
GuardianAgentBench: Where Agents Fail and How to Guard Them
Jul 23, 2026Vishal Ishwar Naik, Chenyu Xu, Donna Dong +5AI Agent Reliability LLM Guardrails
Vectara, Inc., Palo Alto, CA, USA · Iowa State University, Ames, IA, USA
OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills
Jul 22, 2026Qiyuan Liu, Tingfeng Hui, Kun Zhan +2LLM Agent Evaluation AI Agent Safety
Department of Data Science, City University of Hong Kong · Hong Kong Institute of AI for Science, City University of Hong Kong · Beijing University of Posts and Telecommunications +1
JANUS: Foreseeing Latent Risk for Long-Horizon Agent Safety
Jul 22, 2026Yuan Xiong, Linji Hao, Shizhu He +2AI Agent Safety LLM Agent Safety
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Peking University, Beijing, China +2
Engineering Trustworthy Agentic AI for Critical Systems
Jul 20, 2026Omar Al-Refai, Ibrahim Shahbaz, Adam Ali Husseinat +3AI Assurance Agent Reliability
Innovations in Systems Trust and Resilience (iSTAR) Laboratory, Texas A&M University, TX, USA · Texas A&M Institute of Data Science - Security, Privacy and Resilience for Trusted AI (SPARTA) Thematic Lab., Texas A&M University, TX, USA · Texas A&M Global Cyber Research Institute (GCRI), Texas A&M University, TX, USA
Operational Hallucination and Safety Drift in AI Agents
Jul 20, 2026Shasha Yu, Fiona Carroll, Barry L. BentleyLLM Agent Evaluation AI Agent Safety
Cardiff School of Technologies, Cardiff Metropolitan University Cardiff, UK · Harvard Medical School, Harvard University Boston, USA
Specifying the Delegated-Autonomy Boundary: Requirements Engineering for Agentic AI
Jul 19, 2026Chetan Arora, Andreas Vogelsang, Abbi SharmaAI Agent Safety AI Agent Governance
Faculty of IT, Monash University Melbourne, Victoria, Australia · University of Duisburg-Essen Essen, Germany
PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution
Jul 18, 2026Yang Liu, Weixing Chen, Xinshuai Song +8LLM Agent Orchestration AI Agent Safety
1X-Era Lab · 2HCP Lab, Sun Yat-sen University · 3Peng Cheng Laboratory
SeerGuard: A Safety Framework for Mobile GUI Agents via World Model Prediction
Jul 17, 2026Xue Yu, Bo Yuan, Kailin Zhao +3GUI Agents AI Agent Safety
JIUTIAN Research

Aug 27, 2026, cs.LGJ/K move · Enter open · S save

Beyond Task Completion: Training Capable and Safe Computer-Use Agents

Zeyu Kang, Zhenyun Yin, Yang Zhang, Shan He+4

Shanghai Artificial Intelligence Laboratory · Harbin Institute of Technology · Fudan University+2

Computer-Use Agents AI Agent Safety LLM Agent Training RL for Tool Use

Open paper AskPDF arXiv

Computer-use agents (CUAs) have made rapid progress in completing complex tasks through graphical user interfaces, yet post-training centered on task success alone does not induce reliable safety behavior. A reliable CUA must condition its execution on risk: it should complete ordinary benign tasks, avoid environmental hazards and continue when a safe completion path remains, and refuse when the goal is harmful or no safe path exists. To learn this conditional policy, we develop Safety and Capability Optimization for Policy Execution (SCOPE), which jointly post-trains a CUA for task-execution capability and safety-aware decision making. To provide aligned training data for this joint objective, we further introduce SCOPE-Gen, an automated pipeline that synthesizes verifiable capability tasks and converts them into paired environment-risk variants while preserving their original goals. Using the resulting tasks, we construct SATraj-OS, a trajectory dataset comprising capability demonstrations, safe continuations, and explicit refusals. SCOPE first learns from all three trajectory types through supervised fine-tuning and then further improves task completion through online reinforcement learning. Starting from Qwen3.5-9B, SCOPE-RL achieves a 54.17% task success rate on OSWorld and a 64.30% attack-avoidance rate on OS-BLIND, yielding the best aggregate capability--safety score of 58.80% among the evaluated agents. Ablations reveal asymmetric but complementary roles for the two forms of safety supervision: refusal trajectories account for most of the attack-avoidance gain, whereas risk-handling trajectories preserve greater task utility at comparable attack-avoidance levels.

Momentum

Related topics

Latest papers 207