AI Agent Safety

Follow this topic Ask about this topic

AI agent safety asks how to keep systems that plan over multiple steps and use tools, memory, or other agents within authorized and harmless bounds. Researchers test computer-use, coding, web, and embodied agents with prompt-injection, misuse, and long-horizon trajectory benchmarks, measuring executed effects as well as final answers. Approaches include red-teaming and monitoring, safety fine-tuning and refusal training, and runtime controls such as least-privilege access, pre-execution policy gates, sandboxing, human approval, and formal verification. A central challenge is that safety can fail through ambiguous goals, changing context, tool or agent interactions, and pressure over time, while stronger restrictions can block legitimate tasks and assurances may not transfer to deployment.

Latest papers

330 papers47 papers in the last four weeks, up 68% on the four weeks before. 0.3% of all new papers.

LLM Agents Make Collective Belief Dynamics Programmable: Challenges and Research Directions
May 19, 2026Xin He, Junxi Shen, Yuchen Mou +4Opinion Dynamics AI Agent Safety
Centre for Frontier AI Research, Agency for Science, Technology and Research (A*STAR), Singapore · Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore · College of Design and Engineering, National University of Singapore, Singapore +1
Agent Meltdowns: The Road to Hell Is Paved with Helpful Agents
May 18, 2026Rishi Jha, Harold Triedman, Arkaprabha Bhattacharya +1Agent Failure Analysis AI Agent Reliability
Department of Computer Science Cornell University
Surviving the Unseen: Predictive Defense for Novel Multi-Turn Multimodal Attacks
May 18, 2026Doohee YouAdversarial Attacks on VLMs Multimodal Robustness
Trust and Safety, Google
LivePI: More Realistic Benchmarking of Agents Against Indirect Prompt Injection
May 18, 2026Lei Zhao, Abhay Bhaskar, Edgar DobribanAI Agent Security AI Agent Safety
University of Pennsylvania
Ethical Hyper-Velocity (EHV): A Hardware-Rooted Zero-Trust Runtime Enforcement Architecture for Agentic AI Systems
May 18, 2026Riddhi Mohan SharmaAI Agent Safety Runtime Enforcement for AI Agents
IAM Architect & Non-Human Identity Governance Enterprise Identity, M&A Integration, Regulated Environments
Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security
May 17, 2026Jinhu Qi, Muzhi Li, Jiahong Liu +9AI Agent Reliability AI Agent Evaluation
Faculty of Engineering, Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China. · Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, China. · Shanghai Academy of AI for Science, Shanghai, China.
TClone: Low-Latency Forking of Live GUI Environments for Computer-Use Agents
May 17, 2026Yutong Huang, Vikranth Srivatsa, Alex Asch +2Computer-Use Agents AI Agent Safety
University of California, San Diego · University of California, San Diego and GenseeAI
LiSA: Lifelong Safety Adaptation via Conservative Policy Induction
May 14, 2026Minbeom Kim, Lesly Miculicich, Bhavana Dalvi Mishra +6Continual Learning for LLM Agents LLM Guardrails
Google Cloud AI Research · Google · Seoul National University
Conformity Generates Collective Misalignment in AI Agents Societies
May 11, 2026Giordano De Marzo, Alessandro Bellina, Claudio Castellano +2Opinion Dynamics AI Agent Safety
University of Konstanz, Konstanz, Germany · Centro Ricerche Enrico Fermi, Rome, Italy · Complexity Science Hub, Vienna, Austria +5
Beyond Autonomy: A Dynamic Tiered AgentRunner Framework for Governable and Resilient Enterprise AI Execution
May 11, 2026Kai Pan, Rong HouLLM Agent Orchestration AI Agent Safety
A2A Lab
The Authorization-Execution Gap Is a Major Safety and Security Problem in Open-World Agents
May 10, 2026Baoyuan Wu, Qingshan Liu, Adel Bibi +2Agent Failure Analysis AI Agent Security
Chinese University of Hong Kong, Shenzhen, China · Nanjing University of Posts and Telecommunications, China · University of Oxford, UK +2
A Prompt-Aware Structuring Framework for Reliable Reuse of AI-Generated Content in the Agentic Web
May 10, 2026Shusaku Egami, Masahiro HamasakiAI Agent Safety LLM Agent Reliability
National Institute of Advanced Industrial Sciencen and Technology (AIST) · Koto, Tokyo, Japan
Containment Verification: AI Safety Guarantees Independent of Alignment
May 9, 2026Royce Moon, Lav R. VarshneyAI Agent Safety Formal Verification
Enclave Intelligence · AI Innovation Institute, Stony Brook University
Mechanism Design Is Not Enough: Prosocial Agents for Cooperative AI
May 8, 2026Xuanqiang Angelo Huang, Charlie Tharas, Samuele Marro +4Mechanism Design AI Agent Safety
ETH Zürich · University of Oxford · Institute for Decentralized AI +5
Playing games with knowledge: AI-Induced delusions need game theoretic interventions
May 8, 2026Will Beaumaster, Paul SchraterLLM Sycophancy Mechanism Design
University of Minnesota
Why Does Agentic Safety Fail to Generalize Across Tasks?
May 7, 2026Yonatan Slutzky, Yotam Alexander, Tomer Slor +2AI Agent Safety LLM Agent Safety
Tel Aviv University
Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors
May 7, 2026Jonas Wiedermann-Möller, Leonard Dung, Maksym AndriushchenkoAI Agent Evaluation AI Agent Safety
Universität Bielefeld · Ruhr-Universität Bochum · ELLIS Institute Tübingen +2
Safactory: A Scalable Agentic Infrastructure for Training Trustworthy Autonomous Intelligence
May 7, 2026Xinquan Chen, Zhenyun Yin, Shan He +38Agentic RL AI Agent Safety
Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges
May 7, 2026Shihao Weng, Yang Feng, Xiaofei XieLLM-as-a-Judge AI Agent Safety
Nanjing University Nanjing, China · Singapore Management University Singapore
TRACE: A Metrologically-Grounded Engineering Framework for Trustworthy Agentic AI Systems in Operationally Critical Domains
May 5, 2026Serhii ZabolotniiAI Agent Safety AI Agent Governance
Cherkasy State Business College, Cherkasy 18028, Ukraine · State Scientific Research Institute of Armament and Military Equipment Testing and Certification, Cherkasy, Ukraine · healthPrecision, New York, NY, USA
Enhancing Agent Safety Judgment: Controlled Benchmark Rewriting and Analogical Reasoning for Deceptive Out-of-Distribution Scenarios
May 5, 2026Zuoyu Zhang, Yancheng ZhuAnalogical Reasoning for Language Models AI Agent Safety
Shenzhen University, Shenzhen, China.
An Empirical Study of Agent Skills for Healthcare: Practice, Gaps, and Governance
May 4, 2026Gelei Xu, Ningzhi Tang, Xueyang Li +4Healthcare AI Agent Safety
University of Notre Dame · Emory University
Executor-Side Progressive Risk-Gated Actuation for Agentic AI in Wireless Supervisory Control
May 4, 2026Zhenyu Liu, Yi Ma, Rahim TafazolliSafety-Critical Control AI Agent Safety
6GIC, Institute for Communication Systems, University of Surrey, Guildford, United Kingdom, GU2 7XH
MILD: Mediator Agent System with Bidirectional Perception and Multi-Layered Alignment for Human-Vehicle Collaboration
May 2, 2026Jiyao Wang, Yunbiao Wang, Yubo Jiao +6AI Alignment AI Agent Safety
Department of Civil Engineering, McGill University, Montreal H3A 0G4, Quebec, Canada · Systems Hub, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, Guangdong, China · Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg L-4365, Luxembourg
Position: Safety and Fairness in Agentic AI Depend on Interaction Topology, Not on Model Scale or Alignment
May 1, 2026Tanav Singh Bajaj, Nikhil Singh, Karan Anand +1Opinion Dynamics AI Agent Safety
Department of Computer Science, University of British Columbia, Vancouver, Canada · Department of Artificial Intelligence, IIT Hyderabad, Hyderabad, India · Amazon, Delhi, India.
AgenticAITA: A Proof-Of-Concept About Deliberative Multi-Agent Reasoning for Autonomous Trading Systems
May 1, 2026Ivan LetteriMulti-Agent Orchestration AI Agent Safety
Department of Information Engineering, Computer Science, and Mathematics, Center of Excellence for Research DEWS, University of L’Aquila, Via Vetoio, Coppito, 67100 L’Aquila, Italy
Causal Foundations of Collective Agency
Apr 30, 2026Frederik Hytting Jørgensen, Sebastian Weichwald, Lewis HammondAI Agent Safety Causal Abstraction
Copenhagen Causality Lab and Pioneer Centre for AI, University of Copenhagen, Denmark · Cooperative AI Foundation, United Kingdom
Safe Bilevel Delegation (SBD): A Formal Framework for Runtime Delegation Safety in Multi-Agent Systems
Apr 30, 2026Yuan SunAI Agent Safety Multi-Agent Systems
Jilin University
Ambient Persuasion in a Deployed AI Agent: Unauthorized Escalation Following Routine Non-Adversarial Content Exposure
Apr 29, 2026Diego F. Cuadros, Abdoul-Aziz MaigaAgent Failure Analysis AI Agent Security
Digital Epidemiology Laboratory, Digital Futures, University of Cincinnati, Cincinnati, OH, USA · Center for Humanities and Technology (CHaT), University of Cincinnati, Cincinnati, OH, USA · Norwegian University of Science and Technology, Trondheim, Norway
Toward a Science of Intent: Closure Gaps and Delegation Envelopes for Open-World AI Agents
Apr 27, 2026Maximiliano Armesto, Christophe KolbAI Agent Evaluation AI Agent Safety
Taller Technologies

May 19, 2026J/K move · Enter open · S save

LLM Agents Make Collective Belief Dynamics Programmable: Challenges and Research Directions

Xin He, Junxi Shen, Yuchen Mou, David M. Bossens+3

Centre for Frontier AI Research, Agency for Science, Technology and Research (A*STAR), Singapore · Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore · College of Design and Engineering, National University of Singapore, Singapore+1

Opinion Dynamics AI Agent Safety Multi-Agent LLMs

Open paper AskPDF arXiv

Classical models of opinion dynamics assume human participants with bounded rationality and limited coordination. The rise of LLM-based agents introduces a qualitative shift: agents can now participate in online discussions at scale, maintain consistent persuasion strategies, and coordinate systematically. This paper argues that LLM agents make collective belief dynamics programmable, enabling deliberate steering of population-level beliefs. We term this emerging problem programmable collective belief control. Through controlled multi-agent simulations, we provide proof-of-concept evidence that coordinated AI agents can induce measurable belief shifts that stabilize within a few interaction rounds. We identify four structural properties (indistinguishability, persistence, contextuality, and configurability) that make detection and defense fundamentally difficult. Based on these findings, we outline a research agenda spanning theoretical foundations for adversarial belief dynamics, operational methods for system-level detection and intervention, and simulation infrastructure for scalable experimentation. Our goal is not to present a complete solution, but to articulate why this problem demands urgent attention and to provide a conceptual foundation for future work.