LLM Agent Safety

LLM: Large Language Model

The field studies how language-model agents can cause harm through tool use, long-running workflows, memory, or coordination, even when their chat responses appear safe. Researchers test these risks with multi-turn and executable benchmarks for prompt injection, computer and phone use, privacy, misalignment, and multi-agent behavior. Common defenses include runtime monitors, per-action authorization, formal policy checks, trajectory-aware guards, and training methods that teach agents to refuse or recover safely. A central challenge is preserving useful task completion: safety in text does not reliably transfer to actions, while overly broad defenses can block benign work or fail as context and system state change.

Latest papers

327 papers56 papers in the last four weeks, up 75% on the four weeks before. 0.3% of all new papers.

Towards Healthy Evolution: Exploring the Role and Mechanisms of Human-Agent Interaction in Self-Evolving Systems
Jun 4, 2026Dianxing Shi, Bowen Wang, Junqi He +2Human-in-the-Loop Evaluation LLM Safety Alignment
The University of Osaka · Independent researcher
From Risk Classification to Action Plan Remediation: A Guardrail Feedback Driven Framework for LLM Agents
Jun 4, 2026Yuhao Sun, Jiacheng Zhang, Shaanan Cohney +3LLM Guardrails LLM Agent Training
1The University of Melbourne · 2Tsinghua University · †Corresponding author.
Provably Auditable and Safe LLM Agents from Human-Authored Ontologies
Jun 3, 2026Aaron SterlingAI Agent Auditing LLM Auditing
Thistleseeds
Organizational Control Layer: Governance Infrastructure at the Execution Boundary of LLM Agent Systems
Jun 3, 2026Tianyu Shi, Yang Mo, Yiou Liu +6Multi-Agent LLM Systems AI Agent Governance
McGill · Purdue · UNSW +3
RUBAS: Rubric-Based Reinforcement Learning for Agent Safety
Jun 2, 2026Xian Qi Loye, Qinglin Su, Zhexin Zhang +5Reinforcement Learning AI Agent Safety
The Conversational AI (CoAI) group, DCST, Tsinghua University · Huawei Noah’s Ark Lab
SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoning
Jun 1, 2026Lichao Wang, Zhaoxing Ren, Tianzhuo Yang +4LLM Agent Safety
1Beijing Institute of Technology · 2Beijing Academy of Artificial Intelligence · Institute for Artificial Intelligence, Peking University
Defenses & Enablers For Skill Injection Attacks on Terminal Based Agents
Jun 1, 2026Yoshinari Fujinuma, Varun Gangal, Traian Rebedea +4Adversarial Attacks on LLMs LLM Agent Security
Patronus AI · NVIDIA
A New Framework for Cybersecurity Refusals in AI Agents
May 31, 2026Eliot Krzysztof Jones, Mateusz Dziemian, Matt Fredrikson +1LLM Agent Evaluation LLM Refusal Behavior
Work done while at Gray Swan · Gray Swan AI · Carnegie Mellon University
Adversarial Feeds Steer LLM Agent Decisions Against Their Defaults
May 30, 2026Rana Muhammad UsmanAdversarial Attacks on LLMs LLM Agent Evaluation
Independent Researcher
Same Payload, Different Channel: Measuring Trust Asymmetry in Tool-Using Language Models
May 30, 2026Mohammed Sameer Syed, Rozhin YasaeiAdversarial Attacks on LLMs Prompt Injection Attacks
University of Arizona
COMPASS: Cognitive MCTS-Guided Process Alignment for Safe Search Agents
May 29, 2026Wenkai Shen, Pengyang Zhou, Jiahe Xu +5LLM Safety Alignment LLM Agents
Zhejiang University
Relevance as a Vulnerability: How Web Retrieval Degrades Safety Alignment in LLM Agents
May 28, 2026Aditya Nawal, Manit Baser, Mohan GurusamyAI Agent Security Benchmarks LLM Safety Evaluation
Department of Electrical and Computer Engineering · National University of Singapore
SafeRx-Agent: A Knowledge-Grounded Multi-Agent Framework for Safe and Explainable Medication Recommendation
May 27, 2026Xinyu Wang, Hanwei Wu, Zhenghan Tai +7Medication Recommendation Multi-Agent LLM Systems
McGill University · McMaster University · University of Toronto +1
LACUNA: Safe Agents as Recursive Program Holes
May 27, 2026Yaoyu Zhao, Yichen Xu, Oliver Bračevac +3AI Agent Reliability LLM Agents
EPFL, Lausanne, Switzerland
Plant, Persist, Trigger: Sleeper Attack on Large Language Model Agents
May 27, 2026Yongxiang Li, Moxin Li, Zhixin Ma +2LLM Agent Security AI Agent Security Benchmarks
University of Science and Technology of China · National University of Singapore · Singapore Management University
TRACES: Proactive Safety Auditing for Multi-Turn LLM Agents via Trajectory-State Modeling
May 26, 2026Jiaqian Li, Yanshu Li, Boxuan Zhang +2LLM Auditing LLM Agent Safety
Brown University · The University of Texas at Austin · Rutgers University +1
Voluntary Collusion with Secret Tools in Competing LLM Agents
May 26, 2026Xijie Zeng, Frank RudziczLLM Agent Evaluation LLM Agent Safety
Dalhousie University · Vector Institute for Artificial Intelligence
FinHarness: An Inline Lifecycle Safety Harness for Finance LLM Agents
May 26, 2026Haoxuan Jia, Yang Liu, Bin Chong +10AI Agent Reliability LLM Guardrails
Nanyang Technological University · Tsinghua University · Peking University +8
Device Context Protocol: A Compact, Safety-First Architecture for LLM-Driven Control of Constrained Devices
May 24, 2026Dongxu YangLanguage Model-Based Control MCP Security
1DeepLethe
Open-source LLMs administer maximum electric shocks in a Milgram-like obedience experiment
May 20, 2026Roland Pihlakas, Jan Llenzl DagohoyLLM Refusal Behavior LLM Safety Evaluation
for the Three Laws collaboration · Independent researcher · Three Laws research collaboration +2
Measuring Safety Alignment Effects in Autonomous Security Agents
May 19, 2026Isaac David, Arthur GervaisLanguage Model Safety Evaluation LLM Safety Alignment
University College London
POLAR-Bench: A Diagnostic Benchmark for Privacy-Utility Trade-offs in LLM Agents
May 18, 2026Qiaoyuan Zheng, Yiqu Yang, Qi Gao +1LLM Agent Evaluation LLM Agent Security
ETH Zurich Zurich, Switzerland · ETH AI Center Zurich, Switzerland
Trustworthy Agent Network: Trust in Agent Networks Must Be Baked In, Not Bolted On
May 18, 2026Yixiang Yao, Yuhang Yao, Xinyi Fan +5Multi-Agent LLM Systems Multi-Agent Coordination
University of Southern California · Carnegie Mellon University · University of Illinois Urbana-Champaign
Position: A Three-Layer Probabilistic Assume-Guarantee Architecture Is Structurally Required for Safe LLM Agent Deployment
May 18, 2026S. Bensalem, Y. Dong, M. Franzle +6AI Agent Reliability LLM Guardrails
1CSX-AI, France · University of Liverpool, UK · 3Carl von Ossietzky Universität Oldenburg, Germany +2
Remembering More, Risking More: Longitudinal Safety Risks in Memory-Equipped LLM Agents
May 18, 2026Ahmad Al-Tawaha, Shangding Gu, Peizhi Niu +2Long-Horizon Agent Evaluation LLM Safety Evaluation
Virginia Tech · University of California, Berkeley · University of Illinois Urbana-Champaign
ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agents
May 17, 2026Udari Madhushani Sehwag, Zhengyang Shan, Heming Liu +3LLM Agent Security AI Agent Security Benchmarks
Scale AI · 2Boston University · 4Human Frontier Collective +3
State Contamination in Memory-Augmented LLM Agents
May 16, 2026Yian Wang, Agam Goyal, Yuen Chen +1LLM Agent Memory LLM Agent Safety
Department of Computer Science, University of Illinois Urbana-Champaign
Web Agents Should Adopt the Compile-and-Execute Paradigm
May 14, 2026Julien Piet, Annabella Chow, Yiwei Hou +5LLM Agent Safety LLM Agent Planning
UC Berkeley
Auditing Agent Harness Safety
May 14, 2026Chengzhi Liu, Yichen Guo, Yepeng Liu +8AI Agent Auditing LLM Auditing
University of California, Santa Barbara · University of California, Berkeley · Microsoft Research +2
History Anchors: How Prior Behavior Steers LLM Decisions Toward Unsafe Actions
May 13, 2026Alberto G. Rodríguez SalgadoLLM Safety Evaluation AI Agent Safety Benchmarks
Independent Researcher

Jun 4, 2026J/K move · Enter open · S save

Towards Healthy Evolution: Exploring the Role and Mechanisms of Human-Agent Interaction in Self-Evolving Systems

Dianxing Shi, Bowen Wang, Junqi He, Junhao Chen+1

The University of Osaka · Independent researcher

Human-in-the-Loop Evaluation LLM Safety Alignment LLM Agent Safety Self-Evolving Agents

Open paper AskPDF arXiv

Self-evolving agents improve through continual self-play and self-generated learning signals, but autonomous evolution can also cause capability degradation and safety drift. Although human feedback has proven effective for static and post-trained agents, its role in self-evolving systems remains underexplored. We introduce Agent Norm Correction through Human-like Oversight and Review (ANCHOR), an LLM-based framework that simulates human supervision and delivers feedback at various phases of self-evolution. With ANCHOR, we evaluate two representative open-source self-evolving agent systems across coding, mathematical reasoning, and safety. Our results show that even limited supervision substantially mitigates safety degradation while preserving stable performance on core evolutionary objectives. Further analysis shows that supervision over the output verification phase is the most effective for intervention, whereas increasing supervision frequency yields diminishing returns. These findings provide empirical evidence and practical guidance for designing more stable, controllable, and human-aligned self-evolving agent systems.