LLM Agent Safety

LLM: Large Language Model

The field studies how language-model agents can cause harm through tool use, long-running workflows, memory, or coordination, even when their chat responses appear safe. Researchers test these risks with multi-turn and executable benchmarks for prompt injection, computer and phone use, privacy, misalignment, and multi-agent behavior. Common defenses include runtime monitors, per-action authorization, formal policy checks, trajectory-aware guards, and training methods that teach agents to refuse or recover safely. A central challenge is preserving useful task completion: safety in text does not reliably transfer to actions, while overly broad defenses can block benign work or fail as context and system state change.

Latest papers

327 papers56 papers in the last four weeks, up 75% on the four weeks before. 0.3% of all new papers.

The Autonomy Tax: Defense Training Breaks LLM Agents
Mar 19, 2026Shawn Li, Yue ZhaoAI Agent Security LLM Agents
University of Southern California
Security awareness in LLM agents: the NDAI zone case
Mar 19, 2026Enrico Bottazzi, Pia ParkAI Agent Security LLM Agent Security
Leku
I Can't Believe It's Corrupt: Evaluating Corruption in Multi-Agent Governance Systems
Mar 19, 2026Vedanta S P, Ponnurangam KumaraguruAI Agent Governance LLM Agent Safety
IIIT Kottayam · IIIT Hyderabad
Differential Harm Propensity in Personalized LLM Agents: The Curious Case of Mental Health Disclosure
Mar 17, 2026Caglar YildirimLLM Agent Safety LLM Agent Evaluation
Khoury College of Computer Sciences Northeastern University Boston, MA 02115, USA
Invisible Orchestrators Suppress Protective Behavior and Dissociate Power-Holders: Safety Risks in Multi-Agent LLM Systems
Mar 17, 2026Hiroki FukuiMulti-Agent LLM Systems Multi-Agent Orchestration
Criminal Psychiatry Research Institute / Sexual Offender Medical Center Department of Neuropsychiatry, Kyoto University
TrinityGuard: A Unified Framework for Safeguarding Multi-Agent Systems
Mar 16, 2026Kai Wang, Biaojie Zeng, Zeming Wei +7LLM Agent Safety Multi-Agent System Security
Shanghai AI Laboratory
Why Agents Compromise Safety Under Pressure
Mar 16, 2026Hengle Jiang, Ke TangSafety-Utility Trade-Off in Language Models AI Agent Safety
Guangdong Provincial Key Laboratory of Brain-inspired Intelligent Computation, Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China
Emotional Cost Functions for AI Safety: Teaching Agents to Feel the Weight of Irreversible Consequences
Mar 15, 2026Pandurang MopgarAI Agent Safety LLM Agent Safety
Independent Researcher
DeceptGuard :A Constitutional Oversight Framework For Detecting Deception in LLM Agents
Mar 14, 2026Snehasis MukhopadhyayDeception in Language Models AI Agent Monitoring
Indian Institute of Information Technology Kalyani
When the Loop Closes: Architectural Limits of In-Context Isolation, Metacognitive Co-option, and the Two-Target Design Problem in Human-LLM Systems
Mar 14, 2026Z. Cheng, N. SongLLM Context Management Human-in-the-Loop AI
Independent Researcher
Sell Me This Stock: Unsafe Recommendation Drift in LLM Agents
Mar 13, 2026Zekun Wu, Adriano Koshiyama, Sahan Bulathwela +1LLM Agent Safety Language Model Safety Evaluation
Centre for Artificial Intelligence, University College London · Holistic AI
How Emotion Shapes the Behavior of LLMs and Agents: A Mechanistic Study
Mar 9, 2026Moran Sun, Tianlin Li, Yuwei Zheng +4Emotion Representation in Language Models Language Model Steering
School of Computer Science and Engineering, Beihang University, Beijing, China · College of Computing and Data Science, Nanyang Technological University, Singapore
Intentional Deception as Controllable Capability in LLM Agents
Mar 8, 2026Jason Starace, Terence SouleDeception in Language Models LLM Agent Safety
Department of Computer Science, University of Idaho, Moscow, ID 83844
Alignment Backfire: Language-Dependent Reversal of Safety Interventions Across 16 Languages in LLM Multi-Agent Systems
Mar 5, 2026Hiroki FukuiLLM Safety LLM Alignment
Research Institute of Criminal Psychiatry / Sex Offender Medical Center; Department of Neuropsychiatry, Kyoto University
When Agents Persuade: Rhetoric Generation and Mitigation in LLMs
Mar 4, 2026Julia Jose, Ritik Roongta, Rachel GreenstadtPersuasion in Language Models LLM Agent Safety
Department of Computer Science and Engineering, New York University, New York, NY, USA
Learning When to Act or Refuse: Guarding Agentic Reasoning Models for Safe Multi-Step Tool Use
Mar 3, 2026Aradhye Agarwal, Gurdit Siyan, Yash Pandya +3LLM Agent Safety Tool-Using Language Agents
Microsoft Research
Contextualized Privacy Defense for LLM Agents
Mar 3, 2026Yule Wen, Yanzhe Zhang, Jianxun Lian +3Privacy Protection for LLM Agents LLM Agent Security
Tsinghua University · Georgia Tech · Microsoft +1
Constitutional Black-Box Monitoring for Scheming in LLM Agents
Feb 28, 2026Simon Storf, Rich Barton-Cooper, James Peters-Gill +1AI Agent Monitoring LLM Agent Safety
MATS Research · Independent · Apollo Research
SAGE-LLM: Towards Safe and Generalizable LLM Controller with Fuzzy-CBF Verification and Graph-Structured Knowledge Retrieval for UAV Decision
Feb 27, 2026Wenzhe Zhao, Yang Zhao, Ganchao Liu +4Language Model-Based Control LLM Agent Safety
School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University, Xi’an 710072, China · Institute of Artificial Intelligence (TeleAI), China Telecom, China
AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purification
Feb 26, 2026Tian Zhang, Yiwei Xu, Juan Wang +8Prompt Injection Defense Prompt Injection Attacks on AI Agents
Key Laboratory of Aerospace Information Security and Trusted Computing, Ministry of Education, School of Cyber Science and Engineering, Wuhan University, Wuhan 430072, China · Department of Computer Science and Engineering, University at Buffalo, SUNY · Department of Computer Science, College of Information Science and Technology, Jinan University, Guangzhou 510632, China +1
Mind the GAP: Text Safety Does Not Transfer to Tool-Call Safety in LLM Agents
Feb 18, 2026Arnold Cartagena, Ariane TeixeiraLLM Agent Safety AI Agent Safety Benchmarks
Independent Researcher
Helpful to a Fault: Measuring Illicit Assistance in Multi-Turn, Multilingual LLM Agents
Feb 18, 2026Nivya Talokar, Ayush K Tarun, Murari Mandal +2LLM Red Teaming LLM Agent Safety
Independent Researcher · EPFL, Switzerland · KIIT Bhubaneswar, India +1
Risky-Bench: Evaluating Agentic Safety Risks in Real-World Life-Assist Deployments
Feb 3, 2026Jingnan Zheng, Yanzhen Luo, Jingjun Xu +8Language Model Safety Evaluation LLM Agent Safety
National University of Singapore · University of Science and Technology of China · Southern University of Science and Technology +3
Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safety
Oct 18, 2025Vamshi Krishna Bonagiri, Ponnurangam Kumaragurum, Khanh Nguyen +1LLM Agent Evaluation Tool-Using Agents
University of California, Berkeley · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · International Institute of Information Technology, Hyderabad (IIIT Hyderabad)
Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks
Oct 16, 2025Trilok Padhi, Pinxian Lu, Abdulkadir Erol +5Adversarial Attacks on LLMs AI Agent Security Benchmarks
1Georgia State University · 2Georgia Institute of Technology · University of California, Berkeley
Measuring Harmfulness of Computer-Using Agents
Jul 31, 2025Aaron Xuxiang Tian, Ruofan Zhang, Janet Tang +3LLM Agent Safety Computer-Use Agent Benchmarks
Independent Researcher · Arizona State University · Gradient +1
AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents
Jun 4, 2025Akshat Naik, Emma Gouné, Patrick Quinn +4AI Alignment LLM Agent Evaluation
Independent Researcher · Department of Engineering University of Cambridge

Mar 19, 2026J/K move · Enter open · S save

The Autonomy Tax: Defense Training Breaks LLM Agents

Shawn Li, Yue Zhao

University of Southern California

AI Agent Security LLM Agents LLM Agent Safety Prompt Injection Defense Prompt Injection Attacks on AI Agents

Open paper AskPDF arXiv

Large language model (LLM) agents increasingly rely on external tools (file operations, API calls, database transactions) to autonomously complete complex multi-step tasks. Practitioners deploy defense-trained models to protect against prompt injection attacks that manipulate agent behavior through malicious observations or retrieved content. We reveal a fundamental capability-alignment paradox: defense training designed to improve safety systematically destroys agent competence while failing to prevent sophisticated attacks. Evaluating defended models against undefended baselines across 97 agent tasks and 1,000 adversarial prompts, we uncover three systematic biases unique to multi-step agents. Agent incompetence bias manifests as immediate tool execution breakdown, with models refusing or generating invalid actions on benign tasks before observing any external content. Cascade amplification bias causes early failures to propagate through retry loops, pushing defended models to timeout on 99% of tasks compared to 13% for baselines. Trigger bias leads to paradoxical security degradation where defended models perform worse than undefended baselines while straightforward attacks bypass defenses at high rates. Root cause analysis reveals these biases stem from shortcut learning: models overfit to surface attack patterns rather than semantic threat understanding, evidenced by extreme variance in defense effectiveness across attack categories. Our findings demonstrate that current defense paradigms optimize for single-turn refusal benchmarks while rendering multi-step agents fundamentally unreliable, necessitating new approaches that preserve tool execution competence under adversarial conditions.