LLM Agent Safety

LLM: Large Language Model

Overview

The field studies how language-model agents can cause harm through tool use, long-running workflows, memory, or coordination, even when their chat responses appear safe. Researchers test these risks with multi-turn and executable benchmarks for prompt injection, computer and phone use, privacy, misalignment, and multi-agent behavior. Common defenses include runtime monitors, per-action authorization, formal policy checks, trajectory-aware guards, and training methods that teach agents to refuse or recover safely. A central challenge is preserving useful task completion: safety in text does not reliably transfer to actions, while overly broad defenses can block benign work or fail as context and system state change.

Latest papers

CardsList

358 papers55 papers in the last four weeks, down 2% on the four weeks before. 0.4% of all new papers.

  1. Intent Contracts: Taking Tools Away from Agents

    Nov 18, 2026Ziyuan Wang, Genliang Zhu, Xiaobo Chen +1Tool Access Control for LLM AgentsLLM Agent Safety

  2. Recovery Gone Too Far: Understanding and Repairing Risky Fallback in LLM Agents

    Nov 15, 2026Liwen Wang, Jiangrong Wu, Zongjie Li +3LLM Agent SafetyTool Access Control for LLM Agents

  3. MemeEconomy: Do LLM Agents Trade Ethics for Survival?

    Nov 13, 2026Syed Nazmus Sakib, Nafiul Haque, Ahnaf Tahmid Manan +2LLM Agent SafetySafety-Utility Trade-Off in Language Models

  4. Soteria: Formally Verified Planning with Runtime Enforcement for Safe LLM Agents

    Nov 13, 2026Mike He, Ankush Desai, Sharad Malik +1LLM Agent SafetyLLM Agent Planning

  5. What LLM Agents See and What Users Say

    Nov 13, 2026Yue Zhao, Angela Huyue Zhang, S. Alex Yang +2LLM Agent SafetyLLM Agent Evaluation

  6. Only as Safe as the Weakest Agent: A Break That Appears Only When the System Is Multi-Agent

    Nov 12, 2026Keegan Wang, Anantika MannbyLLM Agent SafetyLLM Agent Security

  7. Bait: Mitigating Harmful Agentic Reinforcement Learning via Probing Reward Function

    Nov 12, 2026Tiansheng Huang, Ananth Balashankar, Virat Shejwalkar +2AI Agent SecurityLLM Agent Safety

  8. When Evaluators of AI Agents Disagree: An Empirical Study of Native Scoring, LLM Judges, and Commit-Boundary Checks on tau2-bench

    Nov 10, 2026Sungbin ImAI Agent EvaluationLLM-as-a-Judge

  9. DEPUTYBENCH: Evaluating Contextual Safety in Enterprise Agents

    Nov 9, 2026Hao Zhao, Minghao LinAI Agent Security BenchmarksLLM Agent Safety

  10. ShadowBench-MCP: Measuring Shadow Invocations in MCP-Style Tool Routing

    Nov 9, 2026Eyhab Al-Masri, Wenjun YangLLM Agent SecurityLLM Routing

  11. Treating the Model as an Untrusted Component: Deterministic Policy Enforcement, Human Approval and Taint Tracking Against Prompt Injection in Tool-Using Agents, with Validation on Open-Weights Models

    Nov 8, 2026Mustafa S. Aljumaily, Nawar S. Alseelawi, Raghad AlshabandarPrompt Injection DefenseLLM Agent Safety

  12. Safeguarding Agentic AI in Radiology Workflows: A Multimodel Evaluation of Rule-based and Large Language Model-based Verification

    Nov 8, 2026Semil Eminovic, Robin Schmidt, Bogdan Levita +3LLM Agent SafetyLLM Guardrails

  13. When Experience Becomes a Backdoor: Poisoning Self-Evolving LLM Agents via Failure-Aware Task Optimization

    Nov 7, 2026Minhua Lin, Zongyu Wu, Hanqing Lu +4LLM Backdoor AttacksBackdoor Attacks

  14. Tail-risk-aware Agent Safety: Constraining Tail Risks Across Multi-step LLM Agent Workflows

    Nov 7, 2026Shilong Pan, Zhiliang Tian, Zihan Chen +3LLM Agent Safety

  15. SafeOrchestrate: Runtime Checked Intent-Driven Orchestration for Cyber-Physical Systems

    Nov 6, 2026Muhammad Ashfaq, Ahmed R. Sadik, Ewan Warburton +3Cyber-Physical SystemsLLM Agent Orchestration

  16. Authorization Drift in LLM Agents: Evaluating Boundary Compliance in Multi-Step Tool Use

    Nov 6, 2026Chuxue Cao, Jingru Zeng, Boqin Yin +7LLM Agent SafetyTool Access Control for LLM Agents

  17. When Routine Operations Break Architectural Pattern Relationships in LLM Agent Harnesses

    Nov 6, 2026Boming Xia, Qinghua Lu, Dino Sejdinovic +1LLM Agent HarnessesSoftware Engineering

  18. Trilemma: How Agents choose between Honesty, Confidentiality, and Task Completion

    Nov 5, 2026Johannes Taraz, Abhinav Kumar, Sahar AbdelnabiLLM Agent SafetyPersonal AI Agents

  19. Stress Testing Chain-of-Thought Monitoring Against Covert Misalignment

    Nov 5, 2026Yao Huang, Yifan Wang, Yitong Sun +2LLM Agent SafetyCoT Monitoring

  20. The Memory Trap: Language Agents Overgeneralize from Benign Interactions

    Nov 5, 2026Kaiqu Liang, Jaime Fernández Fisac, Thomas L. Griffiths +1Memory-Augmented Language Model AgentsLLM Agent Safety

  21. When Do Summaries Become Instructions? Unsupported Policy in LLM Agents' Self-Written Handoff Summaries

    Nov 4, 2026Wenhui ChuLLM Agent Policy ComplianceLLM Agent Safety

  22. Voluntary Servitude: Weaponized Safety Warning Mechanisms for Controlling AI Agents

    Nov 3, 2026Ilan Zendel, Matan Levy, Stav Cohen +2AI Agent SecurityLLM Agent Safety

  23. When Debate Amplifies Risk: Social-Bias Vulnerabilities in Multi-Agent Safety Deliberation

    Nov 3, 2026Qinghua Mao, Xi Lin, Yuxin Qi +3LLM Agent SafetyMulti-Agent Debate

  24. VeriHarm-Bench: Measuring Harmful Success in LLM Coding Agents Verifiably

    Nov 2, 2026Wei-Lin Chen, Zhepei Wei, Xinyu Zhu +3AI Agent Safety BenchmarksLLM Agent Safety

  25. Do Agents Still Follow the Rules? Benchmarking Rule Compliance under Agent Context Compaction

    Nov 2, 2026Yiwei Zha, Qingyu Meng, Herbert Bos +1AI Agent Safety BenchmarksLLM Agent Safety

  26. Predictive Safety Guardrails Do Not Always Make LLM Agents Safer

    Nov 2, 2026Weixiang Zhao, Zhihao Liu, Guancheng Li +3LLM Agent SafetySafety Filtering

  27. Cumulative and Audited Guardrail Tool Evolution for LLM Agent Safety

    Nov 2, 2026Xiao Yang, Aishan Liu, Siyuan Liang +5AI Agent SecurityLLM Agent Security