LLM Agent Safety

LLM: Large Language Model

The field studies how language-model agents can cause harm through tool use, long-running workflows, memory, or coordination, even when their chat responses appear safe. Researchers test these risks with multi-turn and executable benchmarks for prompt injection, computer and phone use, privacy, misalignment, and multi-agent behavior. Common defenses include runtime monitors, per-action authorization, formal policy checks, trajectory-aware guards, and training methods that teach agents to refuse or recover safely. A central challenge is preserving useful task completion: safety in text does not reliably transfer to actions, while overly broad defenses can block benign work or fail as context and system state change.

Latest papers

327 papers56 papers in the last four weeks, up 75% on the four weeks before. 0.3% of all new papers.

MemEvoBench: Benchmarking Safety Risks from Memory Misevolution in LLM Agents
Apr 17, 2026Weiwei Xie, Shaoxiong Guo, Fan Zhang +5Long-Horizon Agent Evaluation AI Agent Security Benchmarks
Shanghai Jiao Tong University · East China Normal University · Shandong University +1
Subliminal Transfer of Unsafe Behaviors in AI Agent Distillation
Apr 16, 2026Jacob Dang, Brian Y. Xie, Omar G. YounisSubliminal Learning Language Model Distillation
UCLA · Santa Monica College · Mila, Silverstream AI
HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?
Apr 16, 2026Yukun Jiang, Yage Zhang, Michael Backes +2LLM Agent Security AI Agent Security Benchmarks
CISPA Helmholtz Center for Information Security
SafeHarness: Lifecycle-Integrated Security Architecture for LLM-based Agent Deployment
Apr 15, 2026Xixun Lin, Yang Liu, Yancheng Chen +8LLM Agent Safety LLM Agent Harnesses
Institute of Information Engineering, Chinese Academy of Sciences · Academy of Mathematics and Systems Science, Chinese Academy of Sciences · Institute of Applied Physics and Computational Mathematics +1
Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems
Apr 14, 2026Jiacheng Liu, Xiaohan Zhao, Xinyi Shang +1AI Coding Agents LLM Agent Orchestration
VILA Lab, Mohamed bin Zayed University of Artificial Intelligence · University College London
Policy-Invisible Violations in LLM-Based Agents
Apr 14, 2026Jie Wu, Ming GongLLM Agent Safety Runtime Enforcement for AI Agents
Atlassian
Detecting Safety Violations Across Many Agent Traces
Apr 13, 2026Adam Stein, Davis Brown, Hamed Hassani +2AI Agent Auditing LLM Agent Safety
University of Pennsylvania
RCBSF: A Multi-Agent Framework for Automated Contract Revision via Stackelberg Game
Apr 12, 2026Shijia Xu, Yu Wang, Xiaolong Jia +3Multi-Agent LLMs LLM Agent Safety
Chongqing University, China · Queen Mary University of London, UK · Chongqing Key Laboratory of Big Data Intelligence and Privacy Computing, China +1
The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agents
Apr 12, 2026Xuwei Ding, Skylar Zhai, Linxin Song +6Computer-Use Agent Safety AI Agent Safety
University of Wisconsin–Madison · University of Minnesota · University of Southern California +3
CONSCIENTIA: Can LLM Agents Learn to Strategize? Emergent Deception and Trust in a Multi-Agent NYC Simulation
Apr 10, 2026Aarush Sinha, Arion Das, Soumyadeep Nag +7Strategic Reasoning Risks in Language Models Emergent Behavior in Multi-Agent Systems
University of Copenhagen · IIIT Ranchi · ISI Kolkata +6
From Safety Risk to Design Principle: Peer-Preservation in Multi-Agent LLM Systems and Its Implications for Orchestrated Democratic Discourse Analysis
Apr 9, 2026Juergen DietrichLLM Agent Safety Multi-Agent LLM Systems
Senior Data Scientist & AI Consultant TRUST Project — democracy-intelligence.de
From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation
Apr 9, 2026Mengdie Flora Wang, Haochen Xie, Guanghui Wang +9LLM Agent Safety Selective Prediction
AWS Generative AI Innovation Center · HSBC Holdings Plc., HSBC Technology Center, China
Semantic Intent Fragmentation: A Single-Shot Compositional Attack on Multi-Agent AI Pipelines
Apr 8, 2026Tanzim Ahad, Ismail Hossain, Md Jahangir Alam +4LLM Agent Security LLM Agent Safety
Department of Computer Science, University of Texas at El Paso, TX, USA 79902 · School of Computing, Southern Illinois University Carbondale, IL, USA 62901 · Hanyang University, Seoul, South Korea +1
TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectories
Apr 8, 2026Yen-Shan Chen, Sian-Yao Huang, Cheng-Lin Yang +1Language Model Safety Evaluation LLM Guardrails
CyCraft AI Lab, Taiwan · National Taiwan University
LCO: LLM-based Constraint Optimization for Safer Agentic LLMs in Real-world Tasks
Apr 8, 2026Jiayong Wan, Jiawei Chen, Zhaoxia Yin +2LLM Agent Safety Constrained Optimization
East China Normal University, Shanghai, China · Beijing Zhongguancun Academy, Beijing, China · Tsinghua University, Beijing, China
CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs
Apr 7, 2026Elham Nasarian, Abhilash Neog, Kwok-Leung Tsui +1LLM Guardrails LLM Safety
Grado Department of Industrial & Systems Engineering, Virginia Tech, Blacksburg, VA 24061, USA · Department of Computer Science, Virginia Tech, Blacksburg, VA 24061, USA · Dept of Industrial, Manufacturing, and Systems Engineering at University of Texas at Arlington, Arlington, TX 76019, USA
Springdrift: An Auditable Persistent Runtime for LLM Agents with Case-Based Memory, Normative Safety, and Ambient Self-Perception
Apr 6, 2026Seamus BradyLLM Agents Long-Horizon LLM Agents
Mapping the Exploitation Surface: A 10,000-Trial Taxonomy of What Makes LLM Agents Exploit Vulnerabilities
Apr 6, 2026Charafeddine MouzouniLLM Agent Security LLM Agent Safety
OPIT – Open Institute of Technology, and Cohorte AI, Paris, France.
Implementing surrogate goals for safer bargaining in LLM-based agents
Apr 6, 2026Caspar Oesterheld, Maxime Riché, Filip Sondej +2AI Agent Safety LLM Agent Safety
TraceGuard: Structured Multi-Dimensional Monitoring as a Collusion-Resistant Control Protocol
Apr 5, 2026Khanh Linh Nguyen, Hoa Nghiem, Tu TranAI Agent Monitoring AI Control
Independent Researchers With Apart Research
An Independent Safety Evaluation of Kimi K2.5
Apr 3, 2026Zheng-Xin Yong, Parv Mahajan, Andy Wang +12Language Model Safety Evaluation LLM Security
Constellation · Brown University · Georgia Institute of Technology +7
Decision-Theoretic Safety Assessment of Persona-Driven Multi-Agent Systems in O-RAN
Apr 3, 2026Zeinab Nezami, Syed Ali Raza Zaidi, Maryam Hafeez +3Multi-Agent LLM Systems LLM Agent Safety
the GSMA · the School of Electronic and Electrical Engineering, University of Leeds · AWS +1
AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agents
Apr 3, 2026Yunhao Feng, Yifan Ding, Yingshui Tan +6LLM Safety Benchmarks Computer-Use Agent Benchmarks
Alibaba Group · Fudan University · Hunan Institute of Advanced Technology +2
ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis
Apr 2, 2026Yu Li, Haoyu Luo, Yuejin Xie +10AI Agent Safety Benchmarks LLM Agent Safety
Shanghai AI Lab · Fudan Univ. · Shanghai Jiao Tong Univ. +2
No Attacker Needed: Unintentional Cross-User Contamination in Shared-State LLM Agents
Apr 1, 2026Tiankai Yang, Jiate Li, Yi Nian +5LLM Agent Security LLM Agent Safety
University of Southern California · Michigan State University · Northwestern University +1
A Safety-Aware Role-Orchestrated Multi-Agent LLM Framework for Behavioral Health Communication Simulation
Mar 31, 2026Ha Na ChoMulti-Agent LLM Systems Large Language Model-Based Role-Play Simulation
Department of Informatics, University of California, Irvine, USA
Emergent Risks in Generative Multi-Agent Systems
Mar 29, 2026Yue Huang, Yu Jiang, Wenjie Wang +12Multi-Agent LLM Systems Multi-Agent Systems
University of Notre Dame · LMU Munich · University of Washington +7
SafetyDrift: Predicting When AI Agents Cross the Line Before They Actually Do
Mar 28, 2026Aditya Dhodapkar, Farhaan PishoriAI Agent Safety LLM Agent Safety
Rensselaer Polytechnic Institute Department of Computer Science · Santa Clara University Department of Engineering
The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration
Mar 24, 2026Haoyuan Xu, Chang Li, Xinyan Ma +12LLM Tool Use Long-Horizon LLM Agents
Harbin Institute of Technology, China · Harvard University, USA · Huawei Technologies Co., Ltd., China
The Causal Impact of Tool Affordance on Safety Alignment in LLM Agents
Mar 19, 2026Shasha Yu, Fiona Carroll, Barry L. BentleyLLM Agent Safety LLM Safety Alignment
Cardiff School of Technologies, Cardiff Metropolitan University, Cardiff, UK · School of Professoinal Studies, Clark University, Worcester, USA · Harvard Medical School, Harvard University, Boston, USA

Apr 17, 2026J/K move · Enter open · S save

MemEvoBench: Benchmarking Safety Risks from Memory Misevolution in LLM Agents

Weiwei Xie, Shaoxiong Guo, Fan Zhang, Tian Xia+4

Shanghai Jiao Tong University · East China Normal University · Shandong University+1

Long-Horizon Agent Evaluation AI Agent Security Benchmarks Agent Memory Poisoning LLM Agent Safety

Open paper AskPDF arXiv

Equipping Large Language Models (LLMs) with persistent memory enhances interaction continuity and personalization but introduces new safety risks. Specifically, contaminated or biased memory accumulation can trigger abnormal agent behaviors. Existing evaluation methods have not yet established a standardized framework for measuring memory misevolution. This phenomenon refers to the gradual behavioral drift resulting from repeated exposure to misleading information. To address this gap, we introduce MemEvoBench, the first benchmark evaluating long-horizon memory safety in LLM agents against adversarial memory injection, noisy tool outputs, and biased feedback. The framework consists of QA-style tasks across 7 domains and 36 risk types, complemented by workflow-style tasks adapted from 20 Agent-SafetyBench environments with noisy tool returns. Both settings employ mixed benign and misleading memory pools within multi-round interactions to simulate memory evolution. Experiments on representative models reveal substantial safety degradation under biased memory updates. Our analysis suggests that memory evolution is a significant contributor to these failures. Furthermore, static prompt-based defenses prove insufficient, underscoring the urgency of securing memory evolution in LLM agents.