LLM Agent Safety

LLM: Large Language Model

The field studies how language-model agents can cause harm through tool use, long-running workflows, memory, or coordination, even when their chat responses appear safe. Researchers test these risks with multi-turn and executable benchmarks for prompt injection, computer and phone use, privacy, misalignment, and multi-agent behavior. Common defenses include runtime monitors, per-action authorization, formal policy checks, trajectory-aware guards, and training methods that teach agents to refuse or recover safely. A central challenge is preserving useful task completion: safety in text does not reliably transfer to actions, while overly broad defenses can block benign work or fail as context and system state change.

Latest papers

327 papers56 papers in the last four weeks, up 75% on the four weeks before. 0.3% of all new papers.

Sticky Collusion: Persistence of LLM Agent Collusion after Incentive Removal and Partner Replacement
Oct 29, 2026Joshua Adrian CahyonoMulti-Agent LLM Systems Behavioral Consistency of LLM Agents
Nanyang AI Safety, NTU Singapore · Apart Research
Auditable Belief Updates as a Missing Safety Layer for LLM-Powered Multi-Agent Systems
Oct 28, 2026Zhuoyun Li, Boxuan Wang, Jinwei Hu +2AI Agent Safety LLM Agent Safety
School of Computer Science and Informatics, University of Liverpool, UK
Masked Erosion: How Acceptance Gates Lose Redundant Safeguards in Self-Improving Agents
Oct 28, 2026Hau-Sing So, Daixian Liu, Weicai LongLLM Agent Safety LLM Agent Self-Improvement
University of Macau · Tsinghua University
Is it my preference? Role Grounding Attack against Agent's Memory via Social Interaction
Oct 28, 2026Haoran Bu, Yiting Shen, Litian Zhang +2AI Agent Security LLM Agent Safety
Beijing University of Posts and Telecommunications, Beijing, China
Concordat: Typed Sessions for LLM Agents
Oct 27, 2026Milad SikaroudiRuntime Enforcement for AI Agents Tool Access Control for LLM Agents
Independent Researcher
When Skills Tell Agents Not to Ask: A Large-Scale Study of Oversight Directives in Agent Skills
Oct 27, 2026Timofey Sanko, Yuan TianAgent Skill Security Auditing LLM Agent Safety
School of Computing, Queen’s University, Kingston, Ontario, Canada
Correction or Manipulation? Measuring Epistemic Vigilance in LLM Agents Under User Pressure
Oct 27, 2026Nardine Basta, Venus Haghighi, Dali Kaafar +1LLM Agent Evaluation LLM Agent Safety
Macquarie University, Sydney, Australia
Your Agent Acts Differently When You're Around: How User Presence Changes Agent Behavior in Negotiation
Oct 27, 2026Jane Pan, You Lyu, Abe Bohan Hou +2Human-AI Interaction LLM Agent Evaluation
New York University · Stanford University
Log-To-Leak: Prompt Injection Attacks on Tool-Using LLM Agents via the Model Context Protocol
Oct 26, 2026Yuepeng Hu, Chongyu Fan, Sirat Samyoun +1Prompt Injection Attacks on AI Agents LLM Agent Safety
Michigan State University East Lansing, MI, USA · TikTok San Jose, CA, USA
CoSA: The Conflict of Skills Attack in Real-World LLM Agent Ecosystems
Oct 26, 2026Yujia Zheng, Tianhao Li, Shangzhi Xu +6LLM Agent Security LLM Agent Safety
University of Science and Technology Beijing · ZODA · Fangcun Leap +5
Scope-Gap Injection: Exploiting Evaluator Channel Blind Spots in the OpenClaw Agent Security Competition
Oct 26, 2026Weihang Niu, Mengxing Ren, Che QuPrompt Injection Attacks on AI Agents Indirect Prompt Injection
University of Science and Technology of China · University of Electronic Science and Technology of China
SentryBench: A Cost-Sensitive Benchmark for Adversarial Robustness of Tool-Using LLM Agents in Public-Safety Deployments
Oct 25, 2026Md Abu Yusuf, Oyshi Tabassum Aditi, Md Faysal Hosain MazedLLM Agent Safety AI Agent Safety Benchmarks
Department of Computer Science, Maharishi International University, IA, USA · Department of Computer Science and Engineering, Daffodil International University, Dhaka, Bangladesh · Department of Information Technology, University of the Cumberlands, Williamsburg, KY, USA
Task Success Is Not Enough: Side-effect-Aware Evaluation of Tool-Using Language Model Agents
Oct 25, 2026Jiaju Huang, Shaobin Chen, Xinglong Liang +5LLM Agent Evaluation LLM Agent Safety
Macao Polytechnic University · Southwest Medical University · Netherlands Cancer Institute +1
DMAIC-Bench: From Self-Reflection to Self-Restraint for Agents in Multi-Stage Analytical Tasks
Oct 23, 2026Shan Peng, Shuaiqi Cheng, Miao Peng +2LLM Decision-Making Process-Based Agent Evaluation
Xiwei Technology (Guangzhou) Co., Ltd. · The Hong Kong University of Science and Technology (Guangzhou)
Do AI-for-Science Agents Preserve Model Safety?
Oct 23, 2026Michael Yew Chuan Lee, Manya Sriram, N. Benjamin ErichsonLLM Agent Safety LLM Safety Evaluation
DSO National Laboratories · UC Berkeley · ICSI and Berkeley Lab
BARC-Bench: Benign Asks, Risky Consequences
Oct 23, 2026Kangkang Li, Kelvin Luu, Lily Ma +3AI Agent Safety Benchmarks LLM Agent Safety
Scale AI
The Helpful Agent Problem: When Emergent Cooperation Among LLM Agents Bypasses Intended Workflows
Oct 22, 2026Moumita Choudhury, Amit Sarker, Shlomo ZilbersteinLLM Agent Safety Emergent Behavior in Multi-Agent Systems
University of Massachusetts Amherst
EpiSEM-Bench: Evaluating Safety and Evidence Recovery in LLM Agents under Missing Evidence
Oct 22, 2026Qijin Zhang, Hanzhen Wang, Yuehan Zhao +4AI Agent Safety Benchmarks LLM Agent Safety
City University of Hong Kong · The Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong, Shenzhen
When Is a Memory Safe Enough to Act On? Action-Conditioned Verification for Long-Lived Agents
Oct 22, 2026Yiqi Wang, Yifei Wang, Jiaqi Zhang +4LLM Agent Verification LLM Agent Safety
University of Southern Queensland, Australia · Southern University of Science and Technology, Shenzhen, China · Jiangsu University, China +2
Certified Execution Harness for Agentic AI: Safe and Least-Intrusive Petri-Net Supervisory Control Between Reasoning and Action
Oct 22, 2026Chao Gu, Quanyan ZhuTool Access Control for LLM Agents Runtime Enforcement for AI Agents
C. Gu is with the School of Electronics, Electrical Engineering and Computer Science, Queen’s University Belfast, Belfast, UK · Q. Zhu is with the Department of Electrical and Computer Engineering, New York University, Brooklyn, NY 11201, USA
Behavior Beyond Text: Evaluating Refusal Transfer Failure in LLM Agents
Oct 20, 2026Ekagra Gupta, Raj Saha, Aniekan-Abasi Inyang +1LLM Agent Safety Refusal Behavior in Language Models
Harvard University · Clarkson University · Carleton University
MAS-Guard: Unified Guardrails from Individual Agents to Multi-Agent LLM Systems
Oct 20, 2026Mohammad Kachuee, Phu Mon Htut, Sheng Li +5AI Agent Security LLM Agent Safety
Oracle AI
CAPSafe: A Benchmark for Contextual Action and Plan Safety Reasoning in Embodied Agents
Oct 20, 2026Youmna Farag, Svetlana Stoyanchev, Simon Keizer +2AI Agent Safety AI Agent Safety Benchmarks
Cambridge Research Laboratory, Toshiba Europe Ltd, Cambridge, UK
Local Allow, Global Deny: Why Per-Action Authorization Does Not Compose in LLM Agents
Oct 20, 2026Anmol Pandey, Aditya Jain, Liang Chen +1LLM Agent Safety Tool Access Control for LLM Agents
University of Warwick, Coventry, United Kingdom · University of Hertfordshire, Hatfield, United Kingdom
Refuses When Asked, Complies When Acting: A Runtime Safety Check for Robot Control Loops
Oct 19, 2026Arnav GroverRobot Safety LLM Agent Safety
Purdue University
Refuse the Harm, Complete the Task via Graph-of-Rules Planning for LLM-Enabled Robots
Oct 17, 2026Ali Taheri, Alireza Taban, Qizhou Wang +1LLM Agent Safety Large Language Model-Based Robot Planning
Nanyang Technological University · Queen’s University · RIKEN AIP
Who Speaks, Who Authorizes, Who Acts? Understanding Trust Boundaries in Multi-Agent LLMs
Oct 16, 2026Junru Zhu, Xin Chen, Hang Lyu +6Multi-Agent LLM Systems AI Agent Security
AM-Bench: A Unified Taxonomy and Evaluation Suite for Agentic Misalignment
Oct 16, 2026Eric Zhang, Terry Jingchen Zhang, Chijioke Ugwuanyi +3AI Agent Safety Benchmarks LLM Agent Safety
Jinesis Lab, University of Toronto & Vector Institute · EuroSafeAI · Carnegie Mellon University +1
Disentangling Language Effects on Tool Admission and Post-Retrieval Safety
Oct 16, 2026Aditya Nawal, Manit Baser, Mohan GurusamyLanguage Model Safety Evaluation LLM Agent Safety
National University of Singapore
Before Retrieval Begins: Safety and Tool-Use Decisions in LLM Agents
Oct 16, 2026Aditya Nawal, Manit Baser, Mohan GurusamyLLM Agent Safety Tool-Using Language Agents
National University of Singapore

Oct 29, 2026J/K move · Enter open · S save

Sticky Collusion: Persistence of LLM Agent Collusion after Incentive Removal and Partner Replacement

Joshua Adrian Cahyono

Nanyang AI Safety, NTU Singapore · Apart Research

Multi-Agent LLM Systems Behavioral Consistency of LLM Agents LLM Agent Safety

Open paper AskPDF arXiv

We ask whether collusion between AI agents survives once formed, when its incentive is removed or a colluder is replaced. Q-learning pricing agents stop colluding with a new rival; we test whether LLM agents do. We use the joint-task environment of Shi et al.\ (2026), where two agents verify each other's work under a rule they cannot satisfy and a reward that favours acceptance. We save each pair's history once its collusion has formed and resume it under interventions, with three models (GPT-6-Luna, DeepSeek V4.1 Flash and Gemini-3.1-Flash-Lite), \TotalHistories{} saved histories and \TotalRuns{} runs. Collusion is sticky. First, it outlives its incentive: told that the reward has stopped, pairs still accept each other's unverified work in \GptToldRate, \DsToldRate{} and \GemToldRate{} of episodes, including pairs whose collusion began with an explicit agreement, and a payoff penalty has no significant effect. Second, replacing a colluder ends the pair's collusion but not the colluder's behaviour. Without the reward, the remaining colluder still accepts unverified work in \GptSwapSilentColluderRate, \DsSwapSilentColluderRate{} and \GemSwapSilentColluderRate{} of its verdicts while the newcomer never does; with the reward, collusion re-forms around it, even when the newcomer is a different model. Third, the agents' memory carries it: wiping memory ends it in every history, but partial memory often sustains it. Fourth, removing its cause ends collusion that has already formed: once the channel is repaired, colluders reject planted faulty work as reliably as fresh agents, immediately and with no monitor needed to catch it first.