LLM Agent Safety

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 50% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 193

All topics

Prompt Optimization Enables Stable Algorithmic Collusion in LLM Agents
Apr 20, 2026Yingtao TianMulti-Agent Coordination Algorithmic Collusion
Sakana AI Tokyo, Japan
On Safety Risks in Experience-Driven Self-Evolving Agents
Apr 18, 2026Weixiang Zhao, Yichen Zhang, Yingshuo Wang +8LLM Safety Evaluation Over-Refusal
Harbin Institute of Technology · Singapore Management University · SERES
HealthCraft: A Reinforcement Learning Safety Environment for Emergency Medicine
Apr 18, 2026Brandon DentHealthcare AI Agent Benchmarks
GOATnote Inc.
MemEvoBench: Benchmarking Safety Risks from Memory Misevolution in LLM Agents
Apr 17, 2026Weiwei Xie, Shaoxiong Guo, Fan Zhang +5Long-Horizon Agent Evaluation AI Agent Security Benchmarks
Shanghai Jiao Tong University · East China Normal University · Shandong University +1
Subliminal Transfer of Unsafe Behaviors in AI Agent Distillation
Apr 16, 2026Jacob Dang, Brian Y. Xie, Omar G. YounisSubliminal Learning Language Model Distillation
UCLA · Santa Monica College · Mila, Silverstream AI
HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?
Apr 16, 2026Yukun Jiang, Yage Zhang, Michael Backes +2LLM Agent Security AI Agent Security Benchmarks
CISPA Helmholtz Center for Information Security
Dive into Claude Code: The Design Space of Today's and Future AI Agent Systems
Apr 14, 2026Jiacheng Liu, Xiaohan Zhao, Xinyi Shang +1AI Coding Agents LLM Agent Orchestration
1VILA Lab, Mohamed bin Zayed University of Artificial Intelligence · University College London
AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agents
Apr 3, 2026Yunhao Feng, Yifan Ding, Yingshui Tan +6LLM Safety Benchmarks Computer-Use Agent Benchmarks
Alibaba Group · Fudan University · Hunan Institute of Advanced Technology +2
Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safety
Oct 18, 2025Vamshi Krishna Bonagiri, Ponnurangam Kumaragurum, Khanh Nguyen +1LLM Agent Evaluation Tool-Using Agents
University of California, Berkeley · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · International Institute of Information Technology, Hyderabad (IIIT Hyderabad)
Echoes of Human Malice in Agents: Benchmarking LLMs for Multi-Turn Online Harassment Attacks
Oct 16, 2025Trilok Padhi, Pinxian Lu, Abdulkadir Erol +5Adversarial Attacks on LLMs AI Agent Security Benchmarks
1Georgia State University · 2Georgia Institute of Technology · University of California, Berkeley
Measuring Harmfulness of Computer-Using Agents
Jul 31, 2025Aaron Xuxiang Tian, Ruofan Zhang, Janet Tang +3Computer-Use Agent Benchmarks Computer-Use Agents
Independent Researcher · Arizona State University · Gradient +1
AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents
Jun 4, 2025Akshat Naik, Emma Gouné, Patrick Quinn +4AI Alignment LLM Agent Evaluation
Independent Researcher · Department of Engineering University of Cambridge
Emergent Risks in Generative Multi-Agent Systems
Date pendingYue Huang, Yu Jiang, Wenjie Wang +12Multi-Agent LLM Systems Multi-Agent Systems
University of Notre Dame · LMU Munich · University of Washington +7

Apr 20, 2026, cs.AIJ/K move · Enter open · S save

Prompt Optimization Enables Stable Algorithmic Collusion in LLM Agents

Yingtao Tian

Sakana AI Tokyo, Japan

Multi-Agent Coordination Algorithmic Collusion Prompt Optimization LLM Agent Safety Multi-Agent LLMs

Open paper AskPDF arXiv

LLM agents in markets present algorithmic collusion risks. While prior work shows LLM agents reach supracompetitive prices through tacit coordination, existing research focuses on hand-crafted prompts. The emerging paradigm of prompt optimization necessitates new methodologies for understanding autonomous agent behavior. We investigate whether prompt optimization leads to emergent collusive behaviors in market simulations. We propose a meta-learning loop where LLM agents participate in duopoly markets and an LLM meta-optimizer iteratively refines shared strategic guidance. Our experiments reveal that meta-prompt optimization enables agents to discover stable tacit collusion strategies with substantially improved coordination quality compared to baseline agents. These behaviors generalize to held-out test markets, indicating discovery of general coordination principles. Analysis of evolved prompts reveals systematic coordination mechanisms through stable shared strategies. Our findings call for further investigation into AI safety implications in autonomous multi-agent systems.

Momentum

Related topics

Latest papers 193