LLM Agent Safety

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

24 papers in the last four weeks, up 50% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 193

All topics

Semantic Quorum Assurance: Collective Certification for Non-Deterministic AI Infrastructure
Jun 6, 2026Jun He, Deying YuAI Assurance Cybersecurity
The Cold-Start Safety Gap in LLM Agents
Jun 5, 2026Chung-En Sun, Linbo Liu, Tsui-Wei WengLLM Agent Security AI Agent Safety Benchmarks
University of California, San Diego
Evaluating Agentic Configuration Repair for Computer Networks
Jun 4, 2026Rufat Asadli, Benjamin Hoffman, Ioannis Protogeros +1LLM Agent Evaluation AI Agent Benchmarks
Department of Information Technology and Electrical Engineering, ETH Zurich.
Towards Healthy Evolution: Exploring the Role and Mechanisms of Human-Agent Interaction in Self-Evolving Systems
Jun 4, 2026Dianxing Shi, Bowen Wang, Junqi He +2Human-in-the-Loop Evaluation LLM Safety Alignment
The University of Osaka · Independent researcher
From Risk Classification to Action Plan Remediation: A Guardrail Feedback Driven Framework for LLM Agents
Jun 4, 2026Yuhao Sun, Jiacheng Zhang, Shaanan Cohney +3LLM Guardrails LLM Agent Training
1The University of Melbourne · 2Tsinghua University · †Corresponding author.
Provably Auditable and Safe LLM Agents from Human-Authored Ontologies
Jun 3, 2026Aaron SterlingAI Agent Auditing LLM Auditing
Thistleseeds
Organizational Control Layer: Governance Infrastructure at the Execution Boundary of LLM Agent Systems
Jun 3, 2026Tianyu Shi, Yang Mo, Yiou Liu +6Multi-Agent LLM Systems AI Agent Governance
McGill · Purdue · UNSW +3
RUBAS: Rubric-Based Reinforcement Learning for Agent Safety
Jun 2, 2026Xian Qi Loye, Qinglin Su, Zhexin Zhang +5Reinforcement Learning AI Agent Safety
The Conversational AI (CoAI) group, DCST, Tsinghua University · Huawei Noah’s Ark Lab
SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoning
Jun 1, 2026Lichao Wang, Zhaoxing Ren, Tianzhuo Yang +4LLM Agent Safety
1Beijing Institute of Technology · 2Beijing Academy of Artificial Intelligence · Institute for Artificial Intelligence, Peking University
Defenses & Enablers For Skill Injection Attacks on Terminal Based Agents
Jun 1, 2026Yoshinari Fujinuma, Varun Gangal, Traian Rebedea +4Adversarial Attacks on LLMs LLM Agent Security
Patronus AI · NVIDIA
A New Framework for Cybersecurity Refusals in AI Agents
May 31, 2026Eliot Krzysztof Jones, Mateusz Dziemian, Matt Fredrikson +1LLM Agent Evaluation LLM Refusal Behavior
Work done while at Gray Swan · Gray Swan AI · Carnegie Mellon University
Adversarial Feeds Steer LLM Agent Decisions Against Their Defaults
May 30, 2026Rana Muhammad UsmanAdversarial Attacks on LLMs LLM Agent Evaluation
Independent Researcher
Same Payload, Different Channel: Measuring Trust Asymmetry in Tool-Using Language Models
May 30, 2026Mohammed Sameer Syed, Rozhin YasaeiAdversarial Attacks on LLMs Prompt Injection Attacks
University of Arizona
COMPASS: Cognitive MCTS-Guided Process Alignment for Safe Search Agents
May 29, 2026Wenkai Shen, Pengyang Zhou, Jiahe Xu +5LLM Safety Alignment LLM Agents
Zhejiang University
Relevance as a Vulnerability: How Web Retrieval Degrades Safety Alignment in LLM Agents
May 28, 2026Aditya Nawal, Manit Baser, Mohan GurusamyAI Agent Security Benchmarks LLM Safety Evaluation
Department of Electrical and Computer Engineering · National University of Singapore
SafeRx-Agent: A Knowledge-Grounded Multi-Agent Framework for Safe and Explainable Medication Recommendation
May 27, 2026Xinyu Wang, Hanwei Wu, Zhenghan Tai +7Multi-Agent LLM Systems Healthcare
McGill University · McMaster University · University of Toronto +1
LACUNA: Safe Agents as Recursive Program Holes
May 27, 2026Yaoyu Zhao, Yichen Xu, Oliver Bračevac +3AI Agent Reliability LLM Agents
EPFL, Lausanne, Switzerland
Plant, Persist, Trigger: Sleeper Attack on Large Language Model Agents
May 27, 2026Yongxiang Li, Moxin Li, Zhixin Ma +2LLM Agent Security AI Agent Security Benchmarks
University of Science and Technology of China · National University of Singapore · Singapore Management University
TRACES: Proactive Safety Auditing for Multi-Turn LLM Agents via Trajectory-State Modeling
May 26, 2026Jiaqian Li, Yanshu Li, Boxuan Zhang +2LLM Auditing LLM Agent Safety
Brown University · The University of Texas at Austin · Rutgers University +1
Voluntary Collusion with Secret Tools in Competing LLM Agents
May 26, 2026Xijie Zeng, Frank RudziczLLM Agent Evaluation LLM Agent Safety
Dalhousie University · Vector Institute for Artificial Intelligence
FinHarness: An Inline Lifecycle Safety Harness for Finance LLM Agents
May 26, 2026Haoxuan Jia, Yang Liu, Bin Chong +10AI Agent Reliability LLM Guardrails
Nanyang Technological University · Tsinghua University · Peking University +8
Device Context Protocol: A Compact, Safety-First Architecture for LLM-Driven Control of Constrained Devices
May 24, 2026Dongxu YangLanguage Model-Based Control MCP Security
1DeepLethe
Open-source LLMs administer maximum electric shocks in a Milgram-like obedience experiment
May 20, 2026Roland Pihlakas, Jan Llenzl DagohoyLLM Refusal Behavior LLM Safety Evaluation
for the Three Laws collaboration · Independent researcher · Three Laws research collaboration +2
Measuring Safety Alignment Effects in Autonomous Security Agents
May 19, 2026Isaac David, Arthur GervaisLanguage Model Safety Evaluation LLM Safety Alignment
University College London
POLAR-Bench: A Diagnostic Benchmark for Privacy-Utility Trade-offs in LLM Agents
May 18, 2026Qiaoyuan Zheng, Yiqu Yang, Qi Gao +1LLM Agent Evaluation LLM Agent Security
ETH Zurich Zurich, Switzerland · ETH AI Center Zurich, Switzerland
Trustworthy Agent Network: Trust in Agent Networks Must Be Baked In, Not Bolted On
May 18, 2026Yixiang Yao, Yuhang Yao, Xinyi Fan +5Multi-Agent LLM Systems Multi-Agent Coordination
University of Southern California · Carnegie Mellon University · University of Illinois Urbana-Champaign
Position: A Three-Layer Probabilistic Assume-Guarantee Architecture Is Structurally Required for Safe LLM Agent Deployment
May 18, 2026S. Bensalem, Y. Dong, M. Franzle +6AI Agent Reliability LLM Guardrails
1CSX-AI, France · University of Liverpool, UK · 3Carl von Ossietzky Universität Oldenburg, Germany +2
Remembering More, Risking More: Longitudinal Safety Risks in Memory-Equipped LLM Agents
May 18, 2026Ahmad Al-Tawaha, Shangding Gu, Peizhi Niu +2Long-Horizon Agent Evaluation LLM Safety Evaluation
Virginia Tech · University of California, Berkeley · University of Illinois Urbana-Champaign
ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agents
May 17, 2026Udari Madhushani Sehwag, Zhengyang Shan, Heming Liu +3LLM Agent Security AI Agent Security Benchmarks
Scale AI · 2Boston University · 4Human Frontier Collective +3
State Contamination in Memory-Augmented LLM Agents
May 16, 2026Yian Wang, Agam Goyal, Yuen Chen +1LLM Agent Memory LLM Agent Safety
Department of Computer Science, University of Illinois Urbana-Champaign

Jun 6, 2026, cs.LGJ/K move · Enter open · S save

Semantic Quorum Assurance: Collective Certification for Non-Deterministic AI Infrastructure

Jun He, Deying Yu

AI Assurance Cybersecurity Multi-Agent Systems AI Agent Governance LLM Agent Safety Multi-Agent System Security

Open paper AskPDF arXiv

As large language model (LLM) agents are integrated into autonomous cloud operations, distributed systems face a semantic reliability problem: proposer agents can generate production mutations, such as modifying IAM policies, opening firewall security groups, or executing data exports, that are syntactically valid and statically authorized but operationally unsafe. Classical distributed consensus protocols replicate deterministic state transitions but do not evaluate the safety of the proposed intent. To address this gap, we introduce Semantic Quorum Assurance (SQA), a control-plane primitive for governing non-deterministic agentic infrastructure. SQA represents proposals as declarative execution contracts bound to cryptographic evidence chains and routes them to a diverse panel of read-only, sandboxed validator agents. SQA aggregates their judgments under a risk-adaptive quorum predicate that enforces model and archetype diversity, adjusts weights based on calibrated assurance scores, and respects archetype-specific vetoes. Admitted proposals execute only through a sovereign execution gate. We instantiate SQA in a cloud-native control plane and formalize a correlated cognitive failure model for non-deterministic validators. On 500 infrastructure-inspired mutation scenarios, with safety results reported on held-out safe/unsafe trials excluding ambiguous scenarios, SQA reduces unsafe approval from 18.5% for single-agent validation to 0.3% while adding median validation latency of 1.45--4.12 seconds across the studied risk buckets.

Momentum

Related topics

Latest papers 193