AI Agent Safety
Overview
AI agent safety asks how to keep systems that plan over multiple steps and use tools, memory, or other agents within authorized and harmless bounds. Researchers test computer-use, coding, web, and embodied agents with prompt-injection, misuse, and long-horizon trajectory benchmarks, measuring executed effects as well as final answers. Approaches include red-teaming and monitoring, safety fine-tuning and refusal training, and runtime controls such as least-privilege access, pre-execution policy gates, sandboxing, human approval, and formal verification. A central challenge is that safety can fail through ambiguous goals, changing context, tool or agent interactions, and pressure over time, while stronger restrictions can block legitimate tasks and assurances may not transfer to deployment.
Latest papers
353 papers28 papers in the last four weeks, down 49% on the four weeks before. 0.2% of all new papers.
Survive at All Costs: Exploring LLM's Risky Behaviors under Survival Pressure
Self-Attribution Bias: When AI Monitors Go Easy on Themselves
Asymmetric Goal Drift in Coding Agents Under Value Conflict
Evaluating and Understanding Scheming Propensity in LLM Agents
Tracking Capabilities for Safer Agents
The Alignment Flywheel: A Governance-Centric Hybrid MAS for Architecture-Agnostic Safety
Training Agents to Self-Report Misbehavior
The Ghost in the Grammar: Methodological Anthropomorphism in AI Safety Evaluations
Agents of Chaos
ILION: Deterministic Pre-Execution Safety Gates for Agentic AI Systems
The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems
Towards a Science of AI Agent Reliability
Knowing Isn't Understanding: Re-grounding Generative Proactivity with Epistemic and Behavioral Insight
Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report v1.5
A Trajectory-Based Safety Audit of Clawdbot (OpenClaw)
When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents
Agentic AI for Commercial Insurance Underwriting with Adversarial Self-Critique
When Assisting One Disempowers Another
Measuring Harmfulness of Computer-Using Agents
Artificial Intelligent Disobedience: Rethinking the Agency of Our Artificial Teammates
Shutdownable Agents through POST-Agency
Towards Shutdownable Agents via Stochastic Choice
Discounted Reward for Same-Length Trajectories (DReST)' reward function to train agents to (1) pursue goals effectively conditional on each trajectory-length (be USEFUL'), and (2) choose stochastically between different trajectory-lengths (be `NEUTRAL' about trajectory-lengths). In this paper, we propose evaluation metrics for USEFULNESS and NEUTRALITY. We use a DReST reward function to train simple agents to navigate gridworlds, and we find that these agents learn to be USEFUL and NEUTRAL. Our results thus provide some initial evidence that DReST reward functions could train advanced agents to be USEFUL and NEUTRAL. Our theoretical work suggests that these agents would be useful and shutdownable.