cs.AINov 7, 2026

Tail-risk-aware Agent Safety: Constraining Tail Risks Across Multi-step LLM Agent Workflows

Authors: Shilong Pan, Zhiliang Tian, Zihan Chen, Wentao Zou, Weiguo Chen, DongSheng Li

Organizations: PDL Lab, College of Computer Science and Technology National University of Defense Technology

Abstract

LLM-based agents extend LLMs with tool calls, memory, planning, and retrieval to act in external environments, but they are vulnerable to attacks at different workflow steps. Existing defenses, including prompt engineering and post-training alignment, primarily improve LLM-only safety and often overlook risks that arise from multi-step execution. External verification adds guardrails to check each step separately, but it typically ignores how risks can propagate across steps. We argue that we should consider risk transitions across dependent steps through the agent workflow, rather than separate verification on a single step. In this paper, we propose Tail-risk-aware Agent Safety (TAS), a tail-risk-aware safety framework that models agent workflow safety as a sequence of dependent safety states and constrains the rare but extreme risks (i.e., tail risk) with a shared budget. TAS first models safety-state transitions sequentially along the workflow as an MDP to trace risk propagation, then measures a tail-sensitive risk to target rare but high-cost risks (i.e., tail risk), and constrains the risk with a shared budget to derive a step-wise safety strategy. By constraining tail risk rather than the overall average, TAS suppresses rare high-cost failures while preserving utility in normal cases. Experiments show that TAS achieves better performance compared with eleven baselines across three benchmarks.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. TRACES: Proactive Safety Auditing for Multi-Turn LLM Agents via Trajectory-State Modeling

    May 26, 2026Jiaqian Li, Yanshu Li, Boxuan Zhang +2LLM AuditingLLM Agent Safety

  2. AURA-Eval: Evaluation Framework for Acting Under Risk Awareness in LLM Agent Trajectories

    Sep 6, 2026Ruoxi Shang, Christina-Maria Androna, Orfeas Menis Mastromichalakis +6LLM Agent SafetyAI Agent Safety Benchmarks

  3. DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model

    Aug 6, 2026Wenhao Lin, Chenyu Yu, Xingwei Lin +6LLM World ModelsLLM Guardrails