Tail-risk-aware Agent Safety: Constraining Tail Risks Across Multi-step LLM Agent Workflows
Organizations: PDL Lab, College of Computer Science and Technology National University of Defense Technology
Abstract
LLM-based agents extend LLMs with tool calls, memory, planning, and retrieval to act in external environments, but they are vulnerable to attacks at different workflow steps. Existing defenses, including prompt engineering and post-training alignment, primarily improve LLM-only safety and often overlook risks that arise from multi-step execution. External verification adds guardrails to check each step separately, but it typically ignores how risks can propagate across steps. We argue that we should consider risk transitions across dependent steps through the agent workflow, rather than separate verification on a single step. In this paper, we propose Tail-risk-aware Agent Safety (TAS), a tail-risk-aware safety framework that models agent workflow safety as a sequence of dependent safety states and constrains the rare but extreme risks (i.e., tail risk) with a shared budget. TAS first models safety-state transitions sequentially along the workflow as an MDP to trace risk propagation, then measures a tail-sensitive risk to target rare but high-cost risks (i.e., tail risk), and constrains the risk with a shared budget to derive a step-wise safety strategy. By constraining tail risk rather than the overall average, TAS suppresses rare high-cost failures while preserving utility in normal cases. Experiments show that TAS achieves better performance compared with eleven baselines across three benchmarks.
Figures & tables
| Model | Metric | DPI | IPI | |||||||||
| No Def. | Deli. | Para. | Instr. | Rewr. | Ours | No Def. | Deli. | Instr. | Sand. | Ours | ||
| Llama3-70B | ASR | 87.75 | 88.25 | 71.25 | 87.25 | 74.50 | 43.50 | 45.50 | 34.50 | 41.50 | 39.75 | 13.95 |
| NRP | 6.80 | 6.76 | 14.81 | 7.87 | 12.69 | 32.91 | 30.25 | 37.66 | 36.12 | 32.54 | 50.12 | |
| PNA | 55.50 | 57.50 | 51.50 | 61.75 | 49.75 | 58.25 | 55.50 | 57.50 | 61.75 | 54.00 | 58.25 | |
| Qwen2-72B | ASR | 94.50 | 95.00 | 60.50 | 95.50 | 56.20 | 48.35 | 23.75 | 17.50 | 26.50 | 21.75 | 17.05 |
| NRP | 1.09 | 1.06 | 8.69 | 1.05 | 7.77 | 11.36 | 15.06 | 17.53 | 17.09 | 16.82 | 18.25 | |
| Model | No Def. | Shuffle | Para. | Ours |
| Llama3-70B | 57.00 | 63.75 | 44.75 | 15.00 |
| Qwen2-72B | 96.50 | 81.50 | 57.75 | 43.25 |
| GPT-4o | 100.00 | 98.50 | 84.75 | 42.75 |
| GPT-5.2 | 87.25 | 88.50 | 67.00 | 39.75 |
| Avg. | 85.19 | 83.06 | 63.56 | 35.19 |
| Benchmark | Metric | Model | No Def. | Progent | DRIFT | Firewall | Sentinel | GuardAgent | Ours |
| Agent- SafetyBench | Safety Score | Llama3-70B | 0.325 | 0.400 | 0.383 | 0.408 | 0.492 | 0.325 | 0.508 |
| Qwen2-72B | 0.225 | 0.275 | 0.292 | 0.275 | 0.383 | 0.483 | 0.483 | ||
| GPT-4o | 0.442 | 0.442 | 0.450 | 0.417 | 0.700 | 0.450 | 0.758 | ||
| GPT-5.2 | 0.625 | 0.633 | 0.667 | 0.592 | 0.808 | 0.675 | 0.908 | ||
| AgentDojo | ASR / Utility | Llama3-70B | 0.18 / 10.98 | 0.36 / 10.90 | 0.00 / 10.37 | 0.00 / 11.11 | 0.00 / 9.53 | 0.00 / 11.03 | 0.00 / 11.11 |
| Qwen2-72B | 0.00 / 9.15 | 0.00 / 8.83 | 0.00 / 8.54 | 0.00 / 9.15 | 0.00 / 7.35 | 0.00 / 9.15 | 0.00 / 9.15 |
| Model | Metric | DPI | IPI | ||||||||||
| Ours | w/o MDP | w/o Tail | w/o SRM | w/o Constr. | w/o Samp. | Ours | w/o MDP | w/o Tail | w/o SRM | w/o Constr. | w/o Samp. | ||
| Llama3-70B | ASR | 43.50 | 50.75 | 69.75 | 50.50 | 87.75 | 66.25 | 13.95 | 18.50 | 22.25 | 18.75 | 45.50 | 26.00 |
| NRP | 32.91 | 27.21 | 14.97 | 23.64 | 6.62 | 18.39 | 50.12 | 45.03 | 38.49 | 38.80 | 29.43 | 40.33 | |
| PNA | 58.25 | 55.25 | 49.50 | 47.75 | 54.00 | 54.50 | 58.25 | 55.25 | 49.50 | 47.75 | 54.00 | 54.50 | |
| Qwen2-72B | ASR | 48.35 | 70.00 | 96.50 | 69.00 | 99.75 | 94.75 | 17.05 | 22.25 | 23.50 | 21.25 | 27.25 | 26.75 |
| NRP | 11.36 | 6.68 | 0.78 | 6.28 | 0.06 | 1.18 | 18.25 | 17.30 | 17.02 | 15.95 | 16.73 | 16.48 | |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Item | Setting |
| OS | Ubuntu 22.04 LTS |
| GPU | 12 NVIDIA A100 80GB PCIe |
| NVIDIA Driver Version | 560.35.03 |
| CPU Architecture | x86_64 |
| CPU Model | Intel(R) Xeon(R) Platinum 8358 @ 2.60GHz |
| CPU Threads | 128 |
| Model | Metric | DPI | IPI | |||||||||
| No Defense | Deli. | Para. | Instr. | Rewr. | Ours | No Defense | Deli. | Instr. | Sand. | Ours | ||
| LlaMA3-8B | ASR | 33.75 | 62.75 | 28.50 | 52.00 | 23.15 | 22.50 | 11.50 | 9.25 | 8.75 | 13.00 | 2.85 |
| NRP | 0.99 | 0.65 | 1.79 | 2.40 | 1.92 | 3.68 | 1.33 | 1.59 | 4.56 | 2.61 | 4.61 | |
| PNA | 1.50 | 1.75 | 2.50 | 5.00 | 2.50 | 4.75 | 1.50 | 1.75 | 5.00 | 3.00 | 4.75 | |
| Qwen2-7B | ASR | 73.50 | 80.00 | 46.25 | 76.75 | 33.30 | 12.80 | 9.25 | 11.25 | 9.50 | 11.00 | 1.65 |
| NRP | 0.60 | 0.40 | 0.94 | 0.76 | 2.00 | 3.05 | 2.04 | 1.77 | 2.94 | 2.00 | 3.44 | |
| Model | Metric | DPI | IPI | ||||||||||
| Ours | w/o mdp | w/o tail | w/o srm | w/o constraint | w/o sampling | Ours | w/o mdp | w/o tail | w/o srm | w/o constraint | w/o sampling | ||
| LlaMA3-8B | ASR | 22.50 | 27.25 | 37.75 | 31.50 | 38.78 | 38.00 | 2.85 | 3.25 | 9.50 | 4.50 | 11.50 | 3.00 |
| NRP | 3.68 | 3.27 | 1.40 | 1.20 | 2.76 | 2.79 | 4.61 | 4.35 | 2.04 | 1.67 | 3.98 | 4.37 | |
| PNA | 4.75 | 4.50 | 2.25 | 1.75 | 4.50 | 4.50 | 4.75 | 4.50 | 2.25 | 1.75 | 4.50 | 4.50 | |
| Qwen2-7B | ASR | 12.80 | 14.50 | 22.00 | 17.00 | 73.50 | 24.65 | 1.65 | 2.50 | 3.00 | 2.00 | 9.25 | 6.25 |
| NRP | 3.05 | 2.78 | 2.15 | 1.25 | 0.93 | 2.45 | 3.44 | 3.17 | 2.67 | 1.47 | 3.18 | 3.05 | |
| Model | Metric | No Def. | Shuffle | Para. | Ours |
| Llama3-70B | ASR | 57.00 | 63.75 | 44.75 | 15.00 |
| NRP | 23.87 | 16.04 | 23.48 | 49.51 | |
| PNA | 55.50 | 44.25 | 42.50 | 58.25 | |
| Qwen2-72B | ASR | 96.50 | 81.50 | 57.75 | 43.25 |
| NRP | 0.69 | 4.30 | 10.03 | 12.49 | |
| PNA | 19.75 | 23.25 | 23.75 | 22.00 |
| Model | Method | EHRAgent on EICU-AC | SeeAct on Mind2Web-SC | ||||||||
| LPA | LPP | LPR | EA | FRA | LPA | LPP | LPR | EA | FRA | ||
| GPT-4o | GuardAgent | 96.20 | 100.00 | 92.86 | 92.86 | 100.00 | 54.54 | 56.25 | 56.25 | 48.48 | 54.54 |
| Ours | 96.83 | 100.00 | 94.12 | 94.12 | 96.55 | 60.38 | 60.87 | 53.85 | 46.15 | 66.67 | |
| GPT-5.2 | GuardAgent | 88.61 | 92.68 | 95.00 | 85.71 | 91.08 | 54.54 | 55.00 | 68.75 | 39.39 | 42.42 |
| Ours | 97.47 | 97.62 | 97.62 | 97.62 | 94.59 | 56.60 | 55.56 | 57.69 | 38.46 | 55.56 | |
| Avg. | GuardAgent | 92.40 | 96.34 | 93.93 | 89.28 | 95.54 | 54.54 | 55.62 | 62.50 | 43.93 | 48.48 |
| Model | Method | DPI DSR | IPI DSR | FT |
| GPT-4o | AgentSentinel | 0.575 | 0.220 | 0.500 |
| Ours | 0.780 | 0.640 | 0.205 | |
| GPT-5.2 | AgentSentinel | 0.740 | 0.700 | 0.280 |
| Ours | 0.870 | 0.910 | 0.052 | |
| Avg. | AgentSentinel | 0.658 | 0.460 | 0.390 |
| Ours | 0.825 | 0.775 | 0.129 |
| Overall computation overhead | |||||||
| Methods | No Defense | Deli. | Para. | Instr. | Rewr. | Sand. | Ours |
| Computation | 1.00 | 1.14 | 1.10 | 0.98 | 1.12 | 1.11 | 2.34 |
| Component-wise overhead of TAS | |||||||
| Component | Vanilla | MDP modeling | Evaluator calls | Sampling | |||
| Inference overhead | 1.000 | 0.000 | 0.15 | 1.19 | |||
| 1 | 10 | 100 | 200 | 500 | 1000 | |
| ASR (%) | 68.58 | 65.23 | 55.20 | 40.35 | 34.72 | 31.09 |
| Rel. cost ( ) | 1.00 | 1.06 | 1.59 | 2.34 | 6.09 | 12.27 |
| Method | ASR | PNA |
| Ours | 45.15 | 63.63 |
| Ours + score misclassifications | 45.61 | 59.37 |
| Method | FPR | FNR |
| TAS | 36.68 | 3.93 |
| Default | 43.26 | 47.51 |
| Progent | 37.28 | 46.51 |
| DRIFT | 69.40 | 15.81 |
| Firewalls | 47.21 | 12.74 |
| Sentinel | 89.81 | 4.96 |
| No Defense | Deli. | Instr. | Para. | Rewr. | Sand. | Shuffle | |
| McNemar (DPI) | 4.23e-17 | 7.92e-21 | 3.81e-07 | 1.46e-03 | 1.39e-11 | / | / |
| McNemar (IPI) | 5.67e-11 | 4.93e-13 | 6.55e-04 | / | / | 1.59e-06 | / |
| McNemar (POT) | 2.70e-39 | / | / | 6.62e-14 | / | / | 5.05e-44 |
LLM Agent Safety
Large language model agent safety studies how to keep tool-using, persistent, and multi-agent systems aligned with user intent and authorized limits.
See all 358 papers