LLM-based agents extend LLMs with tool calls, memory, planning, and retrieval to act in external environments, but they are vulnerable to attacks at different workflow steps. Existing defenses, including prompt engineering and post-training alignment, primarily improve LLM-only safety and often overlook risks that arise from multi-step execution. External verification adds guardrails to check each step separately, but it typically ignores how risks can propagate across steps. We argue that we should consider risk transitions across dependent steps through the agent workflow, rather than separate verification on a single step. In this paper, we propose Tail-risk-aware Agent Safety (TAS), a tail-risk-aware safety framework that models agent workflow safety as a sequence of dependent safety states and constrains the rare but extreme risks (i.e., tail risk) with a shared budget. TAS first models safety-state transitions sequentially along the workflow as an MDP to trace risk propagation, then measures a tail-sensitive risk to target rare but high-cost risks (i.e., tail risk), and constrains the risk with a shared budget to derive a step-wise safety strategy. By constraining tail risk rather than the overall average, TAS suppresses rare high-cost failures while preserving utility in normal cases. Experiments show that TAS achieves better performance compared with eleven baselines across three benchmarks.
Figures & tables
Figure 1: Overview of TAS. The upper pipeline shows an LLM-based agent executing a multi-step workflow from queries to outputs. As shown in the bottom half, TAS models the safety state along the workflow as an MDP (blue left part), which formalizes the action-conditioned safety-state transitions. Then, with a candidate action, TAS samples candidate outcomes to estimate a risk distribution and measures the tail risk with SRM (yellow middle part). Next, TAS sets a shared budget constraint to check the risk score and provides a safety strategy under three cases (red right part). The process repeats until the workflow ends or the budget is exhausted, to finally generate outputs.
Model
Metric
DPI
IPI
No Def.
Deli.
Para.
Instr.
Rewr.
Ours
No Def.
Deli.
Instr.
Sand.
Ours
Llama3-70B
ASR ↓
87.75
88.25
71.25
87.25
74.50
43.50
45.50
34.50
41.50
39.75
13.95
NRP ↑
6.80
6.76
14.81
7.87
12.69
32.91
30.25
37.66
36.12
32.54
50.12
PNA ↑
55.50
57.50
51.50
61.75
49.75
58.25
55.50
57.50
61.75
54.00
58.25
Qwen2-72B
ASR ↓
94.50
95.00
60.50
95.50
56.20
48.35
23.75
17.50
26.50
21.75
17.05
NRP ↑
1.09
1.06
8.69
1.05
7.77
11.36
15.06
17.53
17.09
16.82
18.25
Table 1: Main results of five defenses on four LLM backbones under DPI and IPI attacks. Lower ASR indicates better safety, while higher PNA and NRP indicate better utility and overall performance. We focus on ASR and NRP, with PNA reported to ensure utility remains close to No Defense . Best and second-best results are bolded and underlined. All improvements are significant under a two-sided exact McNemar test ( p<0.01 ) (App. B.8 ).
Model
No Def.
Shuffle
Para.
Ours
Llama3-70B
57.00
63.75
44.75
15.00
Qwen2-72B
96.50
81.50
57.75
43.25
GPT-4o
100.00
98.50
84.75
42.75
GPT-5.2
87.25
88.50
67.00
39.75
Avg.
85.19
83.06
63.56
35.19
Table 2: ASR (%) ↓ under POT attacks across four LLM backbones.
Benchmark
Metric
Model
No Def.
Progent
DRIFT
Firewall
Sentinel
GuardAgent
Ours
Agent- SafetyBench
Safety Score ↑
Llama3-70B
0.325
0.400
0.383
0.408
0.492
0.325
0.508
Qwen2-72B
0.225
0.275
0.292
0.275
0.383
0.483
0.483
GPT-4o
0.442
0.442
0.450
0.417
0.700
0.450
0.758
GPT-5.2
0.625
0.633
0.667
0.592
0.808
0.675
0.908
AgentDojo
ASR ↓ / Utility ↑
Llama3-70B
0.18 / 10.98
0.36 / 10.90
0.00 / 10.37
0.00 / 11.11
0.00 / 9.53
0.00 / 11.03
0.00 / 11.11
Qwen2-72B
0.00 / 9.15
0.00 / 8.83
0.00 / 8.54
0.00 / 9.15
0.00 / 7.35
0.00 / 9.15
0.00 / 9.15
Table 3: Additional results on Agent-SafetyBench and AgentDojo across four LLM backbones.
Model
Metric
DPI
IPI
Ours
w/o MDP
w/o Tail
w/o SRM
w/o Constr.
w/o Samp.
Ours
w/o MDP
w/o Tail
w/o SRM
w/o Constr.
w/o Samp.
Llama3-70B
ASR ↓
43.50
50.75
69.75
50.50
87.75
66.25
13.95
18.50
22.25
18.75
45.50
26.00
NRP ↑
32.91
27.21
14.97
23.64
6.62
18.39
50.12
45.03
38.49
38.80
29.43
40.33
PNA ↑
58.25
55.25
49.50
47.75
54.00
54.50
58.25
55.25
49.50
47.75
54.00
54.50
Qwen2-72B
ASR ↓
48.35
70.00
96.50
69.00
99.75
94.75
17.05
22.25
23.50
21.25
27.25
26.75
NRP ↑
11.36
6.68
0.78
6.28
0.06
1.18
18.25
17.30
17.02
15.95
16.73
16.48
Table 4: Ablation results for TAS under DPI and IPI attacks. “w/o” denotes removing the corresponding component.
Figure 2: Tail-risk visualization of our method and four baselines (curves with different colors) under DPI attack across four LLM backbones (Fig. 2(a) , 2(b) , 2(c) , and 2(d) ). The X-axis indicates the tail probability, and the Y-axis indicates the risk cost threshold. Lower x-value and higher y-value indicate high cost with low-probability (i.e., tail risk).
Figure 3: Utility-safety trade-offs under DPI attacks with varying budgets and quantile cutoffs across four LLM backbones. Higher values indicate better utility and safety.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Setting
OS
Ubuntu 22.04 LTS
GPU
12 × NVIDIA A100 80GB PCIe
NVIDIA Driver Version
560.35.03
CPU Architecture
x86_64
CPU Model
Intel(R) Xeon(R) Platinum 8358 @ 2.60GHz
CPU Threads
128
Appendix
Table 5: Implementation details of our experiments.
Model
Metric
DPI
IPI
No Defense
Deli.
Para.
Instr.
Rewr.
Ours
No Defense
Deli.
Instr.
Sand.
Ours
LlaMA3-8B
ASR
33.75
62.75
28.50
52.00
23.15
22.50
11.50
9.25
8.75
13.00
2.85
NRP
0.99
0.65
1.79
2.40
1.92
3.68
1.33
1.59
4.56
2.61
4.61
PNA
1.50
1.75
2.50
5.00
2.50
4.75
1.50
1.75
5.00
3.00
4.75
Qwen2-7B
ASR
73.50
80.00
46.25
76.75
33.30
12.80
9.25
11.25
9.50
11.00
1.65
NRP
0.60
0.40
0.94
0.76
2.00
3.05
2.04
1.77
2.94
2.00
3.44
Appendix
Table 6: Additional ASB results across smaller LLM backbones.
Model
Metric
DPI
IPI
Ours
w/o mdp
w/o tail
w/o srm
w/o constraint
w/o sampling
Ours
w/o mdp
w/o tail
w/o srm
w/o constraint
w/o sampling
LlaMA3-8B
ASR
22.50
27.25
37.75
31.50
38.78
38.00
2.85
3.25
9.50
4.50
11.50
3.00
NRP
3.68
3.27
1.40
1.20
2.76
2.79
4.61
4.35
2.04
1.67
3.98
4.37
PNA
4.75
4.50
2.25
1.75
4.50
4.50
4.75
4.50
2.25
1.75
4.50
4.50
Qwen2-7B
ASR
12.80
14.50
22.00
17.00
73.50
24.65
1.65
2.50
3.00
2.00
9.25
6.25
NRP
3.05
2.78
2.15
1.25
0.93
2.45
3.44
3.17
2.67
1.47
3.18
3.05
Appendix
Table 7: Ablations on the small LLMs for different parts of our method. W/o means the removal of the corresponding part.
Model
Metric
No Def.
Shuffle
Para.
Ours
Llama3-70B
ASR ↓
57.00
63.75
44.75
15.00
NRP ↑
23.87
16.04
23.48
49.51
PNA ↑
55.50
44.25
42.50
58.25
Qwen2-72B
ASR ↓
96.50
81.50
57.75
43.25
NRP ↑
0.69
4.30
10.03
12.49
PNA ↑
19.75
23.25
23.75
22.00
Appendix
Table 8: Supplementary ASR, NRP, and PNA results under POT attacks across four LLM backbones.
Model
Method
EHRAgent on EICU-AC
SeeAct on Mind2Web-SC
LPA
LPP
LPR
EA
FRA
LPA
LPP
LPR
EA
FRA
GPT-4o
GuardAgent
96.20
100.00
92.86
92.86
100.00
54.54
56.25
56.25
48.48
54.54
Ours
96.83
100.00
94.12
94.12
96.55
60.38
60.87
53.85
46.15
66.67
GPT-5.2
GuardAgent
88.61
92.68
95.00
85.71
91.08
54.54
55.00
68.75
39.39
42.42
Ours
97.47
97.62
97.62
97.62
94.59
56.60
55.56
57.69
38.46
55.56
Avg.
GuardAgent
92.40
96.34
93.93
89.28
95.54
54.54
55.62
62.50
43.93
48.48
Appendix
Table 9: Additional comparison with GuardAgent under its original evaluation protocol. We report results on EHRAgent with EICU-AC and SeeAct with Mind2Web-SC across two LLM backbones. Following GuardAgent , we report label prediction accuracy (LPA ↑ ), label prediction precision (LPP ↑ ), label prediction recall (LPR ↑ ), explanation accuracy (EA ↑ ), and final response accuracy (FRA ↑ ). Higher values indicate better performance.
Model
Method
DPI DSR ↑
IPI DSR ↑
FT ↓
GPT-4o
AgentSentinel
0.575
0.220
0.500
Ours
0.780
0.640
0.205
GPT-5.2
AgentSentinel
0.740
0.700
0.280
Ours
0.870
0.910
0.052
Avg.
AgentSentinel
0.658
0.460
0.390
Ours
0.825
0.775
0.129
Appendix
Table 10: Additional comparison with AgentSentinel under its original evaluation protocol. Following AgentSentinel , we report defense success rate under direct prompt injection (DPI DSR ↑ ) and indirect prompt injection (IPI DSR ↑ ), as well as failure rate (FT ↓ ), defined as the percentage of failed tasks where the expected attacks fail. Higher DSR indicates stronger defense performance, while lower FT indicates fewer benign-task failures.
Overall computation overhead
Methods
No Defense
Deli.
Para.
Instr.
Rewr.
Sand.
Ours
Computation
1.00 ×
1.14 ×
1.10 ×
0.98 ×
1.12 ×
1.11 ×
2.34 ×
Component-wise overhead of TAS
Component
Vanilla
MDP modeling
Evaluator calls
Sampling
Inference overhead
1.000 ×
0.000 ×
0.15 ×
1.19 ×
Appendix
Table 11: Computation overhead analysis. The upper panel reports the overall computation overhead of different methods relative to No Defense. The lower panel provides a component-wise breakdown of the overhead introduced by TAS.
K
1
10
100
200
500
1000
ASR (%) ↓
68.58
65.23
55.20
40.35
34.72
31.09
Rel. cost ( × ) ↓
1.00
1.06
1.59
2.34
6.09
12.27
Appendix
Table 12: Effect of sampling size K on ASR and relative computational cost.
Method
ASR ↓
PNA ↑
Ours
45.15
63.63
Ours + 10% score misclassifications
45.61
59.37
Appendix
Table 13: Robustness analysis under evaluator score misclassifications. We intentionally introduce 10% score misclassifications into evaluator outputs.
Method
FPR ↓
FNR ↓
TAS
36.68
3.93
Default
43.26
47.51
Progent
37.28
46.51
DRIFT
69.40
15.81
Firewalls
47.21
12.74
Sentinel
89.81
4.96
Appendix
Table 14: False positive rate and false negative rate of different methods over GPT-4o on AgentDojo. FPR ↓ measures the rate of benign cases incorrectly flagged as risky, while FNR ↓ measures the rate of risky cases incorrectly missed. Lower values indicate better risk evaluation.
No Defense
Deli.
Instr.
Para.
Rewr.
Sand.
Shuffle
McNemar p (DPI)
4.23e-17
7.92e-21
3.81e-07
1.46e-03
1.39e-11
/
/
McNemar p (IPI)
5.67e-11
4.93e-13
6.55e-04
/
/
1.59e-06
/
McNemar p (POT)
2.70e-39
/
/
6.62e-14
/
/
5.05e-44
Appendix
Table 15: ASR significance test (two-sided exact McNemar).