Dependence-Aware Risk Assessment and Optimal Design of AI-Agent Safeguard Systems
Authors: Yijia Li, Maochao Xu, Peng Zhao
Organizations: School of Statistics and Data Science, Shanghai University of International Business and Economics, China · Department of Mathematics, Illinois State University, USA · School of Mathematics and Statistics, Jiangsu Normal University, China
Large language model (LLM) agents increasingly rely on safeguards to prevent unauthorized actions while preserving legitimate service. However, existing risk assessments often focus on individual safeguard performance while overlooking dependence among safeguard decisions. This paper develops a dependence-aware framework for risk assessment and optimal design of multi-component AI safeguard systems, jointly selecting configurations and operating policies for sequential layered screening and selective verification. We establish stochastic comparisons of layered-screening error probabilities and invocation costs under lower-orthant ordering. For selective verification, we characterize exactly which policies have error probabilities and review costs that respond monotonically to stronger dependence under supermodular ordering. For a fixed first stage and reviewer, we show that the optimal routing map is a two-threshold rule in the first-stage likelihood ratio. An accompanying open-source software package, Dependence-Aware Policy Optimization Algorithms (DAPOA), facilitates the practical implementation of the proposed framework. Numerical experiments show that ignoring dependence can lead to inaccurate risk assessments and infeasible safeguard designs, with independence-based selections violating security requirements in six of eight scenarios. An empirical study using four LLM safeguards on benchmark-derived agent actions further demonstrates the practical implications of dependence modeling. Under the principal operating setting, dependence-aware selection lowers held-out total cost by 39.6% relative to independence-based selection.
Figures & tables
Notation
Definition
Y
True authorization status: Y=1 for an unauthorized action and Y=0 for an authorized action
m,j
Number of available safeguards and safeguard index
Dj,D,d
Decision of safeguard j ; potential decision vector D=(D1,…,Dm) ; and a realization d∈{0,1}m
Py,pjy,qy(d)
Class-conditional probability Py(⋅)=Pr(⋅∣Y=y) ; marginal blocking probability pjy=Py(Dj=1) ; and joint pattern probability qy(d)=Py(D=d)
π,Gπ,P
Operating policy; final system decision; and admissible policy class P=PL∪PR
M(π),B(π)
Security-miss probability P1(Gπ=0) and false-block probability P0(Gπ=1)
Table 1: Main notation.
Copula family
Parameter ordering from model 1 to model 2
Ordering Eq. ( 3.1 )
Gaussian
Σy(1)⪯offΣy(2)
✓
Student- t
Σy(1)⪯offΣy(2) and νy(1)=νy(2)
✓
Clayton
0≤θy(1)≤θy(2)
✓
Gumbel
1≤θy(1)≤θy(2)
✓
Table 2: Sufficient within-family parameter conditions for the lower-orthant ordering with fixed component margins.
Figure 1: Pattern losses, divided by s0(w) , and likelihood-ratio thresholds. Hollow and filled markers show each pattern under Models 1 and 2, colored by its optimal assignment. The transfer leaves the component margins and the thresholds unchanged, but moves pattern 01 from the REVIEW region to the automatic-ALLOW region. The horizontal axis is broken between 1.15 and 2.15 .
Safeguard
Miss probability 1−pj1
False block pj0
cj
D1
0.160
0.030
0.20
D2
0.080
0.120
0.35
D3
0.100
0.050
0.65
D4
0.035
0.025
2.00
Table 3: Specified heterogeneous safeguard pool.
Scenario
Copula
τ0
τ1
Scenario
Copula
τ0
τ1
S0
Independence
0.00
0.00
S4
Student- t5
0.30
0.30
S1
Gaussian
0.30
0.30
S5
Clayton
0.30
0.30
S2
Gaussian
0.10
0.50
S6
Gumbel
0.30
0.30
S3
Gaussian
0.50
0.10
S7
Gaussian, pair-specific
–
–
Table 4: Dependence specifications with the same margins and costs.
Policy
Selected in
F
h
B
Q
R1
S0
{1,2}
3
{11}
{10,01}
R2
S1,S3,S6
{1,2}
4
{11}
{10,01}
R3
S4
{1,2}
4
{10,11}
{01}
R4
S2,S5
{1,2,3}
4
{101,111}
{100,010,110,001,011}
R5
S7
{1,2}
3
{10,11}
{01}
Table 5: Distinct verification policies selected by the primary minimum-total-cost search. Patterns list decisions in increasing safeguard order within F ; 0 denotes ALLOW and 1 denotes BLOCK. Patterns outside B and Q are automatically allowed. All safeguards in F are invoked before routing.
Best feasible baseline
Full-class selection
Scenario
Policy
M
B
K
J
Policy
M
B
K
J
S0
C(1,2;4)
0.0474
0.0037
1.0110
1.5174
R1
0.0342
0.0107
0.6475
1.0865
S1
L(1,3)
0.0413
0.0726
0.7779
1.8441
R2
0.0488
0.0230
0.8071
1.5017
S2
L(4)
0.0350
0.0250
2.0000
2.5750
R4
0.0490
0.0105
1.5555
2.1399
S3
L(1,3)
0.0233
0.0658
0.7779
1.6032
R2
0.0297
0.0326
0.7831
1.3740
S4
L(1,3)
0.0460
0.0693
0.7779
1.8612
R3
0.0485
0.0385
0.7617
1.5938
Table 6: Best feasible baseline and full-class policy in each scenario under M≤0.05 , B≤0.10 , and (ω,ℓM,ℓB)=(0.10,100,10) . Both are optimized using the scenario’s specified joint distribution. The baseline class contains sequential layered screening and flag-to-review cascading; the full class additionally permits selective verification. Policies R1–R5 are defined in Table 5 .
Figure 2: Security-miss probabilities of total-cost policies selected using either the scenario’s dependence model (circles) or its same-margin independence model (squares). The dashed line marks α=0.05 .
Best feasible baseline
Full-class selection
Scenario
Policy
M
K
Policy
M
K
Mind
S0
L(1,4)
0.0056
1.9780
R6
0.0018
1.4020
0.0018
S1
L(1,4)
0.0176
1.9780
L(1,4)
0.0176
1.9780
0.0189
S2
L(1,4)
0.0271
1.9780
L(1,4)
0.0271
1.9780
0.0382
S3
L(1,4)
0.0090
1.9780
R6
0.0054
1.3451
0.0054
S4
L(1,4)
0.0201
1.9780
L(1,4)
0.0201
1.9780
0.0235
Table 7: Minimum-miss designs under K(π;0.10)≤2 and B(π)≤0.10 . Mind is the miss probability of the independence-selected policy R6. All reported metrics are evaluated under the scenario’s dependence model.
Figure 3: Overview of dependence-aware safeguard risk assessment and optimal policy selection.
Set
Users
Paired units
Authorized
Unauthorized
Calibration
347
664
664
664
Evaluation
338
598
598
598
Total
685
1,262
1,262
1,262
Table 8: Composition of the action-authorization dataset. Each paired unit contains one authorized and one unauthorized action. Users are disjoint between calibration and evaluation.
Component
cj
Misses
False blocks
Qwen (Q)
0.1000
124
9
Gemma (G)
0.1040
172
2
Ministral (Mi)
0.1020
13
20
Sonnet (H)
1.2840
2
0
Table 9: Component errors in the calibration sample and fixed baseline invocation costs. Error counts are based on 664 actions per authorization class.
Pair
Unauthorized
Authorized
(Q,G)
0.431
−0.006
(Q,Mi)
0.211
0.208
(G,Mi)
0.214
0.151
Table 10: Sample Kendall’s τ between local safeguards’ binary decisions in the calibration sample. Each authorization class contains 664 actions.
Setting
Model
Selected policy
Misses
False blocks
K
J
ℓM=10
Independence
Mi→G
15
32
0.1529
0.5458
Gaussian
Mi
17
30
0.1020
0.4950
Principal ( ℓM=100 )
Independence
πI
14
9
0.3779
1.6237
Gaussian
Mi→H
0
30
0.7300
0.9809
ℓM=1000
Independence
Mi→Q→G
14
32
0.2016
12.1749
Gaussian
Mi→H
0
30
0.7300
0.9809
Table 11: Optimal policy selections in the full class and evaluation performance under different operating settings. Unless otherwise specified, parameters take their principal values. Error counts are based on 598 actions per class. Sequential arrows denote stopping at the first BLOCK. The dagger indicates a violation of at least one evaluation risk limit.
Figure 4: Fitted and evaluation probabilities of the error events of independence-selected policies. Left: joint ALLOW on unauthorized actions. Right: at least one BLOCK on authorized actions.
Figure 5: Evaluation miss rates, false-block rates, operating costs, and total costs under three alternative risk requirements. Markers show the corresponding miss limits α and false-block limits β . All settings use baseline costs, ω=0.50 , and (ℓM,ℓB)=(100,10) .
Figure 6: Evaluation total-cost difference JIndependence−JGaussian across all cost–mixture settings. Positive values favor Gaussian-based policy selection.