Rare Event Constrained Reinforcement Learning with Large Deviations
Authors: Rahul Misra
Organizations: Department of Electronic Systems, Automation and Control section, Aalborg University, Fredrik Bajers Vej 7 C, Aalborg East, 9220, Denmark
We study finite-horizon reinforcement learning with a safety constraint on the probability of reaching a failure set. In contrast to the well-studied safe reinforcement learning literature, we consider problems where the probability of reaching the failure set is vanishingly small, but the consequences of reaching the failure set are catastrophic. Direct simulation and estimation of the optimal policy is naturally ineffective when the failure probability is extremely small. Moreover, on a real system the controller can only choose actions, and this is not enough to make failures frequent. In contrast to the rare event simulation literature, our goal is not to estimate the probability of the rare event occurrence, but to upper bound it such that the failure constraint is satisfied. Thus, instead of sampling failures, we use the large deviations principle. We define a damage variable that on exceeding a certain threshold indicates failure. We study the cumulant generating function (CGF) as its positive root provides an upper bound on the failure probability via Cramer's theorem. We extend this approach to Markov decision processes with hitting times through state-action CGFs, and give finite-horizon and state-dependent versions of the bound. The state-action CGFs do not depend on the policy and can be estimated from regular transitions, so the bound can be checked for any policy a priori. This gives a policy improvement algorithm in which every policy satisfies the failure constraint with high probability.
Figures & tables
Figure 1: Left: CGF of one step of Example 1 with q=0.35 and its positive root, the Lundberg coefficient λ⋆=0.619 . Right: the rate function I and the line λ⋆v , which touches it at the cheapest speed v⋆=1−2q=0.30 (Lemma 1 ).
Figure 2: Bound on pθ(x0) for every policy θ in Example 1 ( T=100 ), from one data set of n steps of the policy θ=0.5 and with the exact CGFs, together with the exact failure probability. The largest certified policy is where the bound crosses δ .
Figure 3: Learning on Example 1 . Left: expected cost J(θk) of the policy run in round k . Right: true failure probability of the policy run in round k (computed from the true model, for checking only). Algorithm 1 (blue) moves only when the data allows it and stays below δ . The primal–dual method (orange, β=0.05 ) moves first and corrects later.
θ
T
pθ(x0)
Bθ
Bθ/pθ
0
30
8.76×10−12
1.46×10−11
1.66
0
100
1.36×10−11
1.49×10−11
1.09
0.75
30
8.88×10−7
2.85×10−6
3.21
0.75
100
9.92×10−6
1.47×10−5
1.48
1
30
1.26×10−5
4.35×10−5
3.46
1
100
3.08×10−4
6.82×10−4
2.21
Table 1: Exact failure probability and certificate ( 13 ), N=20 , x0=2 .
n
certified
median θ
cost above optimum
103
42%
0
14.4%
104
100%
0.466
7.9%
105
100%
0.638
3.8%
exact CGF
–
0.727
0.9%
Table 2: Certificate from n steps of the policy θ=0.5 over 200 data sets: share of data sets in which some policy is certified, median certified θ ( θ=0 if none), and median cost above the optimum J=2.914 .
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Cornell University AI for Science Institute Cornell University Ithaca, NY, USA · Department of Electrical & Computer Engineering University of Washington Seattle, WA, USA