Offline safe reinforcement learning requires reliable long-horizon safety estimates from fixed data. Learning Hamilton-Jacobi (HJ) viability in this setting calls for semantic preservation, convergence, and numerical stability. To address these challenges, we propose Monotone Viability Learning (MoViL), which learns viable sets from offline transitions through monotone finite-horizon set recursion. The recursion preserves finite-horizon safety semantics and converges to the infinite-horizon viability kernel. Binary supervision maintains a fixed target scale across horizons, avoiding discount-induced numerical attenuation. We further establish sufficient conditions for recovering behavior-supported viable subsets. The learned viability estimates are used to certify actions at deployment. Dubins car experiments validate the numerical accuracy of the learned viable sets. Evaluations on safe RL benchmarks demonstrate state-of-the-art reward-cost performance. MoViL satisfies the average cost constraint on every task and achieves the lowest average cost among the compared methods. These results highlight MoViL's potential to extend long-horizon HJ viability analysis to safety-critical systems where reliable dynamics models are unavailable and online exploration is unsafe.
Figures & tables
Figure 1: Monotone refinement of finite-horizon viable sets. As the horizon increases, the green sets Kk shrink toward K∞ , progressively excluding states without sufficiently long safe continuations. Arrows indicate state transitions.
Parameter
Value
Offline transitions
200,000
Maximum trajectory length
64
HJ reference grid
91×91×40
Hk network
2 hidden layers, width 192
Activation function
ReLU
Optimizer
Adam
Table 1: Experimental settings for the Dubins car problem.
Figure 2: Viability approximation on the θ=0 slice of the Dubins car state space. (a) MoViL (Ours), (b) FVI, and (c) FISOR. Colors represent the learned function values. Solid and dashed lines indicate the numerical HJ reference and predicted viability boundaries, respectively.
Figure 3: Evolution of viability learning at layers k=1,2,4,8,20 on the θ=0 slice. Each panel shows the current-layer prediction Hk and its calibrated threshold. Solid and dashed lines indicate the numerical HJ reference and predicted boundaries, respectively.
Method
IoU (%) ↑
FPR (%) ↓
FNR (%) ↓
Time (s) ↓
Ours
99.34±0.25
0.42±0.28
0.62±0.28
45.94±2.68
FISOR
94.48±0.23
0.00±0.00
5.52±0.23
84.40±1.95
FVI
99.07±0.09
0.21±0.05
0.91±0.09
2985.38±5.07
Table 2: Viability approximation results on the Dubins car (mean ± standard deviation over three seeds).
Figure 4: IoU (left) and classification errors (right) across recursion layers on the Dubins car problem. Lines and shading indicate the mean and ±1 standard deviation over three seeds. One run converged at k=20 and the other two at k=23 . The first run’s final values are reused when computing statistics at k=21 – 23 .
Parameter
Value
Network architecture
2 hidden layers, width 256
Batch size
512
Hk steps: first / later layers
10,000 / 2,000
Recursion layers: min. / max.
20 / 100
Convergence-validation states
4,096
QH / BC / cost training steps
10,000 / 20,000 / 100,000
Table 3: Key hyperparameters for the offline safe RL experiments.
Task
BCQL
CPQ
CDT
FISOR
LSPC
GSCO
MoViL
reward ↑
cost ↓
reward ↑
cost ↓
reward ↑
cost ↓
reward ↑
cost ↓
reward ↑
cost ↓
reward ↑
cost ↓
reward ↑
cost ↓
Bullet-Safety-Gym
AntRun
0.05
4.63
0.13
0.01
0.69
1.24
0.45
0.76
0.94
1.46
0.60
0.13
0.61
0.06
BallRun
0.35
0.20
0.85
13.67
0.88
0.86
0.14
0.00
0.08
0.00
0.21
0.00
0.30
0.00
CarRun
0.75
2.51
0.75
0.52
0.99
1.47
0.80
0.00
0.75
0.22
0.84
0.00
0.95
0.00
DroneRun
0.65
0.71
0.26
0.44
0.71
0.60
0.41
0.57
0.62
1.34
0.55
0.00
0.57
0.00
Table 4: Performance comparison on Bullet-Safety-Gym and MetaDrive tasks. ↑ indicates higher is better, and ↓ indicates lower is better. Bold : safe policy; Gray : unsafe policy; Bold blue : best safe policy; Bold : second-best safe policy. Safety is defined by normalized cost ≤1 , and rankings are based on reward.
Figure 5: Viability certification ablation on eight Bullet-Safety-Gym tasks. Average normalized reward and cost are reported.
Figure 6: Ablation of probability calibration and trajectory witness protection on eight Bullet-Safety-Gym tasks. Average normalized reward and cost are reported.
Figure 7: Sensitivity to pmin on selected tasks.
Figure 8: Sensitivity to τ on selected tasks.
Figure 9: Effect of viability training data size on policy performance across selected tasks.