When safety is formulated as a limit of cumulative cost, safe reinforcement learning (RL) aims to learn policies that maximize return subject to the cost constraint in data collection and deployment. Off-policy safe RL methods, although offering high sample efficiency, suffer from constraint violations due to cost-agnostic exploration and estimation bias in cumulative cost. To address this issue, we propose Constrained Optimistic eXploration Q-learning (COX-Q), an off-policy safe RL algorithm that integrates cost-bounded online exploration and conservative offline distributional value learning. First, we introduce a novel cost-constrained optimistic exploration strategy that resolves gradient conflicts between reward and cost in the action space and adaptively adjusts the trust region to control the training cost. Second, we adopt truncated quantile critics to stabilize the cost value learning. Quantile critics also quantify epistemic uncertainty to guide exploration. Experiments on safe velocity, safe navigation, and autonomous driving tasks demonstrate that COX-Q achieves high sample efficiency, competitive test safety performance, and controlled data collection cost. The results highlight COX-Q as a promising RL method for safety-critical applications.
Figures & tables
Figure 1: COX-Q v.s. off-policy (top) and on-policy (bottom) baselines. TrainingEpCost is for data collection, which is expected to stay near or below the threshold throughout the training. Note that training and test costs are identical for on-policy baselines.
Figure 2: Benchmark of COX-Q against off-policy baselines on safe navigation tasks (episode cost limit is 10). The bottom figure is the cost value estimation bias, computed from cost critic outputs and the recorded trajectories in the evaluation phase. Below 0 means underestimation.
Figure 3: Ablations on Safe Velocity and Safe Navigation.
Scenario
Metric
CPPOPID
SACLag
CAL
TQC-ORAC
COX-Q (ours)
Overtaking
Collision
331
194
186
97
99
Off-road
96
2
7
3
4
Rule violation
3
0
0
0
0
Timeout
0
2
1
887
0
Intersection
Collision
183
33
23
18
12
Off-road
22
2
1
1
2
Table 1: Test safety performance on SMARTS (512K steps, 2000 stochastic runs)
Scenario
CPPOPID
SACLag
CAL
TQC-ORAC
COX-Q (ours)
Overtaking
3697
1570
1544
3215
1665
Intersection
4969
1755
739
3589
1123
T-junction
5513
1965
1675
3837
1794
Table 2: Number of unsafe events in data collection (512K steps, excluding the initial 5120 steps)
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure C.1: The four selected robots in SafetyVelocity-v1 benchmark.
ROBOT
(wh,wv,wc)
velocity threshold
Action dimension
hopper
(1, 1, 0.001)
0.7402
3
walker2d
(1, 1, 0.001)
2.3415
6
ant
(1, 1, 0.5)
2.6222
8
humanoid
(5, 1.25, 0.1)
1.4119
17
Appendix
Table C.1: Weight coefficients and velocity threshold for SafetyVelocity-v1
Figure C.2: The robots and the tasks in the safe navigation benchmark.
Figure C.3: The three autonomous driving scenarios in SMARTS benchmarks. Arrows are the entering lane of the ego vehicle, and triangles are goal positions. Highlighted green lanes are the “on-route” areas for the ego vehicle. White boxes are surrounding traffic vehicles.
Figure C.4: The policy and critic network structure for SMARTS.