When safety is formulated as a limit of cumulative cost, safe reinforcement learning (RL) aims to learn policies that maximize return subject to the cost constraint in data collection and deployment. Off-policy safe RL methods, although offering high sample efficiency, suffer from constraint violations due to cost-agnostic exploration and estimation bias in cumulative cost. To address this issue, we propose Constrained Optimistic eXploration Q-learning (COX-Q), an off-policy safe RL algorithm that integrates cost-bounded online exploration and conservative offline distributional value learning. First, we introduce a novel cost-constrained optimistic exploration strategy that resolves gradient conflicts between reward and cost in the action space and adaptively adjusts the trust region to control the training cost. Second, we adopt truncated quantile critics to stabilize the cost value learning. Quantile critics also quantify epistemic uncertainty to guide exploration. Experiments on safe velocity, safe navigation, and autonomous driving tasks demonstrate that COX-Q achieves high sample efficiency, competitive test safety performance, and controlled data collection cost. The results highlight COX-Q as a promising RL method for safety-critical applications.
Figures & tables
Figure 1: COX-Q v.s. off-policy (top) and on-policy (bottom) baselines. TrainingEpCost is for data collection, which is expected to stay near or below the threshold throughout the training. Note that training and test costs are identical for on-policy baselines.
Figure 2: Benchmark of COX-Q against off-policy baselines on safe navigation tasks (episode cost limit is 10). The bottom figure is the cost value estimation bias, computed from cost critic outputs and the recorded trajectories in the evaluation phase. Below 0 means underestimation.
Figure 3: Ablations on Safe Velocity and Safe Navigation.
Scenario
Metric
CPPOPID
SACLag
CAL
TQC-ORAC
COX-Q (ours)
Overtaking
Collision
331
194
186
97
99
Off-road
96
2
7
3
4
Rule violation
3
0
0
0
0
Timeout
0
2
1
887
0
Intersection
Collision
183
33
23
18
12
Off-road
22
2
1
1
2
Table 1: Test safety performance on SMARTS (512K steps, 2000 stochastic runs)
Scenario
CPPOPID
SACLag
CAL
TQC-ORAC
COX-Q (ours)
Overtaking
3697
1570
1544
3215
1665
Intersection
4969
1755
739
3589
1123
T-junction
5513
1965
1675
3837
1794
Table 2: Number of unsafe events in data collection (512K steps, excluding the initial 5120 steps)
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure C.1: The four selected robots in SafetyVelocity-v1 benchmark.
ROBOT
(wh,wv,wc)
velocity threshold
Action dimension
hopper
(1, 1, 0.001)
0.7402
3
walker2d
(1, 1, 0.001)
2.3415
6
ant
(1, 1, 0.5)
2.6222
8
humanoid
(5, 1.25, 0.1)
1.4119
17
Appendix
Table C.1: Weight coefficients and velocity threshold for SafetyVelocity-v1
Figure C.2: The robots and the tasks in the safe navigation benchmark.
Figure C.3: The three autonomous driving scenarios in SMARTS benchmarks. Arrows are the entering lane of the ego vehicle, and triangles are goal positions. Highlighted green lanes are the “on-route” areas for the ego vehicle. White boxes are surrounding traffic vehicles.
Figure C.4: The policy and critic network structure for SMARTS.
Ensuring safety during reinforcement learning (RL) training is critical in real-world applications where unsafe exploration can lead to devastating outcomes. While most safe RL methods mitigate risk through constraints or penalization, they still allow exploration of unsafe states during training. In this work, we adopt a stricter safety requirement that eliminates unsafe state visitation during training. To achieve this goal, we propose a Q-learning-based safe RL framework that leverages a behavior policy supported on a safe set. Under the assumption that the induced trajectories remain within the safe set, this policy enables sufficient exploration within the safe region without requiring near-optimality. We adopt a two-stage framework in which the Q-function and policy are trained separately. Specifically, we introduce a KL-regularized Bellman target that constrains the Q-function to remain close to the behavior policy. We then derive the policy induced from the trained Q-values and propose a parametric policy extraction method to approximate the optimal policy. Our approach provides a unified framework that can be adapted to different action spaces and types of behavior policies. Experimental results demonstrate that the proposed method achieves stable learning and well-calibrated value estimates and yields safer behavior with comparable or better performance than existing baselines.
Yeeun Lim, Narim Jeong, Donghwan Lee
HD Korea Shipbuilding & Offshore Engineering · Department of Electrical Engineering, KAIST, Daejeon, South Korea
This paper addresses the problem of safe offline reinforcement learning, which involves training a policy to satisfy safety constraints using an offline dataset. This problem is inherently challenging as it requires balancing three highly interconnected and competing objectives: satisfying safety constraints, maximizing rewards, and adhering to the behavior regularization imposed by the offline dataset. To tackle this trilogy challenge, we propose Q-learning Penalized Transformer policy (QPT), a training--inference consistent framework that bridges conditional sequence modeling with constraint-aware value estimation. QPT trains a Transformer policy that generates actions conditioned on trajectory context and target return/cost, retaining strong behavior regularization. To inject explicit safety semantics during learning, we augment sequence-model training with a Q-shaped penalty using learned reward and cost Q-functions to favor high return under low constraint violation. At inference, the same Q-functions enforce the cost threshold and choose the highest-reward feasible action, closing the loop between training and deployment. We provide a principled analysis under stylized near-deterministic CMDPs, characterizing how Q-penalized conditional generation improve safety and performance. Empirically, QPT consistently outperforms strong safe offline RL baselines across 38 tasks on the DSRL benchmark, and exhibits robust zero-shot adaptation to different constraint thresholds.
Shengchao Hu, Peng Wang, Jifeng Hu +5
Shanghai Jiao Tong University, China · Shenzhen Campus of Sun Yat-sen University, China · Jilin University, China +1
Safe reinforcement learning (Safe RL) aims to maximize expected return while satisfying safety constraints, typically modeled as Constrained Markov Decision Processes (CMDPs). While primal-dual methods scale well to deep RL, they often suffer from delayed constraint correction, leading to oscillatory behavior and prolonged safety violations. In this paper, we propose Constraint-Sensitive Policy Optimization (CSPO), a first-order primal-dual method that incorporates local constraint sensitivity into policy updates. CSPO augments the primal objective with a constraint-sensitive correction derived from the shortest signed distance to the safety boundary, enabling smarter recovery steps back to safety, compensating for delayed Lagrange multiplier updates, reducing oscillations near the boundary, and preserving the KKT solutions of the original constrained problem. Experiments on navigation and locomotion benchmarks demonstrate that CSPO achieves faster safety recovery and high reward preservation, resulting in higher constrained returns compared to state-of-the-art primal-dual and penalty-based methods