Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning

Janaka Chathuranga Brahmanage; Akshat Kumar

cs.LGMar 8, 2026

Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning

Authors: Janaka Chathuranga Brahmanage, Akshat Kumar

Organizations: School of Computing and Information Systems, Singapore Management University.

Abstract

Sequential decision making using Markov Decision Process underpins many realworld applications. Both model-based and model free methods have achieved strong results in these settings. However, real-world tasks must balance reward maximization with safety constraints, often conflicting objectives, that can lead to unstable min/max, adversarial optimization. A promising alternative is safety reachability analysis, which precomputes a forward-invariant safe state, action set, ensuring that an agent starting inside this set remains safe indefinitely. Yet, most reachability based methods address only hard safety constraints, and little work extends reachability to cumulative cost constraints. To address this, first, we define a safetyconditioned reachability set that decouples reward maximization from cumulative safety cost constraints. Second, we show how this set enforces safety constraints without unstable min/max or Lagrangian optimization, yielding a novel offline safe RL algorithm that learns a safe policy from a fixed dataset without environment interaction. Finally, experiments on standard offline safe RL benchmarks, and a real world maritime navigation task demonstrate that our method matches or outperforms state of the art baselines while maintaining safety.

Figures & tables

Figure 1: Electronic navigation chart for maritime traffic navigation in Singapore strait

Figure 2: State augmentation with budget. A desirable property is set

\bar{S}_{P}

to be persistent (trajectories starting in

\bar{S}_{P}

must end in

\bar{S}_{P}

)

Panel of Figure 3 — Figure 3: Comparison with Optimal Solution: Grid-world results with X-axis as the intended-movement probability $p$ (higher values indicate less noise). Top plots show total return; bottom plots show cost for two budget levels.

	CDT		CAPS		CCAC		LSPC		BCRL (Ours)
	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$
PointButton1	0.53 ±0.01	1.68 ±0.13	0.04 ±0.15	0.58 ±1.42	0.68 ±0.14	4.31 ±3.20	0.09 ±0.20	0.69 ±1.03	0.07 ±0.18	0.79 ±2.10
PointButton2	0.46 ±0.01	1.57 ±0.10	0.12 ±0.20	0.96 ±2.35	0.63 ±0.11	4.82 ±3.61	0.19 ±0.21	1.39 ±2.53	0.12 ±0.22	0.81 ±1.36
PointCircle1	0.59 ±0.00	0.69 ±0.04	0.50 ±0.12	0.68 ±1.51	0.58 ±0.13	2.22 ±1.05	0.56 ±0.35	3.49 ±3.55	0.55 ±0.17	0.86 ±0.21
PointCircle2	0.64 ±0.01	1.05 ±0.08	0.51 ±0.21	0.76 ±0.47	0.18 ±0.50	1.42 ±1.09	0.67 ±0.24	4.18 ±4.56	0.58 ±0.10	0.74 ±0.34
PointGoal1	0.69 ±0.02	1.12 ±0.07	0.46 ±0.28	0.63 ±0.81	0.55 ±0.26	1.88 ±1.72	0.26 ±0.27	0.23 ±0.45	0.56 ±0.25	0.70 ±0.65

Table 1: Results for normalized cost and normalized reward:

\uparrow

indicates that a higher value is better, while

\downarrow

indicates that a lower value is preferable. Bold text denotes safe policies, gray indicates unsafe policies, and blue highlights the highest reward among the safe policies for each task. Each result is obtained by running three random seeds across three distinct cost thresholds and evaluating over 20 episodes. Some tasks are omitted to save space. Complete results are provided in Table 3 of Appendix B .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

	CDT		CAPS		CCAC		LSPC		BCRL (Ours)
	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$
PointButton1	0.53 ±0.01	1.68 ±0.13	0.04 ±0.15	0.58 ±1.42	0.68 ±0.14	4.31 ±3.20	0.09 ±0.20	0.69 ±1.03	0.07 ±0.18	0.79 ±2.10
PointButton2	0.46 ±0.01	1.57 ±0.10	0.12 ±0.20	0.96 ±2.35	0.63 ±0.11	4.82 ±3.61	0.19 ±0.21	1.39 ±2.53	0.12 ±0.22	0.81 ±1.36
PointCircle1	0.59 ±0.00	0.69 ±0.04	0.50 ±0.12	0.68 ±1.51	0.58 ±0.13	2.22 ±1.05	0.56 ±0.35	3.49 ±3.55	0.55 ±0.17	0.86 ±0.21
PointCircle2	0.64 ±0.01	1.05 ±0.08	0.51 ±0.21	0.76 ±0.47	0.18 ±0.50	1.42 ±1.09	0.67 ±0.24	4.18 ±4.56	0.58 ±0.10	0.74 ±0.34
PointGoal1	0.69 ±0.02	1.12 ±0.07	0.46 ±0.28	0.63 ±0.81	0.55 ±0.26	1.88 ±1.72	0.26 ±0.27	0.23 ±0.45	0.56 ±0.25	0.70 ±0.65

Appendix

Table 3: Results for normalized cost and normalized reward:

\uparrow

indicates that a higher value is better, while

\downarrow

CDT

CAPS(IQL)

CCAC

LSPC-Final (Our-Run)

BCRL -Deterministic

BCRL -Stochastic

reward

\uparrow

cost

\downarrow

reward

\uparrow

cost

\downarrow

reward

\uparrow

cost

\downarrow

reward

\uparrow

cost

\downarrow

reward

\uparrow

cost

\downarrow

reward

\uparrow

cost

\downarrow

PointButton1

0.53 ±0.01

1.68 ±0.13

0.04 ±0.15

0.58 ±1.42

0.68 ±0.14

4.31 ±3.20

0.09 ±0.20

0.69 ±1.03

0.16 ±0.20

0.97 ±1.33

0.07 ±0.18

0.79 ±2.10

PointButton2

0.46 ±0.01

1.57 ±0.10

0.12 ±0.20

0.96 ±2.35

0.63 ±0.11

4.82 ±3.61

0.19 ±0.21

1.39 ±2.53

0.14 ±0.21

0.72 ±0.84

0.12 ±0.22

0.81 ±1.36

PointCircle1

0.59 ±0.00

0.69 ±0.04

0.50 ±0.12

0.68 ±1.51

0.58 ±0.13

2.22 ±1.05

0.56 ±0.35

3.49 ±3.55

0.47 ±0.10

0.78 ±0.87

0.55 ±0.17

0.86 ±0.21

PointCircle2

0.64 ±0.01

1.05 ±0.08

0.51 ±0.21

0.76 ±0.47

0.18 ±0.50

1.42 ±1.09

0.67 ±0.24

4.18 ±4.56

0.45 ±0.18

0.76 ±0.41

0.58 ±0.10

0.74 ±0.34

PointGoal1

0.69 ±0.02

1.12 ±0.07

0.46 ±0.28

0.63 ±0.81

0.55 ±0.26

1.88 ±1.72

0.26 ±0.27

0.23 ±0.45

0.61 ±0.21

0.88 ±0.65

0.56 ±0.25

0.70 ±0.65

Appendix

Table 4: Results for normalized cost and normalized reward:

\uparrow

indicates that a higher value is better, while

\downarrow

	BC-Safe		BCQ-Lag		COptiDICE		CPQ		BCRL (Ours)
	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$
PointButton1	0.06 ±0.04	0.52 ±0.21	0.24 ±0.04	1.73 ±1.11	0.13 ±0.02	1.35 ±0.91	0.69 ±0.05	3.20 ±1.57	0.07 ±0.18	0.79 ±2.10
PointButton2	0.16 ±0.04	1.10 ±0.84	0.40 ±0.03	2.66 ±1.47	0.15 ±0.03	1.51 ±0.96	0.58 ±0.07	4.30 ±2.35	0.12 ±0.22	0.81 ±1.36
PointCircle1	0.41 ±0.08	0.16 ±0.11	0.54 ±0.17	2.38 ±1.30	0.86 ±0.01	5.51 ±2.93	0.43 ±0.07	0.75 ±1.86	0.55 ±0.17	0.86 ±0.21
PointCircle2	0.48 ±0.08	0.99 ±0.35	0.66 ±0.13	2.60 ±0.71	0.85 ±0.01	8.61 ±4.62	0.24 ±0.40	3.58 ±3.09	0.58 ±0.10	0.74 ±0.34
PointGoal1	0.43 ±0.12	0.54 ±0.24	0.71 ±0.02	0.98 ±0.46	0.49 ±0.05	1.66 ±1.05	0.57 ±0.08	0.35 ±0.37	0.56 ±0.25	0.70 ±0.65

Appendix

Table 5: Comparison with Non-Adaptive Baselines : This table presents results similar to our main results table (Table 1 ), but includes additional baseline methods that do not generalize to varying cost budgets. The results for these baselines are taken from the DSRL benchmark ( Liu et al. 2024 ) .

Figure 5: The red region is used as the environment area. The gray areas indicate anchorage zones, the green areas represent landmasses, and the arrows and regions with dark blue borders represent the Maritime Traffic Separation Scheme (TSS). The high density of crossing points in the red area makes it a more challenging region for navigation, providing a suitable setting for testing advanced planning techniques.

Expectile $\tau_{r}$	0.5		0.6		0.7
	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$
HopperVelocityGymnasium	0.83	0.79	0.80	0.66	0.85	0.60
PointPush1Gymnasium	0.40	0.17	0.46	0.14	0.34	0.17
PointPush2Gymnasium	0.66	0.12	0.71	0.14	0.68	0.12
SwimmerVelocityGymnasium	0.82	0.36	0.98	0.49	0.99	0.38
AntRun	0.95	0.68	1.00	0.69	0.98	0.68

Appendix

Table 6: Ablation on the Reward Expectile

\tau_{R}

Cost Expectile

\tau_{C}

Figure 6: Percentage of Feasible Action Samples:

	LSPC-Best-Policy		BCAP-Best-Policy(ours)
Task	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$
PointButton1	0.10±0.19	0.72±1.17	0.16±0.21	0.81±1.07
PointButton2	0.17±0.20	1.01±1.52	0.23±0.26	0.89±1.06
PointCircle1	0.32±0.18	0.01±0.07	0.47±0.19	0.97±0.84
PointCircle2	0.44±0.21	0.00±0.00	0.54±0.19	0.81±0.82
PointGoal1	0.41±0.31	0.46±0.76	0.68±0.17	0.98±0.93

Appendix

Table 7: Performance of the best policy evaluated during training.

Figure 7: Grid-World Environment: The agent starts at the green cell and receives a reward of

-1

per step. Reaching the target green cell terminates the episode. We vary the transition probability of the intended movement to make navigation more difficult and stochastic, resulting in a non-zero expected cost.

		% of cost critic learning
task		20%	50%	80%	100%
CarCircle1	reward	0.68	0.55	0.58	0.50
	cost	2.79	1.22	1.72	0.50
CarCircle2	reward	0.59	0.60	0.51	0.47
	cost	2.49	3.36	1.03	0.53
CarGoal1	reward	0.45	0.39	0.36	0.39

Appendix

Table 8: Ablation by partially learning cost critic

	BulletGym	SafetGym	MetaDrive
Average Training Time	7.1 minutes	8.5 minutes	18.4 minutes
Evaluation Time	1.1 minute	4.1 minutes	10.8 minutes
Training Steps	100k	100k	200k

Appendix

Table 9: Average Training and evaluation time per instance (by evaluating on 3 distinct cost thresholds, 20 episodes)

Hyperparameters	BulletGym	SafetGym	MetaDrive
Common
Training Steps	100k	100k	200k
Policy network architecture	[512, 512] MLP	[512, 512] MLP	[512, 512] MLP
Cost thresholds for main results	(10, 20, 40)	(20, 40, 80)	(10, 20, 40)
Seeds for main results	(0, 10, 20)	(0, 10, 20)	(0, 10, 20)
BCR

Appendix

Table 10: Default Hyperparameters of BCRL for tasks in three environments.

Safe RL

Safe reinforcement learning studies how agents can learn and act while respecting safety constraints and limiting harmful outcomes.

See all 213 papers

Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies
Mar 16, 2026Mumuksh Tayal, Manan Tayal, Ravi PrakashFlow Matching Offline RL
Indian Institute of Science, India
Off-Policy Safe Reinforcement Learning with Constrained Optimistic Exploration
Mar 25, 2026Guopeng Li, Matthijs T. J. Spaan, Julian F. P. KooijSafe RL Constrained RL
Faculty of Electrical Engineering, Mathematics and Computer Science, Delft University of Technology, Delft, the Netherlands
Q-learning Penalized Transformer for Safe Offline Reinforcement Learning
Sep 28, 2026Shengchao Hu, Peng Wang, Jifeng Hu +5Reinforcement Learning Transformer-Based RL
Shanghai Jiao Tong University, China · Shenzhen Campus of Sun Yat-sen University, China · Jilin University, China +1

Mar 16, 2026J/K move · Enter open · S save

Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies

Mumuksh Tayal, Manan Tayal, Ravi Prakash

Indian Institute of Science, India

Flow Matching Offline RL Safe RL Flow-Based Policy Learning

Open paper AskPDF arXiv

Offline safe reinforcement learning (RL) seeks reward-maximizing policies from static datasets under strict safety constraints. Existing methods often rely on soft expected-cost objectives or iterative generative inference, which can be insufficient for safety-critical real-time control. We propose Safe Flow Q-Learning (SafeFQL), which extends FQL to safe offline RL by combining a Hamilton--Jacobi reachability-inspired safety value function with an efficient one-step flow policy. SafeFQL learns the safety value via a self-consistency Bellman recursion, trains a flow policy by behavioral cloning, and distills it into a one-step actor for reward-maximizing safe action selection without rejection sampling at deployment. Empirically, SafeFQL trades modestly higher offline training cost for substantially lower inference latency than diffusion-style safe generative baselines, which is advantageous for real-time safety-critical deployment. Across boat navigation, and Safety Gymnasium MuJoCo tasks, SafeFQL matches or exceeds prior offline safe RL performance while substantially reducing constraint violations.

Abstract

Safe RL

Explore similar work

Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies

Off-Policy Safe Reinforcement Learning with Constrained Optimistic Exploration

Q-learning Penalized Transformer for Safe Offline Reinforcement Learning