EpiFlow: Bridging Safety and Performance in Offline Reinforcement Learning via Epigraph-Guided Flow Matching

Manan Tayal; Mumuksh Tayal; Ravi Prakash

cs.LGFeb 8, 2026

EpiFlow: Bridging Safety and Performance in Offline Reinforcement Learning via Epigraph-Guided Flow Matching

Authors: Manan Tayal, Mumuksh Tayal, Ravi Prakash

Organizations: TAU Intelligence

Abstract

Offline reinforcement learning (RL) offers a promising framework for deploying autonomous systems in safety-critical settings without the risks of online exploration. However, learning policies that simultaneously achieve high performance and strong safety guarantees from fixed datasets remains a fundamental challenge. Many existing safe offline RL approaches typically rely on soft constraint formulations, which may permit safety violations and are sensitive to distributional shift. In contrast, formal methods such as Hamilton--Jacobi (HJ) reachability and Control Barrier Functions (CBFs) provide rigorous safety guarantees, but often yield overly conservative solutions, often neglecting performance. In this work, we bridge this gap by formulating safe offline RL as a state-constrained optimal control problem, where safety is enforced through hard state constraints and performance is captured via a reward function. The resulting value function satisfies a Hamilton--Jacobi--Bellman (HJB) equation, which we approximate using offline RL on fixed datasets by learning an epigraph feasibility value function and extracting policies via flow matching. This formulation enables principled integration of safety guarantees with data-driven policy optimization. Empirically, across safety-critical benchmarks including boat navigation, Safety-Gymnasium, and Bullet-Safety-Gym tasks, our approach achieves competitive returns while exhibiting near-zero constraint violations, demonstrating a favorable balance between safety and performance in the offline setting.

Figures & tables

Figure 1: Safety–performance trade-off in offline RL (conceptual illustration): soft-constraint methods violate constraints, while classical safety methods stay safe but follow conservative, lower-performing paths; our method achieves safety with high performance.

Figure 2: Framework Overview. We propose a framework to solve the SC-OCP problem by using offline dataset. We learn $\hat{V}$ , which if satisfies to stay positive, makes sure that the system stays safe while maximizing the reward. We then learn a policy which optimizes for $\hat{V}$ at all time steps to find the most optimal path that the offline data supports.

Figure 3: Evaluation Results: Results plot for all the environments. Points towards left ( $\leftarrow$ ) are more safe than those on right ( $\rightarrow$ ). Whereas those towards top ( $\uparrow$ ) have higher rewards than those towards bottom ( $\downarrow$ ). Evaluated over 500 episodes and 5 seed values ( $\pm$ standard deviation).

Figure 4: Effect of Regularizer parameter (

\lambda

) (

\pm

standard deviation).

Figure 5: Sensitivity to Expectile Parameter (

\tau

) (

\pm

standard deviation).

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Feasibility epigraph value function

\hat{V}_{\theta}(x,z)

. ( Top ) Learned without the decomposition regularizer, where

\hat{V}_{\theta}

exhibits weak sensitivity to the epigraph variable

z

and similar contour structure across different

z

values. ( Bottom ) Learned with the regularizer (

\lambda=0.25

), which restores meaningful dependence on

z

and yields distinct level sets consistent with the epigraph formulation.

Figure 8: Illustration of all evaluation environments. ( Top-Left ) environment illustrates Boat Navigation with obstacles and a goal point. ( Middle ) illustrates SafeVelocity MuJoCo environments with Red sphere denoting unsafe state. ( Bottom ) depicts the SafeCarNavigation MuJoCo environments with various obstacles and objectives.

	Boat		HalfCheetah		Hopper		Ant		Walker2D
Method	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$
BCQ-Lag	$0.65_{\pm 0.03}$	$1.89_{\pm 0.80}$	$1.06_{\pm 0.05}$	$58.50_{\pm 1.00}$	$0.72_{\pm 0.02}$	$19.80_{\pm 1.00}$	$1.00_{\pm 0.03}$	$7.40_{\pm 0.60}$	$0.78_{\pm 0.03}$	$3.50_{\pm 0.30}$
BEAR-Lag	$0.78_{\pm 0.04}$	$1.66_{\pm 0.65}$	$0.97_{\pm 0.06}$	$15.60_{\pm 0.90}$	$0.36_{\pm 0.02}$	$17.30_{\pm 0.90}$	$-1.01_{\pm 0.01}$	$0.00_{\pm 0.00}$	$0.78_{\pm 0.03}$	$1.70_{\pm 0.20}$
CPQ	$0.75_{\pm 0.04}$	$7.41_{\pm 0.70}$	$0.39_{\pm 0.05}$	$7.00_{\pm 0.60}$	$0.13_{\pm 0.01}$	$10.00_{\pm 0.40}$	$-1.01_{\pm 0.01}$	$0.00_{\pm 0.00}$	$0.09_{\pm 0.02}$	$0.35_{\pm 0.21}$
COptiDICE	$0.51_{\pm 0.03}$	$1.72_{\pm 0.45}$	$0.61_{\pm 0.07}$	$0.00_{\pm 0.00}$	$0.16_{\pm 0.03}$	$7.90_{\pm 0.35}$	$1.00_{\pm 0.03}$	$13.30_{\pm 0.70}$	$0.14_{\pm 0.01}$	$3.80_{\pm 0.25}$
C2IQL	$0.66_{\pm 0.03}$	$2.36_{\pm 0.50}$	$1.02_{\pm 0.03}$	$25.20_{\pm 0.80}$	$0.16_{\pm 0.01}$	$3.20_{\pm 0.25}$	$1.00_{\pm 0.03}$	$9.60_{\pm 0.60}$	$0.21_{\pm 0.02}$	$3.10_{\pm 0.30}$

Appendix

Table 1: Results on Boat Navigation and Safety Gymnasium. Normalized reward

R\uparrow

and normalized cost

C\downarrow

, reported as mean

{}_{\pm\text{std}}

over 500 evaluation episodes and 5 seeds. Gray : unsafe under the DSRL convention (

C\geq 1

). Bold : method with the lowest cost on each task (highest reward among ties).

	Ant-Run		Ball-Run		Car-Run		Drone-Run
Method	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$
BCQ-Lag	$0.63_{\pm 0.07}$	$4.65_{\pm 0.85}$	$0.31_{\pm 0.05}$	$2.28_{\pm 0.55}$	$0.97_{\pm 0.04}$	$0.10_{\pm 0.30}$	$0.64_{\pm 0.08}$	$10.32_{\pm 1.55}$
BEAR-Lag	$0.07_{\pm 0.03}$	$0.00_{\pm 0.00}$	$-0.36_{\pm 0.06}$	$9.10_{\pm 1.40}$	$0.66_{\pm 0.05}$	$0.03_{\pm 0.05}$	$0.15_{\pm 0.04}$	$4.31_{\pm 0.80}$
CPQ	$-0.02_{\pm 0.02}$	$0.00_{\pm 0.00}$	$0.27_{\pm 0.05}$	$3.70_{\pm 0.75}$	$1.01_{\pm 0.06}$	$9.40_{\pm 1.35}$	$0.49_{\pm 0.06}$	$1.69_{\pm 0.42}$
COptiDICE	$0.51_{\pm 0.01}$	$0.03_{\pm 0.01}$	$0.42_{\pm 0.08}$	$0.15_{\pm 0.03}$	$0.81_{\pm 0.02}$	$0.00_{\pm 0.00}$	$0.57_{\pm 0.01}$	$0.17_{\pm 0.00}$
C2IQL	$0.67_{\pm 0.01}$	$0.11_{\pm 0.01}$	$0.53_{\pm 0.05}$	$0.11_{\pm 0.01}$	$0.85_{\pm 0.01}$	$0.01_{\pm 0.01}$	$0.64_{\pm 0.02}$	$0.09_{\pm 0.02}$

Appendix

Table 2: Results on Bullet-Safety-Gym (DSRL). Same metrics as Table 1 ; 500 evaluation episodes over 5 seeds. Gray : unsafe under the DSRL convention (

C\geq 1

). Bold : method with the lowest cost on each task (highest reward among ties).

Expectile Parameter $(\boldsymbol{\tau})$	0.5	0.7	0.8	0.9
	(Behaviour Policy)
Mean Episode Reward	-485.86 $\pm$ 0.74	-492.36 $\pm$ 0.21	-496.01 $\pm$ 0.58	-496.10 $\pm$ 0.37
Safety Rate (%)	88.9 $\pm$ 0.23	98.9 $\pm$ 1.01	99.9 $\pm$ 0.18	100.0 $\pm$ 0.0
Mean Episode Cost	1.95 $\pm$ 0.19	0.09 $\pm$ 0.03	0.04 $\pm$ 0.05	0.0 $\pm$ 0.0

Appendix

Table 3: Mean Reward, Safety Rate (%), and Mean Cost reported for Boat Navigation by varying

\tau

in the expectile regression. (

\pm

) represents standard deviation.

Figure 9: Results for Control under Perturbations. The plots show average reward (%) (

\pm

standard deviation) achieved relative to no perturbations case, at different levels of perturbations (5%, 10% and 20%), and their average absolute cost values (

\pm

standard deviation) respectively.

Figure 10: Effect of the regularizer parameter $\lambda$ on the inferred epigraph budget. For the Boat Navigation environment, each panel shows the state-wise optimal budget $z^{\ast}(x)$ defined by the feasibility condition $\hat{V}_{\theta}(x,z^{\ast})\geq 0$ . As $\lambda$ increases, the feasible region contracts and $z^{\ast}(x)$ becomes more conservative; states shaded in purple correspond to infeasible regions.

Metric	Flow Matching	Diffusion Model	Weighted BC
R	-496.23 $\pm$ 0.34	-584.53 $\pm$ 2.37	-716.60 $\pm$ 5.23
C	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	18.56 $\pm$ 4.45

Appendix

Table 4: Mean Episode Reward (R) and Mean Episode Cost (C) (

\pm

standard deviation) reported for proposed framework across different policy backbones. Bold : method with the lowest cost (highest reward among ties).

	Ours		FISOR-FM		FISOR
Task	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$
Boat	$\mathbf{0.82}_{\pm 0.01}$	$\mathbf{0.00}_{\pm 0.00}$	$0.74_{\pm 0.02}$	$0.13_{\pm 0.06}$	$0.73_{\pm 0.02}$	$0.86_{\pm 0.30}$
HalfCheetah	$\mathbf{0.93}_{\pm 0.03}$	$\mathbf{0.00}_{\pm 0.00}$	$0.84_{\pm 0.03}$	$0.00_{\pm 0.00}$	$0.77_{\pm 0.03}$	$0.00_{\pm 0.00}$
Hopper	$\mathbf{0.15}_{\pm 0.00}$	$\mathbf{0.05}_{\pm 0.03}$	$0.06_{\pm 0.00}$	$0.14_{\pm 0.10}$	$0.02_{\pm 0.00}$	$0.16_{\pm 0.10}$
Ant	$\mathbf{0.81}_{\pm 0.03}$	$\mathbf{0.00}_{\pm 0.00}$	$0.72_{\pm 0.02}$	$0.00_{\pm 0.00}$	$0.65_{\pm 0.02}$	$0.00_{\pm 0.00}$
Walker2D	$\mathbf{0.10}_{\pm 0.00}$	$\mathbf{0.10}_{\pm 0.10}$	$0.10_{\pm 0.00}$	$0.11_{\pm 0.10}$	$0.09_{\pm 0.00}$	$0.25_{\pm 0.10}$

Appendix

Table 5: Ablation isolating the epigraph value formulation. Normalized reward

R\uparrow

and cost

C\downarrow

(mean

{}_{\pm\text{std}}

) over 500 evaluation episodes and 5 seeds. Bold : method with the lowest cost on each task (highest reward among ties).

	Ours		FISOR-FM
$\alpha$	$R\uparrow$	$C\downarrow$	$R\uparrow$	$C\downarrow$
0.01	$\mathbf{0.92}_{\pm 0.02}$	$\mathbf{0.00}_{\pm 0.00}$	$0.80_{\pm 0.02}$	$0.00_{\pm 0.00}$
0.1	$\mathbf{0.93}_{\pm 0.03}$	$\mathbf{0.00}_{\pm 0.00}$	$0.84_{\pm 0.03}$	$0.00_{\pm 0.00}$
1.0	$\mathbf{0.94}_{\pm 0.03}$	$\mathbf{0.04}_{\pm 0.01}$	$0.86_{\pm 0.04}$	$0.04_{\pm 0.01}$
10	$\mathbf{0.94}_{\pm 0.02}$	$\mathbf{0.05}_{\pm 0.01}$	$0.87_{\pm 0.04}$	$0.06_{\pm 0.02}$

Appendix

Table 6: Sensitivity to

\alpha

on Safe Velocity HalfCheetah. Normalized reward

R\uparrow

and cost

C\downarrow

(mean

{}_{\pm\text{std}}

) over 500 evaluation episodes and 5 seeds. Bold : method with the lowest cost for each

\alpha

(highest reward among ties).

Hyperparameter	Value
Value Function Network (VFN)	Multi-Layer Perceptron (MLP)
Activation Function	ReLU
Optimizer	Adam optimizer
Learning Rate	$3\times 10^{-4}$
Discount Factor ( $\gamma$ )	0.99
Time Step Intervals (FM Policy)( $\mu_{\zeta}$ )	5 (refer Flow Q-Learning)

Appendix

Table 7: Hyperparameters for the Algorithm.

Baseline	Network Component	Architecture Specification
C2IQL [ Liu et al., 2025 ]	Actor / Critic / Value Nets	MLP, (256, 2)
	Cost Reconstruction Model	MLP, (5 layers, 512 units each)
	Reward / Cost Advantage	MLP, (256, 2)
BCQ-Lag (Lag. dual of [ Fujimoto et al., 2019 ] )	VAE (Encoder/Decoder)	MLP, (750, 2), Latent Dim: 2 $\times$ Action Dim
	Perturbation Model $\xi$	MLP, (400, 2)
	Double Q-Critics	MLP, (400, 2)

Appendix

Table 8: Detailed Hyperparameters for Baseline Special Networks. (Layers, Units) notation refers to hidden layers and units per layer. All baselines utilize the DSRL dataset standards Liu et al. [2024] .

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 304 papers

Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies
Mar 16, 2026Mumuksh Tayal, Manan Tayal, Ravi PrakashFlow Matching Offline RL
Indian Institute of Science, India
Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning
Mar 8, 2026Janaka Chathuranga Brahmanage, Akshat KumarSafe RL Constrained RL
School of Computing and Information Systems, Singapore Management University.
Q-learning Penalized Transformer for Safe Offline Reinforcement Learning
Sep 28, 2026Shengchao Hu, Peng Wang, Jifeng Hu +5Reinforcement Learning Transformer-Based RL
Shanghai Jiao Tong University, China · Shenzhen Campus of Sun Yat-sen University, China · Jilin University, China +1

Mar 16, 2026J/K move · Enter open · S save

Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies

Mumuksh Tayal, Manan Tayal, Ravi Prakash

Indian Institute of Science, India

Flow Matching Offline RL Safe RL Flow-Based Policy Learning

Open paper AskPDF arXiv

Offline safe reinforcement learning (RL) seeks reward-maximizing policies from static datasets under strict safety constraints. Existing methods often rely on soft expected-cost objectives or iterative generative inference, which can be insufficient for safety-critical real-time control. We propose Safe Flow Q-Learning (SafeFQL), which extends FQL to safe offline RL by combining a Hamilton--Jacobi reachability-inspired safety value function with an efficient one-step flow policy. SafeFQL learns the safety value via a self-consistency Bellman recursion, trains a flow policy by behavioral cloning, and distills it into a one-step actor for reward-maximizing safe action selection without rejection sampling at deployment. Empirically, SafeFQL trades modestly higher offline training cost for substantially lower inference latency than diffusion-style safe generative baselines, which is advantageous for real-time safety-critical deployment. Across boat navigation, and Safety Gymnasium MuJoCo tasks, SafeFQL matches or exceeds prior offline safe RL performance while substantially reducing constraint violations.

Abstract

Offline RL

Explore similar work

Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies

Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning

Q-learning Penalized Transformer for Safe Offline Reinforcement Learning