cs.LGFeb 8, 2026

EpiFlow: Bridging Safety and Performance in Offline Reinforcement Learning via Epigraph-Guided Flow Matching

Authors: Manan Tayal, Mumuksh Tayal, Ravi Prakash

Organizations: TAU Intelligence

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Offline reinforcement learning (RL) offers a promising framework for deploying autonomous systems in safety-critical settings without the risks of online exploration. However, learning policies that simultaneously achieve high performance and strong safety guarantees from fixed datasets remains a fundamental challenge. Many existing safe offline RL approaches typically rely on soft constraint formulations, which may permit safety violations and are sensitive to distributional shift. In contrast, formal methods such as Hamilton--Jacobi (HJ) reachability and Control Barrier Functions (CBFs) provide rigorous safety guarantees, but often yield overly conservative solutions, often neglecting performance. In this work, we bridge this gap by formulating safe offline RL as a state-constrained optimal control problem, where safety is enforced through hard state constraints and performance is captured via a reward function. The resulting value function satisfies a Hamilton--Jacobi--Bellman (HJB) equation, which we approximate using offline RL on fixed datasets by learning an epigraph feasibility value function and extracting policies via flow matching. This formulation enables principled integration of safety guarantees with data-driven policy optimization. Empirically, across safety-critical benchmarks including boat navigation, Safety-Gymnasium, and Bullet-Safety-Gym tasks, our approach achieves competitive returns while exhibiting near-zero constraint violations, demonstrating a favorable balance between safety and performance in the offline setting.

Figures & tables

Figure 1: Safety–performance trade-off in offline RL (conceptual illustration): soft-constraint methods violate constraints, while classical safety methods stay safe but follow conservative, lower-performing paths; our method achieves safety with high performance.
Figure 2: Framework Overview. We propose a framework to solve the SC-OCP problem by using offline dataset. We learn V^\hat{V} , which if satisfies to stay positive, makes sure that the system stays safe while maximizing the reward. We then learn a policy which optimizes for V^\hat{V} at all time steps to find the most optimal path that the offline data supports.
Figure 3: Evaluation Results: Results plot for all the environments. Points towards left ( ←\leftarrow ) are more safe than those on right ( →\rightarrow ). Whereas those towards top ( ↑\uparrow ) have higher rewards than those towards bottom ( ↓\downarrow ). Evaluated over 500 episodes and 5 seed values ( ±\pm standard deviation).
Figure 4: Effect of Regularizer parameter ( λ\lambda ) ( ±\pm standard deviation).
Figure 5: Sensitivity to Expectile Parameter ( τ\tau ) ( ±\pm standard deviation).
Figure 6: Computation Time Analysis. (Left) shows training time comparison while (Right) shows inference time ( ±\pm standard deviations) comparison of Our framework against all baselines.
Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Feasibility epigraph value function V^θ(x,z)\hat{V}_{\theta}(x,z) . ( Top ) Learned without the decomposition regularizer, where V^θ\hat{V}_{\theta} exhibits weak sensitivity to the epigraph variable zz and similar contour structure across different zz values. ( Bottom ) Learned with the regularizer ( λ=0.25\lambda=0.25 ), which restores meaningful dependence on zz and yields distinct level sets consistent with the epigraph formulation.
Figure 8: Illustration of all evaluation environments. ( Top-Left ) environment illustrates Boat Navigation with obstacles and a goal point. ( Middle ) illustrates SafeVelocity MuJoCo environments with Red sphere denoting unsafe state. ( Bottom ) depicts the SafeCarNavigation MuJoCo environments with various obstacles and objectives.
 BoatHalfCheetahHopperAntWalker2D
MethodR↑R\uparrowC↓C\downarrowR↑R\uparrowC↓C\downarrowR↑R\uparrowC↓C\downarrowR↑R\uparrowC↓C\downarrowR↑R\uparrowC↓C\downarrow
BCQ-Lag0.65±0.030.65_{\pm 0.03}1.89±0.801.89_{\pm 0.80}1.06±0.051.06_{\pm 0.05}58.50±1.0058.50_{\pm 1.00}0.72±0.020.72_{\pm 0.02}19.80±1.0019.80_{\pm 1.00}1.00±0.031.00_{\pm 0.03}7.40±0.607.40_{\pm 0.60}0.78±0.030.78_{\pm 0.03}3.50±0.303.50_{\pm 0.30}
BEAR-Lag0.78±0.040.78_{\pm 0.04}1.66±0.651.66_{\pm 0.65}0.97±0.060.97_{\pm 0.06}15.60±0.9015.60_{\pm 0.90}0.36±0.020.36_{\pm 0.02}17.30±0.9017.30_{\pm 0.90}−1.01±0.01-1.01_{\pm 0.01}0.00±0.000.00_{\pm 0.00}0.78±0.030.78_{\pm 0.03}1.70±0.201.70_{\pm 0.20}
CPQ0.75±0.040.75_{\pm 0.04}7.41±0.707.41_{\pm 0.70}0.39±0.050.39_{\pm 0.05}7.00±0.607.00_{\pm 0.60}0.13±0.010.13_{\pm 0.01}10.00±0.4010.00_{\pm 0.40}−1.01±0.01-1.01_{\pm 0.01}0.00±0.000.00_{\pm 0.00}0.09±0.020.09_{\pm 0.02}0.35±0.210.35_{\pm 0.21}
COptiDICE0.51±0.030.51_{\pm 0.03}1.72±0.451.72_{\pm 0.45}0.61±0.070.61_{\pm 0.07}0.00±0.000.00_{\pm 0.00}0.16±0.030.16_{\pm 0.03}7.90±0.357.90_{\pm 0.35}1.00±0.031.00_{\pm 0.03}13.30±0.7013.30_{\pm 0.70}0.14±0.010.14_{\pm 0.01}3.80±0.253.80_{\pm 0.25}
C2IQL0.66±0.030.66_{\pm 0.03}2.36±0.502.36_{\pm 0.50}1.02±0.031.02_{\pm 0.03}25.20±0.8025.20_{\pm 0.80}0.16±0.010.16_{\pm 0.01}3.20±0.253.20_{\pm 0.25}1.00±0.031.00_{\pm 0.03}9.60±0.609.60_{\pm 0.60}0.21±0.020.21_{\pm 0.02}3.10±0.303.10_{\pm 0.30}
Appendix
Table 1: Results on Boat Navigation and Safety Gymnasium. Normalized reward R↑R\uparrow and normalized cost C↓C\downarrow , reported as mean ±std{}_{\pm\text{std}} over 500 evaluation episodes and 5 seeds. Gray : unsafe under the DSRL convention ( C≥1C\geq 1 ). Bold : method with the lowest cost on each task (highest reward among ties).
 Ant-RunBall-RunCar-RunDrone-Run
MethodR↑R\uparrowC↓C\downarrowR↑R\uparrowC↓C\downarrowR↑R\uparrowC↓C\downarrowR↑R\uparrowC↓C\downarrow
BCQ-Lag0.63±0.070.63_{\pm 0.07}4.65±0.854.65_{\pm 0.85}0.31±0.050.31_{\pm 0.05}2.28±0.552.28_{\pm 0.55}0.97±0.040.97_{\pm 0.04}0.10±0.300.10_{\pm 0.30}0.64±0.080.64_{\pm 0.08}10.32±1.5510.32_{\pm 1.55}
BEAR-Lag0.07±0.030.07_{\pm 0.03}0.00±0.000.00_{\pm 0.00}−0.36±0.06-0.36_{\pm 0.06}9.10±1.409.10_{\pm 1.40}0.66±0.050.66_{\pm 0.05}0.03±0.050.03_{\pm 0.05}0.15±0.040.15_{\pm 0.04}4.31±0.804.31_{\pm 0.80}
CPQ−0.02±0.02-0.02_{\pm 0.02}0.00±0.000.00_{\pm 0.00}0.27±0.050.27_{\pm 0.05}3.70±0.753.70_{\pm 0.75}1.01±0.061.01_{\pm 0.06}9.40±1.359.40_{\pm 1.35}0.49±0.060.49_{\pm 0.06}1.69±0.421.69_{\pm 0.42}
COptiDICE0.51±0.010.51_{\pm 0.01}0.03±0.010.03_{\pm 0.01}0.42±0.080.42_{\pm 0.08}0.15±0.030.15_{\pm 0.03}0.81±0.020.81_{\pm 0.02}0.00±0.000.00_{\pm 0.00}0.57±0.010.57_{\pm 0.01}0.17±0.000.17_{\pm 0.00}
C2IQL0.67±0.010.67_{\pm 0.01}0.11±0.010.11_{\pm 0.01}0.53±0.050.53_{\pm 0.05}0.11±0.010.11_{\pm 0.01}0.85±0.010.85_{\pm 0.01}0.01±0.010.01_{\pm 0.01}0.64±0.020.64_{\pm 0.02}0.09±0.020.09_{\pm 0.02}
Appendix
Table 2: Results on Bullet-Safety-Gym (DSRL). Same metrics as Table 1 ; 500 evaluation episodes over 5 seeds. Gray : unsafe under the DSRL convention ( C≥1C\geq 1 ). Bold : method with the lowest cost on each task (highest reward among ties).
Expectile Parameter (τ)(\boldsymbol{\tau})0.50.70.80.9
 (Behaviour Policy)   
Mean Episode Reward-485.86 ±\pm 0.74-492.36 ±\pm 0.21-496.01 ±\pm 0.58-496.10 ±\pm 0.37
Safety Rate (%)88.9 ±\pm 0.2398.9 ±\pm 1.0199.9 ±\pm 0.18100.0 ±\pm 0.0
Mean Episode Cost1.95 ±\pm 0.190.09 ±\pm 0.030.04 ±\pm 0.050.0 ±\pm 0.0
Appendix
Table 3: Mean Reward, Safety Rate (%), and Mean Cost reported for Boat Navigation by varying τ\tau in the expectile regression. ( ±\pm ) represents standard deviation.
Figure 9: Results for Control under Perturbations. The plots show average reward (%) ( ±\pm standard deviation) achieved relative to no perturbations case, at different levels of perturbations (5%, 10% and 20%), and their average absolute cost values ( ±\pm standard deviation) respectively.
Figure 10: Effect of the regularizer parameter λ\lambda on the inferred epigraph budget. For the Boat Navigation environment, each panel shows the state-wise optimal budget z∗(x)z^{\ast}(x) defined by the feasibility condition V^θ(x,z∗)≥0\hat{V}_{\theta}(x,z^{\ast})\geq 0 . As λ\lambda increases, the feasible region contracts and z∗(x)z^{\ast}(x) becomes more conservative; states shaded in purple correspond to infeasible regions.
MetricFlow MatchingDiffusion ModelWeighted BC
R-496.23 ±\pm 0.34-584.53 ±\pm 2.37-716.60 ±\pm 5.23
C0.0 ±\pm 0.00.0 ±\pm 0.018.56 ±\pm 4.45
Appendix
Table 4: Mean Episode Reward (R) and Mean Episode Cost (C) ( ±\pm standard deviation) reported for proposed framework across different policy backbones. Bold : method with the lowest cost (highest reward among ties).
 OursFISOR-FMFISOR
TaskR↑R\uparrowC↓C\downarrowR↑R\uparrowC↓C\downarrowR↑R\uparrowC↓C\downarrow
Boat0.82±0.01\mathbf{0.82}_{\pm 0.01}0.00±0.00\mathbf{0.00}_{\pm 0.00}0.74±0.020.74_{\pm 0.02}0.13±0.060.13_{\pm 0.06}0.73±0.020.73_{\pm 0.02}0.86±0.300.86_{\pm 0.30}
HalfCheetah0.93±0.03\mathbf{0.93}_{\pm 0.03}0.00±0.00\mathbf{0.00}_{\pm 0.00}0.84±0.030.84_{\pm 0.03}0.00±0.000.00_{\pm 0.00}0.77±0.030.77_{\pm 0.03}0.00±0.000.00_{\pm 0.00}
Hopper0.15±0.00\mathbf{0.15}_{\pm 0.00}0.05±0.03\mathbf{0.05}_{\pm 0.03}0.06±0.000.06_{\pm 0.00}0.14±0.100.14_{\pm 0.10}0.02±0.000.02_{\pm 0.00}0.16±0.100.16_{\pm 0.10}
Ant0.81±0.03\mathbf{0.81}_{\pm 0.03}0.00±0.00\mathbf{0.00}_{\pm 0.00}0.72±0.020.72_{\pm 0.02}0.00±0.000.00_{\pm 0.00}0.65±0.020.65_{\pm 0.02}0.00±0.000.00_{\pm 0.00}
Walker2D0.10±0.00\mathbf{0.10}_{\pm 0.00}0.10±0.10\mathbf{0.10}_{\pm 0.10}0.10±0.000.10_{\pm 0.00}0.11±0.100.11_{\pm 0.10}0.09±0.000.09_{\pm 0.00}0.25±0.100.25_{\pm 0.10}
Appendix
Table 5: Ablation isolating the epigraph value formulation. Normalized reward R↑R\uparrow and cost C↓C\downarrow (mean ±std{}_{\pm\text{std}} ) over 500 evaluation episodes and 5 seeds. Bold : method with the lowest cost on each task (highest reward among ties).
 OursFISOR-FM
α\alphaR↑R\uparrowC↓C\downarrowR↑R\uparrowC↓C\downarrow
0.010.92±0.02\mathbf{0.92}_{\pm 0.02}0.00±0.00\mathbf{0.00}_{\pm 0.00}0.80±0.020.80_{\pm 0.02}0.00±0.000.00_{\pm 0.00}
0.10.93±0.03\mathbf{0.93}_{\pm 0.03}0.00±0.00\mathbf{0.00}_{\pm 0.00}0.84±0.030.84_{\pm 0.03}0.00±0.000.00_{\pm 0.00}
1.00.94±0.03\mathbf{0.94}_{\pm 0.03}0.04±0.01\mathbf{0.04}_{\pm 0.01}0.86±0.040.86_{\pm 0.04}0.04±0.010.04_{\pm 0.01}
100.94±0.02\mathbf{0.94}_{\pm 0.02}0.05±0.01\mathbf{0.05}_{\pm 0.01}0.87±0.040.87_{\pm 0.04}0.06±0.020.06_{\pm 0.02}
Appendix
Table 6: Sensitivity to α\alpha on Safe Velocity HalfCheetah. Normalized reward R↑R\uparrow and cost C↓C\downarrow (mean ±std{}_{\pm\text{std}} ) over 500 evaluation episodes and 5 seeds. Bold : method with the lowest cost for each α\alpha (highest reward among ties).
HyperparameterValue
Value Function Network (VFN)Multi-Layer Perceptron (MLP)
Activation FunctionReLU
OptimizerAdam optimizer
Learning Rate3×10−43\times 10^{-4}
Discount Factor ( γ\gamma )0.99
Time Step Intervals (FM Policy)( μζ\mu_{\zeta} )5 (refer Flow Q-Learning)
Appendix
Table 7: Hyperparameters for the Algorithm.
BaselineNetwork ComponentArchitecture Specification
C2IQL [ Liu et al., 2025 ]Actor / Critic / Value NetsMLP, (256, 2)
 Cost Reconstruction ModelMLP, (5 layers, 512 units each)
 Reward / Cost AdvantageMLP, (256, 2)
BCQ-Lag (Lag. dual of [ Fujimoto et al., 2019 ] )VAE (Encoder/Decoder)MLP, (750, 2), Latent Dim: 2 ×\times Action Dim
 Perturbation Model ξ\xiMLP, (400, 2)
 Double Q-CriticsMLP, (400, 2)
Appendix
Table 8: Detailed Hyperparameters for Baseline Special Networks. (Layers, Units) notation refers to hidden layers and units per layer. All baselines utilize the DSRL dataset standards Liu et al. [2024] .
More in this topic

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 304 papers

Explore similar work

CardsList
Mar 16, 2026

Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies

Offline safe reinforcement learning (RL) seeks reward-maximizing policies from static datasets under strict safety constraints. Existing methods often rely on soft expected-cost objectives or iterative generative inference, which can be insufficient for safety-critical real-time control. We propose Safe Flow Q-Learning (SafeFQL), which extends FQL to safe offline RL by combining a Hamilton--Jacobi reachability-inspired safety value function with an efficient one-step flow policy. SafeFQL learns the safety value via a self-consistency Bellman recursion, trains a flow policy by behavioral cloning, and distills it into a one-step actor for reward-maximizing safe action selection without rejection sampling at deployment. Empirically, SafeFQL trades modestly higher offline training cost for substantially lower inference latency than diffusion-style safe generative baselines, which is advantageous for real-time safety-critical deployment. Across boat navigation, and Safety Gymnasium MuJoCo tasks, SafeFQL matches or exceeds prior offline safe RL performance while substantially reducing constraint violations.

Mumuksh Tayal, Manan Tayal, Ravi Prakash

Indian Institute of Science, India

Flow MatchingOffline RLSafe RLFlow-Based Policy Learning
AskPDF
Mar 8, 2026

Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning

Sequential decision making using Markov Decision Process underpins many realworld applications. Both model-based and model free methods have achieved strong results in these settings. However, real-world tasks must balance reward maximization with safety constraints, often conflicting objectives, that can lead to unstable min/max, adversarial optimization. A promising alternative is safety reachability analysis, which precomputes a forward-invariant safe state, action set, ensuring that an agent starting inside this set remains safe indefinitely. Yet, most reachability based methods address only hard safety constraints, and little work extends reachability to cumulative cost constraints. To address this, first, we define a safetyconditioned reachability set that decouples reward maximization from cumulative safety cost constraints. Second, we show how this set enforces safety constraints without unstable min/max or Lagrangian optimization, yielding a novel offline safe RL algorithm that learns a safe policy from a fixed dataset without environment interaction. Finally, experiments on standard offline safe RL benchmarks, and a real world maritime navigation task demonstrate that our method matches or outperforms state of the art baselines while maintaining safety.

Janaka Chathuranga Brahmanage, Akshat Kumar

School of Computing and Information Systems, Singapore Management University.

Safe RLConstrained RLOffline RL
AskPDF
Sep 28, 2026

Q-learning Penalized Transformer for Safe Offline Reinforcement Learning

This paper addresses the problem of safe offline reinforcement learning, which involves training a policy to satisfy safety constraints using an offline dataset. This problem is inherently challenging as it requires balancing three highly interconnected and competing objectives: satisfying safety constraints, maximizing rewards, and adhering to the behavior regularization imposed by the offline dataset. To tackle this trilogy challenge, we propose Q-learning Penalized Transformer policy (QPT), a training--inference consistent framework that bridges conditional sequence modeling with constraint-aware value estimation. QPT trains a Transformer policy that generates actions conditioned on trajectory context and target return/cost, retaining strong behavior regularization. To inject explicit safety semantics during learning, we augment sequence-model training with a Q-shaped penalty using learned reward and cost Q-functions to favor high return under low constraint violation. At inference, the same Q-functions enforce the cost threshold and choose the highest-reward feasible action, closing the loop between training and deployment. We provide a principled analysis under stylized near-deterministic CMDPs, characterizing how Q-penalized conditional generation improve safety and performance. Empirically, QPT consistently outperforms strong safe offline RL baselines across 38 tasks on the DSRL benchmark, and exhibits robust zero-shot adaptation to different constraint thresholds.

Shengchao Hu, Peng Wang, Jifeng Hu +5

Shanghai Jiao Tong University, China · Shenzhen Campus of Sun Yat-sen University, China · Jilin University, China +1

Reinforcement LearningTransformer-Based RLConstrained RLOffline RL
AskPDF