cs.LGNov 7, 2026

Safe Offline Reinforcement Learning using Behavior Regularisation and Latent Feasibility Guidance

Authors: Mahesh Keswani, Parth Bhardwaj, Ankur Kumar, Raunak P. Bhattacharyya

Organizations: Yardi School of Artificial Intelligence, IIT Delhi · Dept. of Mathematics and Computing, IIT Delhi

Abstract

Safe offline reinforcement learning (RL) seeks to learn policies that maximize return while satisfying safety constraints, using a static dataset. To characterize state-wise safety, reachability analysis identifies the feasible region, defined as the set of states from which constraints can be satisfied at every future step. Policies are commonly extracted from learned value functions with advantage-weighted regression (AWR). However, AWR relies on exponentiated weights that amplify value estimation errors, whereas behavior-regularized policy gradient methods directly utilize the action gradient of the value function and often make more effective use of learned critics. We propose Feasibility-Optimized Conditional Actor Learning (FOCAL), a method that incorporates behavior-regularized policy gradients into a conditional variational autoencoder (CVAE) actor. FOCAL optimizes this policy using a first-order objective partitioned by feasibility, updating the actor to increase return on transitions estimated to be feasible and to reduce constraint violations on transitions estimated to be infeasible. During inference, FOCAL dynamically adjusts the scale of the latent prior according to the estimated feasibility of the state to adjust the conservatism of action sampling. Theoretically, we show that for AWR, the policy update variance depends exponentially on the advantage error, whereas for FOCAL it scales linearly with the variance of the value estimation error. Across the DSRL benchmark, FOCAL satisfies the constraints on all evaluated tasks while achieving competitive or higher returns compared to baselines.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Q-learning Penalized Transformer for Safe Offline Reinforcement Learning

    Sep 28, 2026Shengchao Hu, Peng Wang, Jifeng Hu +5Reinforcement LearningTransformer-Based RL

  2. Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning

    Mar 8, 2026Janaka Chathuranga Brahmanage, Akshat KumarSafe RLSafe Offline RL

  3. Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policies

    Mar 16, 2026Mumuksh Tayal, Manan Tayal, Ravi PrakashFlow MatchingSafe Offline RL