Organizations: Dept. of Mechanical Engineering Iowa State University Ames, USA · Translational AI Center Iowa State University Ames, USA · Sibley School of Mechanical and Aerospace Engineering Cornell University Ithaca, USA
Offline safe reinforcement learning (RL) is increasingly important for cyber-physical systems (CPS), where safety violations during training are unacceptable and only pre-collected data are available. Existing offline safe RL methods typically balance reward-safety tradeoffs through constraint relaxation or joint optimization, but they often lack structural mechanisms to prevent safety drift. We propose LexiSafe, a lexicographic offline RL framework designed to preserve safety-aligned behavior. We first develop LexiSafe-SC, a single-cost formulation for standard offline safe RL, and derive safety-violation and performance-suboptimality bounds that together yield sample-complexity guarantees. We then extend the framework to hierarchical safety requirements with LexiSafe-MC, which supports multiple safety costs and admits its own sample-complexity analysis. Empirically, LexiSafe demonstrates reduced safety violations and improved task performance compared to constrained offline baselines. By unifying lexicographic prioritization with structural bias, LexiSafe offers a practical and theoretically grounded approach for safety-critical CPS decision-making.
Figures & tables
Fig. 1: LexiSafe : The agent learns from an offline dataset D∼πβ under a distributional shift constraint DKL(π∣∣πβ)≤ε . In Stage 1 (phases marked in yellow boxes), the actor network is trained to minimize cumulative costs under constraints with safety hierarchy. In Stage 2 (phase in pink), the model is retrained to maximize reward. This enforces a lexicographic policy update, preserving safety while optimizing performance. Please see Definition 3 for the formula in the Figure.
Method
Offline Setting
Complexity
PBLRL [ 19 ]
✗
✗
PSQD [ 22 ]
✗
✗
LPA [ 23 ]
✗
✗
LLRL [ 21 ]
✗
O~(ϵ2d2T4)
LexiSafe-SC
✓
Theorem 3
LexiSafe-MC
✓
Corollary 1
TABLE I: Comparison of methods with lexicographic order. d : feature dimension, T : episode length, ϵ : accuracy.
Hyperparameter
Value (Experiment range)
Batch size
2048 (32-2048)
Discount factor
0.995 (0.9-0.995)
βc and βr
1 (1-5)
ξreward and ξcost
0.7 (0.6-0.7)
Actor learning rate
3e-4 (3e-4-1e-3)
Q and value learning rate
3e-5 (1e-5-3e-5)
TABLE II: Hyperparameters
Method
BC-Safe
COptiDICE
CPQ
FISOR
LSPC-O
LexiSafe-SC
Task
reward ↑
cost ↓
reward ↑
cost ↓
reward ↑
cost ↓
reward ↑
cost ↓
reward ↑
cost ↓
reward ↑
cost ↓
Safety Gym:
SwimmerVel
0.51
1.07
0.63
7.58
0.13
2.66
-0.04
0.00
0.44
0.14
0.51
0.92
HopperVel
0.36
0.67
0.13
1.51
0.14
2.11
0.17
0.32
0.69
0.00
0.70
0.50
HalfCheetahVel
0.88
0.54
0.65
0.00
0.29
0.74
0.89
0.00
0.97
0.10
0.97
0.69
Walker2dVel
0.79
0.04
0.12
0.74
0.04
0.21
0.38
0.36
0.76
0.02
0.78
0.40
TABLE III: Comparison of our methods with baselines across benchmark tasks. Bold indicates safety (cost < 1), and blue denotes both safety and high performance. Polices that are safe and is within 0.05 range of the highest reward are also marked blue.
Fig. 2: Ablation study showing LexiSafe’s adherence to sequential lexicographical optimization. For both hierarchy orders, LexiSafe proceeds through the intended phases: first minimizing the primary cost, then the secondary, and finally improving reward while maintaining satisfied constraints.
Fig. 3: Comparison of LexiSafe-MC and weighted IQL across different crash-weight values with MetaDrive. LexiSafe-MC satisfies safety constraints while maintaining high reward by explicitly enforcing the user-specified priority order through sequential lexicographic optimization. In contrast, weighted IQL struggles to satisfy constraints reliably using the traditional weighted-sum strategy, highlighting both the practical tuning challenges and the limitations of flat weighting approaches. The brackets in the legend represents ( wcrash , wspeed )
Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.
Sequential decision making using Markov Decision Process underpins many realworld applications. Both model-based and model free methods have achieved strong results in these settings. However, real-world tasks must balance reward maximization with safety constraints, often conflicting objectives, that can lead to unstable min/max, adversarial optimization. A promising alternative is safety reachability analysis, which precomputes a forward-invariant safe state, action set, ensuring that an agent starting inside this set remains safe indefinitely. Yet, most reachability based methods address only hard safety constraints, and little work extends reachability to cumulative cost constraints. To address this, first, we define a safetyconditioned reachability set that decouples reward maximization from cumulative safety cost constraints. Second, we show how this set enforces safety constraints without unstable min/max or Lagrangian optimization, yielding a novel offline safe RL algorithm that learns a safe policy from a fixed dataset without environment interaction. Finally, experiments on standard offline safe RL benchmarks, and a real world maritime navigation task demonstrate that our method matches or outperforms state of the art baselines while maintaining safety.
Janaka Chathuranga Brahmanage, Akshat Kumar
School of Computing and Information Systems, Singapore Management University.
Offline safe reinforcement learning (Safe RL) enables policy learning without online interactions, making it suitable for safety-critical systems such as robotics systems. However, its reliance on static datasets exposes offline Safe RL to data poisoning attacks, where adversaries inject malicious samples that compromise safety and induce unsafe policy behavior. In this work, we propose a new learning paradigm, named safe reinforcement unlearning (Safe-RULE), used as a defense framework to remove the influence of poisoned data without retraining from scratch or requiring access to the original training environment. We further extend reinforcement unlearning to offline Safe RL by explicitly accounting for both task performance and safety constraints during the unlearning process. Experiments across benchmark Safe RL tasks demonstrate that our approach effectively enhances safety performance against data poisoning attacks.
Shixiong Jiang, Taozheng Zhu, Fanxin Kong
Department of Computer Science and Engineering, University of Notre Dame
Offline reinforcement learning (RL) agents often fail when deployed, as the gap between training datasets and real environments leads to unsafe behavior. To address this, we present SAS (Self-Alignment for Safety), a transformer-based framework that enables test-time adaptation in offline safe RL without retraining. In SAS, the main mechanism is self-alignment: at test time, the pretrained agent generates several imagined trajectories and selects those satisfying the Lyapunov condition. These feasible segments are then recycled as in-context prompts, allowing the agent to realign its behavior toward safety while avoiding parameter updates. In effect, SAS turns Lyapunov-guided imagination into control-invariant prompts, and its transformer architecture admits a hierarchical RL interpretation where prompting functions as Bayesian inference over latent skills. Across Safety Gymnasium and MuJoCo benchmarks, SAS consistently reduces cost and failure while maintaining or improving return.