LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchy

Hsin-Jung Yang; Zhanhong Jiang; Prajwal Koirala; Qisai Liu; Cody Fleming; Soumik Sarkar

cs.LGFeb 19, 2026

LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchy

Authors: Hsin-Jung Yang, Zhanhong Jiang, Prajwal Koirala, Qisai Liu, Cody Fleming, Soumik Sarkar

Organizations: Dept. of Mechanical Engineering Iowa State University Ames, USA · Translational AI Center Iowa State University Ames, USA · Sibley School of Mechanical and Aerospace Engineering Cornell University Ithaca, USA

Abstract

Offline safe reinforcement learning (RL) is increasingly important for cyber-physical systems (CPS), where safety violations during training are unacceptable and only pre-collected data are available. Existing offline safe RL methods typically balance reward-safety tradeoffs through constraint relaxation or joint optimization, but they often lack structural mechanisms to prevent safety drift. We propose LexiSafe, a lexicographic offline RL framework designed to preserve safety-aligned behavior. We first develop LexiSafe-SC, a single-cost formulation for standard offline safe RL, and derive safety-violation and performance-suboptimality bounds that together yield sample-complexity guarantees. We then extend the framework to hierarchical safety requirements with LexiSafe-MC, which supports multiple safety costs and admits its own sample-complexity analysis. Empirically, LexiSafe demonstrates reduced safety violations and improved task performance compared to constrained offline baselines. By unifying lexicographic prioritization with structural bias, LexiSafe offers a practical and theoretically grounded approach for safety-critical CPS decision-making.

Figures & tables

Fig. 1: LexiSafe : The agent learns from an offline dataset $\mathcal{D}\sim\pi_{\beta}$ under a distributional shift constraint $D_{KL}(\pi||\pi_{\beta})\leq\varepsilon$ . In Stage 1 (phases marked in yellow boxes), the actor network is trained to minimize cumulative costs under constraints with safety hierarchy. In Stage 2 (phase in pink), the model is retrained to maximize reward. This enforces a lexicographic policy update, preserving safety while optimizing performance. Please see Definition 3 for the formula in the Figure.

Method	Offline Setting	Complexity
PBLRL [ 19 ]	✗	✗
PSQD [ 22 ]	✗	✗
LPA [ 23 ]	✗	✗
LLRL [ 21 ]	✗	$\tilde{\mathcal{O}}(\frac{d^{2}T^{4}}{\epsilon^{2}})$
LexiSafe-SC	✓	Theorem 3
LexiSafe-MC	✓	Corollary 1

TABLE I: Comparison of methods with lexicographic order.

d

: feature dimension,

T

: episode length,

\epsilon

: accuracy.

Hyperparameter	Value (Experiment range)
Batch size	2048 (32-2048)
Discount factor	0.995 (0.9-0.995)
$\beta_{c}$ and $\beta_{r}$	1 (1-5)
$\xi_{reward}$ and $\xi_{cost}$	0.7 (0.6-0.7)
Actor learning rate	3e-4 (3e-4-1e-3)
Q and value learning rate	3e-5 (1e-5-3e-5)

TABLE II: Hyperparameters

Method	BC-Safe		COptiDICE		CPQ		FISOR		LSPC-O		LexiSafe-SC
Task	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$	reward $\uparrow$	cost $\downarrow$
Safety Gym:
SwimmerVel	0.51	1.07	0.63	7.58	0.13	2.66	-0.04	0.00	0.44	0.14	0.51	0.92
HopperVel	0.36	0.67	0.13	1.51	0.14	2.11	0.17	0.32	0.69	0.00	0.70	0.50
HalfCheetahVel	0.88	0.54	0.65	0.00	0.29	0.74	0.89	0.00	0.97	0.10	0.97	0.69
Walker2dVel	0.79	0.04	0.12	0.74	0.04	0.21	0.38	0.36	0.76	0.02	0.78	0.40

TABLE III: Comparison of our methods with baselines across benchmark tasks. Bold indicates safety (cost

<

1), and blue denotes both safety and high performance. Polices that are safe and is within 0.05 range of the highest reward are also marked blue.

Fig. 2: Ablation study showing LexiSafe’s adherence to sequential lexicographical optimization. For both hierarchy orders, LexiSafe proceeds through the intended phases: first minimizing the primary cost, then the secondary, and finally improving reward while maintaining satisfied constraints.

Fig. 3: Comparison of LexiSafe-MC and weighted IQL across different crash-weight values with MetaDrive. LexiSafe-MC satisfies safety constraints while maintaining high reward by explicitly enforcing the user-specified priority order through sequential lexicographic optimization. In contrast, weighted IQL struggles to satisfy constraints reliably using the traditional weighted-sum strategy, highlighting both the practical tuning challenges and the limitations of flat weighting approaches. The brackets in the legend represents ( $w_{crash}$ , $w_{speed}$ )

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,717 papers

Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning
Mar 8, 2026Janaka Chathuranga Brahmanage, Akshat KumarSafe RL Constrained RL
School of Computing and Information Systems, Singapore Management University.
Safe-RULE: Safe Reinforcement UnLEarning
Jun 8, 2026Shixiong Jiang, Taozheng Zhu, Fanxin KongOffline RL Safe RL
Department of Computer Science and Engineering, University of Notre Dame
Lyapunov-Guided Self-Alignment: Test-Time Adaptation for Offline Safe Reinforcement Learning
Apr 29, 2026Seungyub Han, Hyungjin Kim, Jungwoo LeeReinforcement Learning Offline RL
Seoul National University

Mar 8, 2026J/K move · Enter open · S save

Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning

Janaka Chathuranga Brahmanage, Akshat Kumar

School of Computing and Information Systems, Singapore Management University.

Safe RL Constrained RL Offline RL

Open paper AskPDF arXiv

Sequential decision making using Markov Decision Process underpins many realworld applications. Both model-based and model free methods have achieved strong results in these settings. However, real-world tasks must balance reward maximization with safety constraints, often conflicting objectives, that can lead to unstable min/max, adversarial optimization. A promising alternative is safety reachability analysis, which precomputes a forward-invariant safe state, action set, ensuring that an agent starting inside this set remains safe indefinitely. Yet, most reachability based methods address only hard safety constraints, and little work extends reachability to cumulative cost constraints. To address this, first, we define a safetyconditioned reachability set that decouples reward maximization from cumulative safety cost constraints. Second, we show how this set enforces safety constraints without unstable min/max or Lagrangian optimization, yielding a novel offline safe RL algorithm that learns a safe policy from a fixed dataset without environment interaction. Finally, experiments on standard offline safe RL benchmarks, and a real world maritime navigation task demonstrate that our method matches or outperforms state of the art baselines while maintaining safety.

Abstract

Reinforcement Learning

Explore similar work

Beyond Hard Constraints: Budget-Conditioned Reachability For Safe Offline Reinforcement Learning

Safe-RULE: Safe Reinforcement UnLEarning

Lyapunov-Guided Self-Alignment: Test-Time Adaptation for Offline Safe Reinforcement Learning