Safe Reinforcement Learning with Preference-based Constraint Inference

Chenglin Li; Grant Ruan; Hua Geng

cs.LGMar 24, 2026

Safe Reinforcement Learning with Preference-based Constraint Inference

Authors: Chenglin Li, Grant Ruan, Hua Geng

Organizations: Department of Automation, Tsinghua University, Beijing, China · Laboratory for Information & Decision Systems, Massachusetts Institute of Technology, Cambridge, MA, USA

Abstract

Safe reinforcement learning (RL) is a standard paradigm for safety-critical decision making. However, real-world safety constraints can be complex, subjective, and even hard to explicitly specify. Existing works on constraint inference rely on restrictive assumptions or extensive expert demonstrations, which are not realistic in many real-world applications. How to cheaply and reliably learn these constraints is the major challenge we focus on in this study. While inferring constraints from human preferences offers a data-efficient alternative, we identify popular Bradley-Terry (BT) models fail to capture the asymmetric, heavy-tailed nature of safety costs, resulting in risk underestimation. It is still rare in the literature to understand the impacts of BT models on the downstream policy learning. To address the above knowledge gaps, we propose a novel approach namely Preference-based Constrained Reinforcement Learning (PbCRL). We introduce a novel dead zone mechanism into preference modeling and theoretically prove that it encourages heavy-tailed cost distributions, thereby achieving better constraint alignment. Additionally, we incorporate a Signal-to-Noise Ratio (SNR) loss to encourage exploration by cost variances, which is found to benefit policy learning. Further, two-stage training strategy is deployed to lower online labeling burdens while adaptively enhancing constraint satisfaction. Empirical results demonstrate that PbCRL achieves superior alignment with true safety requirements and outperforms state-of-the-art baselines in terms of safety and reward. Our work explores a promising and effective way for constraint inference in Safe RL, with great potential in various safety-critical applications.

Figures & tables

Panel of Figure 1 — Figure 1: Cost distributions: (a) Ground truth: heavy-tailed with expectation (solid line) exceeding safety threshold (dashed line), indicating unsafe. (b) BT model: symmetric with underestimated expectation below the threshold, induces a Type II Error where the model falsely believes the constraint is satisfied. (c) Our model: similar heavy-tailed distribution, better approximate expectation of ground truth cost.

Tasks (Threshold)	Metrics	PPO-Lag (Oracle)	PbCRL (Ours)	RLSF	PPO-BT
HalfCheetah (5)	Return	$2619\pm 124$	$\bm{2367\pm 138}$	$2084\pm 126$	$2494\pm 195$
HalfCheetah (5)	Cost	$4.82\pm 0.91$	$\bm{4.66\pm 1.03}$	$3.26\pm 0.78$	$5.58\pm 2.14$
Walker2d (5)	Return	$3057\pm 142$	$2659\pm 133$	$2415\pm 120$	$\bm{2807\pm 156}$
Walker2d (5)	Cost	$4.88\pm 1.02$	$4.62\pm 0.97$	$3.66\pm 0.85$	$\bm{4.91\pm 1.11}$
Humanoid (5)	Return	$5876\pm 210$	$\bm{5247\pm 149}$	$4816\pm 180$	$5948\pm 225$
Humanoid (5)	Cost	$5.03\pm 1.93$	$\bm{4.85\pm 1.21}$	$4.05\pm 1.12$	$5.89\pm 2.05$

Table 1: Empirical results on Safety Gymnasium tasks. The best method using learned constraints is highlighted in bold .

Figure 2: Average episode return and cost of four algorithms on robotic navigation tasks. Dashed lines are safety thresholds.

Tasks	PbCRL (Ours)		RLSF		PPO-BT
Tasks	W2	Bias	W2	Bias	W2	Bias
Goal	$\bm{16.8}$	$\bm{1.5}$	$68.1$	$11.8$	$42.7$	$6.9$
Push	$\bm{20.3}$	$\bm{0.1}$	$82.4$	$14.9$	$36.5$	$6.8$
Button	$\bm{44.3}$	$\bm{2.5}$	$165.5$	$35.7$	$78.8$	$10.78$

Table 2: W2 and Bias between converged cost and threshold

Task(Threshold)	M	PPO-Lag(Oracle)	PbCRL(Ours)	RLSF	PPO-BT
Block (0.1)	R	$39.1\pm 0.1$	$\bm{36.4\pm 0.3}$	$31.8\pm 0.3$	$38.5\pm 0.6$
Block (0.1)	C	$0.05\pm 0.01$	$\bm{0.07\pm 0.01}$	$0.05\pm 0.01$	$0.12\pm 0.02$
Lane (0.1)	R	$51.6\pm 0.9$	$\bm{51.4\pm 0.5}$	$44.9\pm 0.8$	$51.5\pm 0.8$
Lane (0.1)	C	$0.01\pm 0.01$	$\bm{0.08\pm 0.01}$	$0.06\pm 0.01$	$0.14\pm 0.02$

Table 3: Results on driving scenarios, R: Return, C: Cost

Metric	PPO	RLSF	Safe RLHF	PbCRL (Ours)
Win Rate (Helpful) $\uparrow$	$72.5\%$	$60.4\%$	$79.4\%$	$\bm{80.7\%}$
Win Rate (Harmless) $\uparrow$	$60.7\%$	$54.2\%$	$76.5\%$	$\bm{82.1\%}$
Reward $\uparrow$	$2.78$	$1.31$	$4.05$	$\bm{4.22}$
Cost $\downarrow$	$-0.57$	$1.10$	$-2.97$	$\bm{-3.03}$

Table 4: Results on Language Model Alignment

Tasks	M	$\delta=0$	$\delta=0.1$	$\delta=1$	$\delta=2$
Goal (20)	W2	$38.9$	$33.5$	$\bm{16.8}$	$29.1$
	R	$24.55\pm 1.01$	$24.79\pm 1.18$	$\bm{23.41\pm 1.08}$	$22.51\pm 1.05$
	C	$28.05\pm 2.55$	$27.46\pm 1.89$	$\bm{18.50\pm 2.03}$	$16.49\pm 2.18$
Push (35)	W2	$36.1$	$35.8$	$\bm{20.3}$	$23.6$
	R	$4.15\pm 0.55$	$3.81\pm 0.21$	$\bm{3.20\pm 0.38}$	$2.96\pm 0.35$
	C	$42.50\pm 5.50$	$40.26\pm 2.15$	$\bm{34.86\pm 3.13}$	$30.91\pm 1.78$

Table 5: Ablation study on dead zone parameter

\delta

M	PbCRL	PPO-BT (varying $d$ )
M	$d=20$	$d=20$	$d=18$	$d=15$	$d=10$
R	$\bm{23.41\pm 1.08}$	$24.26\pm 0.99$	$23.86\pm 1.21$	$23.15\pm 1.05$	$18.27\pm 1.13$
C	$\bm{18.50\pm 2.03}$	$26.97\pm 2.62$	$24.08\pm 2.87$	$20.82\pm 2.05$	$14.02\pm 1.95$

Table 6: Ablation study on varying safety threshold offsets

Metrics	$\zeta=10^{-2}$	$\zeta=10^{-3}$ (Ours)	$\zeta=10^{-5}$	$\zeta=0$
Return (Mid)	$3.8\pm 2.2$	$\bm{22.9\pm 1.5}$	$22.5\pm 1.3$	$22.1\pm 1.1$
Cost (Mid)	$17.9\pm 8.1$	$\bm{18.4\pm 2.3}$	$24.5\pm 2.5$	$24.8\pm 2.8$
Return (End)	$6.5\pm 1.8$	$\bm{23.4\pm 1.1}$	$20.6\pm 1.6$	$20.7\pm 1.4$
Cost (End)	$21.9\pm 5.9$	$\bm{18.5\pm 2.0}$	$19.2\pm 2.2$	$18.4\pm 2.6$

Table 7: Ablation study on SNR loss weight

\zeta

Tasks	Metrics	PbCRL-Offline	PbCRL
Goal (20)	Return	$24.79\pm 1.75$	$\bm{23.41\pm 1.08}$
Goal (20)	Cost	$22.70\pm 0.84$	$\bm{18.50\pm 2.03}$
Push (35)	Return	$3.77\pm 0.67$	$\bm{3.20\pm 0.38}$
Push (35)	Cost	$37.48\pm 5.00$	$\bm{34.86\pm 3.13}$

Table 8: Ablation study on two-stage training strategy

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Locomotion Environments in Safety Gymnasium

Panel of Figure 4 — Figure 4: Navigation Environments in Safety Gymnasium

Panel of Figure 5 — Figure 5: Autonomous Driving Environments

Figure 6: Average episode return (Row 1), cost (Row 2) and learned cost (Row 3) of four algorithms on Goal (left), Push (middle) and Button (right) tasks. Dashed lines are safety thresholds.

Tasks (Threshold)	Metrics	PPO-Lag (Oracle)	PbCRL (Ours)	RLSF	PPO-BT
Goal (20)	Return	$23.50\pm 1.14$	$\bm{23.41\pm 1.08}$	$5.78\pm 1.07$	$24.26\pm 0.99$
	Cost	$20.70\pm 1.91$	$\bm{18.50\pm 2.03}$	$8.20\pm 2.21$	$26.97\pm 2.62$
	Learned Cost	N/A	$0.50\pm 1.26$	$89.52\pm 8.07$	$\bm{0.12\pm 1.36}$
	Accuracy	N/A	$\bm{87\%\pm 2\%}$	$83\%\pm 2\%$	$85\%\pm 1\%$
Push (35)	Return	$4.77\pm 0.67$	$\bm{3.20\pm 0.38}$	$1.07\pm 0.35$	$4.09\pm 0.57$
	Cost	$36.17\pm 5.00$	$\bm{34.86\pm 3.13}$	$20.05\pm 3.56$	$41.85\pm 5.65$

Appendix

Table 9: Empirical results on three navigation tasks in Safety Gymnasium

Tasks	PbCRL (Ours)		RLSF		PPO-BT
Tasks	W2	Bias	W2	Bias	W2	Bias
Goal	$\bm{16.8}$	$\bm{1.5}$	$68.1$	$11.8$	$42.7$	$6.9$
Push	$\bm{20.3}$	$\bm{0.1}$	$82.4$	$14.9$	$36.5$	$6.8$
Button	$\bm{44.3}$	$\bm{2.5}$	$165.5$	$35.7$	$78.8$	$10.78$

Appendix

Table 10: Evaluation metrics: W2 distance and Bias between converged cost and threshold

Panel of Figure 7 — Figure 7: Cost distribution in Goal task. Solid line: expectation; Black dashed line: safety threshold.

Task	Metric	PPO-Lag (Oracle)	PbCRL (Ours)	RLSF	PPO-BT
HalfCheetah	Reward	$2803$	$2565$	$2316$	$2687$
HalfCheetah	Cost	$20.78$	$20.25$	$14.52$	$22.50$
Walker2d	Reward	$3075$	$2790$	$2587$	$2853$
Walker2d	Cost	$19.54$	$17.84$	$13.87$	$20.43$
Humanoid	Reward	$6248$	$5761$	$5483$	$6103$
Humanoid	Cost	$19.68$	$19.20$	$17.27$	$21.84$

Appendix

Table 11: Performance Comparison with Uniform Cost Threshold (20) across Tasks

Task (Threshold)	Metrics	$\delta=0$ (No)	$\delta=0.1$ (Small)	$\delta=1$ (Medium) (Ours)	$\delta=2$ (Large)
Goal (20)	W2 distance	$38.9$	$33.5$	$\bm{16.8}$	$29.1$
	Reward	$24.55\pm 1.01$	$24.79\pm 1.18$	$\bm{23.41\pm 1.08}$	$22.51\pm 1.05$
	Cost	$28.05\pm 2.55$	$27.46\pm 1.89$	$\bm{18.50\pm 2.03}$	$16.49\pm 2.18$
	Accuracy	$84\%\pm 2\%$	$85\%\pm 2\%$	$87\%\pm 3\%$	$\bm{88\%\pm 2\%}$
Push (35)	W2 distance	$36.1$	$35.8$	$\bm{20.3}$	$23.6$
	Reward	$4.15\pm 0.55$	$3.81\pm 0.21$	$\bm{3.20\pm 0.38}$	$2.96\pm 0.35$

Appendix

Table 12: Ablation studies on dead zone parameter

\delta

Metrics	$\zeta=10^{-2}$	$\zeta=10^{-3}$ (Ours)	$\zeta=10^{-5}$	$\zeta=0$
Return (Mid)	$3.8\pm 2.2$	$\bm{22.9\pm 1.5}$	$22.5\pm 1.3$	$22.1\pm 1.1$
Cost (Mid)	$17.9\pm 8.1$	$\bm{18.4\pm 2.3}$	$24.5\pm 2.5$	$24.8\pm 2.8$
Return (End)	$6.5\pm 1.8$	$\bm{23.4\pm 1.1}$	$20.6\pm 1.6$	$20.7\pm 1.4$
Cost (End)	$21.9\pm 5.9$	$\bm{18.5\pm 2.0}$	$19.2\pm 2.2$	$18.4\pm 2.6$

Appendix

Table 13: Ablation study on SNR loss weight

\zeta

Figure 8: Ablation study on the two-stage training strategy in Goal (Row 1) and Push (Row 2) tasks.

Task	Noise Level	PbCRL (Ours)	PPO-BT
Goal	$0\%$ (Clean)	$\bm{16.8}$	$42.7$
	$5\%$	$\bm{24.5}$	$53.4$
	$10\%$	$\bm{36.8}$	$60.1$
	$30\%$	$82.8$	$\bm{79.6}$
Block	$0\%$ (Clean)	$\bm{5.9}$	$6.0$
	$5\%$	$\bm{6.9}$	$\bm{6.9}$

Appendix

Table 14: W2 distance under different preference label noise levels in robotic ( Goal ) and driving ( Block ) tasks.

Metric	PPO	RLSF	Safe RLHF	PbCRL (Ours)
Win Rate (Helpfulness) $\uparrow$	$72.5\%$	$60.4\%$	$79.4\%$	$\bm{80.7\%}$
Win Rate (Harmlessness) $\uparrow$	$60.7\%$	$54.2\%$	$76.5\%$	$\bm{82.1\%}$
Reward $\uparrow$	$2.78$	$1.31$	$4.05$	$\bm{4.22}$
Cost $\downarrow$	$-0.57$	$1.10$	$-2.97$	$\bm{-3.03}$

Appendix

Table 15: Language Model Alignment Results

Explore similar work

Dec 29, 2025

SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints

In safety-critical domains, reinforcement learning (RL) systems must satisfy strict, zero-cost safety constraints while achieving meaningful task performance. Existing model-free methods can struggle to achieve high safety without substantially compromising task performance. We introduce Safety-Biased Trust Region Policy Optimisation (SB-TRPO), a principled approach to RL with zero-cost constraints, which requires only a fixed fraction of the maximal cost reduction achievable within the trust region, thus retaining flexibility for reward optimisation. We show that the idealised update nevertheless converges to zero cost and maximal reward amongst zero-cost policies in finite MDPs. A practical gradient-based approximation provides local improvements in both safety and reward under suitable gradient alignment. Experiments on Safety Gymnasium demonstrate high safety alongside strong task performance.

Dominik Wagner, Ankit Kanwar, Luke Ong

NTU Singapore · Sony Corporation

Safety-Critical Control Policy Optimization Constrained RL Trust Region Policy Optimization Safe RL Policy Learning

Ask PDF

May 7, 2026

Multi-Objective Constraint Inference using Inverse reinforcement learning

Constraint inference is widely considered essential to align reinforcement learning agents with safety boundaries and operational guidelines by observing expert demonstrations. However, existing approaches typically assume homogeneous demonstrations (i.e., generated by a single expert or multiple experts with identical objectives). They also have limited ability to capture individual preferences and often suffer from computational inefficiencies. In this paper, we introduce Multi-Objective Constraint Inference (MOCI), a novel framework designed to jointly extract shared constraints and individual preferences from heterogeneous expert trajectories, where multiple experts pursue different objectives. MOCI effectively models and learns from diverse, and potentially conflicting, behaviors. Empirical evaluations demonstrate that MOCI significantly outperforms existing baselines, achieving improved predictive performance, and maintaining competitive computational efficiency on a standard grid-world benchmark. These results establish MOCI as an accurate, flexible, and computationally practical approach for real-world constraint inference and preference learning tasks.

Syed Ihtesham Hussain Shah, Floris den Hengst, Aneta Lisowska +1

Faculty of Sciences Vrije Universiteit Amsterdam

Reinforcement Learning Constrained RL Preference-Based RL

Ask PDF

Sep 28, 2026

Q-learning Penalized Transformer for Safe Offline Reinforcement Learning

This paper addresses the problem of safe offline reinforcement learning, which involves training a policy to satisfy safety constraints using an offline dataset. This problem is inherently challenging as it requires balancing three highly interconnected and competing objectives: satisfying safety constraints, maximizing rewards, and adhering to the behavior regularization imposed by the offline dataset. To tackle this trilogy challenge, we propose Q-learning Penalized Transformer policy (QPT), a training--inference consistent framework that bridges conditional sequence modeling with constraint-aware value estimation. QPT trains a Transformer policy that generates actions conditioned on trajectory context and target return/cost, retaining strong behavior regularization. To inject explicit safety semantics during learning, we augment sequence-model training with a Q-shaped penalty using learned reward and cost Q-functions to favor high return under low constraint violation. At inference, the same Q-functions enforce the cost threshold and choose the highest-reward feasible action, closing the loop between training and deployment. We provide a principled analysis under stylized near-deterministic CMDPs, characterizing how Q-penalized conditional generation improve safety and performance. Empirically, QPT consistently outperforms strong safe offline RL baselines across 38 tasks on the DSRL benchmark, and exhibits robust zero-shot adaptation to different constraint thresholds.

Shengchao Hu, Peng Wang, Jifeng Hu +5

Shanghai Jiao Tong University, China · Shenzhen Campus of Sun Yat-sen University, China · Jilin University, China +1

Reinforcement Learning Transformer-Based RL Constrained RL Offline RL

Ask PDF