Safe Reinforcement Learning with Preference-based Constraint Inference

Chenglin Li; Grant Ruan; Hua Geng

cs.LGMar 24, 2026

Safe Reinforcement Learning with Preference-based Constraint Inference

Authors: Chenglin Li, Grant Ruan, Hua Geng

Organizations: Department of Automation, Tsinghua University, Beijing, China · Laboratory for Information & Decision Systems, Massachusetts Institute of Technology, Cambridge, MA, USA

Abstract

Safe reinforcement learning (RL) is a standard paradigm for safety-critical decision making. However, real-world safety constraints can be complex, subjective, and even hard to explicitly specify. Existing works on constraint inference rely on restrictive assumptions or extensive expert demonstrations, which are not realistic in many real-world applications. How to cheaply and reliably learn these constraints is the major challenge we focus on in this study. While inferring constraints from human preferences offers a data-efficient alternative, we identify popular Bradley-Terry (BT) models fail to capture the asymmetric, heavy-tailed nature of safety costs, resulting in risk underestimation. It is still rare in the literature to understand the impacts of BT models on the downstream policy learning. To address the above knowledge gaps, we propose a novel approach namely Preference-based Constrained Reinforcement Learning (PbCRL). We introduce a novel dead zone mechanism into preference modeling and theoretically prove that it encourages heavy-tailed cost distributions, thereby achieving better constraint alignment. Additionally, we incorporate a Signal-to-Noise Ratio (SNR) loss to encourage exploration by cost variances, which is found to benefit policy learning. Further, two-stage training strategy is deployed to lower online labeling burdens while adaptively enhancing constraint satisfaction. Empirical results demonstrate that PbCRL achieves superior alignment with true safety requirements and outperforms state-of-the-art baselines in terms of safety and reward. Our work explores a promising and effective way for constraint inference in Safe RL, with great potential in various safety-critical applications.

Figures & tables

Panel of Figure 1 — Figure 1: Cost distributions: (a) Ground truth: heavy-tailed with expectation (solid line) exceeding safety threshold (dashed line), indicating unsafe. (b) BT model: symmetric with underestimated expectation below the threshold, induces a Type II Error where the model falsely believes the constraint is satisfied. (c) Our model: similar heavy-tailed distribution, better approximate expectation of ground truth cost.

Tasks (Threshold)	Metrics	PPO-Lag (Oracle)	PbCRL (Ours)	RLSF	PPO-BT
HalfCheetah (5)	Return	$2619\pm 124$	$\bm{2367\pm 138}$	$2084\pm 126$	$2494\pm 195$
HalfCheetah (5)	Cost	$4.82\pm 0.91$	$\bm{4.66\pm 1.03}$	$3.26\pm 0.78$	$5.58\pm 2.14$
Walker2d (5)	Return	$3057\pm 142$	$2659\pm 133$	$2415\pm 120$	$\bm{2807\pm 156}$
Walker2d (5)	Cost	$4.88\pm 1.02$	$4.62\pm 0.97$	$3.66\pm 0.85$	$\bm{4.91\pm 1.11}$
Humanoid (5)	Return	$5876\pm 210$	$\bm{5247\pm 149}$	$4816\pm 180$	$5948\pm 225$
Humanoid (5)	Cost	$5.03\pm 1.93$	$\bm{4.85\pm 1.21}$	$4.05\pm 1.12$	$5.89\pm 2.05$

Table 1: Empirical results on Safety Gymnasium tasks. The best method using learned constraints is highlighted in bold .

Figure 2: Average episode return and cost of four algorithms on robotic navigation tasks. Dashed lines are safety thresholds.

Tasks	PbCRL (Ours)		RLSF		PPO-BT
Tasks	W2	Bias	W2	Bias	W2	Bias
Goal	$\bm{16.8}$	$\bm{1.5}$	$68.1$	$11.8$	$42.7$	$6.9$
Push	$\bm{20.3}$	$\bm{0.1}$	$82.4$	$14.9$	$36.5$	$6.8$
Button	$\bm{44.3}$	$\bm{2.5}$	$165.5$	$35.7$	$78.8$	$10.78$

Table 2: W2 and Bias between converged cost and threshold

Task(Threshold)	M	PPO-Lag(Oracle)	PbCRL(Ours)	RLSF	PPO-BT
Block (0.1)	R	$39.1\pm 0.1$	$\bm{36.4\pm 0.3}$	$31.8\pm 0.3$	$38.5\pm 0.6$
Block (0.1)	C	$0.05\pm 0.01$	$\bm{0.07\pm 0.01}$	$0.05\pm 0.01$	$0.12\pm 0.02$
Lane (0.1)	R	$51.6\pm 0.9$	$\bm{51.4\pm 0.5}$	$44.9\pm 0.8$	$51.5\pm 0.8$
Lane (0.1)	C	$0.01\pm 0.01$	$\bm{0.08\pm 0.01}$	$0.06\pm 0.01$	$0.14\pm 0.02$

Table 3: Results on driving scenarios, R: Return, C: Cost

Metric	PPO	RLSF	Safe RLHF	PbCRL (Ours)
Win Rate (Helpful) $\uparrow$	$72.5\%$	$60.4\%$	$79.4\%$	$\bm{80.7\%}$
Win Rate (Harmless) $\uparrow$	$60.7\%$	$54.2\%$	$76.5\%$	$\bm{82.1\%}$
Reward $\uparrow$	$2.78$	$1.31$	$4.05$	$\bm{4.22}$
Cost $\downarrow$	$-0.57$	$1.10$	$-2.97$	$\bm{-3.03}$

Table 4: Results on Language Model Alignment

Tasks	M	$\delta=0$	$\delta=0.1$	$\delta=1$	$\delta=2$
Goal (20)	W2	$38.9$	$33.5$	$\bm{16.8}$	$29.1$
	R	$24.55\pm 1.01$	$24.79\pm 1.18$	$\bm{23.41\pm 1.08}$	$22.51\pm 1.05$
	C	$28.05\pm 2.55$	$27.46\pm 1.89$	$\bm{18.50\pm 2.03}$	$16.49\pm 2.18$
Push (35)	W2	$36.1$	$35.8$	$\bm{20.3}$	$23.6$
	R	$4.15\pm 0.55$	$3.81\pm 0.21$	$\bm{3.20\pm 0.38}$	$2.96\pm 0.35$
	C	$42.50\pm 5.50$	$40.26\pm 2.15$	$\bm{34.86\pm 3.13}$	$30.91\pm 1.78$

Table 5: Ablation study on dead zone parameter

\delta

M	PbCRL	PPO-BT (varying $d$ )
M	$d=20$	$d=20$	$d=18$	$d=15$	$d=10$
R	$\bm{23.41\pm 1.08}$	$24.26\pm 0.99$	$23.86\pm 1.21$	$23.15\pm 1.05$	$18.27\pm 1.13$
C	$\bm{18.50\pm 2.03}$	$26.97\pm 2.62$	$24.08\pm 2.87$	$20.82\pm 2.05$	$14.02\pm 1.95$

Table 6: Ablation study on varying safety threshold offsets

Metrics	$\zeta=10^{-2}$	$\zeta=10^{-3}$ (Ours)	$\zeta=10^{-5}$	$\zeta=0$
Return (Mid)	$3.8\pm 2.2$	$\bm{22.9\pm 1.5}$	$22.5\pm 1.3$	$22.1\pm 1.1$
Cost (Mid)	$17.9\pm 8.1$	$\bm{18.4\pm 2.3}$	$24.5\pm 2.5$	$24.8\pm 2.8$
Return (End)	$6.5\pm 1.8$	$\bm{23.4\pm 1.1}$	$20.6\pm 1.6$	$20.7\pm 1.4$
Cost (End)	$21.9\pm 5.9$	$\bm{18.5\pm 2.0}$	$19.2\pm 2.2$	$18.4\pm 2.6$

Table 7: Ablation study on SNR loss weight

\zeta

Tasks	Metrics	PbCRL-Offline	PbCRL
Goal (20)	Return	$24.79\pm 1.75$	$\bm{23.41\pm 1.08}$
Goal (20)	Cost	$22.70\pm 0.84$	$\bm{18.50\pm 2.03}$
Push (35)	Return	$3.77\pm 0.67$	$\bm{3.20\pm 0.38}$
Push (35)	Cost	$37.48\pm 5.00$	$\bm{34.86\pm 3.13}$

Table 8: Ablation study on two-stage training strategy

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Locomotion Environments in Safety Gymnasium

Panel of Figure 4 — Figure 4: Navigation Environments in Safety Gymnasium

Panel of Figure 5 — Figure 5: Autonomous Driving Environments

Figure 6: Average episode return (Row 1), cost (Row 2) and learned cost (Row 3) of four algorithms on Goal (left), Push (middle) and Button (right) tasks. Dashed lines are safety thresholds.

Tasks (Threshold)	Metrics	PPO-Lag (Oracle)	PbCRL (Ours)	RLSF	PPO-BT
Goal (20)	Return	$23.50\pm 1.14$	$\bm{23.41\pm 1.08}$	$5.78\pm 1.07$	$24.26\pm 0.99$
	Cost	$20.70\pm 1.91$	$\bm{18.50\pm 2.03}$	$8.20\pm 2.21$	$26.97\pm 2.62$
	Learned Cost	N/A	$0.50\pm 1.26$	$89.52\pm 8.07$	$\bm{0.12\pm 1.36}$
	Accuracy	N/A	$\bm{87\%\pm 2\%}$	$83\%\pm 2\%$	$85\%\pm 1\%$
Push (35)	Return	$4.77\pm 0.67$	$\bm{3.20\pm 0.38}$	$1.07\pm 0.35$	$4.09\pm 0.57$
	Cost	$36.17\pm 5.00$	$\bm{34.86\pm 3.13}$	$20.05\pm 3.56$	$41.85\pm 5.65$

Appendix

Table 9: Empirical results on three navigation tasks in Safety Gymnasium

Tasks	PbCRL (Ours)		RLSF		PPO-BT
Tasks	W2	Bias	W2	Bias	W2	Bias
Goal	$\bm{16.8}$	$\bm{1.5}$	$68.1$	$11.8$	$42.7$	$6.9$
Push	$\bm{20.3}$	$\bm{0.1}$	$82.4$	$14.9$	$36.5$	$6.8$
Button	$\bm{44.3}$	$\bm{2.5}$	$165.5$	$35.7$	$78.8$	$10.78$

Appendix

Table 10: Evaluation metrics: W2 distance and Bias between converged cost and threshold

Panel of Figure 7 — Figure 7: Cost distribution in Goal task. Solid line: expectation; Black dashed line: safety threshold.

Task	Metric	PPO-Lag (Oracle)	PbCRL (Ours)	RLSF	PPO-BT
HalfCheetah	Reward	$2803$	$2565$	$2316$	$2687$
HalfCheetah	Cost	$20.78$	$20.25$	$14.52$	$22.50$
Walker2d	Reward	$3075$	$2790$	$2587$	$2853$
Walker2d	Cost	$19.54$	$17.84$	$13.87$	$20.43$
Humanoid	Reward	$6248$	$5761$	$5483$	$6103$
Humanoid	Cost	$19.68$	$19.20$	$17.27$	$21.84$

Appendix

Table 11: Performance Comparison with Uniform Cost Threshold (20) across Tasks

Task (Threshold)	Metrics	$\delta=0$ (No)	$\delta=0.1$ (Small)	$\delta=1$ (Medium) (Ours)	$\delta=2$ (Large)
Goal (20)	W2 distance	$38.9$	$33.5$	$\bm{16.8}$	$29.1$
	Reward	$24.55\pm 1.01$	$24.79\pm 1.18$	$\bm{23.41\pm 1.08}$	$22.51\pm 1.05$
	Cost	$28.05\pm 2.55$	$27.46\pm 1.89$	$\bm{18.50\pm 2.03}$	$16.49\pm 2.18$
	Accuracy	$84\%\pm 2\%$	$85\%\pm 2\%$	$87\%\pm 3\%$	$\bm{88\%\pm 2\%}$
Push (35)	W2 distance	$36.1$	$35.8$	$\bm{20.3}$	$23.6$
	Reward	$4.15\pm 0.55$	$3.81\pm 0.21$	$\bm{3.20\pm 0.38}$	$2.96\pm 0.35$

Appendix

Table 12: Ablation studies on dead zone parameter

\delta

Metrics	$\zeta=10^{-2}$	$\zeta=10^{-3}$ (Ours)	$\zeta=10^{-5}$	$\zeta=0$
Return (Mid)	$3.8\pm 2.2$	$\bm{22.9\pm 1.5}$	$22.5\pm 1.3$	$22.1\pm 1.1$
Cost (Mid)	$17.9\pm 8.1$	$\bm{18.4\pm 2.3}$	$24.5\pm 2.5$	$24.8\pm 2.8$
Return (End)	$6.5\pm 1.8$	$\bm{23.4\pm 1.1}$	$20.6\pm 1.6$	$20.7\pm 1.4$
Cost (End)	$21.9\pm 5.9$	$\bm{18.5\pm 2.0}$	$19.2\pm 2.2$	$18.4\pm 2.6$

Appendix

Table 13: Ablation study on SNR loss weight

\zeta

Figure 8: Ablation study on the two-stage training strategy in Goal (Row 1) and Push (Row 2) tasks.

Task	Noise Level	PbCRL (Ours)	PPO-BT
Goal	$0\%$ (Clean)	$\bm{16.8}$	$42.7$
	$5\%$	$\bm{24.5}$	$53.4$
	$10\%$	$\bm{36.8}$	$60.1$
	$30\%$	$82.8$	$\bm{79.6}$
Block	$0\%$ (Clean)	$\bm{5.9}$	$6.0$
	$5\%$	$\bm{6.9}$	$\bm{6.9}$

Appendix

Table 14: W2 distance under different preference label noise levels in robotic ( Goal ) and driving ( Block ) tasks.

Metric	PPO	RLSF	Safe RLHF	PbCRL (Ours)
Win Rate (Helpfulness) $\uparrow$	$72.5\%$	$60.4\%$	$79.4\%$	$\bm{80.7\%}$
Win Rate (Harmlessness) $\uparrow$	$60.7\%$	$54.2\%$	$76.5\%$	$\bm{82.1\%}$
Reward $\uparrow$	$2.78$	$1.31$	$4.05$	$\bm{4.22}$
Cost $\downarrow$	$-0.57$	$1.10$	$-2.97$	$\bm{-3.03}$

Appendix

Table 15: Language Model Alignment Results

Explore similar work

SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints
Dec 29, 2025Dominik Wagner, Ankit Kanwar, Luke OngSafety-Critical Control Policy Optimization
NTU Singapore · Sony Corporation
Multi-Objective Constraint Inference using Inverse reinforcement learning
May 7, 2026Syed Ihtesham Hussain Shah, Floris den Hengst, Aneta Lisowska +1Reinforcement Learning Constrained RL
Faculty of Sciences Vrije Universiteit Amsterdam
Q-learning Penalized Transformer for Safe Offline Reinforcement Learning
Sep 28, 2026Shengchao Hu, Peng Wang, Jifeng Hu +5Reinforcement Learning Transformer-Based RL
Shanghai Jiao Tong University, China · Shenzhen Campus of Sun Yat-sen University, China · Jilin University, China +1

Dec 29, 2025J/K move · Enter open · S save

SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints

Dominik Wagner, Ankit Kanwar, Luke Ong

NTU Singapore · Sony Corporation

Safety-Critical Control Policy Optimization Constrained RL Trust Region Policy Optimization Safe RL Policy Learning

Open paper AskPDF arXiv

In safety-critical domains, reinforcement learning (RL) systems must satisfy strict, zero-cost safety constraints while achieving meaningful task performance. Existing model-free methods can struggle to achieve high safety without substantially compromising task performance. We introduce Safety-Biased Trust Region Policy Optimisation (SB-TRPO), a principled approach to RL with zero-cost constraints, which requires only a fixed fraction of the maximal cost reduction achievable within the trust region, thus retaining flexibility for reward optimisation. We show that the idealised update nevertheless converges to zero cost and maximal reward amongst zero-cost policies in finite MDPs. A practical gradient-based approximation provides local improvements in both safety and reward under suitable gradient alignment. Experiments on Safety Gymnasium demonstrate high safety alongside strong task performance.

Abstract

Explore similar work

SB-TRPO: Towards Safe Reinforcement Learning with Hard Constraints

Multi-Objective Constraint Inference using Inverse reinforcement learning

Q-learning Penalized Transformer for Safe Offline Reinforcement Learning