Learning to maintain safety through expert demonstrations in settings with unknown constraints: A Q-learning perspective

George Papadopoulos; George A. Vouros

cs.LGFeb 27, 2026

Learning to maintain safety through expert demonstrations in settings with unknown constraints: A Q-learning perspective

Authors: George Papadopoulos, George A. Vouros

Organizations: University of Piraeus, Piraeus, Greece

Abstract

Given a set of trajectories demonstrating the execution of a task safely in a constrained MDP with observable rewards but with unknown constraints and non-observable costs, we aim to find a policy that maximizes the likelihood of demonstrated trajectories trading the balance between being conservative and increasing significantly the likelihood of high-rewarding trajectories but with potentially unsafe steps. Having these objectives, we aim towards learning a policy that maximizes the probability of the most $promising$ trajectories with respect to the demonstrations. In so doing, we formulate the ``promise" of individual state-action pairs in terms of $Q$ values, which depend on task-specific rewards as well as on the assessment of states' safety, mixing expectations in terms of rewards and safety. This entails a safe Q-learning perspective of the inverse learning problem under constraints: The devised Safe $Q$ Inverse Constrained Reinforcement Learning (SafeQIL) algorithm is compared to state-of-the art inverse constraint reinforcement learning algorithms to a set of challenging benchmark tasks, showing its merits.

Figures & tables

Figure 1. Snapshots of: (top-left) SafetyCarPush2 - v0, (top-right) SafetyPointCircle2-v0, (bottom-left) SafetyPointGoal1-v0, (bottom-right) SafetyCarButton1-v0.

Algorithm	Reward $\uparrow$	Cost $\downarrow$
SafeQIL	5.27 $\pm$ 1.85	34.22 $\pm$ 2.71
ICRL 2	23.21 $\pm$ 1.28	62.60 $\pm$ 9.87
ICRL 7	-0.50 $\pm$ 4.93	42.80 $\pm$ 35.43
VICRL	21.87 $\pm$ 1.20	62.97 $\pm$ 10.81
SAC-GAIL	7.17 $\pm$ 1.65	44.80 $\pm$ 18.60
SAC	27.47 $\pm$ 0.21	49.15 $\pm$ 2.21

Table 1. SafetyPointGoal1-v0: Comparative results.

Panel of Figure 2 — Figure 2. Learning curves for SafetyPointGoal1-v0.

Algorithm	Reward $\uparrow$	Cost $\downarrow$
SafeQIL	27.06 $\pm$ 4.10	29.28 $\pm$ 8.41
ICRL 2	28.36 $\pm$ 17.37	177.88 $\pm$ 6.70
ICRL 7	8.21 $\pm$ 17.07	34.07 $\pm$ 16.74
VICRL	26.29 $\pm$ 0.69	5.49 $\pm$ 0.81
SAC-GAIL	11.89 $\pm$ 9.55	33.95 $\pm$ 28.02
SAC	58.81 $\pm$ 0.70	392.20 $\pm$ 4.38

Table 2. SafetyPointCircle2-v0: Comparative results.

Algorithm	Reward $\uparrow$	Cost $\downarrow$
SafeQIL	-3.81 $\pm$ 3.05	70.11 $\pm$ 72.96
ICRL 2	0.31 $\pm$ 2.23	222.01 $\pm$ 173.61
ICRL 7	-15.83 $\pm$ 12.98	145.94 $\pm$ 23.54
VICRL	-14.12 $\pm$ 10.83	34.65 $\pm$ 23.36
SAC-GAIL	0.14 $\pm$ 4.72	98.15 $\pm$ 81.54
SAC	19.49 $\pm$ 3.50	299.27 $\pm$ 20.61

Table 3. SafetyCarButton1-v0: Comparative results.

Algorithm	Reward $\uparrow$	Cost $\downarrow$
SafeQIL	-0.62 $\pm$ 0.65	57.20 $\pm$ 30.02
ICRL 2	0.00 $\pm$ 1.73	203.10 $\pm$ 37.24
ICRL 7	1.15 $\pm$ 0.49	178.81 $\pm$ 60.69
VICRL	-9.05 $\pm$ 8.58	55.80 $\pm$ 71.45
SAC-GAIL	0.76 $\pm$ 0.92	87.97 $\pm$ 10.57
SAC	1.50 $\pm$ 0.01	245.36 $\pm$ 28.52

Table 4. SafetyCarPush2-v0: Comparative results.

Algorithm	Reward $\uparrow$	Cost $\downarrow$
Original	5.27 $\pm$ 1.85	34.22 $\pm$ 2.71
w/o cosine similarity	3.74 $\pm$ 2.52	30.25 $\pm$ 19.10
w/o constraint term	4.34 $\pm$ 4.41	29.02 $\pm$ 14.03

Table 5. Ablation results for key SafeQIL’s variations.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Algorithm	Mean Reward	Mean Cost	Cost Drop ( $\Delta C$ )	Reward Drop ( $\Delta R$ )	Ratio ( $\rho$ )
SafetyPointGoal1-v0 (SAC: R 27.47, C 49.15)	SafeQIL	5.27	34.22	30.4%	80.8%	0.38
	ICRL 2	23.21	62.60	-27.4%	15.5%	N/A (Unsafe)
	ICRL 7	-0.50	42.80	12.9%	101.8%	0.13
	VICRL	21.87	62.97	-28.1%	20.4%	N/A (Unsafe)
	SAC-GAIL	7.17	44.80	8.9%	73.9%	0.12
SafetyPointCircle2-v0 (SAC: R 58.81, C 392.20)	SafeQIL	27.06	29.28	92.5%	54.0%	1.71

Appendix

Table 6. Standard Trade-off Analysis (Mean Values Only). The Ratio

\rho

indicates % Cost Drop per 1% Reward Drop.

Benchmark	Algorithm	Pess. Reward ( $\tilde{R}$ )	Pess. Cost ( $\tilde{C}$ )	Robust $\Delta\tilde{C}$	Robust $\Delta\tilde{R}$	Ratio ( $\rho$ )
SafetyPointGoal1-v0 (SAC Pess: R 27.26, C 51.36)	SafeQIL	3.42	36.93	28.1%	87.5%	0.32
	ICRL 2	21.93	72.47	-41.1%	19.6%	N/A (Unsafe)
	ICRL 7	-5.43	78.23	-52.3%	119.9%	N/A (Unsafe)
	VICRL	20.67	73.78	-43.6%	24.2%	N/A (Unsafe)
	SAC-GAIL	5.52	63.40	-23.4%	79.7%	N/A (Unsafe)
SafetyPointCircle2-v0 (SAC Pess: R 58.11, C 396.58)	SafeQIL	22.96	37.69	90.5%	60.5%	1.50

Appendix

Table 7. Robust Trade-off Analysis (Mean

\pm

Std). Pessimistic metrics penalize high variance in cost and reward.

Figure 3. Learning curves for the SafetyPointCircle2-v0 benchmark.

Figure 4. Learning curves for the SafetyCarButton1-v0 benchmark.

Figure 5. Learning curves for the SafetyCarPush2-v0 benchmark.

Algorithm	Reward $\uparrow$	Cost $\downarrow$
Original	5.27 $\pm$ 1.85	34.22 $\pm$ 2.71
w/o cosine similarity	3.74 $\pm$ 2.52	30.25 $\pm$ 19.10
w/o max term	-10.62 $\pm$ 7.81	15.47 $\pm$ 8.67
w/o constraint term	4.34 $\pm$ 4.41	29.02 $\pm$ 14.03
w/o OOD term	1.07 $\pm$ 0.91	44.99 $\pm$ 4.95
w/o demonstrations term	-3.71 $\pm$ 1.00	8.41 $\pm$ 3.76

Appendix

Table 8. Complete ablation study evaluation results for SafeQIL.

1x Dataset
Algorithm	Reward $\uparrow$	Cost $\downarrow$
SafeQIL	5.27 $\pm$ 1.85	34.22 $\pm$ 2.71
ICRL 2	23.21 $\pm$ 1.28	62.60 $\pm$ 9.87
ICRL 7	-0.50 $\pm$ 4.93	42.80 $\pm$ 35.43
VICRL	21.87 $\pm$ 1.20	62.97 $\pm$ 10.81
SAC-GAIL	7.17 $\pm$ 1.65	44.80 $\pm$ 18.60

Appendix

Table 9. SafetyPointGoal1-v0: Comparative results of Dataset Size Ablation.

1x Dataset
Algorithm	Reward $\uparrow$	Cost $\downarrow$
SafeQIL	-0.62 $\pm$ 0.65	57.20 $\pm$ 30.02
ICRL 2	0.00 $\pm$ 1.73	203.10 $\pm$ 37.24
ICRL 7	1.15 $\pm$ 0.49	178.81 $\pm$ 60.69
VICRL	-9.05 $\pm$ 8.58	55.80 $\pm$ 71.45
SAC-GAIL	0.76 $\pm$ 0.92	87.97 $\pm$ 10.57

Appendix

Table 10. SafetyCarPush2-v0: Comparative results of Dataset Size Ablation.

Dataset Size	Algorithm	Mean Reward	Mean Cost	Cost Drop ( $\Delta C$ )	Reward Drop ( $\Delta R$ )	Ratio ( $\rho$ )
1x Dataset (SAC: R 27.47, C 49.15)	SafeQIL	5.27	34.22	30.4%	80.8%	0.38
	ICRL 2	23.21	62.60	-27.4%	15.5%	N/A (Unsafe)
	ICRL 7	-0.50	42.80	12.9%	101.8%	0.13
	VICRL	21.87	62.97	-28.1%	20.4%	N/A (Unsafe)
	SAC-GAIL	7.17	44.80	8.9%	73.9%	0.12
2x Dataset (SAC: R 27.47, C 49.15)	SafeQIL	3.17	29.50	40.0%	88.5%	0.45

Appendix

Table 11. SafetyPointGoal1-v0: Standard Trade-off Analysis (Dataset Size Ablation). Calculated using Mean Values Only. The Ratio

\rho

indicates % Cost Drop per 1% Reward Drop.

Dataset Size	Algorithm	Pess. Reward ( $\tilde{R}$ )	Pess. Cost ( $\tilde{C}$ )	Robust $\Delta\tilde{C}$	Robust $\Delta\tilde{R}$	Ratio ( $\rho$ )
1x Dataset (SAC Pess: R 27.26, C 51.36)	SafeQIL	3.42	36.93	28.1%	87.5%	0.32
	ICRL 2	21.93	72.47	-41.1%	19.6%	N/A (Unsafe)
	ICRL 7	-5.43	78.23	-52.3%	119.9%	N/A (Unsafe)
	VICRL	20.67	73.78	-43.6%	24.2%	N/A (Unsafe)
	SAC-GAIL	5.52	63.40	-23.4%	79.7%	N/A (Unsafe)
2x Dataset (SAC Pess: R 27.26, C 51.36)	SafeQIL	2.22	39.45	23.2%	91.9%	0.25

Appendix

Table 12. SafetyPointGoal1-v0: Robust Trade-off Analysis (Dataset Size Ablation). Calculated using worst-case bounds (Mean

\pm

Std). Pessimistic metrics penalize high variance in cost and reward.

Dataset Size	Algorithm	Mean Reward	Mean Cost	Cost Drop ( $\Delta C$ )	Reward Drop ( $\Delta R$ )	Ratio ( $\rho$ )
1x Dataset (SAC: R 1.50, C 245.36)	SafeQIL	-0.62	57.20	76.7%	141.3%	0.54
	ICRL 2	0.00	203.10	17.2%	100.0%	0.17
	ICRL 7	1.15	178.81	27.1%	23.3%	1.16
	VICRL	-9.05	55.80	77.3%	703.3%	0.11
	SAC-GAIL	0.76	87.97	64.1%	49.3%	1.30
2x Dataset (SAC: R 1.50, C 245.36)	SafeQIL	-3.38	58.10	76.3%	325.3%	0.23

Appendix

Table 13. SafetyCarPush2-v0: Standard Trade-off Analysis (Dataset Size Ablation). Calculated using Mean Values Only. The Ratio

\rho

indicates % Cost Drop per 1% Reward Drop.

Dataset Size	Algorithm	Pess. Reward ( $\tilde{R}$ )	Pess. Cost ( $\tilde{C}$ )	Robust $\Delta\tilde{C}$	Robust $\Delta\tilde{R}$	Ratio ( $\rho$ )
1x Dataset (SAC Pess: R 1.49, C 273.88)	SafeQIL	-1.27	87.22	68.2%	185.2%	0.37
	ICRL 2	-1.73	240.34	12.2%	216.1%	0.06
	ICRL 7	0.66	239.50	12.6%	55.7%	0.23
	VICRL	-17.63	127.25	53.5%	1283.2%	0.04
	SAC-GAIL	-0.16	98.54	64.0%	110.7%	0.58
2x Dataset (SAC Pess: R 1.49, C 273.88)	SafeQIL	-6.35	75.96	72.3%	526.2%	0.14

Appendix

Table 14. SafetyCarPush2-v0: Robust Trade-off Analysis (Dataset Size Ablation). Calculated using worst-case bounds (Mean

\pm

Std). Pessimistic metrics penalize high variance in cost and reward.

Parameter	Value
SAC (Actor / Critics)
Actor/Critics Learning Rate	$3\times 10^{-4}$
Actor/Critics Optimizer	Adam
Discount Factor ( $\gamma$ )	$0.99$
Batch Size	$256$
Target Networks Smoothing Coefficient ( $\eta$ )	$0.005$

Appendix

Table 15. Hyperparameter settings for SafeQIL.

Parameter	Value
General PPO (Forward Step)
Policy/Critics Learning Rate	$3\times 10^{-4}$
Policy/Critics Optimizer	Adam
Reward GAE $\gamma$	$0.99$
Reward GAE $\lambda$	$0.95$
Reward Value Function Coefficient	$0.5$

Appendix

Table 16. Hyperparameter setting 1 for ICRL.

Parameter	Value
General PPO (Forward Step)
Policy/Critics Learning Rate	$3\times 10^{-4}$
Policy/Critics Optimizer	Adam
Reward GAE $\gamma$	$0.99$
Reward GAE $\lambda$	$0.95$
Reward Value Function Coefficient	$0.5$

Appendix

Table 17. Hyperparameter setting 2 for ICRL.

Parameter	Value
General PPO (Forward Step)
Policy/Critics Learning Rate	$3\times 10^{-5}$
Policy/Critics Optimizer	Adam
Reward GAE $\gamma$	$0.99$
Reward GAE $\lambda$	$0.9$
Reward Value Function Coefficient	$0.5$

Appendix

Table 18. Hyperparameter setting 3 for ICRL.

Parameter	Value
General PPO (Forward Step)
Policy/Critics Learning Rate	$3\times 10^{-5}$
Policy/Critics Optimizer	Adam
Reward GAE $\gamma$	$0.99$
Reward GAE $\lambda$	$0.95$
Reward Value Function Coefficient	$0.5$

Appendix

Table 19. Hyperparameter setting 1 for VICRL.

Parameter	Value
General PPO (Forward Step)
Policy/Critics Learning Rate	$3\times 10^{-5}$
Policy/Critics Optimizer	Adam
Reward GAE $\gamma$	$0.99$
Reward GAE $\lambda$	$0.95$
Reward Value Function Coefficient	$0.5$

Appendix

Table 20. Hyperparameter setting 2 for VICRL.

General PPO (Forward Step)
Parameter	Value
Policy/Critics Learning Rate	$3\times 10^{-5}$
Policy/Critics Optimizer	Adam
Reward GAE $\gamma$	$0.99$
Reward GAE $\lambda$	$0.9$
Reward Value Function Coefficient	$0.5$

Appendix

Table 21. Hyperparameter setting 3 for VICRL.

Parameter	Value
General PPO (Forward Step)
Policy/Critics Learning Rate	$3\times 10^{-5}$
Policy/Critics Optimizer	Adam
Reward GAE $\gamma$	$0.99$
Reward GAE $\lambda$	$0.9$
Reward Value Function Coefficient	$0.5$

Appendix

Table 22. Hyperparameter setting 4 for VICRL.

Parameter	Value
General PPO (Forward Step)
Policy/Critics Learning Rate	$3\times 10^{-5}$
Policy/Critics Optimizer	Adam
Reward GAE $\gamma$	$0.99$
Reward GAE $\lambda$	$0.9$
Reward Value Function Coefficient	$0.5$

Appendix

Table 23. Hyperparameter setting 5 for VICRL.

Parameter	Value
General PPO (Forward Step)
Policy/Critics Learning Rate	$3\times 10^{-5}$
Policy/Critics Optimizer	Adam
Reward GAE $\gamma$	$0.99$
Reward GAE $\lambda$	$0.9$
Reward Value Function Coefficient	$0.5$

Appendix

Table 24. Hyperparameter setting 6 for VICRL.

Parameter	Value
General PPO (Forward Step)
Policy/Critics Learning Rate	$1\times 10^{-4}$
Policy/Critics Optimizer	Adam
Reward GAE $\gamma$	$0.99$
Reward GAE $\lambda$	$0.95$
Reward Value Function Coefficient	$0.5$

Appendix

Table 25. Hyperparameter setting 7 for VICRL.

Parameter	Value
General PPO (Forward Step)
Policy/Critics Learning Rate	$1\times 10^{-4}$
Policy/Critics Optimizer	Adam
Reward GAE $\gamma$	$0.99$
Reward GAE $\lambda$	$0.95$
Reward Value Function Coefficient	$0.5$

Appendix

Table 26. Hyperparameter setting 8 for VICRL.

Parameter	Value
General PPO (Forward Step)
Policy/Critics Learning Rate	$1\times 10^{-4}$
Policy/Critics Optimizer	Adam
Reward GAE $\gamma$	$0.99$
Reward GAE $\lambda$	$0.95$
Reward Value Function Coefficient	$0.5$

Appendix

Table 27. Hyperparameter setting 9 for VICRL.

Parameter	Value
SAC (Actor / Critics)
Actor/Critics Learning Rate	$3\times 10^{-4}$
Actor/Critics Optimizer	Adam
Discount Factor ( $\gamma$ )	$0.99$
Batch Size	$256$
Target Networks Smoothing Coefficient ( $\eta$ )	$0.005$

Appendix

Table 28. Hyperparameter settings for SAC-GAIL.

Safe RL

Safe reinforcement learning studies how agents can learn and act while respecting safety constraints and limiting harmful outcomes.

See all 215 papers

Q-learning Penalized Transformer for Safe Offline Reinforcement Learning
Sep 28, 2026Shengchao Hu, Peng Wang, Jifeng Hu +5Reinforcement Learning Transformer-Based RL
Shanghai Jiao Tong University, China · Shenzhen Campus of Sun Yat-sen University, China · Jilin University, China +1
Safe-Support Q-Learning: Learning without Unsafe Exploration
Apr 28, 2026Yeeun Lim, Narim Jeong, Donghwan LeeReinforcement Learning KL-Regularized RL
HD Korea Shipbuilding & Offshore Engineering · Department of Electrical Engineering, KAIST, Daejeon, South Korea
Off-Policy Safe Reinforcement Learning with Constrained Optimistic Exploration
Mar 25, 2026Guopeng Li, Matthijs T. J. Spaan, Julian F. P. KooijSafe RL Constrained RL
Faculty of Electrical Engineering, Mathematics and Computer Science, Delft University of Technology, Delft, the Netherlands

Sep 28, 2026J/K move · Enter open · S save

Q-learning Penalized Transformer for Safe Offline Reinforcement Learning

Shengchao Hu, Peng Wang, Jifeng Hu, Qiyang Zhou+4

Shanghai Jiao Tong University, China · Shenzhen Campus of Sun Yat-sen University, China · Jilin University, China+1

Reinforcement Learning Transformer-Based RL Constrained RL Offline RL

Open paper AskPDF arXiv

This paper addresses the problem of safe offline reinforcement learning, which involves training a policy to satisfy safety constraints using an offline dataset. This problem is inherently challenging as it requires balancing three highly interconnected and competing objectives: satisfying safety constraints, maximizing rewards, and adhering to the behavior regularization imposed by the offline dataset. To tackle this trilogy challenge, we propose Q-learning Penalized Transformer policy (QPT), a training--inference consistent framework that bridges conditional sequence modeling with constraint-aware value estimation. QPT trains a Transformer policy that generates actions conditioned on trajectory context and target return/cost, retaining strong behavior regularization. To inject explicit safety semantics during learning, we augment sequence-model training with a Q-shaped penalty using learned reward and cost Q-functions to favor high return under low constraint violation. At inference, the same Q-functions enforce the cost threshold and choose the highest-reward feasible action, closing the loop between training and deployment. We provide a principled analysis under stylized near-deterministic CMDPs, characterizing how Q-penalized conditional generation improve safety and performance. Empirically, QPT consistently outperforms strong safe offline RL baselines across 38 tasks on the DSRL benchmark, and exhibits robust zero-shot adaptation to different constraint thresholds.

Abstract

Safe RL

Explore similar work

Q-learning Penalized Transformer for Safe Offline Reinforcement Learning

Safe-Support Q-Learning: Learning without Unsafe Exploration

Off-Policy Safe Reinforcement Learning with Constrained Optimistic Exploration