Offline Safe Reinforcement Learning

Offline safe reinforcement learning (RL) focuses on training robots and AI agents to make optimal decisions while adhering to safety constraints, using only pre-collected data. Current research emphasizes developing algorithms that address the challenges of limited and potentially imperfect data, focusing on techniques like pessimistic value iteration, conditional distribution shaping, and risk-sensitive approaches within model architectures such as Decision Transformers and diffusion models. This field is crucial for deploying RL agents in real-world settings where online trial-and-error is too risky, with applications ranging from robotics and autonomous driving to healthcare.

Papers

July 19, 2024

OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning
Yihang Yao, Zhepeng Cen, Wenhao Ding, Haohong Lin, Shiqi Liu, Tingnan Zhang, Wenhao Yu, Ding Zhao
Safe Reinforcement Learning Offline Safe Reinforcement Learning Feature Shaping

July 10, 2024

Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning
Dake Zhang, Boxiang Lyu, Shuang Qiu, Mladen Kolar, Tong Zhang
High Quality Risk Description Value Iteration Risk Sensitive Reinforcement Learning Offline Safe Reinforcement Learning

March 4, 2024

Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policy
Chenyang Cao, Zichen Yan, Renhao Lu, Junbo Tan, Xueqian Wang
Reinforcement Learning Recovery Guarantee FETCH Robot Offline Safe Reinforcement Learning Offline Goal Conditioned Reinforcement Learning Safety Critical Task

February 27, 2024

Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learning
Zijian Guo, Weichao Zhou, Wenchao Li
Temporal Logic Safe Reinforcement Learning Decision Transformer Signal Temporal Logic Offline Safe Reinforcement Learning

February 13, 2024

Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning
Eslam Eldeeb, Houssem Sifaou, Osvaldo Simeone, Mohammad Shehab, Hirley Alves
Multi Agent Reinforcement Learning Multi Agent System Digital Twin Quantile Regression Distributional Reinforcement Learning Multi Agent Reinforcement Conservative Q Learning Offline Safe Reinforcement Learning

February 8, 2024

Offline Risk-sensitive RL with Partial Observability to Enhance Performance in Human-Robot Teaming
Giorgio Angelotti, Caroline P. C. Chanel, Adam H. M. Pinto, Christophe Lounis, Corentin Chauffaut, Nicolas Drougard
System Performance Physiological Signal Observable Markov Decision Process Partial Observability Human Robot Teaming Human Operator Offline Safe Reinforcement Learning Human State Estimation

January 19, 2024

Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model
Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang, Shengbo Eben Li, Xianyuan Zhan, Jingjing Liu
Diffusion Model Safety Constraint Safe Policy Offline Policy Learning Offline Safe Reinforcement Learning

January 1, 2024

Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning
Honghao Wei, Xiyue Peng, Arnob Ghosh, Xin Liu
Offline Reinforcement Learning Robust Deep Reinforcement Learning Offline Safe Reinforcement Learning

October 31, 2023

Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Driving
Haohong Lin, Wenhao Ding, Zuxin Liu, Yaru Niu, Jiacheng Zhu, Yuming Niu, Ding Zhao
Autonomous Driving Autonomous Vehicle Offline Reinforcement Learning Causal Representation Critical Scenario Generation Offline Safe Reinforcement Learning

September 18, 2023

Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration
Jinning Li, Xinyi Liu, Banghua Zhu, Jiantao Jiao, Masayoshi Tomizuka, Chen Tang, Wei Zhan
Safe Reinforcement Learning Offline Policy Learning Offline Safe Reinforcement Learning Safe Reinforcement Learning Algorithm

June 15, 2023

Datasets and Benchmarks for Offline Safe Reinforcement Learning
Zuxin Liu, Zijian Guo, Haohong Lin, Yihang Yao, Jiacheng Zhu, Zhepeng Cen, Hanjiang Hu, Wenhao Yu, Tingnan Zhang, Jie Tan, Ding Zhao
Data Set New Benchmark Safe Reinforcement Learning Offline Safe Reinforcement Learning

June 1, 2023

Safe Offline Reinforcement Learning with Real-Time Budget Constraints
Qian Lin, Bo Tang, Zifan Wu, Chao Yu, Shangqin Mao, Qianlong Xie, Xingxing Wang, Dong Wang
Reinforcement Learning Budget Constraint Offline Safe Reinforcement Learning Trajectory Distribution Multi Session Budget Optimization

February 14, 2023

Constrained Decision Transformer for Offline Safe Reinforcement Learning
Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, Ding Zhao
Safe Reinforcement Learning Decision Transformer Policy Constraint Offline Safe Reinforcement Learning

January 28, 2023

SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning
Qin Zhang, Linrui Zhang, Haoran Xu, Li Shen, Bowen Wang, Yongzhe Chang, Xueqian Wang, Bo Yuan, Dacheng Tao
Constructive Approach Sequence Model Policy Constraint Offline Safe Reinforcement Learning

January 13, 2023

Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning
Taylor W. Killian, Sonali Parbhoo, Marzyeh Ghassemi
Person Identification Risk Sensitive Worst Case Offline Safe Reinforcement Learning Risk Preference

Offline Safe Reinforcement Learning

Papers

OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learning

Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning

Offline Goal-Conditioned Reinforcement Learning for Safety-Critical Tasks with Recovery Policy

Temporal Logic Specification-Conditioned Decision Transformer for Offline Safe Reinforcement Learning

Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning

Offline Risk-sensitive RL with Partial Observability to Enhance Performance in Human-Robot Teaming

Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model

Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning

Safety-aware Causal Representation for Trustworthy Offline Reinforcement Learning in Autonomous Driving

Guided Online Distillation: Promoting Safe Reinforcement Learning by Offline Demonstration

Datasets and Benchmarks for Offline Safe Reinforcement Learning

Safe Offline Reinforcement Learning with Real-Time Budget Constraints

Constrained Decision Transformer for Offline Safe Reinforcement Learning

SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning

Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning