Conservative Policy

Conservative policy optimization in reinforcement learning aims to develop robust and safe learning agents by limiting the magnitude of policy updates at each iteration, thereby preventing drastic and potentially unsafe changes. Current research focuses on improving the efficiency and sample complexity of these methods, exploring techniques like model-based approaches, iterative refinement of behavior regularization, and the use of novel divergence measures (e.g., Tsallis KL divergence) to control policy updates. These advancements are significant because they enhance the reliability and safety of reinforcement learning agents, particularly in real-world applications where unexpected behavior can have serious consequences.

Papers

January 26, 2024

Off-Policy Primal-Dual Safe Reinforcement Learning
Zifan Wu, Bo Tang, Qian Lin, Chao Yu, Shangqin Mao, Qianlong Xie, Xingxing Wang, Dong Wang
Reward Maximization Conservative Policy Convex Q Learning

December 7, 2023

MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator
Xiao-Yin Liu, Xiao-Hu Zhou, Guotao Li, Hao Li, Mei-Jiang Gui, Tian-Yu Xiang, De-Xing Huang, Zeng-Guang Hou
Offline Reinforcement Learning Model Based Offline Reinforcement Learning Bellman Operator Conservative Policy

June 9, 2023

Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
Xiaohan Hu, Yi Ma, Chenjun Xiao, Yan Zheng, Jianye Hao
Offline Reinforcement Learning Conservative Policy Behavior Regularization

January 27, 2023

Generalized Munchausen Reinforcement Learning using Tsallis KL Divergence
Lingwei Zhu, Zheng Chen, Matthew Schlegel, Martha White
Reinforcement Learning Kullback Leibler Divergence Vq Vae Conservative Policy

December 12, 2022

Variance-Reduced Conservative Policy Iteration
Naman Agarwal, Brian Bullins, Karan Singh
Reinforcement Learning Policy Gradient Global Optimality Conservative Policy Local Optimality

November 1, 2022

Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
Paria Rashidinejad, Hanlin Zhu, Kunhe Yang, Stuart Russell, Jiantao Jiao
Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Conservative Policy Augmented Lagrangian Single Policy Concentrability

September 16, 2022

Conservative Dual Policy Optimization for Efficient Model-Based Reinforcement Learning
Shenao Zhang
High Efficiency Model Based Reinforcement Learning Model Complexity Global Optimality Policy Improvement Conservative Policy

May 20, 2022

The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measure
Xing Chen, Dongcui Diao, Hechang Chen, Hengshuai Yao, Haiyin Piao, Zhixiao Sun, Zhiwei Yang, Randy Goebel, Bei Jiang, Yi Chang
Policy Gradient Proximal Policy Optimization Policy Evaluation Policy Optimization High Dimensional Clipping Conservative Policy Policy Data

April 26, 2022

RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning
Marc Rigter, Bruno Lacerda, Nick Hawes
Reinforcement Learning Offline Reinforcement Learning Robust Version Model Based Offline Reinforcement Learning Adversarial Environment Conservative Policy Robust Adversarial

February 15, 2022

CUP: A Conservative Update Policy Algorithm for Safe Reinforcement Learning
Long Yang, Jiaming Ji, Juntao Dai, Yu Zhang, Pengfei Li, Gang Pan
Safe Reinforcement Learning Safety Constraint Safe Reinforcement Learning Algorithm Conservative Policy

December 20, 2021

Learning Robust Policy against Disturbance in Transition Dynamics via State-Conservative Policy Optimization
Yufei Kuang, Miao Lu, Jie Wang, Qi Zhou, Bin Li, Houqiang Li
Robust Policy Unknown Disturbance Transition Dynamic Conservative Policy