Policy Regularization

Policy regularization in reinforcement learning aims to improve the stability, robustness, and efficiency of learned policies by constraining their behavior during training. Current research focuses on developing novel regularization techniques, often leveraging diffusion models, variational autoencoders, or Q-function estimates to guide policy learning and mitigate issues like out-of-distribution actions and catastrophic forgetting in continual learning settings. These advancements are significant because they enhance the reliability and applicability of reinforcement learning agents across diverse domains, from robotics and autonomous driving to personalized recommendations and human-AI collaboration.

Papers

November 7, 2024

Hypercube Policy Regularization Framework for Offline Reinforcement Learning
Yi Shen, Hanyan Huang
LeArning Abstract Offline Reinforcement Learning Constrained Policy Optimization Policy Regularization

May 31, 2024

Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning
Linjiajie Fang, Ruoxue Liu, Jing Zhang, Wenjia Wang, Bing-Yi Jing
Offline Reinforcement Learning Diffusion Policy Policy Regularization Actor Critic Framework Diffusion Noise

May 30, 2024

Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
Tenglong Liu, Yang Li, Yixing Lan, Hao Gao, Wei Pan, Xin Xu
Offline Reinforcement Learning Adaptive Importance Policy Regularization Behavior Policy

May 29, 2024

Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation
Fengshuo Bai, Rui Zhao, Hongming Zhang, Sijia Cui, Ying Wen, Yaodong Yang, Bo Xu, Lei Han
Reinforcement Learning Policy Learning Preference Based Reinforcement Learning Policy Regularization Experience Pool Reward Estimation Accuracy

April 29, 2024

MRIC: Model-Based Reinforcement-Imitation Learning with Mixture-of-Codebooks for Autonomous Driving Simulation
Baotian He, Yibing Li
Imitation Learning Codebook Learning Highway Driving Autonomous Driving Simulation Policy Regularization Regularized Reinforcement Learning

February 22, 2024

COPR: Continual Human Preference Learning via Optimal Policy Regularization
Han Zhang, Lin Gui, Yu Lei, Yuanzhao Zhai, Yehong Zhang, Yulan He, Hui Wang, Yue Yu, Kam-Fai Wong, Bin Liang, Ruifeng Xu
Reinforcement Learning Continual LEArning Optimal Policy Preference Learning Policy Regularization

October 26, 2023

Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotion
Laura Smith, Yunhao Cao, Sergey Levine
Deep Reinforcement Learning Continuum Limit Quadrupedal Robot Efficient Exploration Legged Locomotion Robot Locomotion Real World Reinforcement Learning Policy Regularization Continuous Improvement

October 24, 2023

COPR: Continual Learning Human Preference through Optimal Policy Regularization
Han Zhang, Lin Gui, Yuanzhao Zhai, Hui Wang, Yu Lei, Ruifeng Xu
Reinforcement Learning Pre Trained Language Model Continual Learning Preference Learning Policy Regularization

October 11, 2023

Score Regularized Policy Optimization through Diffusion Behavior
Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, Jun Zhu
Diffusion Model Offline Reinforcement Learning Policy Gradient Diffusion Explainer Policy Regularization

October 9, 2023

Increasing Entropy to Boost Policy Gradient Performance on Personalization Tasks
Andrew Starnes, Anton Dereventsov, Clayton Webster
Policy Gradient Regularization Model Stable Entropy Policy Regularization Personalization Task Diversity Reinforcement Learning

July 25, 2023

Offline Reinforcement Learning with On-Policy Q-Function Regularization
Laixi Shi, Robert Dadashi, Yuejie Chi, Pablo Samuel Castro, Matthieu Geist
Offline Reinforcement Learning Estimation Task Q Function Policy Regularization Poor Extrapolation Performance

June 11, 2023

Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, Yang Yu
Offline Reinforcement Learning Policy Regularization Action Sampling

June 6, 2023

March 12, 2023

Twice Regularized Markov Decision Processes: The Equivalence between Robustness and Regularization
Esther Derman, Yevgeniy Men, Matthieu Geist, Shie Mannor
Native Robustness Markov Decision Process Policy Regularization Regularized Reinforcement Learning Robust Markov Decision Process

October 11, 2022

Human-AI Coordination via Human-Regularized Search and Learning
Hengyuan Hu, David J Wu, Adam Lerer, Jakob Foerster, Noam Brown
Reinforcement Learning LeArning Abstract AI Agent Behavior Cloning Policy Regularization Learning Search Human AI Coordination

August 26, 2022

Symbolic Explanation of Affinity-Based Reinforcement Learning Agents with Markov Models
Charl Maree, Christian W. Omlin
State of the Art Reinforcement Policy Regularization Model Understanding Personalized Incentive Symbolic Language

May 30, 2022

Truly Deterministic Policy Optimization
Ehsan Saleh, Saba Ghaffari, Timothy Bretl, Matthew West
Policy Gradient Interacting System Policy Regularization

May 23, 2022

POLTER: Policy Trajectory Ensemble Regularization for Unsupervised Reinforcement Learning
Frederik Schubert, Carolin Benjamins, Sebastian Döhler, Bodo Rosenhahn, Marius Lindauer
Model Free Reinforcement Learning Benchmark Unsupervised Reinforcement Learning Policy Regularization Phantom Threat

March 23, 2022

Your Policy Regularizer is Secretly an Adversary
Rob Brekelmans, Tim Genewein, Jordi Grau-Moya, Grégoire Delétang, Markus Kunesch, Shane Legg, Pedro Ortega
Reinforcement Learning Adversary Agent Policy Regularization Adversarial Reward

Policy Regularization

Papers

Hypercube Policy Regularization Framework for Offline Reinforcement Learning

Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning

Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning

Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation

MRIC: Model-Based Reinforcement-Imitation Learning with Mixture-of-Codebooks for Autonomous Driving Simulation

COPR: Continual Human Preference Learning via Optimal Policy Regularization

Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotion

COPR: Continual Learning Human Preference through Optimal Policy Regularization

Score Regularized Policy Optimization through Diffusion Behavior

Increasing Entropy to Boost Policy Gradient Performance on Personalization Tasks

Offline Reinforcement Learning with On-Policy Q-Function Regularization

Policy Regularization with Dataset Constraint for Offline Reinforcement Learning

State Regularized Policy Optimization on Data with Dynamics Shift

COPR: Consistency-Oriented Pre-Ranking for Online Advertising

Twice Regularized Markov Decision Processes: The Equivalence between Robustness and Regularization

Human-AI Coordination via Human-Regularized Search and Learning

Symbolic Explanation of Affinity-Based Reinforcement Learning Agents with Markov Models

Truly Deterministic Policy Optimization

POLTER: Policy Trajectory Ensemble Regularization for Unsupervised Reinforcement Learning

Your Policy Regularizer is Secretly an Adversary