Regularized Reinforcement Learning

Regularized reinforcement learning (RL) aims to improve the stability, efficiency, and robustness of RL algorithms by incorporating penalty terms or constraints into the learning process. Current research focuses on addressing issues like over-regularization in various action spaces, improving the transferability of learned rewards in inverse RL, and mitigating estimation biases in offline RL through techniques such as robust averaging and selective penalization. These advancements enhance the reliability and sample efficiency of RL, leading to improved performance in diverse applications, including robotics, autonomous driving, and language model control.

Papers

December 15, 2024

Entropy-Regularized Process Reward Model
Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, Tong Zhang
Reward Model Regularized Reinforcement Learning

November 28, 2024

Convex Regularization and Convergence of Policy Gradient Flows under Safety Constraints
Pekka Malo, Lauri Viitasaari, Antti Suominen, Eeva Vilkkumaa, Olli Tahvonen
Early Stage Convergence Gradient Flow Safety Constraint State Action Space Regularized Reinforcement Learning Convex Regularization

October 10, 2024

GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment
Yuancheng Xu, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, Sumitra Ganesh
Reward Model Test Time Best Arm Autoregressive Text Generation Regularized Reinforcement Learning

June 10, 2024

Decoupling regularization from the action space
Sobhan Mohammadpour, Emma Frejinger, Pierre-Luc Bacon
Action Space Regularization Model Inverse Reinforcement Learning Entropy Regularized Regularized Reinforcement Learning

June 3, 2024

Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning
Andreas Schlaginhaufen, Maryam Kamgarpour
Optimal Policy Inverse Reinforcement Learning Task Transferability Reward Report Reward Learning Transition Dynamic Regularized Reinforcement Learning

May 23, 2024

Exclusively Penalized Q-learning for Offline Reinforcement Learning
Junghyuk Yeom, Yonghyeon Jo, Jungmo Kim, Sanghyeon Lee, Seungyul Han
Offline Reinforcement Learning Q Learning Estimation Bias Policy Constraint Regularized Reinforcement Learning

May 3, 2024

Regularized Q-learning through Robust Averaging
Peter Schmitt-Förster, Tobias Sutter
Q Learning Regularized Reinforcement Learning

April 29, 2024

MRIC: Model-Based Reinforcement-Imitation Learning with Mixture-of-Codebooks for Autonomous Driving Simulation
Baotian He, Yibing Li
Imitation Learning Codebook Learning Highway Driving Autonomous Driving Simulation Policy Regularization Regularized Reinforcement Learning

March 27, 2024

Bridging the Gap: Regularized Reinforcement Learning for Improved Classical Motion Planning with Safety Modules
Elias Goldsztejn, Ronen I. Brafman
Reinforcement Learning Autonomous Navigation Safe Navigation Regularized Reinforcement Learning Human Navigation Classical Motion Planning

March 10, 2024

Optimal Policy Sparsification and Low Rank Decomposition for Deep Reinforcement Learning
Vikram Goddla
Deep Reinforcement Learning Low Rank Decomposition Regularized Reinforcement Learning Deep Policy

October 25, 2023

Controlled Decoding from Language Models
Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, Ahmad Beirami
Language Model Reinforcement Learning Regularized Reinforcement Learning Compound Token Reinforcement Learning Objective

March 12, 2023

Twice Regularized Markov Decision Processes: The Equivalence between Robustness and Regularization
Esther Derman, Yevgeniy Men, Matthieu Geist, Shie Mannor
Native Robustness Markov Decision Process Policy Regularization Regularized Reinforcement Learning Robust Markov Decision Process

February 27, 2023

Optimistic Planning by Regularized Dynamic Programming
Antoine Moulin, Gergely Neu
Markov Decision Process Near Optimal Policy Approximate Dynamic Programming Regularized Reinforcement Learning Linear Mixture Markov Decision Process

December 28, 2022

On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrations
Tim G. J. Rudner, Cong Lu, Michael A. Osborne, Yarin Gal, Yee Whye Teh
Expert Demonstration Policy Parameterization Regularized Reinforcement Learning Pathological Feature Behavior Policy

October 11, 2022

Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning
Anton Bakhtin, David J Wu, Adam Lerer, Jonathan Gray, Athul Paul Jacob, Gabriele Farina, Alexander H Miller, Noam Brown
Multi Agent Task Planning Game Content Regularized Reinforcement Learning Imitation Learned Policy Self Play Reinforcement Learning Reward Maximizing Policy

August 27, 2022

Object Goal Navigation using Data Regularized Q-Learning
Nandiraju Gireesh, D. A. Sasi Kiran, Snehasis Banerjee, Mohan Sridharan, Brojeshwar Bhowmick, Madhava Krishna
Semantic Map Object Goal Navigation Regularized Reinforcement Learning

June 2, 2022

Finite-Time Analysis of Entropy-Regularized Neural Natural Actor-Critic Algorithm
Semih Cayci, Niao He, R. Srikant
Markov Decision Process Policy OpTimization Finite Time Global Optimality Entropy Regularized Regularized Reinforcement Learning Sub Optimal Policy

May 23, 2022

RL with KL penalties is better viewed as Bayesian inference
Tomasz Korbak, Ethan Perez, Christopher L Buckley
Language Model Reinforcement Learning Bayesian Inference LLM Fine Tuning Regularized Reinforcement Learning Kullback Leibler

February 21, 2022

Accelerating Primal-dual Methods for Regularized Markov Decision Processes
Haoya Li, Hsiang-fu Yu, Lexing Ying, Inderjit Dhillon
Reinforcement Learning Markov Decision Process Primal Dual Natural Gradient Descent Primal Dual Algorithm Exponential Convergence Rate Regularized Reinforcement Learning

February 11, 2022

Regularized Q-learning
Han-Dong Lim, Donghwan Lee
Q Learning Linear Function Approximation Regularized Reinforcement Learning

Regularized Reinforcement Learning

Papers

Entropy-Regularized Process Reward Model

Convex Regularization and Convergence of Policy Gradient Flows under Safety Constraints

GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment

Decoupling regularization from the action space

Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning

Exclusively Penalized Q-learning for Offline Reinforcement Learning

Regularized Q-learning through Robust Averaging

MRIC: Model-Based Reinforcement-Imitation Learning with Mixture-of-Codebooks for Autonomous Driving Simulation

Bridging the Gap: Regularized Reinforcement Learning for Improved Classical Motion Planning with Safety Modules

Optimal Policy Sparsification and Low Rank Decomposition for Deep Reinforcement Learning

Controlled Decoding from Language Models

Twice Regularized Markov Decision Processes: The Equivalence between Robustness and Regularization

Optimistic Planning by Regularized Dynamic Programming

On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrations

Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning

Object Goal Navigation using Data Regularized Q-Learning

Finite-Time Analysis of Entropy-Regularized Neural Natural Actor-Critic Algorithm

RL with KL penalties is better viewed as Bayesian inference

Accelerating Primal-dual Methods for Regularized Markov Decision Processes

Regularized Q-learning