Policy Parametrization

Policy parametrization in reinforcement learning focuses on how to represent the agent's decision-making policy, aiming to optimize its performance and efficiency. Current research emphasizes developing algorithms that achieve improved convergence rates and sample complexity, often employing techniques like natural policy gradients, primal-dual methods, and variance reduction, alongside exploring alternative policy distributions beyond the common Gaussian, such as Bingham distributions for rotational tasks. These advancements are significant because they enable more efficient training of reinforcement learning agents, leading to better performance in various applications, including robotics, resource allocation, and language model alignment.

Papers

September 3, 2024

Learning State-Dependent Policy Parametrizations for Dynamic Technician Routing with Rework
Jonas Stein, Florentin D Hildebrandt, Barrett W Thomas, Marlin W Ulmer
Reinforcement Learning LeArning Abstract Sequential Decision Routing Problem Task Specification Diverse Skill Policy Parametrization

August 21, 2024

Last-Iterate Convergence of General Parameterized Policies in Constrained MDPs
Washim Uddin Mondal, Vaneet Aggarwal
Primal Dual Constrained Markov Decision Process Policy Parameterization Last Iterate Convergence Policy Parametrization

July 26, 2024

Order-Optimal Global Convergence for Average Reward Reinforcement Learning via Actor-Critic Approach
Swetha Ganesh, Washim Uddin Mondal, Vaneet Aggarwal
Monte Carlo Multi Agent Actor Critic Policy Parametrization Average Reward Reinforcement Learning

July 23, 2024

Functional Acceleration for Policy Mirror Descent
Veronica Chelu, Doina Precup
Reinforcement Learning Policy Mirror Descent Policy Parametrization Functional Acceleration

May 29, 2024

Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF
Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, Bo Dai
Reinforcement Learning Unified Framework Preference Optimization Reinforcement Learning From Human Feedback Online Service Standard Reinforcement Learning Implicit Reward Policy Parametrization

March 11, 2024

Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts
Onur Celik, Aleksandar Taranovic, Gerhard Neumann
Reinforcement Learning Mixture Component Expert Knowledge Curriculum Reinforcement Learning Diverse Skill Policy Parametrization Context Distribution

February 3, 2024

Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithm
Qinbo Bai, Washim Uddin Mondal, Vaneet Aggarwal
Optimal Policy Primal Dual Constrained Markov Decision Process Regret Guarantee Simple Regret Policy Gradient Algorithm Average Reward Markov Decision Process Policy Parametrization

February 1, 2024

Towards Efficient Exact Optimization of Language Model Alignment
Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, Minlie Huang
Reinforcement Learning Direct Preference Optimization Language Model Alignment Policy Parametrization Step Optimization

January 2, 2024

Global Convergence of Natural Policy Gradient with Hessian-aided Momentum Variance Reduction
Jie Feng, Ke Wei, Jinchi Chen
Global Convergence Natural Policy Gradient Policy Parametrization Gradient Dominance Property

November 21, 2022

Best of Both Worlds in Online Control: Competitive Ratio and Policy Regret
Gautam Goel, Naman Agarwal, Karan Singh, Elad Hazan
Optimal Policy World Event Sublinear Regret Disturbance Rejection Counterfactual Regret Online Control Algorithm Policy Parametrization Competitive Ratio

November 15, 2022

An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods
Yanli Liu, Kaiqing Zhang, Tamer Başar, Wotao Yin
Policy Gradient Variance Reduction Natural Policy Gradient Improved Analysis Policy Parametrization Variance Aware

November 3, 2022

Geometry and convergence of natural policy gradient methods
Johannes Müller, Guido Montúfar
Markov Decision Process Early Stage Convergence Geometric Analysis Natural Policy Gradient Local Convergence Policy Parametrization

June 12, 2022

Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Conservative Natural Policy Gradient Primal-Dual Algorithm
Qinbo Bai, Amrit Singh Bedi, Vaneet Aggarwal
Primal Dual Constrained Reinforcement Learning Constrained Markov Decision Process Natural Policy Gradient Constraint Violation Policy Parametrization Pf Agp

February 8, 2022

Bingham Policy Parameterization for 3D Rotations in Reinforcement Learning
Stephen James, Pieter Abbeel
Reinforcement Learning Policy Parameterization 3D Rotation Policy Parametrization

Policy Parametrization

Papers

Learning State-Dependent Policy Parametrizations for Dynamic Technician Routing with Rework

Last-Iterate Convergence of General Parameterized Policies in Constrained MDPs

Order-Optimal Global Convergence for Average Reward Reinforcement Learning via Actor-Critic Approach

Functional Acceleration for Policy Mirror Descent

Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF

Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Experts

Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithm

Towards Efficient Exact Optimization of Language Model Alignment

Global Convergence of Natural Policy Gradient with Hessian-aided Momentum Variance Reduction

Best of Both Worlds in Online Control: Competitive Ratio and Policy Regret

An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods

Geometry and convergence of natural policy gradient methods

Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Conservative Natural Policy Gradient Primal-Dual Algorithm

Bingham Policy Parameterization for 3D Rotations in Reinforcement Learning