Policy Gradient

Policy gradient methods are a core component of reinforcement learning, aiming to optimize policies by directly estimating the gradient of expected cumulative rewards. Current research emphasizes improving sample efficiency and addressing challenges like high-dimensional state spaces and non-convex optimization landscapes through techniques such as residual policy learning, differentiable simulation, and novel policy architectures (e.g., tree-based, low-rank matrix models). These advancements are significant for both theoretical understanding of reinforcement learning algorithms and practical applications in robotics, control systems, and other domains requiring efficient and robust decision-making under uncertainty.

Papers

December 5, 2023

Score-Aware Policy-Gradient Methods and Performance Guarantees using Local Lyapunov Conditions: Applications to Product-Form Stochastic Networks and Queueing Systems
Céline Comte, Matthieu Jonckheere, Jaron Sanders, Albert Senen-Cerda
Policy Gradient Model Based Reinforcement Learning Lyapunov Function Performance Guarantee Stochastic Network

November 30, 2023

Anomaly Detection via Learning-Based Sequential Controlled Sensing
Geethu Joseph, Chen Zhong, M. Cenk Gursoy, Senem Velipasalar, Pramod K. Varshney
Anomaly Detection Markov Decision Process Policy Gradient Actor Critic Algorithm Sequential Selection Sequential Hypothesis Testing

November 26, 2023

Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learning
Changyu Chen, Ramesha Karunasena, Thanh Hong Nguyen, Arunesh Sinha, Pradeep Varakantham
Reinforcement Learning Generative Model Policy Gradient Action Space Stochastic Policy Stochastic Action

November 24, 2023

Directly Attention Loss Adjusted Prioritized Experience Replay
Zhuoying Chen, Huiping Li, Zhaoxu Wang
Reinforcement Learning Multi Agent Reinforcement Learning Policy Gradient Experience Replay Attention Loss Preference Based Reward

November 23, 2023

Fast Policy Learning for Linear Quadratic Control with Entropy Regularization
Xin Guo, Xinyu Li, Renyuan Xu
Policy Gradient Optimal Policy Policy Iteration Entropy Regularization Policy Learning Method

November 15, 2023

Efficiently Escaping Saddle Points for Non-Convex Policy Optimization
Sadegh Khorasani, Saber Salehkaleybar, Negar Kiyavash, Niao He, Matthias Grossglauser
Policy Gradient Variance Reduction Saddle Point Variance Reduced Policy Gradient

November 14, 2023

November 10, 2023

Clipped-Objective Policy Gradients for Pessimistic Policy Optimization
Jared Markowitz, Edward W. Staley
Policy Gradient Proximal Policy Optimization Policy Optimization Natural Policy Gradient Trust Region Policy Optimization Multi Objective Policy

November 7, 2023

November 5, 2023

On the Second-Order Convergence of Biased Policy Gradient Algorithms
Siqiao Mu, Diego Klabjan
Policy Gradient Actor Critic Algorithm Vanilla Reinforcement Learning

November 1, 2023

Emergence of Collective Open-Ended Exploration from Decentralized Meta-Reinforcement Learning
Richard Bornemann, Gautier Hamon, Eleni Nisioti, Clément Moulin-Frier
Policy Gradient Path Breaking Emergence Meta Reinforcement Learning Cooperative Behavior Collaborative Exploration Decentralized Reinforcement Learning

October 31, 2023

Vanishing Gradients in Reinforcement Finetuning of Language Models
Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thilak, Arwen Bradley, Preetum Nakkiran, Joshua Susskind, Etai Littwin
Language Model Fine Tuning Policy Gradient Pretrained Language Model Reward Maximization Supervised Finetuning Gradient Vanishing

October 30, 2023

Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithms
Shenao Zhang, Boyi Liu, Zhaoran Wang, Tuo Zhao
Practical Algorithm Policy Gradient Theoretical Understanding Reparameterization Method

October 29, 2023

Optimization Landscape of Policy Gradient Methods for Discrete-time Static Output Feedback
Jingliang Duan, Jie Li, Xuyang Chen, Kai Zhao, Shengbo Eben Li, Lin Zhao
Reinforcement Learning Policy Gradient Discrete Time Natural Policy Gradient Optimization Landscape Vanilla Reinforcement Learning

October 28, 2023

Pessimistic Off-Policy Multi-Objective Optimization
Shima Alizadeh, Aniruddha Bhargava, Karthick Gopalswamy, Lalit Jain, Branislav Kveton, Ge Liu
Policy Gradient Multi Objective Optimization Inverse Propensity Multi Objective Policy

October 24, 2023

Fractal Landscapes in Policy Optimization
Tao Wang, Sylvia Herbert, Sicun Gao
Deep Reinforcement Learning Policy Gradient Policy OpTimization Fractal Structure

October 23, 2023

Policy Gradient with Kernel Quadrature
Satoshi Hayakawa, Tetsuro Morimura
Gaussian Process Policy Gradient Reinforcement Learning Task Reward Value Kernel Quadrature