Softmax Policy

Softmax policy gradient methods optimize policies in reinforcement learning by parameterizing action probabilities using a softmax function, aiming to find optimal strategies for maximizing cumulative rewards. Current research focuses on improving the convergence speed and robustness of these methods, exploring techniques like Nesterov's accelerated gradient, dynamic policy gradients, and novel estimators to address issues such as suboptimal policy saturation and the challenges posed by non-concave objective functions. These advancements enhance the efficiency and applicability of reinforcement learning algorithms across various domains, including multi-agent systems and constrained Markov decision processes, leading to improved performance in practical applications.

Papers

May 21, 2024

Towards Principled, Practical Policy Gradient for Bandits and Tabular MDPs
Michael Lu, Matin Aghaei, Anant Raj, Sharan Vaswani
Markov Decision Process Multi Armed Bandit Policy Gradient General Principle Tabular Markov Decision Process Softmax Policy

April 4, 2024

Elementary Analysis of Policy Gradient Methods
Jiacai Liu, Wenye Li, Ke Wei
Policy Gradient Policy Iteration Natural Policy Gradient Softmax Policy

October 18, 2023

Accelerated Policy Gradient: On the Convergence Rates of the Nesterov Momentum for Reinforcement Learning
Yen-Ju Chen, Nai-Chieh Huang, Ching-Pei Lee, Ping-Chun Hsieh
Reinforcement Learning Policy Gradient Policy OpTimization Convergence Rate Nesterov Momentum Accelerated Gradient Softmax Policy Momentum Acceleration

October 4, 2023

Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methods
Sara Klein, Simon Weissmann, Leif Döring
Markov Decision Process Policy Gradient Optimal Policy Convergence Analysis Softmax Policy

June 2, 2023

Context-Aware Bayesian Network Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning
Dingyang Chen, Qi Zhang
Actor Critic Cooperative Multi Agent Reinforcement Learning Markov Game Multi Agent Policy Softmax Policy

March 22, 2023

Matryoshka Policy Gradient for Entropy-Regularized RL: Convergence and Global Optimality
François Ged, Maria Han Veiga
Policy Gradient Early Stage Convergence Maximum Entropy Global Optimality Entropy Regularized Maximum Entropy Reinforcement Learning Softmax Policy

June 15, 2022

Convergence and Price of Anarchy Guarantees of the Softmax Policy Gradient in Markov Potential Games
Dingyang Chen, Qi Zhang, Thinh T. Doan
Policy Gradient Early Stage Convergence Markov Game Curious Price Natural Policy Gradient Stability Guarantee Softmax Policy Markov Potential Game

June 6, 2022

Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPs
Dongsheng Ding, Kaiqing Zhang, Jiali Duan, Tamer Başar, Mihailo R. Jovanović
Sample Complexity Early Stage Convergence Primal Dual Constrained Markov Decision Process Natural Policy Gradient Softmax Policy

June 4, 2022

Interpolating Between Softmax Policy Gradient and Neural Replicator Dynamics with Capped Implicit Exploration
Dustin Morrill, Esra'a Saleh, Michael Bowling, Amy Greenwald
Adversarial Bandit Replicator Dynamic Softmax Policy Implicit Exploration

February 2, 2022

On the Global Convergence Rates of Decentralized Softmax Gradient Play in Markov Potential Games
Runyu Zhang, Jincheng Mei, Bo Dai, Dale Schuurmans, Na Li
Policy OpTimization Single Agent Reinforcement Learning Softmax Policy Markov Potential Game Global Convergence Rate Decentralized Policy Gradient

January 20, 2022

A Prescriptive Dirichlet Power Allocation Policy with Deep Reinforcement Learning
Yuan Tian, Minghao Han, Chetan Kulkarni, Olga Fink
Reinforcement Learning Deep Reinforcement Learning Power Allocation Softmax Policy Optimal Operation Gaussian Policy

December 22, 2021

An Alternate Policy Gradient Estimator for Softmax Policies
Shivam Garg, Samuele Tosatto, Yangchen Pan, Martha White, A. Rupam Mahmood
Policy Gradient Optimal Policy Softmax Policy

Softmax Policy

Papers

Towards Principled, Practical Policy Gradient for Bandits and Tabular MDPs

Elementary Analysis of Policy Gradient Methods

Accelerated Policy Gradient: On the Convergence Rates of the Nesterov Momentum for Reinforcement Learning

Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methods

Context-Aware Bayesian Network Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning

Matryoshka Policy Gradient for Entropy-Regularized RL: Convergence and Global Optimality

Convergence and Price of Anarchy Guarantees of the Softmax Policy Gradient in Markov Potential Games

Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPs

Interpolating Between Softmax Policy Gradient and Neural Replicator Dynamics with Capped Implicit Exploration

On the Global Convergence Rates of Decentralized Softmax Gradient Play in Markov Potential Games

A Prescriptive Dirichlet Power Allocation Policy with Deep Reinforcement Learning

An Alternate Policy Gradient Estimator for Softmax Policies