Policy Gradient

Follow this topic Ask about this topic

Policy gradient methods optimize a policy directly, using reward or advantage estimates to adjust the likelihood of actions that perform well. Common approaches include REINFORCE, actor–critic and natural policy gradient, alongside deterministic or pathwise gradients and variance-reduction techniques. They are used in bandits, continuous control, multi-agent learning and language-model post-training with verifiable rewards. Theory gives convergence and sample-efficiency guarantees in settings such as tabular MDPs, but exploration, function approximation and noisy or biased gradient estimates remain central challenges.

Latest papers

69 papers11 papers in the last four weeks, up 175% on the four weeks before. 0.1% of all new papers.

Global Convergence of Average Reward Constrained MDPs with Neural Critic and General Policy Parameterization
Mar 8, 2026Anirudh Satheesh, Pankaj Kumar Barman, Washim Uddin Mondal +1Natural Policy Gradient Constrained RL
Department of Computer Science, University of Maryland, College Park, MD 20740, USA · Indian Institute of Technology, Kanpur, Uttar Pradesh, 208016, India · Department of Electrical Engineering, Indian Institute of Technology, Kanpur, Uttar Pradesh, 208016, India +1
Post-Training with Policy Gradients: Optimality and the Base Model Barrier
Mar 7, 2026Alireza Mousavi-Hosseini, Murat A. ErdogduLanguage Model Post-Training Policy Gradient
University of Toronto, Vector Institute. · University of Toronto, Vector Institute, and Numurho.
Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies
Feb 27, 2026Xiang Li, Yuheng Zhang, Nan JiangNatural Policy Gradient Offline RL
Nanjing University · UIUC
Distributed scalable coupled policy algorithm for networked multi-agent reinforcement learning
Dec 5, 2025Pengcheng Dai, Dongming Wang, Wenwu Yu +1Cooperative MARL Decentralized MARL
Engineering Systems and Design Pillar, Singapore University of Technology and Design, Singapore 487372 · Department of Electrical and Computer Engineering, University of California, Riverside, CA 92521, USA · Frontiers Science Center for Mobile Information Communication and Security, School of Mathematics, Southeast University, Nanjing 210096, China, and also with Purple Mountain Laboratories, Nanjing 211102, China
Policy Gradient with Self-Attention for Model-Free Distributed Nonlinear Multi-Agent Games
Sep 22, 2025Eduardo Sebastián, Maitrayee Keskar, Eeman Iqbal +3Policy Gradient Self-Attention
Department of Computer Science and Technology, University of Cambridge, UK · Department of Electrical and Computer Engineering, University of California San Diego, USA · RoPeRt group, at DIIS - I3A, Universidad de Zaragoza, Spain
Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation: The Case of Multi-Armed Bandits
May 6, 2025Max Qiushi Lin, Jincheng Mei, Matin Aghaei +6Multi-Armed Bandits Policy Gradient
Simon Fraser University · Google DeepMind · Google Research +1
Analysis of On-policy Policy Gradient Methods under the Distribution Mismatch
Mar 28, 2025Weizhen Wang, Xinzhe Zhou, Jianping He +1Policy Gradient Policy Gradient Methods
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai, China
Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradients
Mar 28, 2024Johannes Müller, Semih Çaycı, Guido MontúfarPolicy Gradient Optimization Convergence Analysis
Department of Mathematics, RWTH Aachen University, Aachen, 52062, Germany · Departments of Mathematics and Statistics & Data Science, University of California, Los Angeles, 90095, USA · Max Planck Institute for Mathematics in the Sciences, Leipzig, 04103, Germany
Is the Policy Gradient a Gradient?
Jun 17, 2019Chris Nota, Philip S. ThomasPolicy Gradient Policy Gradient Methods
College of Information and Computer Sciences University of Massachusetts Amherst

Mar 8, 2026J/K move · Enter open · S save

Global Convergence of Average Reward Constrained MDPs with Neural Critic and General Policy Parameterization

Anirudh Satheesh, Pankaj Kumar Barman, Washim Uddin Mondal, Vaneet Aggarwal

Department of Computer Science, University of Maryland, College Park, MD 20740, USA · Indian Institute of Technology, Kanpur, Uttar Pradesh, 208016, India · Department of Electrical Engineering, Indian Institute of Technology, Kanpur, Uttar Pradesh, 208016, India+1

Natural Policy Gradient Constrained RL Actor-Critic Methods Policy Gradient Average-Reward RL

Open paper AskPDF arXiv

We study infinite-horizon Constrained Markov Decision Processes (CMDPs) with general policy parameterizations and multi-layer neural network critics. Existing theoretical analyses for constrained reinforcement learning largely rely on tabular policies or linear critics, which limits their applicability to high-dimensional and continuous control problems. We propose a primal-dual natural actor-critic algorithm that integrates neural critic estimation with natural policy gradient updates and leverages Neural Tangent Kernel (NTK) theory to control function-approximation error under Markovian sampling, without requiring access to mixing-time oracles. We establish global convergence and cumulative constraint violation rates of

\tilde{\mathcal{O}}(T^-1/4)

up to approximation errors induced by the policy and critic classes. Our results provide the first such guarantees for CMDPs with general policies and multi-layer neural critics, substantially extending the theoretical foundations of actor-critic methods beyond the linear-critic regime.