Policy Gradient Methods

Follow this topic Ask about this topic

They estimate gradients from sampled trajectories using score-function methods such as REINFORCE, actor–critic value baselines, natural gradients, or pathwise differentiation through models. PPO-style clipping and trust regions constrain updates, while variance reduction and finer-grained credit assignment aim to use fewer samples. These methods are studied in bandits, robotics, continuous control, multi-agent and safe decision-making, and language-model post-training, where rewards can be sparse and trajectories long. A central tension is improving sample efficiency and stability without biasing updates or losing exploration; convergence guarantees exist in restricted settings, while high-dimensional and long-horizon problems remain challenging.

Latest papers

165 papers25 papers in the last four weeks, up 178% on the four weeks before. 0.1% of all new papers.

A Unified Analysis of Importance-Corrected Momentum Actor-Critic over a Markovian Trajectory
Oct 24, 2026Yanjie Dong, Zhou Su, Wei Wang +1Reinforcement Learning Actor-Critic Methods
School of Cyber-Science and Engineering, Xi’an Jiaotong University · School of Information and Communications Engineering, Xi’an Jiaotong University · State Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology
Direction-Aware Token Credit Assignment with Reward-Model Gradients
Oct 23, 2026Yikuan Hu, Jifeng Zhu, Minle Su +6Token-Level Credit Assignment Policy Gradient Methods
Nanjing University · National University of Singapore · Tsinghua University +1
Effectiveness of Curriculum Learning Depends on Reward Sparsity and Competing Optima
Oct 22, 2026John J. Vastola, Ann Huang, Satpreet H. Singh +2Curriculum RL Reinforcement Learning
Department of Neurobiology, Harvard Medical School · Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University · Department of Psychology and Center for Brain Science, Harvard University
Characterizing Oblivious Common-Reward Multi-Player Bandits: Universal Stationarity, Coordination Traps, and Certified Global Rates
Oct 21, 2026Mingxiao Li, Wenhai Lai, Hongyu WangMulti-Agent Bandits Stochastic Optimization Convergence
Anatomy of Off-Policy Policy Gradient: Importance Sampling, KL Regularization, and Baselines
Oct 19, 2026Haoqun Cao, Yurun Yuan, Tengyang XiePolicy Gradient Methods Off-Policy RL
University of Wisconsin–Madison
On Reparameterizing the Score Function
Oct 16, 2026Mohammad Hossein Amani, Leonardo Bocchieri, Maxime Peyrard +2Policy Gradient Policy Gradient Methods
EPFL · CNRS, Univ. Grenoble Alpes · XENT Labs
Baselines and Score Centering in Policy Gradient Estimation
Oct 15, 2026Shuo Han, Ziqing HuPolicy Gradient Policy Gradient Methods
University of Illinois Chicago · Perplexity AI
MIPS: Mitigating Mode Collapse in Reinforcement Learning for Large Outcome Spaces
Oct 14, 2026Armaan Sandhu, Abhijeet Sinha, Dianbo LiuMolecule Design Reinforcement Learning
University of Massachusetts Amherst · National University of Singapore
Variance-Optimal State Resampling for Reinforcement Learning with Verifiable Rewards
Oct 11, 2026Yibo Wang, Yuanxin Liu, Zhaoran WangReinforcement Learning with Verifiable Rewards Policy Gradient Methods
Ascent Conditions for Length-Normalized Single-Rollout Policy Updates
Oct 10, 2026Levin David SchwabU-Statistics Policy Gradient Methods
Target-ESS Tempered Policy Optimization (TETPO): Smooth Variance Control via Bridge Importance Sampling
Oct 10, 2026Shayan Mohajer Hamidi, Kayhan Behdin, Wenhui Zhu +4RL for Language Models RL Post-Training
LinkedIn
Variance-Reduced Policy Gradient is Nearly Minimax-Optimal
Oct 9, 2026Leonardo Cesani, Matteo Papini, Marcello RestelliPolicy Gradient Policy Gradient Methods
Politecnico di Milano · Universit`a degli Studi di Milano
Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Control
Oct 8, 2026Adonis Jamal, Samy Mekkaoui, Yadh Hafsi +1Policy Gradient Methods Model-Free RL
École Polytechnique
Temporally Interpretable Differentiable Decision Trees
Oct 7, 2026Eisuke Hirota, Aarav Sane, Rohan PalejaInterpretable ML Interpretability in RL
Department of Computer Science, Purdue University · Thomas Lord Department of Computer Science, University of Southern California
A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping
Oct 7, 2026Junwei Su, Mengfan Liu, Yanyong Zhang +1Proximal Policy Optimization Actor-Critic Methods
School of Artificial Intelligence and Data Science, University of Science and Technology of China · Department of Computer Science, University of Hong Kong
Convex-Concave Reinforcement Learning
Oct 6, 2026Shripad V. Deshmukh, Yaswanth Chittepu, Dhawal Gupta +2Reinforcement Learning Policy Gradient Methods
University of Massachusetts · Institute of Foundation Models
Variance-Optimal Control Variates for Learning with Black-box Feedback
Oct 3, 2026Zihao Zhao, Shuhan Zhang, Kai WangVariance Reduction Policy Gradient Methods
School of Computational Science and Engineering Georgia Institute of Technology
Planning to Learn
Oct 2, 2026Ian OsbandPolicy Gradient Methods Policy Gradient
Google DeepMind
Sample complexity of variance-reduced policy gradient: weaker assumptions and lower bounds
Oct 2, 2026Gabor Paczolay, Matteo Papini, Alberto Maria Metelli +2Policy Gradient Policy Gradient Methods
Politecnico di Milano. · Budapest University of Technology and Economics. · University of Milan.
Multi-Fidelity Policy Gradients Stabilize Data-Scarce Reinforcement Learning
Oct 1, 2026Xinjie Liu, Ruihan Zhao, Anirban Chaudhuri +3Policy Gradient Methods Policy Learning
University of Texas at Austin · University of British Columbia
OPTS-TTPO: Enhancing Finite-Sample Policy-Gradient Learning with Tree Search
Sep 30, 2026Junyu Lu, Shichao Weng, Zhiqiang Wang +10Tree Search Policy Gradient Methods
Dobot Robotics · Independent Researcher · Zhejiang University +2
Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness
Sep 29, 2026Zhiwei Wang, Yanxi Chen, Yaliang Li +1Policy Gradient Methods Off-Policy RL
Department of Mathematical Sciences Tsinghua University · Alibaba Group
Regularized policy gradient with learned mixtures of Gaussians for games with continuous actions
Sep 29, 2026Ondřej Kubíček, Viliam Lisý, Tuomas SandholmGame-Playing Agents Self-Play RL
Czech Technical University in Prague · Carnegie Mellon University · Artificial Intelligence Center
ORPG: Reconciling Multiple Reward Objectives through Objective-wise Policy Gradients
Sep 28, 2026Shicheng Fang, Yiwen Zhao, Wenbo Tian +4Multi-Objective Reinforcement Learning Gradient Interference
Fudan University · Shanghai Innovation Institute
Sufficiency of Zeroth-Order Reward Shaping for Policy Gradient in Stabilization Control
Sep 28, 2026Yisheng Zhang, Tao Wang, Sicun GaoReward Shaping Robotic RL
University of California San Diego, La Jolla, CA, USA · Tsinghua University, Beijing, China
OSCC: Certified Observation-Safe Coupling Optimization for Gradient-Noise Control in Imperfect-Information Learning
Sep 27, 2026Miaobo Hu, Shuhao Hu, Xiaobo Guo +4Partially Observable RL Policy Gradient Methods
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China · Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China
Learning Robot Policies from Sparse Success Signals via STL-Guided Stein Variational Policy Gradient
Sep 25, 2026Hongrui Zheng, Cristian Ioan Vasile, Antonio Loquercio +1Robot Policy Learning Reinforcement Learning
Department of Electrical and Systems Engineering, University of Pennsylvania, Philadelphia, PA, USA · Mechanical Engineering and Mechanics Department, Lehigh University, Bethlehem, PA, USA
Bundled Contact Gradients: Stabilizing Differentiable Simulation for Deployable Dynamic Tasks
Sep 25, 2026Dyuman Aditya, Jin Cheng, Clemens Schwarke +4Robotics Simulation Differentiable Physics
University of Southern California, LA, USA. · ETH Zurich, Switzerland. · NVIDIA. +1
Learning Vision-Based Agile Gap Traversal: Differentiable Simulation with a Warm-Started Critic
Sep 25, 2026Nuthasith Gerdpratoom, Tianchen Sun, Yichao Gao +1Visuomotor Policy Learning Robotic RL
Department of Electrical and Computer Engineering, National University of Singapore, 4 Engineering Drive 3, Singapore 117583, Singapore
Luck Is Not Skill: When Do Paired Rollouts Help Group-Relative RL of LLM Agents?
Sep 21, 2026Nazmus SakibGroup Relative Policy Optimization Policy Gradient Methods
BRAC University, Dhaka, Bangladesh

Oct 24, 2026J/K move · Enter open · S save

A Unified Analysis of Importance-Corrected Momentum Actor-Critic over a Markovian Trajectory

Yanjie Dong, Zhou Su, Wei Wang, Gang Wang

School of Cyber-Science and Engineering, Xi’an Jiaotong University · School of Information and Communications Engineering, Xi’an Jiaotong University · State Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing Institute of Technology

Reinforcement Learning Actor-Critic Methods Policy Gradient Methods Temporal-Difference Learning Importance Sampling for RL

Open paper AskPDF arXiv

We present a generic actor-critic method that combines a ridge-regularized pathwise least-squares temporal-difference (LSTD) critic with importance-corrected momentum (ICM) actors. The critic solves a ridge-regularized system per frame by Newton-Schulz inversion and needs no stepsize, no knowledge of the mixing time, and no rank condition. Averaging stochastic policy gradients over frames of

T

consecutive transitions, the ICM actors incur a frame-dependent gradient bias of order

\frac{1}{T}

. To our knowledge, we establish the first finite-time bounds for ICM actor-critic methods with normalized and unnormalized steps along a single Markovian trajectory. Up to a stationary-to-discounted distribution mismatch, the averaged gradient norm over

K

updates is bounded by

{\cal O}(K^{-1/3}+\frac{1}{T})

for normalized steps and by

{\cal O}(K^{-1/4}+\frac{1}{T})

for unnormalized steps. We specialize these bounds to Muon, normalized stochastic gradient descent (SGD) with momentum, SGD with momentum, and AdaNorm, and compare them in the nuclear norm. With a dimension-free stepsize, Muon confines the factor

\min\{m,n\}^{1/2}

associated with an

m\times n

actor parameter matrix to the noise and correction terms. The comparison also characterizes when orthogonalization guarantees a larger noiseless ascent than normalization: exactly when the gradient's nuclear rank exceeds the spectral-to-Frobenius curvature ratio.