Policy Gradient

Follow this topic Ask about this topic

Policy gradient methods optimize a policy directly, using reward or advantage estimates to adjust the likelihood of actions that perform well. Common approaches include REINFORCE, actor–critic and natural policy gradient, alongside deterministic or pathwise gradients and variance-reduction techniques. They are used in bandits, continuous control, multi-agent learning and language-model post-training with verifiable rewards. Theory gives convergence and sample-efficiency guarantees in settings such as tabular MDPs, but exploration, function approximation and noisy or biased gradient estimates remain central challenges.

Latest papers

69 papers12 papers in the last four weeks, up 200% on the four weeks before. 0.1% of all new papers.

On Reparameterizing the Score Function
Oct 16, 2026Mohammad Hossein Amani, Leonardo Bocchieri, Maxime Peyrard +2Policy Gradient Policy Gradient Methods
EPFL · CNRS, Univ. Grenoble Alpes · XENT Labs
Baselines and Score Centering in Policy Gradient Estimation
Oct 15, 2026Shuo Han, Ziqing HuPolicy Gradient Policy Gradient Methods
University of Illinois Chicago · Perplexity AI
Backpropagating Verifier Rewards Through Continuous Flow Language Models
Oct 13, 2026Xu Han, Jiajing Hu, Li-Ping LiuRL for Language Model Reasoning Language Modeling
Tufts University
Ascent Conditions for Length-Normalized Single-Rollout Policy Updates
Oct 10, 2026Levin David SchwabU-Statistics Policy Gradient Methods
Variance-Reduced Policy Gradient is Nearly Minimax-Optimal
Oct 9, 2026Leonardo Cesani, Matteo Papini, Marcello RestelliPolicy Gradient Policy Gradient Methods
Politecnico di Milano · Universit`a degli Studi di Milano
When KL Regularization Misfires in Group Policy Optimization
Oct 8, 2026Fei DingKL-Regularized RL Group Relative Policy Optimization
Alibaba Group
TRIAGE: Direction-Aware Mismatch Stabilization of Native NVFP4 Reinforcement Learning
Oct 5, 2026Zhen Li, Shuai Zhang, Yanggan Gu +7Reinforcement Learning RL Post-Training
InfiX.ai, Singapore · NVIDIA
Planning to Learn
Oct 2, 2026Ian OsbandPolicy Gradient Methods Policy Gradient
Google DeepMind
Sample complexity of variance-reduced policy gradient: weaker assumptions and lower bounds
Oct 2, 2026Gabor Paczolay, Matteo Papini, Alberto Maria Metelli +2Policy Gradient Policy Gradient Methods
Politecnico di Milano. · Budapest University of Technology and Economics. · University of Milan.
From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation
Oct 1, 2026Siqi Zhu, Suozhi Huang, Kaixuan Zhang +4Policy Gradient Language Model Distillation
University of Illinois Urbana-Champaign · Princeton University · Westlake University
Reward-rate Policy Gradient for Efficient Machine Learning Engineering Agents
Sep 28, 2026Muhang Tian, Sherry YangPolicy Gradient Agentic RL
New York University
Marginally Correct Tool Caches Can Reverse Group-Normalized Policy Updates
Sep 22, 2026Shivam GuptaPolicy Gradient Group Relative Policy Optimization
Independent Researcher
EPIG-Tree: Compute-Optimal Branching for Gradient-Efficient Reinforcement Learning
Sep 17, 2026Nikita Khomich, Leopold Hermansson, Ido HakimiRL for Language Models Policy Gradient
High-Probability Nash Regret for Decentralized Learning in Markov $α$ -Potential Games: Episodic and Fully Online Asynchronous Algorithms with Applications to Markov Congestion Games
Sep 14, 2026S. Rasoul EtesamiPolicy Gradient Asynchronous RL
Department of Industrial and Systems Engineering Coordinated Science Laboratory University of Illinois Urbana-Champaign, Urbana, IL, USA
Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
Sep 8, 2026Youngrok Park, Sangmin Bae, Hojung Jung +6Policy Gradient Weak-to-Strong Generalization
KAIST AI · Microsoft · University of Toronto +3
Beyond the Stability-Exploration Dilemma: Environmental Regularization for LLM Policy Optimization
Aug 24, 2026Xianlei Zhou, Xiangdi Meng, Yu He +7RL for Language Model Reasoning RL Post-Training
AMAP, Alibaba Group · Xi’an Jiaotong University · Beijing Normal University +1
Vector Symbolic Policy Gradient
Aug 19, 2026Ryozo Masukawa, Sanggeon Yun, SungHeon Jeong +6Reinforcement Learning Policy Learning
University of California, Irvine · Intel Corporation · Johns Hopkins University +1
Policy Gradient Steering: Interventions from Behavioral Objectives
Jul 30, 2026Yoann Poupart, Aurélie Beynier, Nicolas MaudetPolicy Gradient Activation Steering
LIP6, Sorbonne University Paris, France
On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards
Jul 25, 2026Fei Ding, Yongkang Zhang, Yuhao Liao +2Policy Gradient RL for Language Model Reasoning
Alibaba Group · Tsinghua University
In-Context Learning as Implicit Policy Gradient
Jul 25, 2026Masahiro Kaneko, Timothy BaldwinRL for Language Models Policy Gradient
MBZUAI
Finite-Time Analysis of the Natural Policy Gradient in Finite-Horizon Markov Decision Processes
Jul 25, 2026Asha Barua, Sajad KhodadadianPolicy Gradient Markov Decision Processes
Grado Department of Industrial and Systems Engineering, Virginia Polytechnic Institute and State University, Blacksburg, Virginia 24061, USA.
A Self-Evolving Default Action for Cooperative Tasks with Continuous Action Space
Jul 21, 2026Shuangyao HuangPolicy Gradient Credit Assignment
Xi’an Jiaotong-Liverpool University School of Internet of Things
Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies
Jul 13, 2026Ziheng Cheng, Xin Guo, Huyên Pham +1Policy Gradient Continuous-Time RL
University of California, Berkeley · Ecole Polytechnique, CMAP · Imperial College London
Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RL
Jul 1, 2026Juliette Decugis, Sean O'Brien, Francis Bach +2Policy Gradient RL for Language Model Reasoning
FAIR at Meta · Work done at Meta, now working at the University of California, San Diego · Inria, Ecole Normale Supérieure
ELMP: Efficient Learning for Motion Planning via Analytical Policy Gradients
Jun 30, 2026Yixiao Li, Tifanny Portela, Jordis Herrmann +2Policy Gradient Robot Policy Adaptation
Robotic Systems Lab, ETH Zürich · ABB · ANYbotics
On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse
Jun 28, 2026Amritansh Mishra, Supriyo Chakraborty, Berkcan KapusuzogluPolicy Gradient Token-Level Credit Assignment
Capital One
FlowDPG: Deterministic Policy Gradient on Flow Matching Policies for Real-World Manipulation
Jun 21, 2026Kexin Shi, Junyao Shi, Poorvi Hebbar +5Deep Deterministic Policy Gradient Policy Gradient
Skild AI · Carnegie Mellon University · University of Pennsylvania
ReFPO: Reflow Regularization for Flow Matching Policy Gradients
Jun 19, 2026Ge Wang, Yibo Peng, Fan Feng +10Policy Gradient Reinforcement Learning
Ising AI · CUHK-Shenzhen · FNii-Shenzhen
Deterministic Policy Gradient for Learning Equilibrium in Time-Inconsistent Control Problems
Jun 10, 2026Xin Guo, Yijie Huang, Xiang YuPolicy Gradient Stochastic Optimal Control
Department of Industrial Engineering and Operations Research, University of California, Berkeley, USA. · Department of Applied Mathematics, The Hong Kong Polytechnic University, Kowloon, Hong Kong.
On Advantage Estimates for Max@K Policy Gradients
Jun 4, 2026Shota Takashiro, Soichiro Nishimori, Paavo Parmas +6Policy Gradient Policy Gradient Methods
The University of Tokyo

Oct 16, 2026J/K move · Enter open · S save

On Reparameterizing the Score Function

Mohammad Hossein Amani, Leonardo Bocchieri, Maxime Peyrard, Clément Hongler+1

EPFL · CNRS, Univ. Grenoble Alpes · XENT Labs

Policy Gradient Policy Gradient Methods Reinforcement Learning Step-Level Credit Assignment in RL

Open paper AskPDF arXiv

Policy gradients for autoregressive sequence models differentiate each token's log-probability while holding the conditioning history fixed. This discards information about how earlier decisions shape later conditional distributions, creating a credit-assignment bottleneck when rewards are sparse. We introduce History-Reparameterized (HR) policy gradients, which replace the frozen prefix in the score function with a differentiable reparameterized or relaxed history, allowing gradients of downstream log-probabilities to propagate backward through the generated trajectory. The resulting history-pathwise term is generally biased when added directly, so we derive Optimal HR (OHR), a centered control-variate estimator with a closed-form coefficient that minimizes the trace of the gradient variance at the population level. The variance reduction is governed by temporal structure in the policy dynamics: random or weakly structured policies provide little benefit, while policies with learned sequential dependencies yield informative history-pathwise corrections. For discrete tokens, the choice of relaxation is crucial: embedding-level relaxation yields substantial variance reduction that grows with horizon, whereas straight-through Gumbel-Softmax yields essentially none in our diagnostics. We validate the method on a Gaussian linear Markov model, a sparse-reward T-maze with a recurrent policy, and an autoregressive Transformer model for small Traveling Salesman Problem (TSP) instances.