Policy Gradient Methods

Follow this topic Ask about this topic

They estimate gradients from sampled trajectories using score-function methods such as REINFORCE, actor–critic value baselines, natural gradients, or pathwise differentiation through models. PPO-style clipping and trust regions constrain updates, while variance reduction and finer-grained credit assignment aim to use fewer samples. These methods are studied in bandits, robotics, continuous control, multi-agent and safe decision-making, and language-model post-training, where rewards can be sparse and trajectories long. A central tension is improving sample efficiency and stability without biasing updates or losing exploration; convergence guarantees exist in restricted settings, while high-dimensional and long-horizon problems remain challenging.

Latest papers

165 papers25 papers in the last four weeks, up 178% on the four weeks before. 0.1% of all new papers.

COSAC: Counterfactual Credit Assignment in Sequential Cooperative Teams
Apr 20, 2026Shripad Deshmukh, Jayakumar Subramanian, Raghavendra Addanki +1Sequential MARL Credit Assignment in RL
College of Information and Computer Sciences University of Massachusetts Amherst · Adobe Research San Jose, CA
Beyond Importance Sampling: Rejection-Gated Policy Optimization
Apr 16, 2026Ziwu Sun, Zhen Gao, Jiyong Zhang +1Importance Sampling Policy Optimization
GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectification
Apr 15, 2026Wangjie Gan, Miao Pan, Linbo Xi +4RL Post-Training RL for Language Models
School of Software Technolog,Zhejiang University
On the "Causality" Step in Policy Gradient Derivations: A Pedagogical Reconciliation of Full Return and Reward-to-Go
Apr 6, 2026Nima H. SiboniPolicy Gradient Methods Reinforcement Learning
Juna.ai, Kastanienallee 32, 10435 Berlin
Softmax gradient policy for variance minimization and risk-averse multi armed bandits
Mar 31, 2026Gabriel TuriniciMulti-Armed Bandits Policy Gradient Methods
CEREMADE, Université Paris Dauphine - PSL, CNRS, Paris, France
Where-to-Learn: Analytical Policy Gradient Directed Exploration for On-Policy Robotic Reinforcement Learning
Mar 28, 2026Leixin Chang, Xinchen Yao, Ben Liu +2RL Exploration Robotic RL
A Lyapunov Analysis of Softmax Policy Gradient for Stochastic Bandits
Mar 27, 2026Tor LattimoreMulti-Armed Bandits Policy Gradient
Google DeepMind
Delightful Distributed Policy Gradient
Mar 20, 2026Ian OsbandPolicy Gradient Methods Policy Learning
Google DeepMind
Operator-Theoretic Foundations and Policy Gradient Methods for General MDPs with Unbounded Costs
Mar 18, 2026Abhishek Gupta, Aditya MahajanMarkov Decision Processes Policy Gradient Methods
The Ohio State University, Columbus, Ohio. · McGill University, Montreal, Canada.
How Log-Barrier Helps Exploration in Policy Optimization
Mar 16, 2026Leonardo Cesani, Matteo Papini, Marcello RestelliRL Exploration Policy Gradient Methods
Politecnico di Milano, Milan, Italy · Università degli Studi di Milano, Milan, Italy
Delightful Policy Gradient
Mar 15, 2026Ian OsbandPolicy Gradient Policy Gradient Methods
Google DeepMind
From $\log π$ to $π$ : Taming Divergence in Soft Clipping via Bilateral Decoupled Decay of Probability Gradient Weight
Mar 15, 2026Xiaoliang Fu, Jiaye Lin, Yangyi Fang +6RL for Language Model Reasoning Policy Gradient Methods
Meituan · Fudan University · Tsinghua University +2
MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learning
Mar 14, 2026Hongjun Wang, Wei Liu, Weibo Gu +2Policy Gradient Methods RL for Language Model Reasoning
The University of Hong Kong · Tencent Youtu Lab
Entropy-Preserving Reinforcement Learning
Mar 12, 2026Aleksei Petrenko, Ben Lipkin, Kevin Chen +4Policy Gradient Methods Behavioral Diversity in RL
Apple · MIT
$V_{0.5}$ : Generalist Value Model as a Prior for Sparse RL Rollouts
Mar 11, 2026Yi-Kai Zhang, Yueqing Sun, Hongyan Hao +4Reinforcement Learning with Verifiable Rewards Value Function Estimation
School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University · Meituan, China
Overcoming Valid Action Suppression in Unmasked Policy Gradient Algorithms
Mar 10, 2026Renos Zabounidis, Roy Siegelmann, Mohamad Qadri +3Policy Gradient Methods
Carnegie Mellon University · Massachusetts Institute of Technology · Virginia Tech, Department of Mechanical Engineering
Breaking the Bias Barrier in Concave Multi-Objective Reinforcement Learning
Mar 9, 2026Swetha Ganesh, Vaneet AggarwalNatural Policy Gradient Multi-Objective Reinforcement Learning
Purdue University, USA 47907
Partial Policy Gradients for RL in LLMs
Mar 6, 2026Puneet Mathur, Branislav Kveton, Subhojyoti Mukherjee +1RL for Language Models Policy Gradient Methods
Adobe Research.
Wasserstein Proximal Policy Gradient
Mar 3, 2026Zhaoyu Zhu, Shuhan Zhang, Rui Gao +1Policy Optimization Policy Gradient Methods
Zhiyuan College, Shanghai Jiao Tong University, Shanghai 200240, China · School of Data Science, The Chinese University of Hong Kong, Shenzhen, Guangdong, China · McCombs School of Business, The University of Texas at Austin, Austin, TX, USA
Rethinking Policy Diversity in Ensemble Policy Gradient in Large-Scale Reinforcement Learning
Mar 2, 2026Naoki Shitanda, Motoki Omura, Tatsuya Harada +1Policy Gradient Methods RL Exploration
The University of Tokyo, Tokyo, Japan · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan
GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control
Mar 2, 2026Haofeng Xu, Junwei Su, Yukun Tian +3RL Stability Asynchronous Advantage Actor-Critic
School of Computing and Data Science, The University of Hong Kong · Alibaba Group · Southeast University
Align and Filter: Improving Performance in Asynchronous On-Policy RL
Mar 2, 2026Homayoun Honari, Roger Creus Castanyer, Michael Przystupa +3Asynchronous RL Policy Gradient Methods
Mila – Quebec AI Institute, Canada · Université de Montréal, Canada · University of Alberta, Canada
Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
Mar 1, 2026Hongyi Zhou, Kai Ye, Erhan Xu +4U-Statistics Group Relative Policy Optimization
Department of Mathematics, Tsinghua University · Department of Statistics, London School of Economics and Political Science · School of Mathematics, University of Birmingham +2
Stabilizing Policy Optimization via Logits Convexity
Mar 1, 2026Hongzhan Chen, Tao Yang, Yuhua Zhu +3Policy Optimization Reinforcement Learning
School of Computer Science and Engineering, Sun Yat-sen University, China · Shanghai Innovation Institute · Wechat Search, Tencent Inc, China +1
ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning
Feb 25, 2026Xiaoxuan Wang, Han Zhang, Haixin Wang +11Agentic RL Reinforcement Learning
Department of Computer Science, University of California, Los Angeles, California, USA · University of Wisconsin–Madison, Madison, Wisconsin, USA
Recurrent Structural Policy Gradient for Partially Observable Mean Field Games
Feb 23, 2026Clarisse Wibault, Johannes Forkel, Sebastian Towers +9Mean Field Games Policy Gradient Methods
FLAIR, University of Oxford · MLRG, University of Oxford · ifo Institute, LMU Munich +5
Descent-Guided Policy Gradient for Scalable Cooperative Multi-Agent Learning
Feb 23, 2026Shan Yang, Yang LiuCooperative MARL Multi-Agent Reinforcement Learning
National University of Singapore
How to Train Your Deep Research Agent? Prompt, Reward, and Policy Optimization in Search-R1
Feb 23, 2026Yinuo Xu, Shuo Lu, Jianjie Cheng +5RL for LLM Agents Autonomous Research Agents
NLPR & MAIS, CASIA · School of AI, UCAS · Meituan Inc.
Not all tokens are needed(NAT): token efficient reinforcement learning
Feb 20, 2026Hejian Sang, Yuanda Xu, Zhengze Zhou +2RL for Language Models Reinforcement Learning
LinkedIn Corporation, CA, USA
Learning Distributed Equilibria in Linear-Quadratic Stochastic Differential Games: An $α$ -Potential Approach
Feb 18, 2026Philipp Plank, Yufei ZhangGame Theory Nash Equilibrium Computation
Department of Mathematics, Imperial College London, London, UK

Apr 20, 2026J/K move · Enter open · S save

COSAC: Counterfactual Credit Assignment in Sequential Cooperative Teams

Shripad Deshmukh, Jayakumar Subramanian, Raghavendra Addanki, Nikos Vlassis

College of Information and Computer Sciences University of Massachusetts Amherst · Adobe Research San Jose, CA

Sequential MARL Credit Assignment in RL Policy Gradient Methods Credit Assignment Cooperative MARL Counterfactual Credit Assignment+1

Open paper AskPDF arXiv

In cooperative teams where agents act in a fixed order and share a single team-level reward (multi-agent language systems, sequential robotic tasks), per-agent credit assignment is under-determined. Critic-based approaches scale poorly as the number of agents grows owing to the costly maintenance of joint/factored critic(s), whereas the existing critic-free alternatives have other issues: common credit across agents that couples every agent's signal to teammate noise, importance-sampling corrections for upstream-update staleness that incur variance exponential in team size, or per-agent counterfactual replay that isolates each agent's effect at the price of extra environment or reward calls. We propose COSAC, a critic-free per-agent policy gradient for sequential cooperative teams. COSAC fits an additive per-agent decomposition of the team reward by a single ridge regression on the rollout batch (giving each agent a learning signal decoupled from teammate noise), and computes each agent's counterfactual advantage from fictitious continuations of the current policy (policy forward passes that replace both importance-sampling reweighting and per-agent environment replay, at no extra environment or reward cost). The estimator instantiates the Sequential Aristocrat Utility (SeqAU), our extension of Wolpert and Tumer's (2001) aristocrat utility to sequential teams. We prove bias and variance bounds on SeqAU credits that stay controlled as the team grows. Our controlled study on sequential bandits demonstrates that COSAC attains the lowest advantage MSE and consistently low learning regret across team sizes up to

K = 16

. On the AI2 Reasoning Challenge (ARC) task, where four Qwen3-0.6B agents reason in turn about a grade-school science question, COSAC attains faster convergence than the other critic-free baselines.