Policy Gradient Methods

Follow this topic Ask about this topic

They estimate gradients from sampled trajectories using score-function methods such as REINFORCE, actor–critic value baselines, natural gradients, or pathwise differentiation through models. PPO-style clipping and trust regions constrain updates, while variance reduction and finer-grained credit assignment aim to use fewer samples. These methods are studied in bandits, robotics, continuous control, multi-agent and safe decision-making, and language-model post-training, where rewards can be sparse and trajectories long. A central tension is improving sample efficiency and stability without biasing updates or losing exploration; convergence guarantees exist in restricted settings, while high-dimensional and long-horizon problems remain challenging.

Latest papers

165 papers25 papers in the last four weeks, up 178% on the four weeks before. 0.1% of all new papers.

Reinforcement learning with an expectile-based objective
Feb 10, 2026Shrey Rakeshkumar Patel, Sumedh Gupte, Soumen Pachal +2Reinforcement Learning Risk-Sensitive RL
Indian Institute of Technology Madras · TCS Research, India
The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL
Feb 6, 2026Yingru Li, Jiawei Xu, Ziniu Li +10RL for Language Model Reasoning Policy Gradient Methods
The Chinese University of Hong Kong, Shenzhen · Fudan University · Hong Kong University of Science and Technology +4
Rethinking the Design Space of Reinforcement Learning for Diffusion Models: On the Importance of Likelihood Estimation Beyond Loss Design
Feb 4, 2026Jaemoo Choi, Yuchen Zhu, Wei Guo +6T2I Generation Diffusion Model Fine-Tuning
Georgia Institute of Technology · National University of Singapore · Nanjing university.
Model-Free Output Feedback Stabilization via Policy Gradient Methods
Jan 27, 2026Ankang Zhang, Ming Chi, Xiaoling Wang +1Dynamical Systems Partially Observable RL
School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China · College of Automation, Nanjing University of Posts and Telecommunications, Nanjing 210023, China
Trust Region Masking for Long-Horizon LLM Reinforcement Learning
Dec 28, 2025Yingru Li, Jiacai Liu, Jiawei Xu +4RL for Language Models Reinforcement Learning
Fudan University · The Chinese University of Hong Kong, Shenzhen
Understanding Negative Learning in Large Language Model Fine-tuning
Nov 30, 2025Yue Wang, Qizhou Wang, Zizhuo Zhang +3LLM Fine-Tuning RL Fine-Tuning
TMLR Group, Department of Computer Science, Hong Kong Baptist University · RIKEN AIP · The University of Tokyo
Context Value Informed In-Context Reinforcement Learning
Sep 27, 2025Wenhao Zhang, Shao Zhang, Xihuai Wang +2In-Context RL Test-Time RL
Shanghai Jiao Tong University
On the Gradient Domination of the LQG Problem
Jul 11, 2025Kasra Fallah, Leonardo F. Toso, James AndersonPolicy Gradient Methods Stochastic Optimal Control
The authors are with the Department of Electrical Engineering at Columbia University, New York, USA.
Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation: The Case of Multi-Armed Bandits
May 6, 2025Max Qiushi Lin, Jincheng Mei, Matin Aghaei +6Multi-Armed Bandits Policy Gradient
Simon Fraser University · Google DeepMind · Google Research +1
GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
Apr 28, 2025Kaichen Zhang, Yuzhong Hong, Junwei Bao +4LLM Post-Training LLM Alignment
Thrust of Artificial Intelligence, Hong Kong University of Science and Technology (Guangzhou) · Zuoyebang Education Technology · Department of Computer Science and Engineering, HKUST
Analysis of On-policy Policy Gradient Methods under the Distribution Mismatch
Mar 28, 2025Weizhen Wang, Xinzhe Zhou, Jianping He +1Policy Gradient Policy Gradient Methods
School of Automation and Intelligent Sensing, Shanghai Jiao Tong University · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai, China
The Price of Failure Risk: Clipped Barrier Policy Gradients with Convex Regularization
Nov 28, 2024Pekka Malo, Lauri Viitasaari, Antti Suominen +2Policy Gradient Methods Constrained RL
Aalto University, Department of Information and Service Management, Finland. · University of Helsinki, Department of Forest Sciences, Finland.
Reflective Policy Optimization
Jun 6, 2024Yaozhong Gan, Renye Yan, Zhe Wu +1Policy Optimization Policy Gradient Methods
QiYuan Lab.
On the Approximation and Convergence of Distributional Policy Gradient Algorithms for Risk-Sensitive Reinforcement Learning
May 23, 2024Xian Yu, Minheng Xiao, Lei YingStochastic Optimization Distributional RL
Department of Integrated Systems Engineering, The Ohio State University, Columbus, OH, USA · Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI, USA
Is the Policy Gradient a Gradient?
Jun 17, 2019Chris Nota, Philip S. ThomasPolicy Gradient Policy Gradient Methods
College of Information and Computer Sciences University of Massachusetts Amherst

Feb 10, 2026J/K move · Enter open · S save

Reinforcement learning with an expectile-based objective

Shrey Rakeshkumar Patel, Sumedh Gupte, Soumen Pachal, Prashanth L. A.+1

Indian Institute of Technology Madras · TCS Research, India

Reinforcement Learning Risk-Sensitive RL Policy Gradient Methods

Open paper AskPDF arXiv

We consider the policy evaluation and control in a finite horizon reinforcement learning (RL) setting under an expectile-based objective. First, we derive the mean-squared error (MSE) and concentration bounds for the classic estimator of expectiles based on independent and identically distributed (i.i.d.) samples. To the best of our knowledge, expectiles have not been analyzed in the non-asymptotic regime and the bounds we derive may be of independent interest. Next, we analyze a Monte Carlo type estimator of expectile of the Markov chain underlying a given policy. We derive upper bounds that hold in expectation as well as with high probability for this estimator. Further, we show the order-optimality of our estimator by deriving a lower bound on expectile-based policy evaluation. For the problem of control, we adopt a policy gradient approach and derive a policy gradient theorem for expectiles. Using this result, we propose a gradient estimator with a

O\left(1/m\right)

mean-squared error bounds, where

m

is the number of trajectories. Further, under standard assumptions for policy gradient-type algorithms, we establish smoothness of the expectile-sensitive objective, in turn leading to stationary convergence rate bounds for the overall risk-sensitive policy gradient algorithm that we propose. Finally, we conduct numerical experiments to show the utility of expectiles on popular RL benchmarks.