Policy Gradient Methods

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, up 133% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 108

All topics

Sample Complexity of Policy Gradient for Log-Growth Control
May 26, 2026Qiuhua Pan, Yukai Shen, Liwei Zhang +2Stochastic Linear Dynamical Systems Stochastic Optimal Control
State Key Laboratory of Submarine Geoscience, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai 200240, China · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai 200240, China · Shanghai Key Laboratory of Perception and Control in Industrial Network Systems, Shanghai 200240, China +1
Credit-assigned Policy Gradient for Early Stage Retrieval in Two-stage Ranking
May 25, 2026Haruka Kiyohara, Mihaela Curmei, Ariel Evnine +7Learning to Rank Information Retrieval
Computer Science Department, Cornell University, Ithaca, NY, USA · Central Applied Science, Meta, Menlo Park, CA, USA.
Global Convergence of Wasserstein Policy Gradient for Entropy-Regularized Reinforcement Learning
May 25, 2026Zhaoyu Zhu, Rui Gao, Shuang LiOptimization Convergence Analysis Wasserstein Gradient Flows
Shanghai Jiao Tong University · The University of Texas at Austin · The Chinese University of Hong Kong, Shenzhen
Refined Analysis of Entropy-Regularized Actor-Critic
May 23, 2026Safwan Labbi, Paul Mangold, Daniil Tiapkin +1Reinforcement Learning Maximum Entropy RL
CMAP, CNRS, École Polytechnique, Institut Polytechnique de Paris, 91120 Palaiseau, France · Université Paris-Saclay, CNRS, LMO, 91405, Orsay, France · Google DeepMind +2
Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimization
May 20, 2026Zheyuan Zhang, Kaiwen Shi, Han Bao +3RL for Language Model Reasoning Policy Optimization
University of Notre Dame · †Corresponding Author
Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagation
May 18, 2026Mingfei SunPolicy Gradient Natural Gradient Methods
The University of Manchester, United Kingdom.
Equilibrium Selection in Multi-Agent Policy Gradients via Opponent-Aware Basin Entry
May 18, 2026Yevhen Shcherbinin, Arina Redina, Maxim Kalpin +1Nash Equilibrium Multi-Agent Coordination
Bloomsbury Technology, London, United Kingdom · London School of Economics and Political Science, London, United Kingdom · University of Bristol, Bristol, United Kingdom +2
Second-Order Actor-Critic Methods for Discounted MDPs via Policy Hessian Decomposition
May 14, 2026Sanjeev Manivannan, Shuban VReinforcement Learning Policy Gradient Methods
Indian Institute of Technology Madras Chennai, India
Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training
May 12, 2026Rasool Fakoor, Murdock Aubry, Nicholas Stranges +1Importance Sampling Policy Optimization
Boson AI
Delightful Gradients Accelerate Corner Escape
May 12, 2026Jincheng Mei, Ian OsbandMulti-Armed Bandits Policy Gradient
Google DeepMind
Revisiting Policy Gradients for Restricted Policy Classes: Escaping Myopic Local Optima with $k$ -step Policy Gradients
May 11, 2026Alex DeWeese, Guannan QuPolicy Gradient Policy Optimization
Department of Electrical and Computer Engineering Carnegie Mellon University Pittsburgh, PA 15213-3890, USA
Policy Gradient Methods for Non-Markovian Reinforcement Learning
May 11, 2026Avik Kar, Siddharth Chandak, Rahul Singh +4Reinforcement Learning Policy Gradient Methods
Department of Computer Science and Automation, Indian Institute of Science, Bengaluru, India. · Department of Electrical Engineering, Stanford University, Stanford, CA, USA. · Department of Electrical and Electronics Engineering, Nanyang Technological University, Singapore. +1
What should post-training optimize? A test-time scaling law perspective
May 11, 2026Muheng Li, Jian Qian, Wenlong MouTest-Time Scaling Best-of-N Sampling
Department of Statistical Sciences, University of Toronto† · Department of AI and Data Science, The University of Hong Kong ⋄
Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework
May 11, 2026Phalguni Nanda, Zaiwei ChenPolicy Gradient Reinforcement Learning
Edwardson School of Industrial Engineering, Purdue University
The Reciprocity Gradient
May 8, 2026Yue Lin, Pascal Poupart, Shuhui Zhu +5Credit Assignment in RL Policy Gradient Methods
The Chinese University of Hong Kong, Shenzhen · LIGHTSPEED · University of Waterloo +2
KL for a KL: On-Policy Distillation with Control Variate Baseline
May 8, 2026Minjae Oh, Sangjun Song, Gyubin Choi +2Language Model Distillation Policy Gradient Methods
Graduate School of Data Science Seoul National University
Rethinking Importance Sampling in LLM Policy Optimization: A Cumulative Token Perspective
May 8, 2026Yuheng Zhang, Chenlu Ye, Shuowei Jin +4RL for Language Model Reasoning Importance Sampling
UIUC · University of Michigan · Amazon
Actor-Critic with Active Importance Sampling
May 8, 2026Majid Molaei, Gabor Paczolay, Matteo Papini +2Reinforcement Learning Policy Gradient Methods
Politecnico di Milano Milan, Italy
Beyond Negative Rollouts: Positive-Only Policy Optimization with Implicit Negative Gradients
May 7, 2026Mingwei Xu, Hao FangReinforcement Learning RL for Language Model Reasoning
University of Washington, Seattle, USA
Skip What You Can Predict: Predictive Repositioning for Policy Optimization for Efficient LLM Training
May 7, 2026Ismam Nur Swapnil, Aranya Saha, Tasneea Zahra +3RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
Bangladesh University of Engineering and Technology · University of Maryland, College Park · Illinois Institute of Technology +1
Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level
May 7, 2026Nan Jia, Haojin Yang, Xing Ma +6RL for Language Model Reasoning Language Model Distillation
Huazhong University of Science and Technology · Peking University · Meituan
Soft Deterministic Policy Gradient with Gaussian Smoothing
May 7, 2026Hyunjun Na, Donghwan LeeRL Control Policy Gradient
School of Electrical Engineering, KAIST 291 Daehak-ro, Yuseong-gu, Daejeon 34141, Korea
A $^2$ TGPO: Agentic Turn-Group Policy Optimization with Adaptive Turn-level Clipping
May 7, 2026Dingwei Chen, Zefang Zong, Zhipeng Ma +5Credit Assignment in RL Policy Gradient Methods
Tencent Inc · The Chinese University of Hong Kong · Shenzhen MSU-BIT University
On Training in Imagination
May 7, 2026Nadav Timor, Ravid Shwartz-Ziv, Micah Goldblum +2Reward Modeling Reinforcement Learning
Weizmann Institute of Science · New York University · Columbia University
Non-Myopic Active Feature Acquisition via Pathwise Policy Gradients
May 6, 2026Linus Aronsson, Morteza Haghir ChehreghaniPolicy Gradient Active Learning
Department of Computer Science and Engineering · Chalmers University of Technology & University of Gothenburg · Gothenburg, Sweden
Vanishing L2 regularization for the softmax Multi Armed Bandit
May 5, 2026Stefana-Lucia Anita, Gabriel TuriniciMulti-Armed Bandits Policy Gradient
CEREMADE, Universit´e Paris Dauphine - PSL, CNRS, Paris, France · ‘Octav Mayer” Institute of Mathematics of the Romanian Academy, Bd. Carol I 8, Ias¸i 700505, Romania
ANO: Robust Policy Optimization via Bounded, Redescending Gain Fields
May 4, 2026Yiheng Zhang, Yiming Wang, Kaiyan Zhao +3Policy Gradient Methods Proximal Policy Optimization
University of Macau · Wuhan University · National University of Singapore +1
Global Optimality for Constrained Exploration via Penalty Regularization
Apr 30, 2026Florian Wolf, Ilyas Fatkhullin, Niao HeConstrained RL Maximum Entropy RL
The Computing & Mathematical Sciences Department, California Institute of Technology, Pasadena, CA. · Department of Computer Science, ETH Zurich, Switzerland. · 3ETH AI Center, ETH Zurich, Switzerland.
Cost-Aware Learning
Apr 30, 2026Clara Mohri, Amir Globerson, Haim Kaplan +2Stochastic Optimization RL for Language Models
Kempner Institute · Harvard University · Google Research +1
Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoning
Apr 30, 2026Shijin Gong, Kai Ye, Jin Zhu +3Value Function Estimation RL for Language Model Reasoning
School of Management, University of Science and Technology of China · Department of Statistics, London School of Economics and Political Science · School of Mathematics, University of Birmingham +2

May 26, 2026, eess.SYJ/K move · Enter open · S save

Sample Complexity of Policy Gradient for Log-Growth Control

Qiuhua Pan, Yukai Shen, Liwei Zhang, Cailian Chen+1

State Key Laboratory of Submarine Geoscience, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai 200240, China · Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai 200240, China · Shanghai Key Laboratory of Perception and Control in Industrial Network Systems, Shanghai 200240, China+1

Stochastic Linear Dynamical Systems Stochastic Optimal Control Policy Gradient Methods Optimal Control Sample-Efficient RL

Open paper AskPDF arXiv

We study the sample complexity of policy gradient for log-growth control -- the problem of learning, from observed state transitions, a feedback gain that optimally stabilizes a scalar linear system driven through a multiplicative-noise actuation channel. The objective

J(K) = \mathbb{E}[\log|1+BK|]

is the top Lyapunov exponent of the closed loop. This problem carries a structural difficulty we call the cusp obstruction: the optimal gain

K^*

always places the noise singularity

b_{\rm sing}(K) = -1/K

in the interior of the support. At this singular optimum the policy gradient exists only as a Cauchy principal value, not as a Lebesgue integral, and the natural single-sample gradient estimator has infinite variance. Standard first-order stochastic-optimization analysis is thus inapplicable at the optimum, and merely smoothing the objective does not resolve the difficulty. The obstruction, however, has an exploitable symmetry: the Cauchy kernel is an odd function of the displacement from the moving pole, so pairing each observation with its reflection through the pole cancels the divergent part. This one cancellation simultaneously controls the population curvature, the gradient-estimator variance, and the bias incurred when the noise density is estimated. Combining these bounds with a closed-form single-transition gradient oracle, we prove that projected mini-batch policy gradient, initialized in any compact subset of the stabilizing region, attains total sample complexity

\tilde{O}(1/η)

when the noise density is known and

\tilde{O}(η^{-(2s+1)/(2s)})

when it must be estimated, for

C^s

noise densities with

s \geq 2

Momentum

Related topics

Latest papers 108