Policy Gradient Methods

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, up 133% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 108

All topics

EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games
Jun 22, 2026Tristan Maidment, JB Lanier, Chase McDonald +5Self-Play RL Imperfect-Information Games
1Riot Games · 3New York University · University of California, Irvine
KLip-PPO: A per-sample KL perspective on PPO-Clip
Jun 22, 2026Riccardo Colletti, Robin HolzingerKL-Regularized RL Policy Gradient Methods
University of California, Berkeley
Drowning in Routine: Signal Dilution in Multi-Turn Agent Training
Jun 20, 2026Yann Pernot, Vi RetaultReinforcement Learning Credit Assignment in RL
Mila - Québec AI Institute · Polytechnique Montréal
Backpropagating Through Simulation: Analytic Policy Gradients for Sample and Learning Efficient Differentiable Continuous Control
Jun 19, 2026Yueci DengRobotic RL Policy Gradient Methods
School of Data Science, The Chinese University of Hong Kong, Shenzhen
STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability
Jun 17, 2026Haipeng Luo, Qingfeng Sun, Songli Wu +4RL for Language Model Reasoning Maximum Entropy RL
Shenzhen International Graduate School, Tsinghua University · Tencent Hunyuan
GD $^2$ PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimization
Jun 15, 2026Haotian Liu, Yihao Liu, Jingwei Ni +11Multi-Objective Reinforcement Learning RL for Language Models
Alibaba · Renmin University of China · Peking University +3
A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions
Jun 15, 2026Jianghan Shen, Siqi Luo, Yue Li +9RL for Language Models Group Relative Policy Optimization
1Nanjing University · 2Shanghai Artificial Intelligence Laboratory · 3Shanghai Jiao Tong University +4
A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization
Jun 15, 2026Prasanth YSS, Zhichen Ren, Rasa Hosseinzadeh +6RL for Language Model Reasoning Policy Optimization
Layer 6 AI
Proximal Policy Optimization for Amortized Discrete Sampling
Jun 14, 2026Anna Zykova-Myzina, Timofei Gritsaev, Daniil Tiapkin +1Molecular Graph Generation Generative Flow Networks
HSE University · Constructor University · CMAP, CNRS, École polytechnique, IPP
DiPOD: Diffusion Policy Optimization without Drifting Apart
Jun 11, 2026Haozhe Jiang, Haiwen Feng, Pieter Abbeel +3Policy Gradient Methods RL for Diffusion Models
UC Berkeley · NVIDIA
The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning
Jun 6, 2026Zhanke Zhou, Xiangyu Lu, Chentao Cao +4RL for Language Model Reasoning Policy Optimization
1TMLR Group, Department of Computer Science, Hong Kong Baptist University · 2Stanford University · 3Sydney AI Centre, The University of Sydney
Generative Modeling of Discrete Latent Structures via Dynamic Policy Gradients
Jun 5, 2026Stefan Ivanovic, Ge Liu, Mohammed El-KebirLatent Variable Models Policy Gradient Methods
Siebel School of Computing and Data Science, University of Illinois at Urbana-Champaign, IL 61801, USA · Cancer Center at Illinois, University of Illinois Urbana-Champaign, IL 61801, USA
GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios
Jun 5, 2026Ke Hu, Shutong Ding, Panxin Tao +2RL Control Policy Gradient Methods
ShanghaiTech University
OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation
Jun 4, 2026Paavo Parmas, Yongmin Kim, Kohsei Matsutani +5Stochastic Optimization Risk-Sensitive RL
The University of Tokyo
On Advantage Estimates for Max@K Policy Gradients
Jun 4, 2026Shota Takashiro, Soichiro Nishimori, Paavo Parmas +6Policy Gradient Policy Gradient Methods
The University of Tokyo
Retry Policy Gradients in Continuous Action Spaces
Jun 4, 2026Soichiro Nishimori, Paavo ParmasPolicy Optimization RL Exploration
The University of Tokyo, Japan
When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training
Jun 4, 2026Yuanfan Li, Qi Zhou, Wenjing Duan +1Credit Assignment in RL Step-Level Credit Assignment in RL
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · Faculty of Electronic and Information Engineering, Xi’an Jiaotong University
Policy Gradient for Continuous-Time Robust Markov Decision Processes
Jun 3, 2026Tanya Veeravalli, David M. Bossens, Atsushi NitandaRobust Markov Decision Processes Markov Decision Processes
Centre for Frontier AI Research, Agency for Science, Technology and Research (A*STAR) · Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR) · College of Computing and Data Science, Nanyang Technological University
Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions
Jun 2, 2026Bingxu Liu, Jiashun Liu, Johan Obando-Ceron +5Non-Stationary RL Reinforcement Learning
Hong Kong University of Science and Technology · Fudan University · Mila - Québec AI Institute +2
Self-Distilled Policy Gradient
Jun 2, 2026Yifeng Liu, Shiyuan Zhang, Yifan Zhang +1Policy Gradient On-Policy Self-Distillation
Department of Computer Science, University of California, Los Angeles, CA, USA · Princeton AI Laboratory, Princeton University, Princeton, NJ, USA
Task-Induced Representational Invariances Depend on Learning Objective in Deep RL
Jun 1, 2026Manu Srinath Halvagal, Sebastian Lee, SueYeon ChungMarkov Decision Processes Deep Q-Learning
Department of Physics, Harvard University · 2Kempner Institute, Harvard University · Center for Computational Neuroscience, Flatiron Institute
Decision-Focused On-Policy Learning for Contextual Linear Optimization with Partial Feedback
May 31, 2026Wyame Benslimane, Tinghan Ye, Pascal Van Hentenryck +1Predict-Then-Optimize Decision-Focused Learning
Department of Industrial Engineering and Operations Research University of California, Berkeley · H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology
OPD+: Rethinking the Advantage Design for On-Policy Distillation
May 31, 2026Hanyang Zhao, Haoxian Chen, Han Lin +3Language Model Distillation Policy Gradient Methods
Columbia University · Amazon · Meta +1
Zero Collapse: A Failure Mode of Policy Gradient Methods in Discontinuous Reward Environments
May 29, 2026Nishant Kumar, Enrique Areyan Viqueira, Amy GreenwaldReinforcement Learning Policy Gradient Methods
Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying
May 29, 2026Soichiro Nishimori, Paavo Parmas, Sotetsu Koyamada +4RL Exploration Policy Gradient Methods
The University of Tokyo, Tokyo, Japan · Kobe University, Kobe, Japan · Kyoto University, Kyoto, Japan +3
Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization
May 28, 2026Shufan Li, Konstantinos Kallidromitis, Akash Gokul +2Token-Level Credit Assignment RL for Language Model Reasoning
UCLA · Panasonic AI Research · NVIDIA
Modeling Vehicle-Type-Specific Pedestrian Crash Avoidance Behavior in Safety-Critical Interactions Using Smooth-Mamba Deep Reinforcement Learning
May 27, 2026Qingwen Pu, Kun Xie, Hong Yang +2Collision Avoidance Pedestrian Behavior Modeling
Transportation Informatics Lab, Department of Civil and Environmental Engineering, Old Dominion University, Norfolk, VA 23529, United States · Department of Electrical and Computer Engineering, Old Dominion University, Norfolk, VA, United States · Department of Transportation and Urban Infrastructure Studies, SMARTER Center, Morgan State University, 1700 E Cold Spring Ln, Baltimore, MD 21251, USA
ProRL: Effective Reinforcement Learning for Proactive Recommendation via Rectified Policy Gradient Estimation
May 27, 2026Hongru Hou, Tiehua Mei, Denghui Geng +5Policy Gradient Methods Recommender Systems
BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
May 26, 2026Shijin Gong, Erhan Xu, Kai Ye +3RL for Language Model Reasoning Critic-Free RL
School of Management, University of Science and Technology of China · Department of Statistics, London School of Economics and Political Science · Department of Engineering Science, University of Oxford
Bilevel Optimization over Saddle Points of Zero-Sum Markov Games
May 26, 2026Zihao Zheng, Irwin King, Songtao LuGame Theory Zero-Sum Games
Department of Computer Science and Engineering, The Chinese University of Hong Kong · Shun Hing Institute of Advanced Engineering, The Chinese University of Hong Kong

Jun 22, 2026, cs.LGJ/K move · Enter open · S save

EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Games

Tristan Maidment, JB Lanier, Chase McDonald, Nathan Tsang+4

1Riot Games · 3New York University · University of California, Irvine

Self-Play RL Imperfect-Information Games Zero-Sum Games Policy Gradient Methods Proximal Policy Optimization Policy Learning

Open paper AskPDF arXiv

Recent work has established that regularized policy gradient methods such as PPO, when used in self-play, can match or exceed specialized game-theoretic algorithms for solving two-player zero-sum imperfect-information games. The uniform distribution has emerged as a strong policy regularization target for this purpose, but it regularizes equally toward all actions regardless of their viability. We introduce EMAgnet, which instead regularizes toward an exponential moving average (EMA) of the last-iterate policy's parameters, providing an adaptive regularization target that evolves with the agent's improving strategy. We evaluate EMAgnet on both standard two-player zero-sum benchmarks and modified benchmarks with exploration challenges and large numbers of strictly dominated strategies. Relative to PPO self-play with uniform-magnet regularization under both linear and power-law annealing schedules, EMAgnet achieves lower exploitability in the majority of tested environments, with consistent performance gains across games containing strictly dominated strategies.

Momentum

Related topics

Latest papers 108