Policy Optimization

Follow this topic Ask about this topic

Policy optimization studies how to update a decision-making rule so it earns higher rewards while meeting constraints such as safety, fairness, or limited data. Common approaches include policy gradients, actor-critic methods, trust regions and mirror descent, as well as policy iteration and newer generative-policy methods. Work spans continuous-control benchmarks such as MuJoCo, offline and multi-agent settings, and language-model training for mathematical reasoning, coding, and tool use, where token-, segment-, or trajectory-level rewards guide updates. A central challenge is balancing effective exploration and sample reuse with stable updates, useful credit assignment, and improvements that carry over from training to deployment.

Latest papers

224 papers32 papers in the last four weeks, up 52% on the four weeks before. 0.2% of all new papers.

Revisiting Policy Gradients for Restricted Policy Classes: Escaping Myopic Local Optima with $k$ -step Policy Gradients
May 11, 2026Alex DeWeese, Guannan QuPolicy Gradient Policy Optimization
Department of Electrical and Computer Engineering Carnegie Mellon University Pittsburgh, PA 15213-3890, USA
Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework
May 11, 2026Phalguni Nanda, Zaiwei ChenPolicy Gradient Reinforcement Learning
Edwardson School of Industrial Engineering, Purdue University
Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates
May 10, 2026Anish Diwan, Davide Tateo, Christopher E. Mower +3Reinforcement Learning Policy Optimization
Technical University of Darmstadt · Robotics Institute Germany (RIG) · Lund University +3
Rethinking Ratio-Based Trust Regions for Policy Optimization in Multi-Agent Reinforcement Learning
May 9, 2026Chulabhaya Wijesundara, Andrea Baisero, Zhongheng Li +3Multi-Agent System Optimization Policy Optimization
Northeastern University · STR · University of California, Irvine
Revisiting Mixture Policies in Entropy-Regularized Actor-Critic
May 9, 2026Jiamin He, Samuel Neumann, Jincheng Mei +2RL Control Policy Optimization
University of Alberta & Amii · Google DeepMind · Canada CIFAR AI Chair
A Single Deep Preference-Conditioned Policy for Learning Pareto Coverage Sets
May 9, 2026Akihiro Kubo, Kosuke Nakanishi, Shin IshiiMulti-Objective Reinforcement Learning Policy Optimization
International Research Center for Neurointelligence, The University of Tokyo, Tokyo, Japan · ATR Neural Information Analysis Laboratories, Kyoto, Japan · Department of Information Science, Kyoto University, Kyoto, Japan
Generative Actor-Critic with Soft Bridge Policies
May 9, 2026Ke He, Le He, Shunpu Tang +2Policy Optimization Maximum Entropy RL
Guangzhou University · Zhejiang University · Southeast University
Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria
May 8, 2026Juanxi Tian, Fengyuan Liu, Jiaming Han +6Policy Optimization Multimodal Reward Modeling
1Nanyang Technological University · 2Ant Group · 3MMLab, The Chinese University of Hong Kong +1
Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias
May 8, 2026Rahaf Abu Hara, Vaibbhav Murarri, Claudio ZitoPolicy Optimization LLM-Guided RL
Laboratory of AI and Robotics Research (LAIRR) Heriot-Watt University Dubai, United Arab Emirates
Drifting Field Policy: A One-Step Generative Policy via Wasserstein Gradient Flow
May 8, 2026Juil Koo, Mingue Park, Jiwon Choi +2Robot Policy Learning Wasserstein Gradient Flows
KAIST
Model-Driven Policy Optimization in Differentiable Simulators via Stochastic Exploration
May 8, 2026Yuval Aroosh, Ayal TaitlerPolicy Optimization RL Exploration
Department of Industrial Engineering and Management Ben-Gurion University of the Negev Beer Sheva, Israel
SNAPO: Smooth Neural Adjoint Policy Optimization for Optimal Control via Differentiable Simulation
May 7, 2026Dmitri Goloubentsev, Natalija KarpichinaPolicy Optimization Differentiable Programming
MatLogica, U.K.
Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex
May 7, 2026Yun Qu, Qi Wang, Yixiu Mao +11RL for Language Model Reasoning Group Relative Policy Optimization
Department of Automation, Tsinghua University · 2LLM Department, Tencent
AGPO: Asymmetric Group Policy Optimization for Verifiable Reasoning and Search Ads Relevance at JD
May 7, 2026Yang Xu, Kun Yao, Yiming Deng +3RL for Language Model Reasoning Policy Optimization
Nanjing University, Nanjing, China · Peking University, Beijing, China · JD.com, Beijing, China
Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement
May 7, 2026Haodong Liang, Lifeng LaiReinforcement Learning Transformer-Based RL
Department of Electrical and Computer Engineering, University of California, Davis
Approximate Next Policy Sampling: Replacing Conservative Target Policy Updates in Deep RL
May 6, 2026Dillon Sandhu, Ronald ParrPolicy Optimization Policy Iteration
Department of Computer Science, Duke University, USA
Vanishing L2 regularization for the softmax Multi Armed Bandit
May 5, 2026Stefana-Lucia Anita, Gabriel TuriniciMulti-Armed Bandits Policy Gradient
CEREMADE, Universit´e Paris Dauphine - PSL, CNRS, Paris, France · ‘Octav Mayer” Institute of Mathematics of the Romanian Academy, Bd. Carol I 8, Ias¸i 700505, Romania
Reference-Sampled Boltzmann Projection for KL-Regularized RLVR: Target-Matched Weighted SFT, Finite One-Shot Gaps, and Policy Mirror Descent
May 4, 2026Yao Shu, Chenxing Wei, Hongbin Lin +2Supervised Fine-Tuning Reinforcement Learning
Segment-Aligned Policy Optimization for Multi-Modal Reasoning
May 2, 2026Lei Gao, Zhuoming Li, Mengxi Jia +4RL for Language Model Reasoning Policy Optimization
Fudan University · China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd. · Southeast University +1
SAVGO: Learning State-Action Value Geometry with Cosine Similarity for Continuous Control
May 1, 2026Stavros Orfanoudakis, Pedro P. VergaraRepresentation Learning Policy Optimization
Delft University of Technology, The Netherlands
SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
Apr 26, 2026Alexis Limozin, Eduard Durech, Torsten Hoefler +2RL for Language Model Reasoning Policy Optimization
ETH AI Center, ETH Zürich · EPFL · Allen Institute for AI
Tempered Sequential Monte Carlo for Trajectory and Policy Optimization with Differentiable Dynamics
Apr 23, 2026Heng YangMarkov Chain Monte Carlo Trajectory Optimization
Harvard University
Bounded Ratio Reinforcement Learning
Apr 20, 2026Yunke Ao, Le Chen, Bruce D. Lee +5Reinforcement Learning Group Relative Policy Optimization
ETH Zurich · MPI for Intelligent Systems · University of Alberta +2
StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning
Apr 20, 2026Daoyu Wang, Qingchuan Li, Mingyue Cheng +4Agentic RL Credit Assignment in RL
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China
OmniVLA-RL: A Vision-Language-Action Model with Spatial Understanding and Online RL
Apr 20, 2026Haoxiang Jie, Yaoyuan Yan, Xiangyu Wei +4Flow Matching Policy Optimization
1AI Lab, Country Garden Services · 3VBot · 2Omni AI +1
Beyond Importance Sampling: Rejection-Gated Policy Optimization
Apr 16, 2026Ziwu Sun, Zhen Gao, Jiyong Zhang +1Importance Sampling Policy Optimization
Wasserstein Formulation of Reinforcement Learning. An Optimal Transport Perspective on Policy Optimization
Apr 16, 2026Mathias DusReinforcement Learning Wasserstein Gradient Flows
IRMA
Positive-Only Drifting Policy Optimization
Apr 15, 2026Qi ZhangPolicy Optimization RL for Generative Models
Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood
Apr 14, 2026Xingyu Lin, Yilin Wen, Du Su +5RL for Language Model Reasoning Policy Optimization
College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University · Baidu Inc. +2
Olfactory pursuit: catching a moving odor source in complex flows
Apr 13, 2026Maurizio Carbone, Lorenzo Piro, Robin A. Heinonen +3Policy Optimization Partially Observable Markov Decision Processes
Istituto dei Sistemi Complessi, CNR, Via dei Taurini 19, 00185 Rome, Italy and INFN “Tor Vergata", Via della Ricerca Scientifica 1, 00133 Rome, Italy · Department of Physics & INFN, Tor Vergata University of Rome, Via della Ricerca Scientifica 1, 00133 Rome, Italy · Machine Learning Genoa Center (MaLGa) & Department of Civil, Chemical and Environmental Engineering, University of Genoa, Genoa, Italy +2

May 11, 2026J/K move · Enter open · S save

Revisiting Policy Gradients for Restricted Policy Classes: Escaping Myopic Local Optima with $k$ -step Policy Gradients

Alex DeWeese, Guannan Qu

Department of Electrical and Computer Engineering Carnegie Mellon University Pittsburgh, PA 15213-3890, USA

Policy Gradient Policy Optimization Policy Gradient Methods Policy Learning

Open paper AskPDF arXiv

This work revisits standard policy gradient methods used on restricted policy classes, which are known to get stuck in suboptimal critical points. We identify an important cause for this phenomenon to be that the policy gradient is itself fundamentally myopic, i.e. it only improves the policy based on the one-step

Q

-function. In this work, we propose a generalized

k

-step policy gradient method that couples the randomness within a

k

-step time window and can escape the myopic local optima in MDPs with restricted policy classes. We show this new method is theoretically guaranteed to converge to a solution that is exponentially close in performance to the optimal deterministic policy with respect to

k

. Further, we show projected gradient descent and mirror descent with this

k

-step policy gradient can achieve this exponential guarantee in

O(\frac{1}{T})

iterations, despite only assuming smoothness and differentiability of the value function. This will provide near optimal solutions to previously elusive applications like state aggregation and partially observable cooperative multi-agent settings. Moreover, our bounds avoid the ubiquitous distribution mismatch factors

||d_μ^{π^*} / d_μ^π||_\infty

and

||d_μ^{π^*} / μ||_\infty

enabling the

k

-step policy gradient method to escape suboptimal critical points that emerge from poor exploration in fully observable settings.