Policy Optimization

Follow this topic Ask about this topic

Policy optimization studies how to update a decision-making rule so it earns higher rewards while meeting constraints such as safety, fairness, or limited data. Common approaches include policy gradients, actor-critic methods, trust regions and mirror descent, as well as policy iteration and newer generative-policy methods. Work spans continuous-control benchmarks such as MuJoCo, offline and multi-agent settings, and language-model training for mathematical reasoning, coding, and tool use, where token-, segment-, or trajectory-level rewards guide updates. A central challenge is balancing effective exploration and sample reuse with stable updates, useful credit assignment, and improvements that carry over from training to deployment.

Latest papers

221 papers32 papers in the last four weeks, up 52% on the four weeks before. 0.2% of all new papers.

Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates
May 10, 2026Anish Diwan, Davide Tateo, Christopher E. Mower +3Reinforcement Learning Policy Optimization
Technical University of Darmstadt · Robotics Institute Germany (RIG) · Lund University +3
Rethinking Ratio-Based Trust Regions for Policy Optimization in Multi-Agent Reinforcement Learning
May 9, 2026Chulabhaya Wijesundara, Andrea Baisero, Zhongheng Li +3Multi-Agent System Optimization Policy Optimization
Northeastern University · STR · University of California, Irvine
Revisiting Mixture Policies in Entropy-Regularized Actor-Critic
May 9, 2026Jiamin He, Samuel Neumann, Jincheng Mei +2RL Control Policy Optimization
University of Alberta & Amii · Google DeepMind · Canada CIFAR AI Chair
A Single Deep Preference-Conditioned Policy for Learning Pareto Coverage Sets
May 9, 2026Akihiro Kubo, Kosuke Nakanishi, Shin IshiiMulti-Objective Reinforcement Learning Policy Optimization
International Research Center for Neurointelligence, The University of Tokyo, Tokyo, Japan · ATR Neural Information Analysis Laboratories, Kyoto, Japan · Department of Information Science, Kyoto University, Kyoto, Japan
Generative Actor-Critic with Soft Bridge Policies
May 9, 2026Ke He, Le He, Shunpu Tang +2Policy Optimization Maximum Entropy RL
Guangzhou University · Zhejiang University · Southeast University
Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria
May 8, 2026Juanxi Tian, Fengyuan Liu, Jiaming Han +6Policy Optimization Multimodal Reward Modeling
1Nanyang Technological University · 2Ant Group · 3MMLab, The Chinese University of Hong Kong +1
Reflective Prompted Policy Optimization: Trajectory-Grounded Revision and Salience Bias
May 8, 2026Rahaf Abu Hara, Vaibbhav Murarri, Claudio ZitoPolicy Optimization LLM-Guided RL
Laboratory of AI and Robotics Research (LAIRR) Heriot-Watt University Dubai, United Arab Emirates
Drifting Field Policy: A One-Step Generative Policy via Wasserstein Gradient Flow
May 8, 2026Juil Koo, Mingue Park, Jiwon Choi +2Robot Policy Learning Wasserstein Gradient Flows
KAIST
Model-Driven Policy Optimization in Differentiable Simulators via Stochastic Exploration
May 8, 2026Yuval Aroosh, Ayal TaitlerPolicy Optimization RL Exploration
Department of Industrial Engineering and Management Ben-Gurion University of the Negev Beer Sheva, Israel
SNAPO: Smooth Neural Adjoint Policy Optimization for Optimal Control via Differentiable Simulation
May 7, 2026Dmitri Goloubentsev, Natalija KarpichinaPolicy Optimization Differentiable Programming
MatLogica, U.K.
Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex
May 7, 2026Yun Qu, Qi Wang, Yixiu Mao +11RL for Language Model Reasoning Group Relative Policy Optimization
Department of Automation, Tsinghua University · 2LLM Department, Tencent
AGPO: Asymmetric Group Policy Optimization for Verifiable Reasoning and Search Ads Relevance at JD
May 7, 2026Yang Xu, Kun Yao, Yiming Deng +3RL for Language Model Reasoning Policy Optimization
Nanjing University, Nanjing, China · Peking University, Beijing, China · JD.com, Beijing, China
Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement
May 7, 2026Haodong Liang, Lifeng LaiReinforcement Learning Transformer-Based RL
Department of Electrical and Computer Engineering, University of California, Davis
Approximate Next Policy Sampling: Replacing Conservative Target Policy Updates in Deep RL
May 6, 2026Dillon Sandhu, Ronald ParrPolicy Optimization Policy Iteration
Department of Computer Science, Duke University, USA
Vanishing L2 regularization for the softmax Multi Armed Bandit
May 5, 2026Stefana-Lucia Anita, Gabriel TuriniciMulti-Armed Bandits Policy Gradient
CEREMADE, Universit´e Paris Dauphine - PSL, CNRS, Paris, France · ‘Octav Mayer” Institute of Mathematics of the Romanian Academy, Bd. Carol I 8, Ias¸i 700505, Romania
Reference-Sampled Boltzmann Projection for KL-Regularized RLVR: Target-Matched Weighted SFT, Finite One-Shot Gaps, and Policy Mirror Descent
May 4, 2026Yao Shu, Chenxing Wei, Hongbin Lin +2Supervised Fine-Tuning Reinforcement Learning
Segment-Aligned Policy Optimization for Multi-Modal Reasoning
May 2, 2026Lei Gao, Zhuoming Li, Mengxi Jia +4RL for Language Model Reasoning Policy Optimization
Fudan University · China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd. · Southeast University +1
SAVGO: Learning State-Action Value Geometry with Cosine Similarity for Continuous Control
May 1, 2026Stavros Orfanoudakis, Pedro P. VergaraRepresentation Learning Policy Optimization
Delft University of Technology, The Netherlands
SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
Apr 26, 2026Alexis Limozin, Eduard Durech, Torsten Hoefler +2RL for Language Model Reasoning Policy Optimization
ETH AI Center, ETH Zürich · EPFL · Allen Institute for AI
Tempered Sequential Monte Carlo for Trajectory and Policy Optimization with Differentiable Dynamics
Apr 23, 2026Heng YangMarkov Chain Monte Carlo Trajectory Optimization
Harvard University
Bounded Ratio Reinforcement Learning
Apr 20, 2026Yunke Ao, Le Chen, Bruce D. Lee +5Reinforcement Learning Group Relative Policy Optimization
ETH Zurich · MPI for Intelligent Systems · University of Alberta +2
StepPO: Step-Aligned Policy Optimization for Agentic Reinforcement Learning
Apr 20, 2026Daoyu Wang, Qingchuan Li, Mingyue Cheng +4Agentic RL Credit Assignment in RL
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China
OmniVLA-RL: A Vision-Language-Action Model with Spatial Understanding and Online RL
Apr 20, 2026Haoxiang Jie, Yaoyuan Yan, Xiangyu Wei +4Flow Matching Policy Optimization
1AI Lab, Country Garden Services · 3VBot · 2Omni AI +1
Beyond Importance Sampling: Rejection-Gated Policy Optimization
Apr 16, 2026Ziwu Sun, Zhen Gao, Jiyong Zhang +1Importance Sampling Policy Optimization
Wasserstein Formulation of Reinforcement Learning. An Optimal Transport Perspective on Policy Optimization
Apr 16, 2026Mathias DusReinforcement Learning Wasserstein Gradient Flows
IRMA
Positive-Only Drifting Policy Optimization
Apr 15, 2026Qi ZhangPolicy Optimization RL for Generative Models
Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood
Apr 14, 2026Xingyu Lin, Yilin Wen, Du Su +5RL for Language Model Reasoning Policy Optimization
College of Computer Science and Technology, Jilin University · Key Laboratory of Symbolic Computation and Knowledge Engineering of MOE, Jilin University · Baidu Inc. +2
Olfactory pursuit: catching a moving odor source in complex flows
Apr 13, 2026Maurizio Carbone, Lorenzo Piro, Robin A. Heinonen +3Policy Optimization Partially Observable Markov Decision Processes
Istituto dei Sistemi Complessi, CNR, Via dei Taurini 19, 00185 Rome, Italy and INFN “Tor Vergata", Via della Ricerca Scientifica 1, 00133 Rome, Italy · Department of Physics & INFN, Tor Vergata University of Rome, Via della Ricerca Scientifica 1, 00133 Rome, Italy · Machine Learning Genoa Center (MaLGa) & Department of Civil, Chemical and Environmental Engineering, University of Genoa, Genoa, Italy +2
DDO-RM: Distribution-Level Policy Improvement after Reward Learning
Apr 13, 2026Tiantian Zhang, Jierui Zuo, Michael Chen +1Policy Optimization KL-Regularized RL
Department of Computer Science Columbia University, New York, NY 10027, USA · Department of Management Science and Engineering Tsinghua University, Beijing, China · Department of Computer Science Carnegie Mellon University, Pittsburgh, PA 15213, USA
Adapt to Thrive! Adaptive Power-Mean Policy Optimization for Improved LLM Reasoning
Apr 11, 2026Yiming Huang, Zhenbo Shi, Shuzheng Gao +3RL for Language Model Reasoning Policy Optimization
Harbin Institute of Technology, Shenzhen · The Chinese University of Hong Kong · Peng Cheng Laboratory

May 10, 2026J/K move · Enter open · S save

Trust Region Inverse Reinforcement Learning: Explicit Dual Ascent using Local Policy Updates

Anish Diwan, Davide Tateo, Christopher E. Mower, Haitham Bou-Ammar+2

Technical University of Darmstadt · Robotics Institute Germany (RIG) · Lund University+3

Reinforcement Learning Policy Optimization Imitation Learning Trust Region Policy Optimization Policy Learning

Open paper AskPDF arXiv

Inverse reinforcement learning (IRL) is typically formulated as maximizing entropy subject to matching the distribution of expert trajectories. Classical (dual-ascent) IRL guarantees monotonic performance improvement but requires fully solving an RL problem each iteration to compute dual gradients. More recent adversarial methods avoid this cost at the expense of stability and monotonic dual improvement, by directly optimizing the primal problem and using a discriminator to provide rewards. In this work, we bridge the gap between these approaches by enabling monotonic improvement of the reward function and policy without having to fully solve an RL problem at every iteration. Our key theoretical insight is that a trust-region-optimal policy for a reward function update can be globally optimal for a smaller update in the same direction. This smaller update allows us to explicitly optimize the dual objective while only relying on a local search around the current policy. In doing so, our approach avoids the training instabilities of adversarial methods, offers monotonic performance improvement, and learns a reward function in the traditional sense of IRL--one that can be globally optimized to match expert demonstrations. Our proposed algorithm, Trust Region Inverse Reinforcement Learning (TRIRL), outperforms state-of-the-art imitation learning methods across multiple challenging tasks by a factor of 2.4x in terms of aggregate inter-quartile mean, while recovering reward functions that generalize to system dynamics shifts.