Policy Optimization

Follow this topic Ask about this topic

Policy optimization studies how to update a decision-making rule so it earns higher rewards while meeting constraints such as safety, fairness, or limited data. Common approaches include policy gradients, actor-critic methods, trust regions and mirror descent, as well as policy iteration and newer generative-policy methods. Work spans continuous-control benchmarks such as MuJoCo, offline and multi-agent settings, and language-model training for mathematical reasoning, coding, and tool use, where token-, segment-, or trajectory-level rewards guide updates. A central challenge is balancing effective exploration and sample reuse with stable updates, useful credit assignment, and improvements that carry over from training to deployment.

Latest papers

221 papers33 papers in the last four weeks, up 83% on the four weeks before. 0.2% of all new papers.

When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning
Oct 23, 2026Zongyao Zhao, Cheng-Jie Zhao, Ruihang Jiang +3Policy Optimization Group Relative Policy Optimization
The University of Hong Kong · Tsinghua University
Dual-Adaptive Z-Score PPO: Dynamic Clipping Using Action Confidence and Normalized Advantage Magnitude
Oct 23, 2026Taha M. Mahmoud, Naima KaabouchProximal Policy Optimization Policy Optimization
Artificial Intelligence Research (AIR) Center, University of North Dakota, Grand Forks, ND, USA
MOER: Mix-Policy Optimization under Expert Prefix Guidance with Selective Rollback
Oct 22, 2026Xuyang Zhong, Jiuding Yang, Yiwen Guo +1RL for Language Model Reasoning Policy Optimization
City University of Hong Kong · LIGHTSPEED · Independent Researcher
One Act, Many Masks: Effect-Marginal Policy Optimization for Long-Horizon Agents
Oct 21, 2026Shaojin Ma, Ruitao Wang, Pengyi Li +6Policy Optimization RL for LLM Agents
Tianjin University · Fudan University · Renmin University of China
Doob-Based Generative Policy Optimization
Oct 20, 2026Maojiang Su, Qijie Zhu, Yu Xiao +4Policy Optimization RL for Diffusion Models
Department of Computer Science, Northwestern University · Center for Foundation Models and Generative AI, Northwestern University · Department of Statistics and Data Science, Northwestern University +1
Bellman Language Models: Solving the Recurring Decisions of LLM Agents
Oct 17, 2026Adarsh Vatsa, William EiersLLM Decision-Making LLM Agent Efficiency
Stevens Institute of Technology
Lifted State Policy Optimization: Internalizing Self-Correction through Transition Geometry
Oct 14, 2026Ren Zhuang, Ben Wang, Shuifa SunRL for Language Model Reasoning Policy Optimization
Hangzhou Normal University
DR-APO: Decoupled-Representation Advantage-Guided Policy Optimization from Continuous Control to Embodied Navigation
Oct 12, 2026Runhua Zhang, Yingjie Niu, Chao LiPolicy Optimization RL for Generative Models
DEEP Robotics, Hangzhou, China · Zhejiang University, Hangzhou, China
Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence
Oct 9, 2026Zhicheng Cai, Xinyuan Guo, Hanlin Wu +5RL for Language Models RL for Language Model Reasoning
Institute for AI Industry Research (AIR), Tsinghua University SIA-Lab of Tsinghua AIR and ByteDance Seed
Cross-Behavior Policy Optimization: Unified Learning from Offline Demonstrations and On- and Off-Policy Rollouts via a Shared Regression Target
Oct 9, 2026Du Cheng, Junyu Lu, Jingfan Zhang +3Policy Optimization RL for Language Model Reasoning
Closing the Horizon Gap in Policy Optimization for Adversarial MDPs
Oct 8, 2026Mingyi Li, Taira TsuchiyaRegret Minimization in RL Markov Decision Processes
The University of Tokyo · The University of Tokyo, The University of Osaka, and RIKEN
Pessimistic Minimax Learning for Public-Private Information Games under Unilateral Coverage
Oct 4, 2026Shuze Daniel Liu, Claire Chen, Jiuqi Wang +1Imperfect-Information Games Zero-Sum Games
Massachusetts Institute of Technology · Purdue University · California Institute of Technology +1
Sharp Convergence and Sample Complexity of Policy Mirror Descent for Average-Reward MDPs
Oct 2, 2026Enes Arda, Atilla EryilmazMarkov Decision Processes Average-Reward RL
The Ohio State University
On the Convergence of Success Conditioning for Policy Optimization
Oct 2, 2026Matthew Brun, Xu Andy SunPolicy Optimization Markov Decision Processes
Operations Research Center, Massachusetts Institute of Technology, Cambridge, MA 02139. · Sloan School of Management, Massachusetts Institute of Technology, Cambridge, MA 02139.
Sample complexity of variance-reduced policy gradient: weaker assumptions and lower bounds
Oct 2, 2026Gabor Paczolay, Matteo Papini, Alberto Maria Metelli +2Policy Gradient Policy Gradient Methods
Politecnico di Milano. · Budapest University of Technology and Economics. · University of Milan.
FERPO: Forward Entropy-Regularized Policy Optimization
Oct 1, 2026Sebastian Sanokowski, Alireza Sarmadi, Majid KhadivKL-Regularized RL Policy Optimization
Applied and Theoretical Aspects of Robot Intelligence (ATARI) Lab, Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich
Iterative Policy Refinement through Semantic Rollout Analysis
Oct 1, 2026Feiyu Gavin Zhu, Qi Xu, Zhifei Deng +4Policy Optimization LLM-Guided RL
Carnegie Mellon University · Meta
Towards Optimal Policy Improvement
Oct 1, 2026Yaniv Oren, Viliam Vadocz, Wiktor Zabka +7Markov Decision Processes Policy Optimization
Department of Intelligent Systems, TU Delft · Centrum Wiskunde & Informatica, Amsterdam · Department of Computer Science, ETH Zürich +3
Grounding Large Language Models in DSGE Simulators for Policy Generation and Forecasting
Oct 1, 2026Aditya Dubey, Namah Gupta, Vinti AgarwalRL for Language Models Large Language Model-Guided Optimization
Birla Institute of Technology and Science, Pilani, India
Role-Adaptive Policy Optimization for Offline Reinforcement Learning
Sep 30, 2026Seonvin Cho, Soohyun Choi, Songnam HongPolicy Optimization Offline RL
Department of Electronic Engineering, Hanyang University Seoul, Republic of Korea
Fast Regularized Policy Mirror Descent with One-Step TD Updates
Sep 30, 2026Qipei Chen, Wenye Li, Yule Sun +1Markov Decision Processes Policy Optimization
School of Mathematical Sciences, Fudan University · School of Data Science, Fudan University
Self-Evolving Algorithm-Design Agents: Escaping In-Context Evolutionary Stagnation via Population-Curated Policy Optimization
Sep 30, 2026Chen Lu, Ke Xue, Siyuan Xu +2Automated Algorithm Discovery Large Language Model-Guided Optimization
State Key Laboratory of Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · Huawei Noah’s Ark Lab, China
Achieving an $O(1/N)$ Optimality Gap in Average-Reward Weakly-Coupled MDPs
Sep 29, 2026Yige Hong, Xiangcheng Zhang, Qiaomin Xie +2Markov Decision Processes Average-Reward RL
H. Milton Stewart School of Industrial & Systems Engineering, Georgia Institute of Technology · John A. Paulson School of Engineering and Applied Sciences, Harvard University · Department of Industrial and Systems Engineering, University of Wisconsin–Madison +2
Bellman-Certified Rounding for Sparse Policy Deployment in MDPs
Sep 29, 2026Zhaojun PengMarkov Decision Processes Policy Optimization
Markovian Nonconvex ADMM for Reinforcement Learning: Bellman-Resolvent Stability Beyond Smooth Blocks
Sep 29, 2026Zhaojun PengAlternating Direction Method of Multipliers RL Control
Towards Better Training Signal: Advantage Clipped Policy Optimization
Sep 29, 2026Ruichuan Huang, Jinghan Liu, Congliang ChenPolicy Optimization RL for Language Model Reasoning
Shenzhen Loop Area Institute
Diffusion Policy Improvement with Proposal-Conditioned Refinement Flows
Sep 29, 2026Junhyun Ha, Juho Lee, Byoungwoo ParkPolicy Optimization Offline RL
KAIST
Regularized policy gradient with learned mixtures of Gaussians for games with continuous actions
Sep 29, 2026Ondřej Kubíček, Viliam Lisý, Tuomas SandholmGame-Playing Agents Self-Play RL
Czech Technical University in Prague · Carnegie Mellon University · Artificial Intelligence Center
Factorized Scheduling Principle: Learning Interpretable and Transferable Policies via Structured Additive Functions
Sep 29, 2026Hong Je-Gal, Hyun-Suk LeeInterpretable ML Policy Optimization
Department of Artificial Intelligence and Robotics, Sejong University, South Korea · Artificial Intelligence Robotics Institute (AIRI), Sejong University, South Korea
RareDx: Controlled Knowledge Integration and Graph-Grounded Policy Optimization for Rare-Disease Diagnosis
Sep 28, 2026Bo Zhang, Yuchen Wang, Dongbai Li +6Rare Disease Diagnosis LLM Reasoning with Graphs
Xi’an Jiaotong University · UIUC · Tsinghua University +4

Oct 23, 2026J/K move · Enter open · S save

When to Stop Penalizing Actions? Martingale GRPO for VLA Reinforcement Learning

Zongyao Zhao, Cheng-Jie Zhao, Ruihang Jiang, Bing Zheng+2

The University of Hong Kong · Tsinghua University

Policy Optimization Group Relative Policy Optimization Credit Assignment in RL RL for VLMs

Open paper AskPDF arXiv

A failed vision-language-action (VLA) rollout does not imply that every action contributed to failure. Yet in group-relative policy optimization (GRPO), a single trajectory-level advantage is assigned to all actions in a rollout. Potentially useful recovery attempts can thus be penalized alongside the errors they aim to correct. Ideal stopping conditions for failure feedback are characterized using success-probability and policy-score martingales. At these stops, the removed gradient contribution has zero conditional expectation given the retained information. Truncation therefore preserves the expectation of a reference GRPO group-gradient estimator and can reduce its covariance. One such stop occurs when the conditional probability of success is zero, motivating error and recovery cues for selecting candidate feedback boundaries. Martingale GRPO (MART-GRPO) is introduced with a vision-language model (VLM) for event-guided truncation. The VLM is fine-tuned on our annotated training set using low-rank adaptation (LoRA). The last visible error without observed recovery is localized by the frozen VLM in complete rollout videos. Failure feedback is retained through the selected error interval and removed afterward, while successful-rollout loss terms and original loss normalization are preserved. For retrospective boundaries, changes in expectation and variance are analyzed to assess departures from ideal stopping. Experiments with OpenVLA-OFT on RoboTwin 2.0 show that MART-GRPO improves success rates over full-trajectory GRPO across multiple tasks.