Policy Optimization

Follow this topic Ask about this topic

Policy optimization studies how to update a decision-making rule so it earns higher rewards while meeting constraints such as safety, fairness, or limited data. Common approaches include policy gradients, actor-critic methods, trust regions and mirror descent, as well as policy iteration and newer generative-policy methods. Work spans continuous-control benchmarks such as MuJoCo, offline and multi-agent settings, and language-model training for mathematical reasoning, coding, and tool use, where token-, segment-, or trajectory-level rewards guide updates. A central challenge is balancing effective exploration and sample reuse with stable updates, useful credit assignment, and improvements that carry over from training to deployment.

Latest papers

224 papers32 papers in the last four weeks, up 52% on the four weeks before. 0.2% of all new papers.

Predictable GRPO: A Closed-Form Model of Training Dynamics
Jun 29, 2026Rajat Ghosh, Datta Nimmaturi, Aryan Singhal +4Group Relative Policy Optimization Policy Optimization
Nutanix San Jose, USA · Unsloth Hyderabad, India · Nutanix2026 San Jose, USA +1
Hierarchical Policy Learning via Spectral Decomposition
Jun 28, 2026Shuxin Cao, Liquan Wang, Walker Byrnes +3Robot Policy Learning Robot Skill Learning
Georgia Institute of Technology, Atlanta, Georgia, USA · Georgia Tech Research Institute, Atlanta, Georgia, USA · Harvard University, Cambridge, Massachusetts, USA
The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning
Jun 28, 2026Jing Liang, Hongyao Tang, Yi Ma +9Reinforcement Learning RL for Language Model Reasoning
1Tianjin · 2Alibaba
MiniOpt: Reasoning to Model and Solve General Optimization Problems with Limited Resources
Jun 24, 2026Ke Zhao, Zixiang Di, Hong Qian +9Large Language Model-Guided Optimization RL for Language Model Reasoning
East China Normal University, Shanghai 200062, China · AntGroup, Hangzhou 310000, China · Southern University of Science and Technology, Shenzhen 518055, China +1
Entropy Objectives in Markov Decision Processes
Jun 19, 2026S. Akshay, Raghav Goyal, Aditya Neeraje +1Markov Decision Processes Markov Models
Indian Institute of Technology Bombay, Mumbai, India. · Tata Institute of Fundamental Research, Mumbai, India.
Robot Self-Improvement via Human-Video Dynamics Models
Jun 19, 2026Hanzhi Chen, Anran Zhang, Simon Schaefer +5Robot Skill Learning Policy Optimization
1ETH Zurich · 2Technical University of Munich · 4MCML +1
VIMPO: Value-Implicit Policy Optimization for LLMs
Jun 18, 2026Zhewei Kang, Aosong Feng, Sergey Levine +2RL for Language Model Reasoning Policy Optimization
UC Berkeley · Yale University
Co-policy: Responsive Human-Robot Co-Creation for Musical Performances
Jun 18, 2026Xuetao Li, Wenke Huang, Mang Ye +4Policy Optimization Robot Motion Generation
School of Computer Science, Wuhan University, Wuhan, China · College of Computing and Data Science, Nanyang Technological University, Singapore · School of Automation, Wuhan University of Technology, Wuhan, China +2
A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization
Jun 15, 2026Prasanth YSS, Zhichen Ren, Rasa Hosseinzadeh +6RL for Language Model Reasoning Policy Optimization
Layer 6 AI
Scalar-Stepsize Nonuniform Monte Carlo Optimistic Policy Iteration: A Certified Counterexample
Jun 14, 2026Yuanlong ChenMarkov Decision Processes Policy Optimization
$α$ -fair heterogeneous agent reinforcement learning
Jun 11, 2026Yao-hua Franck Xu, Tayeb Lemlouma, Jean-Marie Bonnin +1Algorithmic Fairness Policy Optimization
Orange Innov Lannion, France · IRISA Université de Rennes Lannion, France · IRISA IMT Atlantique Rennes, France
IntElicit: Eliciting and Assessing Contextualized Creativity via Dialogue Policy Optimization
Jun 10, 2026Mingjia Li, Jin Wu, Hong Qian +7Creativity Assessment Policy Optimization
East China Normal University · Shanghai Innovation Institute
Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning
Jun 9, 2026Zhiyuan Zhou, Andy Peng, Charles Xu +4Policy Optimization Offline RL
UC Berkeley · Physical Intelligence
On-sky demonstration of reinforcement learning for adaptive optics control
Jun 9, 2026Jalo Nousiainen, Vincent Chambouleyron, Benoit Neichel +7RL Control Reinforcement Learning
European Southern Observatory · Aix Marseille University, CNRS, CNES, LAM, Marseille, France · Wakea Consulting, 4 Rue Joseph Bouchayer, 38100 Grenoble +1
Convergence of Monte Carlo Optimistic Policy Iteration: Beyond Uniform State-Action Updates
Jun 9, 2026Octave Oliviers, Glenn VinnicombeMarkov Decision Processes Reinforcement Learning
Department of Engineering University of Cambridge Cambridge, UK
Rethinking the Divergence Regularization in LLM RL
Jun 8, 2026Jiarui Yao, Xiangxin Zhou, Penghui Qi +3RL for Language Models Policy Optimization
1Tencent Hunyuan · 2UIUC · 3NUS
Structure-Conditioned Actor-Critic Branches for Quality-Diversity Reinforcement Learning
Jun 7, 2026Lianrong Zuo, Peilan Xu, Yong Liu +1Reinforcement Learning Policy Optimization
Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies, Institute of Cyberspace Security, School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 518055, Guangdong, China
The Easy, the Hard, and the Learnable: Confidence and Difficulty-Adaptive Policy Optimization for LLM Reasoning
Jun 6, 2026Zhanke Zhou, Xiangyu Lu, Chentao Cao +4RL for Language Model Reasoning Policy Optimization
1TMLR Group, Department of Computer Science, Hong Kong Baptist University · 2Stanford University · 3Sydney AI Centre, The University of Sydney
Retry Policy Gradients in Continuous Action Spaces
Jun 4, 2026Soichiro Nishimori, Paavo ParmasPolicy Optimization RL Exploration
The University of Tokyo, Japan
Neetyabhas: A Framework for Uncertainty-Aware Public Policy Optimization in Rational Agent-Based Models
Jun 3, 2026Janani Venugopalan, Gaurav Deshkar, Rishabh Gaur +2Policy Optimization Agent-Based Modeling
Engineering for Research (e4r), Thoughtworks Technologies, Binarius Building, Deepak Complex, Shastrinagar, Yerawada, Pune, Maharashtra, India - 411006
Physics-Guided Policy Optimization with Self-Distillation
Jun 2, 2026Ke Wang, Yuning Wu, Haoran Liu +3On-Policy Self-Distillation Policy Optimization
Amazon, Arlington, USA · Amazon, New York, USA · Amazon, San Jose, USA +1
From Global Policies to Local Strategies: Multi-Objective Optimization of Resource-Specific Handover Policies
Jun 1, 2026Lukas Kirchdorfer, Artemis Doumeni, Han van der Aa +1Policy Optimization Multi-Objective Optimization
SAP Signavio, Walldorf, Germany · DTU Compute, Technical University of Denmark, Kongens Lyngby, Denmark · Faculty of Computer Science, University of Vienna, Vienna, Austria
Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Belief
May 30, 2026Hongqiang Lin, Pengfei Wang, Nenggan ZhengReinforcement Learning Bayesian RL
Zhejiang University, Hangzhou, China.
When are LLMs Sufficient Policy Optimizers for Sequential RL Tasks?
May 29, 2026Stephane Hatgis-Kessell, Emma BrunskillLanguage Model-Based Control Policy Optimization
Department of Computer Science, Stanford University
HPO: Hysteretic Policy Optimization for Stable and Efficient Training under Sparse-Reward Regime
May 28, 2026Mohamed Sana, Nicola Piovesan, Antonio De Domenico +2Group Relative Policy Optimization Policy Optimization
Paris Research Center, Huawei Technologies, Boulogne-Billancourt, France · Huawei Technologies, China
Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization
May 28, 2026Shufan Li, Konstantinos Kallidromitis, Akash Gokul +2Token-Level Credit Assignment RL for Language Model Reasoning
UCLA · Panasonic AI Research · NVIDIA
Insurance Pricing Optimization via Off-Policy Evaluation
May 27, 2026Sascha Günther, Dimitri Semenovich, Mario V. WüthrichOff-Policy Evaluation Policy Optimization
Department of Mathematics, ETH Zurich
Long Live The Balance: Information Bottleneck Driven Tree-based Policy Optimization
May 27, 2026Hao Jiang, Shurui Li, Tianpeng Bu +7RL for Language Models Exploration-Exploitation Tradeoff
Alibaba Cloud Computing, Alibaba Group.
Ratio-Variance Regularized Policy Optimization
May 26, 2026Yu Luo, Shuo Han, Yihan Hu +5RL for Language Model Reasoning Policy Optimization
Department of Foundation Model, 2012 Labs, Huawei Shanghai · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · Department of Computer Science and Technology, Tsinghua University +1
AgensFlow: A Coordination-Policy Substrate for Multi-Agent Systems
May 26, 2026Nicole KoenigsteinMulti-Agent LLM Systems Multi-Agent System Optimization
Independent researcher.

Jun 29, 2026J/K move · Enter open · S save

Predictable GRPO: A Closed-Form Model of Training Dynamics

Rajat Ghosh, Datta Nimmaturi, Aryan Singhal, Vaishnavi Bhargava+3

Nutanix San Jose, USA · Unsloth Hyderabad, India · Nutanix2026 San Jose, USA+1

Group Relative Policy Optimization Policy Optimization Neural Network Training Dynamics Policy Learning Gradient Descent Dynamics

Open paper AskPDF arXiv

We develop a first-principles reduced-order model of these dynamics. Under a single mean-field assumption that summarizes the policy by its expected reward, we reduce the GRPO update to a stochastically-forced damped oscillator whose mass, damping, and stiffness are fixed in closed form by the optimizer hyperparameters together with a single measured curvature scale -- momentum supplies the inertia, off-policy lag erodes the damping, and the group size enters, to leading order, as a noise temperature. The reduction has three consequences. First, it subsumes the empirical single-exponential saturation law as its overdamped limit, recasting the fitted plateau, timescale, and size exponent as the fixed point, inverse stiffness, and curvature-scaling exponent of the underlying potential, and adding, through the retained inertial term, the slow-start phase the single exponential cannot represent. Second, it yields predictions tied to independently measurable quantities rather than fitted ones: group-size invariance of the deterministic trajectory with a

1/G

stationary fluctuation, a sharp stability threshold in the refresh interval, and an overdamped-to-oscillatory transition. Third, it furnishes diagnostics that separate failure modes a reward curve alone conflates -- reward hacking, advantage degeneracy, policy concentration, and dynamical instability. Across three models and two group sizes, the closed-form trajectory fits training reward to

R^2 \geq 0.91

and the mean trajectory is group-size invariant to leading order -- on both the reward curve and out-of-distribution transfer to eight math benchmarks -- while the within-group reward spread retains a residual

G

-dependence that the leading-order temperature picture does not capture.