Proximal Policy Optimization

Also known as PPO

Proximal Policy Optimization (PPO) is an actor–critic method that learns actions from rewards while limiting how much the policy changes at each update, commonly through clipped probability ratios and generalized advantage estimation. Current approaches refine this update rule with divergence-based trust regions, adaptive clipping, better value critics, and selective reuse of past experience; safety constraints, demonstrations, and sequence models are also used to guide learning. PPO is applied to benchmarks such as Atari and MuJoCo, as well as robotics, multi-agent control, and language-model reasoning. A central challenge is improving sample efficiency without making updates unstable: clipping and on-policy data can help control risk, but may restrict useful learning, while critic errors and stale data can undermine performance.

Latest papers

116 papers20 papers in the last four weeks, up 150% on the four weeks before. 0.1% of all new papers.

Priority-Preserving Advantage Aggregation for Multi-Reward Policy Optimization
Oct 26, 2026Xiangchi Yuan, Xingjian Diao, Zheyuan Liu +7Multi-Objective Reinforcement Learning RL Post-Training
Georgia Institute of Technology · Dartmouth College · University of Notre Dame
Dual-Adaptive Z-Score PPO: Dynamic Clipping Using Action Confidence and Normalized Advantage Magnitude
Oct 23, 2026Taha M. Mahmoud, Naima KaabouchProximal Policy Optimization Policy Optimization
Artificial Intelligence Research (AIR) Center, University of North Dakota, Grand Forks, ND, USA
EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping
Oct 21, 2026Chengxi Li, Hanlin Wu, Zhicheng Cai +4RL for Language Model Reasoning Proximal Policy Optimization
Institute for AI Industry Research (AIR), Tsinghua University · Department of Computer Science and Technology, Tsinghua University · SIA-Lab of Tsinghua AIR and ByteDance Seed
Reinforcement learning on the EPRI AI-ccelerating Unit Commitment benchmark: diagnosing over-commitment and guiding MILP with learned policies
Oct 18, 2026Maharani Devira Pramita, Aidan O'SullivanLearning to Optimize RL for Combinatorial Optimization
UCL Energy Institute, Bartlett School of Environment, Energy and Resources, University College London, Central House, 14 Upper Woburn Place, London WC1H 0NN, United Kingdom
VASPO: Self-Distillation with Privileged Value Estimation
Oct 14, 2026Fei Wu, Zhenrong Zhang, Jianshu Zhang +1RL for Language Model Reasoning On-Policy Self-Distillation
University of Science and Technology of China · IFLYTEK Research
Design Space Exploration of Reinforcement Learning Configurations for Post-Legalization Placement
Oct 14, 2026Foteini Oikonomou, Maria Pantazi-Kypraiou, George Goudroumanis +5RL for Combinatorial Optimization Reward Design for RL
Department of Electrical and Computer Engineering, University of Thessaly, Volos, Greece · School of Computing Science, University of Glasgow, UK · Department of Electronic and Electrical Engineering, Trinity College Dublin, Ireland
TPPO: Tracking-Value PPO For Language Model Reasoning
Oct 12, 2026Futing Wang, Chenxi Li, Jianhao Yan +6RL for Language Models RL for Language Model Reasoning
Zhejiang University · Westlake University · Shanghai AI Laboratory +4
Structure-Aware Numerical Stabilization for Half-Precision Proximal Policy Optimization
Oct 9, 2026Linxuan Wu, Liang ChenRL Stability Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences Beijing, China
Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence
Oct 9, 2026Zhicheng Cai, Xinyuan Guo, Hanlin Wu +5RL for Language Models RL for Language Model Reasoning
Institute for AI Industry Research (AIR), Tsinghua University SIA-Lab of Tsinghua AIR and ByteDance Seed
Constrained Command-Conditioned Reinforcement Learning with Bandit Strategy Selection in Real-Time Strategy Games
Oct 8, 2026Nick Leenders, Roy Lindelauf, Joost van Oijen +1Reinforcement Learning Game-Playing Agents
NLR Royal Netherlands Aerospace Centre Amsterdam, The Netherlands · Data Science Center of Excellence Faculty of Military Sciences, Breda, NL · Departement Intelligent Systems Tilburg University, Tilburg, NL
A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping
Oct 7, 2026Junwei Su, Mengfan Liu, Yanyong Zhang +1Proximal Policy Optimization Actor-Critic Methods
School of Artificial Intelligence and Data Science, University of Science and Technology of China · Department of Computer Science, University of Hong Kong
Energy-Aware Path Following: Comparative Analysis of Reinforcement Learning and NMPC for Electric Vehicles
Oct 6, 2026Mohamed Sabaa, Mostafa EmamRobot Trajectory Tracking Nonlinear MPC
Computer Science Department, Najran University, Najran, Saudi Arabia · Institute of Applied Mathematics and Scientific Computing, University of the Bundeswehr Munich, Neubiberg, 85579, Germany
When a Correct Reward Is Not Enough: Diagnosing and Guiding PPO in an Analytically Solved Broker-Trader Game
Oct 2, 2026Siu Tung Wong, Carlo CampajolaRL Control Proximal Policy Optimization
Institute of Finance and Technology, University College London · Institute of Finance and Technology University College London London, United Kingdom · UZH Blockchain Center +1
Multi-Fidelity Policy Gradients Stabilize Data-Scarce Reinforcement Learning
Oct 1, 2026Xinjie Liu, Ruihan Zhao, Anirban Chaudhuri +3Policy Gradient Methods Policy Learning
University of Texas at Austin · University of British Columbia
Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?
Oct 1, 2026Alessandro Montenegro, Riccardo Venturelli, Marco Mussi +2Experience Replay Proximal Policy Optimization
Politecnico di Milano, Milan, Italy · Università degli Studi di Milano, Milan, Italy
PPO-HRAP: Proximal Policy Optimization with a Hybrid Regime-Aware Policy for Risk-Controlled Trading
Oct 1, 2026Duong Hien Chi Kien, Thanh Trung HuynhReinforcement Learning Risk-Sensitive RL
College of Engineering and Computer Science, VinUniversity Ha Noi, Viet Nam
Free Everywhere, Exact on Trees: PPO's Dropped Correction Buys Sample Efficiency Under Aggressive Reuse
Sep 30, 2026Nima H. SiboniImportance Sampling Proximal Policy Optimization
Juna.ai, Kastanienallee 32, 10435 Berlin, Germany
DexPolicy: Scheduled Exploration for Trajectory-Guided Dexterous Manipulation
Sep 30, 2026Haoyu Wang, Siyuan Qian, Yanjun Li +4Robotic Manipulation RL Exploration
School of Computer Science, Peking University · AI2 Robotics
EasyPPO: Stabilizing the Critic Is Key
Sep 29, 2026Xuanyi Zhou, Qiuyang Mang, Huanzhi Mao +7RL for Language Models Proximal Policy Optimization
University of California, Berkeley · Princeton University
Action Chunking Proximal Policy Optimization with Feedback Correction
Sep 28, 2026Sanghyun Hahn, Jonghyun ChoiRobotic RL Feedback Control
Cornell University · Seoul National University
Safe Greenhouse Climate Control Using Lagrangian-Constrained PPO with Kolmogorov-Arnold Networks
Sep 28, 2026Hangzun Liu, Yuling Fan, Fang Tian +3Constrained RL Proximal Policy Optimization
College of Informatics, Huazhong Agricultural University, Wuhan 430070, China · Yunnan Modern Agricultural Industry Research Institute Co., Ltd.,Kunming, Yunnan 650000, China · College of Biological and Agricultural Engineering, Jilin University, Changchun 130022, China +1
Information-Time Proximal Policy Optimization
Sep 21, 2026Yongcheng Zeng, Xinyu Cui, Yan Song +9Reinforcement Learning RL for Language Model Reasoning
Institute of Automation, Chinese Academy of Sciences, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China · University College London +5
Spiking Neural Network Actor-Critic Proximal Policy Optimization Control for Autonomous UAV Navigation Through Constrained Openings in Civil Infrastructure and Buildings
Sep 20, 2026Francis Noah Walugembe, Maciej Wielgosz, Tomaž Goričan +1UAV Navigation Spiking Neural Networks
Applied Intelligence PhD Programme, Alma Mater Europaea University, Maribor, Slovenia · Faculty of Computer Science, Electrical and Telecommunications, AGH University of Krakow, 30-059 Krakow, Poland · Alma Mater Europaea University, Maribor, Slovenia
Cost-Aware Reinforcement Learning with Action Masking and Projection for Battery Energy Storage Dispatch under Suppressed-Spread Market Shifts
Sep 20, 2026Kuanlin Chen, Chen-Wei Kuo, Cheng-En OuConstrained RL Proximal Policy Optimization
Independent Researcher Taoyuan, Taiwan · National Tsing Hua University Hsinchu, Taiwan · Independent Researcher
Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening
Sep 16, 2026Yizhuo Li, Jianhao Yan, Yun Luo +9RL for Language Models Value Function Estimation
Shanghai Jiao Tong University · Shanghai AI Laboratory · Westlake University +4
Statistical Turbulence and High-Fidelity Disturbance Fields for Quadrotor Flight Control
Sep 9, 2026Xun HuangQuadrotor Control Robust RL
State Key Laboratory of Turbulence and Complex Systems, Department of Aeronautics and Astronautics, Peking University, Beijing, 100871, People’s Republic of China
SLA-Safe Energy Control for AI-Native NG-RAN Using Stability-Aware Constrained PPO
Sep 5, 2026Dharmendra KumarConstrained RL Proximal Policy Optimization
Independent Researcher Dallas, TX, USA
Train What You Deploy:Token-Faithful Post-Training of a Production Coding
Sep 4, 2026Cheng Li, Jiexiong Liu, Yixuan Chen +1RL for Code Generation Proximal Policy Optimization
KunlunMeta
BCPPO: Bachelier-Inspired Constrained Proximal Policy Optimization for Tail-Risk-Aware Safe Reinforcement Learning
Aug 31, 2026Dongsheng Hou, Yanqiao Chen, Yuhan RuiReinforcement Learning Constrained RL
Southern University of Science and Technology
Sense Once, Serve Many: Common-Trace Factorized Constrained PPO for Online Sensing-Session Consolidation in Multi-Tenant ISAC Networks
Aug 29, 2026Dang-Dung VuIntegrated Sensing and Communication Proximal Policy Optimization
Faculty of Information Technology, VNU University of Engineering and Technology (VNU-UET), Hanoi 100000, Vietnam

Oct 26, 2026J/K move · Enter open · S save

Priority-Preserving Advantage Aggregation for Multi-Reward Policy Optimization

Xiangchi Yuan, Xingjian Diao, Zheyuan Liu, Chunhui Zhang+6

Georgia Institute of Technology · Dartmouth College · University of Notre Dame

Multi-Objective Reinforcement Learning RL Post-Training Proximal Policy Optimization

Open paper AskPDF arXiv

LLM post-training often optimizes multiple objectives with unequal priorities, such as correctness and reasoning efficiency. However, the widely used additive reward aggregation in RL can violate these priorities: when primary and auxiliary advantages conflict, a sufficiently large auxiliary signal can overturn the preference induced by the primary objective. We term this problem advantage priority inversion. Although reward-level gating can protect priority more directly, it still removes potentially useful auxiliary distinctions among rollout samples that fail the primary condition. To address this trade-off, we propose Primary Advantage Policy Optimization (PAPO), a priority-aware advantage aggregation method that asymmetrically combines a primary and an auxiliary advantage. PAPO retains auxiliary supervision when the objectives agree and smoothly attenuates conflicting auxiliary signals, preserving the primary sign during fusion for primary-consistent ordering. This rule applies to sequence-level advantages in GRPO and token-level advantages in actor--critic PPO. Experiments on multiple domains and models show that PAPO consistently achieves the best primary objective while maintaining best or competitive auxiliary-objective performance across both GRPO- and PPO-based optimization. Code is available.