Proximal Policy Optimization

Also known as PPO

Overview

Proximal Policy Optimization (PPO) is an actor–critic method that learns actions from rewards while limiting how much the policy changes at each update, commonly through clipped probability ratios and generalized advantage estimation. Current approaches refine this update rule with divergence-based trust regions, adaptive clipping, better value critics, and selective reuse of past experience; safety constraints, demonstrations, and sequence models are also used to guide learning. PPO is applied to benchmarks such as Atari and MuJoCo, as well as robotics, multi-agent control, and language-model reasoning. A central challenge is improving sample efficiency without making updates unstable: clipping and on-policy data can help control risk, but may restrict useful learning, while critic errors and stale data can undermine performance.

Latest papers

CardsList

126 papers14 papers in the last four weeks, down 22% on the four weeks before. 0.1% of all new papers.

  1. Stack more levels: how to get general and human-like Mario playing

    Nov 12, 2026Shinichi Arimura, Yuchen Li, Julian TogeliusGame-Playing AgentsRL for Game AI

  2. A physics-constrained deep reinforcement learning method for autonomous mission planning of Titan deep space probes

    Nov 11, 2026Hanmin Li, Tao Wang, Shuangliang Liu +2Constrained RLPhysics-Informed RL

  3. Ordered Policy Optimization

    Nov 6, 2026Dongkyu Derek Cho, Pan XuPolicy OptimizationProximal Policy Optimization

  4. Efficient Detailed Scheduling of Multiproduct Pipelines using Action-Decoupled Deep Reinforcement Learning

    Nov 2, 2026Kai Shao, Ling Jian, Liming Liu +2Proximal Policy OptimizationHybrid Action Space RL

  5. MAPO: Mixture of Experts with Alternating Policy Optimization for Universal Humanoid Motion Tracking

    Nov 2, 2026Seonghyeon Lim, Hyeonwoo Lee, Mincheol Kim +2Humanoid Motion TrackingRobot Policy Learning

  6. Learning from Relative Action Credit for Escaping Local Minima

    Oct 27, 2026Giovanni Minelli, Giulio Turrisi, Lorenzo Amatucci +2Reinforcement LearningRobotic RL

  7. Repulsive Population Policy Gradients for Behaviorally Diverse Reinforcement Learning

    Oct 27, 2026Saeed Hedayatian, Harsh Shah, Celine Mneimneh +1Reinforcement LearningBehavioral Diversity in RL

  8. Bridging Exploration and Loss of Plasticity in On-Policy Reinforcement Learning

    Oct 26, 2026Andrea Vasco Grieco, Evelyn D'Elia, Edoardo Caldarelli +1Loss of PlasticityProximal Policy Optimization

  9. Learning Multi-Phase Robot Loco-Manipulation via Phase-Conditioned Policy Optimization

    Oct 26, 2026Simone Manni, Annika Wollschläger, Dario Bellicoso +4Robotic Loco-ManipulationLong-Horizon Robotic Manipulation

  10. Priority-Preserving Advantage Aggregation for Multi-Reward Policy Optimization

    Oct 26, 2026Xiangchi Yuan, Xingjian Diao, Zheyuan Liu +7Multi-Objective Reinforcement LearningRL Post-Training

  11. Dual-Adaptive Z-Score PPO: Dynamic Clipping Using Action Confidence and Normalized Advantage Magnitude

    Oct 23, 2026Taha M. Mahmoud, Naima KaabouchProximal Policy OptimizationPolicy Optimization

  12. EPIC: Rethinking Trust Regions for Autoregressive LLM Policies via Exact-KL Prefix Importance-Ratio Clipping

    Oct 21, 2026Chengxi Li, Hanlin Wu, Zhicheng Cai +4RL for Language Model ReasoningProximal Policy Optimization

  13. Reinforcement learning on the EPRI AI-ccelerating Unit Commitment benchmark: diagnosing over-commitment and guiding MILP with learned policies

    Oct 18, 2026Maharani Devira Pramita, Aidan O'SullivanLearning to OptimizeRL for Combinatorial Optimization

  14. VASPO: Self-Distillation with Privileged Value Estimation

    Oct 14, 2026Fei Wu, Zhenrong Zhang, Jianshu Zhang +1RL for Language Model ReasoningOn-Policy Self-Distillation

  15. Design Space Exploration of Reinforcement Learning Configurations for Post-Legalization Placement

    Oct 14, 2026Foteini Oikonomou, Maria Pantazi-Kypraiou, George Goudroumanis +5RL for Combinatorial OptimizationReward Design for RL

  16. TPPO: Tracking-Value PPO For Language Model Reasoning

    Oct 12, 2026Futing Wang, Chenxi Li, Jianhao Yan +6RL for Language ModelsRL for Language Model Reasoning

  17. Structure-Aware Numerical Stabilization for Half-Precision Proximal Policy Optimization

    Oct 9, 2026Linxuan Wu, Liang ChenRL StabilityReinforcement Learning

  18. Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence

    Oct 9, 2026Zhicheng Cai, Xinyuan Guo, Hanlin Wu +5RL for Language ModelsRL for Language Model Reasoning

  19. Constrained Command-Conditioned Reinforcement Learning with Bandit Strategy Selection in Real-Time Strategy Games

    Oct 8, 2026Nick Leenders, Roy Lindelauf, Joost van Oijen +1Reinforcement LearningGame-Playing Agents

  20. A Closed-Loop Non-Asymptotic Convergence Analysis of PPO with Learned Critics and Clipping

    Oct 7, 2026Junwei Su, Mengfan Liu, Yanyong Zhang +1Proximal Policy OptimizationActor-Critic Methods

  21. Energy-Aware Path Following: Comparative Analysis of Reinforcement Learning and NMPC for Electric Vehicles

    Oct 6, 2026Mohamed Sabaa, Mostafa EmamRobot Trajectory TrackingNonlinear MPC

  22. When a Correct Reward Is Not Enough: Diagnosing and Guiding PPO in an Analytically Solved Broker-Trader Game

    Oct 2, 2026Siu Tung Wong, Carlo CampajolaRL ControlProximal Policy Optimization

  23. Multi-Fidelity Policy Gradients Stabilize Data-Scarce Reinforcement Learning

    Oct 1, 2026Xinjie Liu, Ruihan Zhao, Anirban Chaudhuri +3Policy Gradient MethodsPolicy Learning

  24. Reusing Past Samples in Proximal Policy Optimization: When and How Does It Help?

    Oct 1, 2026Alessandro Montenegro, Riccardo Venturelli, Marco Mussi +2Experience ReplayProximal Policy Optimization

  25. PPO-HRAP: Proximal Policy Optimization with a Hybrid Regime-Aware Policy for Risk-Controlled Trading

    Oct 1, 2026Duong Hien Chi Kien, Thanh Trung HuynhReinforcement LearningRisk-Sensitive RL

  26. DexPolicy: Scheduled Exploration for Trajectory-Guided Dexterous Manipulation

    Sep 30, 2026Haoyu Wang, Siyuan Qian, Yanjun Li +4Robotic ManipulationRL Exploration

  27. EasyPPO: Stabilizing the Critic Is Key

    Sep 29, 2026Xuanyi Zhou, Qiuyang Mang, Huanzhi Mao +7RL for Language ModelsProximal Policy Optimization

  28. Action Chunking Proximal Policy Optimization with Feedback Correction

    Sep 28, 2026Sanghyun Hahn, Jonghyun ChoiRobotic RLFeedback Control

  29. Safe Greenhouse Climate Control Using Lagrangian-Constrained PPO with Kolmogorov-Arnold Networks

    Sep 28, 2026Hangzun Liu, Yuling Fan, Fang Tian +3Constrained RLProximal Policy Optimization