Proximal Policy Optimization

Also known as PPO

Overview

Proximal Policy Optimization (PPO) is an actor–critic method that learns actions from rewards while limiting how much the policy changes at each update, commonly through clipped probability ratios and generalized advantage estimation. Current approaches refine this update rule with divergence-based trust regions, adaptive clipping, better value critics, and selective reuse of past experience; safety constraints, demonstrations, and sequence models are also used to guide learning. PPO is applied to benchmarks such as Atari and MuJoCo, as well as robotics, multi-agent control, and language-model reasoning. A central challenge is improving sample efficiency without making updates unstable: clipping and on-policy data can help control risk, but may restrict useful learning, while critic errors and stale data can undermine performance.

Latest papers

CardsList

126 papers14 papers in the last four weeks, down 22% on the four weeks before. 0.1% of all new papers.

  1. Information-Time Proximal Policy Optimization

    Sep 21, 2026Yongcheng Zeng, Xinyu Cui, Yan Song +9Reinforcement LearningRL for Language Model Reasoning

  2. Spiking Neural Network Actor-Critic Proximal Policy Optimization Control for Autonomous UAV Navigation Through Constrained Openings in Civil Infrastructure and Buildings

    Sep 20, 2026Francis Noah Walugembe, Maciej Wielgosz, Tomaž Goričan +1UAV NavigationSpiking Neural Networks

  3. Cost-Aware Reinforcement Learning with Action Masking and Projection for Battery Energy Storage Dispatch under Suppressed-Spread Market Shifts

    Sep 20, 2026Kuanlin Chen, Chen-Wei Kuo, Cheng-En OuConstrained RLProximal Policy Optimization

  4. Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening

    Sep 16, 2026Yizhuo Li, Jianhao Yan, Yun Luo +9RL for Language ModelsValue Function Estimation

  5. Statistical Turbulence and High-Fidelity Disturbance Fields for Quadrotor Flight Control

    Sep 9, 2026Xun HuangQuadrotor ControlRobust RL

  6. Sense Once, Serve Many: Common-Trace Factorized Constrained PPO for Online Sensing-Session Consolidation in Multi-Tenant ISAC Networks

    Aug 29, 2026Dang-Dung VuIntegrated Sensing and CommunicationProximal Policy Optimization

  7. Goodput Maximization for Large Language Model Edge Inference: A Two-Phase Maskable PPO Approach

    Aug 26, 2026Xiaojing Chen, Qi Zhang, Wei Ni +2LLM Inference SchedulingEdge Computing

  8. Where Entropy Is Measured Matters: Policy Geometry in Bounded Continuous-Control PPO

    Aug 25, 2026Yiyang He, Zhichun Zhou, Ziwei Wang +2Policy OptimizationProximal Policy Optimization

  9. Advantage-level Aggregation Reinforcement Learning for X-point Target Magnetic Configuration Control in an EXL-50U Experiment-Calibrated Simulation Environment

    Aug 21, 2026Siqi Ding, Xuanhe Wang, Pei Guo +10RL ControlFeedback Control

  10. SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning

    Aug 20, 2026Dayang Liang, Lang Feng, Bo An +1RL for Language ModelsAgentic RL

  11. Knowledge-Distilled End-to-End Reinforcement Learning for Smooth 6-DOF Thrust Control and Rapid Adaptation to Ocean Currents in Remotely Operated Vehicles

    Aug 9, 2026Tiankuang Wen, Huiping Li, Gang Liu +1Robotic RLProximal Policy Optimization

  12. Per-Shipment Multi-Agent Reinforcement Learning for Intermodal Freight Routing Under Hurricane Disruption

    Aug 7, 2026Aliza Sharmin, Xudong Wang, Mustafa Can Camur +1Vehicle RoutingProximal Policy Optimization

  13. SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling

    Aug 3, 2026Evan Assmus, Qining Zhang, Lei YingReinforcement LearningRobotic RL

  14. Start Classifying: Categorical Critics for LLM Reinforcement Learning

    Aug 3, 2026Zhijian Zhou, Long Li, Xuan Zhang +7RL for Language Model ReasoningReinforcement Learning with Verifiable Rewards

  15. Learning-Based Collaborative MEC for LLM Inference with Soft-Deadline Awareness via Transformer-Enhanced PPO

    Aug 3, 2026Ngoc Hung Nguyen, Bjorn LandfeldtReinforcement LearningTransformer-Based RL

  16. ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning

    Jul 30, 2026Zhenrong Zhang, Fei Wu, Jun Du +2RL for Language Model ReasoningCredit Assignment in RL

  17. P3: Probabilistic Policy Propagation for Stable VAE-Based Robot Learning

    Jul 28, 2026Liyun Yan, Jianming Ma, Yang Zhang +5Variational AutoencodersRobot Policy Learning

  18. Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning

    Jul 21, 2026Junyao Yang, Yucheng Shi, Zongxia Li +6Asynchronous RLProximal Policy Optimization

  19. OR Else: A Differentiable Trust Region for Policy Optimization

    Jul 20, 2026Chinmay Rane, Kanishka Tyagi, Michael ManryRL for Language ModelsGroup Relative Policy Optimization

  20. Aggregate in the Advantage, Not the Ratio: A Canonical-Form Analysis of Cooperative Multi-Agent Policy Optimization

    Jul 20, 2026Zijian Zhao, Sen LiProximal Policy OptimizationCooperative MARL

  21. Predictive Divergence Masks for LLM RL

    Jul 12, 2026Xiangxin Zhou, Jiarui Yao, Penghui Qi +4RL for Language ModelsReinforcement Learning

  22. Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization

    Jul 11, 2026Zhicheng Cai, Xinyuan Guo, Hanlin Wu +3Reinforcement LearningRL for Language Model Reasoning

  23. SafeExplorer: An Unbiased Mixed-Policy Gradient for Reinforcement Learning with Recovery Interventions

    Jul 9, 2026Elham Daneshmand, Majid Khadiv, Glen Berseth +1Robot SafetyProximal Policy Optimization

  24. Federated Physics-Grounded Reinforcement Learning for Distributed Stability Control in Smart Grids

    Jul 6, 2026Omar Al-Refai, Ibrahim Shahbaz, Adam Ali Husseinat +1Reinforcement LearningFederated RL

  25. PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF

    Jun 29, 2026Doo Hwan Hwang, Kee-Eung KimReinforcement LearningRL for Language Model Reasoning

  26. PPO-EAL: Exact Augmented Lagrangian Proximal Policy Optimization for Safe Robotic Control

    Jun 26, 2026Jiatao Ding, Songqun Gao, Andrea Del Prete +1Robotic ControlConstrained RL