Proximal Policy Optimization
Also known as PPO
Overview
Proximal Policy Optimization (PPO) is an actor–critic method that learns actions from rewards while limiting how much the policy changes at each update, commonly through clipped probability ratios and generalized advantage estimation. Current approaches refine this update rule with divergence-based trust regions, adaptive clipping, better value critics, and selective reuse of past experience; safety constraints, demonstrations, and sequence models are also used to guide learning. PPO is applied to benchmarks such as Atari and MuJoCo, as well as robotics, multi-agent control, and language-model reasoning. A central challenge is improving sample efficiency without making updates unstable: clipping and on-policy data can help control risk, but may restrict useful learning, while critic errors and stale data can undermine performance.
Latest papers
126 papers14 papers in the last four weeks, down 22% on the four weeks before. 0.1% of all new papers.
Spiking Neural Network Actor-Critic Proximal Policy Optimization Control for Autonomous UAV Navigation Through Constrained Openings in Civil Infrastructure and Buildings
Cost-Aware Reinforcement Learning with Action Masking and Projection for Battery Energy Storage Dispatch under Suppressed-Spread Market Shifts
Rethinking Critic Learning in PPO: Understanding and Mitigating Value Flattening
Statistical Turbulence and High-Fidelity Disturbance Fields for Quadrotor Flight Control
SLA-Safe Energy Control for AI-Native NG-RAN Using Stability-Aware Constrained PPO
Train What You Deploy:Token-Faithful Post-Training of a Production Coding
BCPPO: Bachelier-Inspired Constrained Proximal Policy Optimization for Tail-Risk-Aware Safe Reinforcement Learning
Sense Once, Serve Many: Common-Trace Factorized Constrained PPO for Online Sensing-Session Consolidation in Multi-Tenant ISAC Networks
Goodput Maximization for Large Language Model Edge Inference: A Two-Phase Maskable PPO Approach
Where Entropy Is Measured Matters: Policy Geometry in Bounded Continuous-Control PPO
Advantage-level Aggregation Reinforcement Learning for X-point Target Magnetic Configuration Control in an EXL-50U Experiment-Calibrated Simulation Environment
SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
Knowledge-Distilled End-to-End Reinforcement Learning for Smooth 6-DOF Thrust Control and Rapid Adaptation to Ocean Currents in Remotely Operated Vehicles
Per-Shipment Multi-Agent Reinforcement Learning for Intermodal Freight Routing Under Hurricane Disruption
SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling
Start Classifying: Categorical Critics for LLM Reinforcement Learning
Learning-Based Collaborative MEC for LLM Inference with Soft-Deadline Awareness via Transformer-Enhanced PPO
ReDiPPO: Reference-Guided Value Calibration and Discrepancy-Aware Token Reweighting for Mathematical Reasoning
P3: Probabilistic Policy Propagation for Stable VAE-Based Robot Learning
Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning
OR Else: A Differentiable Trust Region for Policy Optimization
Llama-3.2-1B-Instruct on Anthropic hh-rlhf with one shared reward model and three seeds per method. Under GAE, PPO-OR has a mean final training-time reward-model score higher than PPO-clip, with a larger observed across-seed spread. Under group-relative advantages, GRPO-OR does not have a higher mean score, but shows a smaller observed spread, a near-zero terminal OR residual, and a declining overshoot fraction, while the matched GRPO clipped-objective trace remains variable. Both group-relative methods exhibit substantially larger rollout-to-current log-ratio displacement than the GAE methods, and OR does not consistently reduce it. Thus, OR changes optimization behavior in both matched comparisons, but the observed reward effect differs between them. At , the GRPO-OR diagnostics do not translate into a reward-score gain. Whether larger groups change this outcome remains open. The reported scores are training-time reward-model measurements, not held-out human-preference performance.