Proximal Policy Optimization

Also known as PPO

Proximal Policy Optimization (PPO) is an actor–critic method that learns actions from rewards while limiting how much the policy changes at each update, commonly through clipped probability ratios and generalized advantage estimation. Current approaches refine this update rule with divergence-based trust regions, adaptive clipping, better value critics, and selective reuse of past experience; safety constraints, demonstrations, and sequence models are also used to guide learning. PPO is applied to benchmarks such as Atari and MuJoCo, as well as robotics, multi-agent control, and language-model reasoning. A central challenge is improving sample efficiency without making updates unstable: clipping and on-policy data can help control risk, but may restrict useful learning, while critic errors and stale data can undermine performance.

Latest papers

116 papers20 papers in the last four weeks, up 150% on the four weeks before. 0.1% of all new papers.

Variational Proximal Policy Optimization
Jun 6, 2026Ousmane Amadou DiaRL for Language Model Reasoning Proximal Policy Optimization
Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions
Jun 2, 2026Bingxu Liu, Jiashun Liu, Johan Obando-Ceron +5Non-Stationary RL Reinforcement Learning
Hong Kong University of Science and Technology · Fudan University · Mila - Québec AI Institute +2
Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying
May 29, 2026Soichiro Nishimori, Paavo Parmas, Sotetsu Koyamada +4RL Exploration Policy Gradient Methods
The University of Tokyo, Tokyo, Japan · Kobe University, Kobe, Japan · Kyoto University, Kyoto, Japan +3
ESPO: Early-Stopping Proximal Policy Optimization
May 28, 2026Zihang Li, Rui Zhou, Yingcheng Shi +8LLM Reasoning Reinforcement Learning
1Tongyi Lab , Alibaba Group · 2Peking University
Confidence-Orchestrated Self-Evolution against Uncertain LLM Feedback
May 27, 2026Bowen Wei, Nan Wang, Yuqing Zhou +2Self-Training for Language Models RL for Language Model Reasoning
George Mason University · Amazon
Ratio-Variance Regularized Policy Optimization
May 26, 2026Yu Luo, Shuo Han, Yihan Hu +5RL for Language Model Reasoning Policy Optimization
Department of Foundation Model, 2012 Labs, Huawei Shanghai · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University · Department of Computer Science and Technology, Tsinghua University +1
Completion vs Optimality: Policy Gradient in Long-Horizon Cumulative-Damage Problems
May 26, 2026Wolfgang Maass, Sabine JanzenPolicy Gradient Proximal Policy Optimization
Saarland University & German Research Center for Artificial Intelligence (DFKI) · German Research Center for Artificial Intelligence (DFKI)
Joint Optimization of Training and Inference in Federated Edge Learning via Constrained Multi-Objective Deep Reinforcement Learning
May 25, 2026Zhen Li, Jun Cai, Chao Yang +1Multi-Objective Reinforcement Learning Edge Inference
Department of Electrical and Computer Engineering, Concordia University, Montreal, QC, H3G 1M8, Canada · School of Automation, Guangdong University of Technology, China
DeepSeekMath Meets Order Book: Group-Aware Policy Optimization for High-Frequency Directional Trading
May 25, 2026Sayak Charabarty, Souradip PalGroup Relative Policy Optimization Proximal Policy Optimization
Department of Computer Science Northwestern University Evanston, IL 60208, USA · School of Electrical and Computer Engineering Purdue University West Lafayette, IN 47906, USA
Not All Transitions Matter: Evidence from PPO
May 22, 2026Ajhesh BasnetProximal Policy Optimization Neural Network Training Dynamics
Department of Artificial Intelligence and Data Science KPR Institute of Engineering and Technology, Coimbatore
Self-Play Reinforcement Learning under Imperfect Information in Big 2
May 21, 2026Aalok PatwaGame-Playing Agents Self-Play RL
University of Pennsylvania
Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards
May 20, 2026Deokgyu Yoon, Hyungkyu Kang, Joongkyu Lee +4RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards
1Seoul National University · 2Upstage
GAE Falls Short in Imperfect-Information Self-Play Reinforcement Learning
May 19, 2026Zhiyuan Fan, Gabriele FarinaReinforcement Learning Self-Play RL
MIT
Shared Backbone PPO for Multi-UAV Communication Coverage with Connection Preservation
May 18, 2026Z. JiangMulti-Robot Systems Proximal Policy Optimization
Zhejiang University
Progressive Generalization Augmentation with Deeply Coupled RND-PPO and Domain-Prioritized Noise Injection for Robust Crop Management Reinforcement Learning
May 17, 2026Wu YangReinforcement Learning Proximal Policy Optimization
Chongho Bridge Group Limited
ERPPO: Entropy Regularization-based Proximal Policy Optimization
May 13, 2026Changha Lee, Gyusang ChoMulti-Agent Systems Proximal Policy Optimization
Korea Advanced Institute of Science and Technology, Daejeon, South Korea
Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction
May 12, 2026Zhong Guan, Yongjian Guo, Haoran Sun +5Asynchronous RL Reinforcement Learning
Tianjin University · Tsinghua University · JDT AI Infra +1
TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing
May 12, 2026Yuanpeng Li, Gefei Lin, Annie Qu +1Multi-Task RL Proximal Policy Optimization
UC Irvine · George Washington University · UC Santa Barbara +1
Neuromorphic Reinforcement Learning for Quadruped Locomotion Control on Uneven Terrain
May 10, 2026Zhuangyu Han, Abhronil SenguptaEquilibrium Propagation Terrain-Aware Robot Locomotion
School of Electrical Engineering and Computer Science Penn State University University Park, PA 16802
Rethinking Ratio-Based Trust Regions for Policy Optimization in Multi-Agent Reinforcement Learning
May 9, 2026Chulabhaya Wijesundara, Andrea Baisero, Zhongheng Li +3Multi-Agent System Optimization Policy Optimization
Northeastern University · STR · University of California, Irvine
Approximate Next Policy Sampling: Replacing Conservative Target Policy Updates in Deep RL
May 6, 2026Dillon Sandhu, Ronald ParrPolicy Optimization Policy Iteration
Department of Computer Science, Duke University, USA
ANO: Robust Policy Optimization via Bounded, Redescending Gain Fields
May 4, 2026Yiheng Zhang, Yiming Wang, Kaiyan Zhao +3Policy Gradient Methods Proximal Policy Optimization
University of Macau · Wuhan University · National University of Singapore +1
Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning
May 1, 2026Chengshuai Shi, Wenzhe Li, Xinran Liang +10Sequential Decision Making Proximal Policy Optimization
Princeton Language and Intelligence, Princeton University · Fudan University · Tsinghua University
Sample-efficient Neuro-symbolic Proximal Policy Optimization
Apr 28, 2026Simone Murari, Celeste Veronese, Daniele MeliReinforcement Learning Proximal Policy Optimization
Dept. of Computer Science, University of Verona, Italy
Bounded Ratio Reinforcement Learning
Apr 20, 2026Yunke Ao, Le Chen, Bruce D. Lee +5Reinforcement Learning Group Relative Policy Optimization
ETH Zurich · MPI for Intelligent Systems · University of Alberta +2
Representation over Routing: Diagnosing Temporal Routing Pathologies in Multi-Timescale PPO
Apr 15, 2026Jing SunReinforcement Learning Credit Assignment in RL
Information Engineering School, Chengyi College, Jimei University, Xiamen 361000
SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks
Apr 10, 2026Tianyi Wang, Yixia Li, Long Li +6RL for Language Model Reasoning Proximal Policy Optimization
Southern University of Science and Technology · Beijing University of Posts and Telecommunications · INFLY TECH +3
Territory Paint Wars: Diagnosing and Mitigating Failure Modes in Competitive Multi-Agent PPO
Apr 4, 2026Diyansha SinghMulti-Agent Reinforcement Learning Self-Play RL
Independent Researcher
Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training
Apr 2, 2026Dong Shu, Denghui Zhang, Jessica HullmanRL Post-Training RL for Language Model Reasoning
Northwestern University · Stevens Institute of Technology
Policy Improvement Reinforcement Learning
Apr 1, 2026Huaiyang Wang, Xiaojie Li, Xiaohan Wang +10RL for Language Models RL for Language Model Reasoning
Beihang University · Meituan · Peking University

Jun 6, 2026J/K move · Enter open · S save

Variational Proximal Policy Optimization

Ousmane Amadou Dia

RL for Language Model Reasoning Proximal Policy Optimization Variational Inference Stein Variational Gradient Descent RL from Human Feedback

Open paper AskPDF arXiv

Reinforcement Learning from Human Feedback via Proximal Policy Optimization often suffers from policy mode collapse, brittle exploration loops, and distribution drift. This paper introduces Variational Proximal Policy Optimization (

\textsc{VP}_2\textsc{O}

), a particle-based variational inference framework that maps policy optimization to Stein Variational Gradient Descent within a Mixture-of-Experts architecture. By leveraging functional kernels over localized expert prototypes alongside an expert orthogonalization loss,

\textsc{VP}_2\textsc{O}

introduces a geometry-based proximal-control mechanism that can reduce reliance on fixed clipping or KL schedules. Our results on a 33B/4B sparse Mixture-of-Experts model show several improvements across complex reasoning benchmarks, establishing a

+\mathbf{179}

ELO gain on Codeforces and a

\mathbf{32\%}

reduction in token count on AIME mathematical reasoning tasks.