Proximal Policy Optimization

Also known as PPO

Proximal Policy Optimization (PPO) is an actor–critic method that learns actions from rewards while limiting how much the policy changes at each update, commonly through clipped probability ratios and generalized advantage estimation. Current approaches refine this update rule with divergence-based trust regions, adaptive clipping, better value critics, and selective reuse of past experience; safety constraints, demonstrations, and sequence models are also used to guide learning. PPO is applied to benchmarks such as Atari and MuJoCo, as well as robotics, multi-agent control, and language-model reasoning. A central challenge is improving sample efficiency without making updates unstable: clipping and on-policy data can help control risk, but may restrict useful learning, while critic errors and stale data can undermine performance.

Latest papers

116 papers20 papers in the last four weeks, up 150% on the four weeks before. 0.1% of all new papers.

Match or Replay: Self Imitating Proximal Policy Optimization
Mar 29, 2026Gaurav Chaudhary, Laxmidhar Behera, Washim Uddin MondalRL Exploration Sample-Efficient RL
Department of Electrical Engineering, Indian Institute of Technology Kanpur
Bitboard version of Tetris AI
Mar 24, 2026Xingguo Chen, Pingshou Xiong, Zhenyu Luo +6Game-Playing Agents RL Benchmarks
School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing, Jiangsu, 210023, China
ContractionPPO: Certified Reinforcement Learning via Differentiable Contraction Layers
Mar 20, 2026Vrushabh Zinage, Narek Harutyunyan, Eric Verheyden +2Robust RL Reinforcement Learning
California Institute of Technology · Brown University
Bridging Network Fragmentation: A Semantic-Augmented DRL Framework for UAV-aided VANETs
Mar 19, 2026Gaoxiang Cao, Wenke Yuan, Huasen He +4Intelligent Transportation Systems Wireless Communications
Department of Automation, University of Science & Technology of China
Multi-Agent Reinforcement Learning Counteracts Delayed CSI in Multi-Satellite Systems
Mar 17, 2026Marios Aristodemou, Yasaman Omid, Sangarapillai Lambotharan +2Wireless Communications Proximal Policy Optimization
Department of Computer Science, University of York, York, U.K. · Wolfson School of Mechanical, Electrical and Manufacturing Engineering, Loughborough University, Loughborough, U.K. · Institute for Digital Technologies, Loughborough University (London Campus), London, U.K. +1
Efficient Morphology-Control Co-Design via Stackelberg Proximal Policy Optimization
Mar 16, 2026Yanning Dai, Yuhui Wang, Dylan R. Ashley +1Robot Morphology-Control Co-Design Bilevel Optimization
Center of Excellence for Generative AI, King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia. · Dalle Molle Institute for Artificial Intelligence Research (IDSIA), Lugano, Switzerland. · Università della Svizzera italiana (USI), Lugano, Switzerland. +1
Optimize Wider, Not Deeper: Consensus Aggregation for Policy Optimization
Mar 13, 2026Zelal Su, Mustafaoglu, Sungyoung Lee +3Policy Optimization Proximal Policy Optimization
Lain · Department of Computer Science, University of Texas at Austin · Department of Electrical and Computer Engineering, University of Texas at Austin
UAV-MARL: Multi-Agent Reinforcement Learning for Time-Critical and Dynamic Medical Supply Delivery
Mar 11, 2026Islam Guven, Mehmet ParlakMulti-Agent Coordination Multi-Agent Reinforcement Learning
ICTEAM, Université catholique de Louvain Ottignies-Louvain-la-Neuve, Belgium
When Learning Rates Go Wrong: Early Structural Signals in PPO Actor-Critic
Mar 10, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +3Reinforcement Learning Proximal Policy Optimization
Universitat Polit`ecnica de Val`encia, Valencia, Spain · Universitat Jaume I, Castell´o de la Plana, Spain · Openchip & Software Technologies, Barcelona, Spain
Adaptive Double-Booking Strategy for Outpatient Scheduling Using Multi-Objective Reinforcement Learning
Mar 7, 2026Ninda Nurseha Amalina, Heungjo AnHealthcare Multi-Objective Reinforcement Learning
Department of Industrial Engineering, Kumoh National Institute of Technology, Gumi City, South Korea
Optimistic Policy Regularization: Elite-Statistic Reward Shaping
Mar 6, 2026Mai Pham, Vikrant Vaze, Peter ChinRegularized RL Sample-Efficient RL
Thayer School of Engineering, Dartmouth College, USA
Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environments
Mar 6, 2026Michael Beukman, Khimya Khetarpal, Zeyu Zheng +4Reinforcement Learning Proximal Policy Optimization
FLAIR, University of Oxford · Google DeepMind
TADPO: Reinforcement Learning Goes Off-road
Mar 6, 2026Zhouchonghao Wu, Raymond Song, Vedant Mundheda +3End-to-End Autonomous Driving Autonomous Driving
Robotics Institute, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213 · National Robotics Engineering Center, Robotics Institute, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15201
BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning
Mar 5, 2026Yuan Li, Bo Wang, Yufei Gao +4RL for Language Models Proximal Policy Optimization
Fudan University · Shanghai Innovation Institute
Diffusion Policy through Conditional Proximal Policy Optimization
Mar 5, 2026Ben Liu, Shunpeng Yang, Hua ChenDiffusion Policy RL for Diffusion Models
Southern University of Science and Technology, Shenzhen, China · Limx Dynamics · Hong Kong University of Science and Technology, Hongkong, China +1
When Sensors Fail: Temporal Sequence Models for Robust PPO under Sensor Drift
Mar 4, 2026Kevin Vogt-Lowell, Theodoros Tsiligkaridis, Rodney Lafuente-Mercado +4Robust RL Partially Observable RL
MIT Lincoln Laboratory · Harvard · MIT
Swimming Under Constraints: A Safe Reinforcement Learning Framework for Quadrupedal Bio-Inspired Propulsion
Mar 4, 2026Xinyu Cui, Fei Han, Hang Xu +9Constrained RL Underwater Robotics
FSI lab, Westlake University, Hangzhou, 310030, China. · Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China. · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China +3
Integrating LTL Constraints into PPO for Safe Reinforcement Learning
Mar 1, 2026Maifang Zhang, Hang Yu, Qian Zuo +3Safe RL Constrained RL
School of Informatics, University of Edinburgh, Edinburgh EH8 9AB, Scotland. · School of Computer Science, Faculty of Engineering, University of Sydney, Darlington NSW 2008, Australia. · School of Engineering and Physical Sciences, Heriot-Watt University, Edinburgh EH14 4AS, Scotland.
Stabilizing Policy Optimization via Logits Convexity
Mar 1, 2026Hongzhan Chen, Tao Yang, Yuhua Zhu +3Policy Optimization Reinforcement Learning
School of Computer Science and Engineering, Sun Yat-sen University, China · Shanghai Innovation Institute · Wechat Search, Tencent Inc, China +1
Actor-Critic Pretraining for Proximal Policy Optimization
Feb 27, 2026Andreas Kernbach, Amr Elsheikh, Nicolas Grupp +2Robotic RL Sample-Efficient RL
Fraunhofer Institute for Manufacturing Engineering and Automation IPA, Stuttgart, Germany · Institute of Industrial Manufacturing and Management IFF, University of Stuttgart, Germany · Institute for Control Engineering of Machine Tools and Manufacturing Units ISW, University of Stuttgart, Germany +1
Learning Robust Penetration Testing Policies under Partial Observability: A systematic evaluation
Sep 24, 2025Raphael Simon, Pieter Libin, Wim MeesReinforcement Learning Partially Observable RL
Cyber Defence Lab, CISS Department Royal Military Academy · AI Lab, Department of Computer Science Vrije Universiteit Brussel
Policy Improvement with Style-Specific Demonstrations
Jun 20, 2025Lingfeng Li, Yunlong Lu, Yongyi Wang +1Reinforcement Learning Game-Playing Agents
School of Computer Science, Peking University, Beijing 100871, China
Learning Large-Scale Competitive Team Behaviors with Mean-Field Interactions
Apr 29, 2025Bhavini Jeloka, Yue Guan, Panagiotis TsiotrasMean-Field MARL Multi-Agent Reinforcement Learning
School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA, USA
Atomic Proximal Policy Optimization for Electric Robo-Taxi Dispatch and Charger Allocation
Feb 19, 2025Jim Dai, Manxi Wu, Zhanhao ZhangProximal Policy Optimization Electric Vehicle Charging
Operations Research and Information Engineering, Cornell University · Department of Civil and Environmental Engineering, University of California, Berkeley
Enhancing Hardware Fault Tolerance in Machines with Reinforcement Learning Policy Gradient Algorithms
Jul 21, 2024Sheila Schoepp, Mehran Taghian, Shotaro Miwa +3RL Control Reinforcement Learning
Department of Computing Science, University of Alberta, 8900 114 St NW, Edmonton, Alberta, T6G 2S4, Canada · Advanced Technology R&D Center, Mitsubishi Electric Corporation, 8-1-1,Tsukaguchi-honmachi, Amagasaki-shi, Hyogo, 661-8661, Japan · Information Technology R&D Center, Mitsubishi Electric Corporation, 5-1-1, Ofuna, Kamakura-shi, Kanagawa, 247-8501, Japan +1
Transductive Off-policy Proximal Policy Optimization
Jun 6, 2024Yaozhong Gan, Xiaoyang Tan, Zhe Wu +1Policy Optimization Proximal Policy Optimization
Independent Researcher

Mar 29, 2026J/K move · Enter open · S save

Match or Replay: Self Imitating Proximal Policy Optimization

Gaurav Chaudhary, Laxmidhar Behera, Washim Uddin Mondal

Department of Electrical Engineering, Indian Institute of Technology Kanpur

RL Exploration Sample-Efficient RL Sparse-Reward RL Proximal Policy Optimization Imitation Learning

Open paper AskPDF arXiv

Reinforcement Learning (RL) agents often struggle with inefficient exploration, particularly in environments with sparse rewards. Traditional exploration strategies can lead to slow learning and suboptimal performance because agents fail to systematically build on previously successful experiences, thereby reducing sample efficiency. To tackle this issue, we propose a self-imitating on-policy algorithm that enhances exploration and sample efficiency by leveraging past high-reward state-action pairs to guide policy updates. Our method incorporates self-imitation by using optimal transport distance in dense reward environments to prioritize state visitation distributions that match the most rewarding trajectory. In sparse-reward environments, we uniformly replay successful self-encountered trajectories to facilitate structured exploration. Experimental results across diverse environments demonstrate substantial improvements in learning efficiency, including MuJoCo for dense rewards and the partially observable 3D Animal-AI Olympics and multi-goal PointMaze for sparse rewards. Our approach achieves faster convergence and significantly higher success rates compared to state-of-the-art self-imitating RL baselines. These findings underscore the potential of self-imitation as a robust strategy for enhancing exploration in RL, with applicability to more complex tasks.