Proximal Policy Optimization

Proximal Policy Optimization (PPO) is a reinforcement learning algorithm used to train agents to make optimal decisions in complex environments, with a current research focus on improving its efficiency and robustness. Recent work explores enhancements such as refined credit assignment methods (e.g., VinePPO), incorporation of human feedback and safety mechanisms (e.g., HI-PPO, PRPO), and addressing challenges in high-dimensional spaces and sample efficiency through techniques like diffusion model integration. These advancements are significant for various applications, including robotics, autonomous systems, and large language model alignment, where PPO's ability to learn effective policies from interactions with the environment is crucial.

Papers

October 10, 2024

On Reward Transferability in Adversarial Inverse Reinforcement Learning: Insights from Random Matrix Theory
Yangchun Zhang, Wang Zhou, Yirui Zhou
Proximal Policy Optimization Reinforcement Learning Algorithm Inverse Reinforcement Learning Tilting Angle Policy Actor Critic Imitation Policy Adversarial Inverse Reinforcement Learning

October 7, 2024

As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss
Xin Mao, Feng-Lin Li, Huimin Xu, Wei Zhang, Wang Chen, Anh Tuan Luu
Fine Tuning Proximal Policy Optimization Direct Preference Optimization Preference Optimization LLM Alignment Semantic Loss

October 2, 2024

VinePPO: Unlocking RL Potential For LLM Reasoning Through Refined Credit Assignment
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, Nicolas Le Roux
Large Language Model Proximal Policy Optimization Complex Reasoning Task LLM Reasoning Credit Assignment State of the Art Reinforcement Value Network

September 25, 2024

Revisiting Space Mission Planning: A Reinforcement Learning-Guided Approach for Multi-Debris Rendezvous
Agni Bandyopadhyay, Guenther Waxenegger-Wilfing
Deep Reinforcement Learning Constructive Approach Proximal Policy Optimization Positive Reinforcement Reinforcement Learning Approach Rendezvous Problem Space Debris Mission Planning Space Debris Removal

September 24, 2024

September 23, 2024

A novel agent with formal goal-reaching guarantees: an experimental study with a mobile robot
Grigory Yaremenko, Dmitrii Dobriborsci, Roman Zashchitin, Ruben Contreras Maestre, Ngoc Quoc Huy Hoang, Pavel Osinenko
Reinforcement Learning Robotics Domain Mobile Robot Proximal Policy Optimization Experimental Study State Action Pair Formal Guarantee Agent Capability

September 17, 2024

September 15, 2024

September 13, 2024

AIPO: Improving Training Objective for Iterative Preference Optimization
Yaojie Shen, Xinyao Wang, Yulei Niu, Ying Zhou, Lexin Tang, Libo Zhang, Fan Chen, Longyin Wen
Proximal Policy Optimization Preference Optimization Training Objective

September 11, 2024

Policy Filtration in RLHF to Fine-Tune LLM for Code Generation
Wei Shen, Chuheng Zhang
Reinforcement Learning Medical LLM Code Generation Proximal Policy Optimization Reward Model Policy Learning Reinforcement Learning From Human Feedback Policy Search

September 9, 2024

Developing Path Planning with Behavioral Cloning and Proximal Policy Optimization for Path-Tracking and Static Obstacle Nudging
Mingyan Zhou, Biao Wang, Tian Tan, Xiatao Sun
Reinforcement Learning Autonomous Driving Imitation Learning Proximal Policy Optimization Trajectory Planning Behavior Cloning Static Obstacle

September 7, 2024

Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn
Hongyao Tang, Glen Berseth
Reinforcement Learning Proximal Policy Optimization Learning Dynamic Net Present Value Policy Iteration Churn Prediction Customer Churn

September 2, 2024

August 28, 2024

Comparison of Model Predictive Control and Proximal Policy Optimization for a 1-DOF Helicopter System
Georg Schäfer, Jakob Rehrl, Stefan Huber, Simon Hirlaender
Deep Reinforcement Learning Predictive Control Consistent Comparison Proximal Policy Optimization Linear Quadratic Regulator

August 27, 2024

Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data
Han Xia, Songyang Gao, Qiming Ge, Zhiheng Xi, Qi Zhang, Xuanjing Huang
Reinforcement Learning Optimal Policy Proximal Policy Optimization Direct Preference Optimization Token Level

August 9, 2024

Kolmogorov-Arnold Network for Online Reinforcement Learning
Victor Augusto Kich, Jair Augusto Bottega, Raul Steinmetz, Ricardo Bedin Grando, Ayano Yorozu, Akihisa Ohya
Proximal Policy Optimization Kolmogorov Arnold Network Online Reinforcement Learning Reinforcement Learning Model

Proximal Policy Optimization

Papers

On Reward Transferability in Adversarial Inverse Reinforcement Learning: Insights from Random Matrix Theory

As Simple as Fine-tuning: LLM Alignment via Bidirectional Negative Feedback Loss

VinePPO: Unlocking RL Potential For LLM Reasoning Through Refined Credit Assignment

Revisiting Space Mission Planning: A Reinforcement Learning-Guided Approach for Multi-Debris Rendezvous

Safe Navigation for Robotic Digestive Endoscopy via Human Intervention-based Reinforcement Learning

NavRL: Learning Safe Flight in Dynamic Environments

A novel agent with formal goal-reaching guarantees: an experimental study with a mobile robot

Leveraging Symmetry to Accelerate Learning of Trajectory Tracking Controllers for Free-Flying Robotic Systems

On-policy Actor-Critic Reinforcement Learning for Multi-UAV Exploration

Proximal Ranking Policy Optimization for Practical Safety in Counterfactual Learning to Rank

Mitigating Dimensionality in 2D Rectangle Packing Problem under Reinforcement Learning Schema

AIPO: Improving Training Objective for Iterative Preference Optimization

Policy Filtration in RLHF to Fine-Tune LLM for Code Generation

Developing Path Planning with Behavioral Cloning and Proximal Policy Optimization for Path-Tracking and Static Obstacle Nudging

Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn

Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimization

Solving Integrated Process Planning and Scheduling Problem via Graph Neural Network Based Deep Reinforcement Learning

Comparison of Model Predictive Control and Proximal Policy Optimization for a 1-DOF Helicopter System

Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data

Kolmogorov-Arnold Network for Online Reinforcement Learning