Proximal Policy Optimization

Proximal Policy Optimization (PPO) is a reinforcement learning algorithm used to train agents to make optimal decisions in complex environments, with a current research focus on improving its efficiency and robustness. Recent work explores enhancements such as refined credit assignment methods (e.g., VinePPO), incorporation of human feedback and safety mechanisms (e.g., HI-PPO, PRPO), and addressing challenges in high-dimensional spaces and sample efficiency through techniques like diffusion model integration. These advancements are significant for various applications, including robotics, autonomous systems, and large language model alignment, where PPO's ability to learn effective policies from interactions with the environment is crucial.

Papers

October 26, 2023

October 23, 2023

Enhancing Robotic Manipulation: Harnessing the Power of Multi-Task Reinforcement Learning and Single Life Reinforcement Learning in Meta-World
Ghadi Nehme, Ishan Sabane, Tejas Y. Deo
Real Power Proximal Policy Optimization Robotic Manipulation Multi Task Reinforcement Learning Soft Actor Critic Single Agent Reinforcement Learning Trust Region Policy Optimization Possible World

October 20, 2023

Absolute Policy Optimization
Weiye Zhao, Feihan Li, Yifan Sun, Rui Chen, Tianhao Wei, Changliu Liu
Proximal Policy Optimization Trust Region Policy Optimization Control Benchmark

October 16, 2023

ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, Zhi-Quan Luo
Reinforcement Learning Proximal Policy Optimization Efficient Reinforcement Learning Many Property

October 8, 2023

FP3O: Enabling Proximal Policy Optimization in Multi-Agent Cooperation with Parameter-Sharing Versatility
Lang Feng, Dong Xing, Junru Zhang, Gang Pan
Multi Agent Reinforcement Learning Proximal Policy Optimization Parameter Sharing Multi Agent Cooperation Multi Agent MuJoCo PPO Algorithm

September 30, 2023

Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment
Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, Jiantao Jiao
Reinforcement Learning Proximal Policy Optimization Trajectory Optimization LLM Alignment External Feedback

September 29, 2023

Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agents
Marco Pleines, Matthias Pallasch, Frank Zimmer, Mike Preuss
Reinforcement Learning Agent Smith Proximal Policy Optimization Episodic Memory Motion Memory

September 26, 2023

Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding
Jiacheng Liu, Andrew Cohen, Ramakanth Pasunuru, Yejin Choi, Hannaneh Hajishirzi, Asli Celikyilmaz
Language Model Proximal Policy Optimization Monte Carlo Tree Search Inference Time Valuation Model State of the Art Reinforcement Hybrid Text

September 25, 2023

Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learning
Lukas Schneider, Jonas Frey, Takahiro Miki, Marco Hutter
Proximal Policy Optimization Distributional Reinforcement Learning Quadrupedal Locomotion

September 13, 2023

Statistical Rejection Sampling Improves Preference Optimization
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, Jialu Liu
Large Language Model Language Model Proximal Policy Optimization Direct Preference Optimization Preference Optimization Preference Pair Rejection Sampling

September 11, 2023

Task-Oriented Cross-System Design for Timely and Accurate Modeling in the Metaverse
Zhen Meng, Kan Chen, Yufeng Diao, Changyang She, Guodong Zhao, Muhammad Ali Imran, Branka Vucetic
Deep Reinforcement Learning Proximal Policy Optimization Robotic Arm Ubiquitous Semantic Metaverse Accurate Model

September 6, 2023

Learning to Recharge: UAV Coverage Path Planning through Deep Reinforcement Learning
Mirco Theile, Harald Bayerlein, Marco Caccamo, Alberto L. Sangiovanni-Vincentelli
LeArning Abstract Deep Reinforcement Learning Unmanned Aerial Vehicle Proximal Policy Optimization Coverage Path Planning

September 1, 2023

Efficient RLHF: Reducing the Memory Usage of PPO
Michael Santacroce, Yadong Lu, Han Yu, Yuanzhi Li, Yelong Shen
Reinforcement Learning Proximal Policy Optimization Reward Model Supervised Fine Tuning Policy Optimization Reinforcement Learning From Human Feedback Memory Management

August 31, 2023

August 28, 2023

Maneuver Decision-Making Through Proximal Policy Optimization And Monte Carlo Tree Search
Zhang Hong-Peng
Reinforcement Learning Proximal Policy Optimization Monte Carlo Tree Search Maneuver Decision

August 23, 2023

Aligning Language Models with Offline Learning from Human Feedback
Jian Hu, Li Tao, June Yang, Chandler Zhou
Language Model Human Feedback Proximal Policy Optimization Supervised Fine Tuning

August 15, 2023

A Reinforcement Learning Approach for Performance-aware Reduction in Power Consumption of Data Center Compute Nodes
Akhilesh Raj, Swann Perarnau, Aniruddha Gokhale
Reinforcement Learning Proximal Policy Optimization Constructive Reduction Power Consumption Compute Node Exascale Computing

August 11, 2023

A Deep Recurrent-Reinforcement Learning Method for Intelligent AutoScaling of Serverless Functions
Siddharth Agarwal, Maria A. Rodriguez, Rajkumar Buyya
Proximal Policy Optimization Model Free Serverless Computing Auto Scaling Reinforcement Learning Architecture

Proximal Policy Optimization

Papers

Learning Regularized Graphon Mean-Field Games with Unknown Graphons

Coalitional Bargaining via Reinforcement Learning: An Application to Collaborative Vehicle Routing

Enhancing Robotic Manipulation: Harnessing the Power of Multi-Task Reinforcement Learning and Single Life Reinforcement Learning in Meta-World

Absolute Policy Optimization

ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models

FP3O: Enabling Proximal Policy Optimization in Multi-Agent Cooperation with Parameter-Sharing Versatility

Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment

Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agents

Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decoding

Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learning

Statistical Rejection Sampling Improves Preference Optimization

Task-Oriented Cross-System Design for Timely and Accurate Modeling in the Metaverse

Learning to Recharge: UAV Coverage Path Planning through Deep Reinforcement Learning

Efficient RLHF: Reducing the Memory Usage of PPO

Reinforcement learning for safety-critical control of an automated vehicle

Curriculum Proximal Policy Optimization with Stage-Decaying Clipping for Self-Driving at Unsignalized Intersections

Maneuver Decision-Making Through Proximal Policy Optimization And Monte Carlo Tree Search

Aligning Language Models with Offline Learning from Human Feedback

A Reinforcement Learning Approach for Performance-aware Reduction in Power Consumption of Data Center Compute Nodes

A Deep Recurrent-Reinforcement Learning Method for Intelligent AutoScaling of Serverless Functions