Proximal Policy Optimization

Proximal Policy Optimization (PPO) is a reinforcement learning algorithm used to train agents to make optimal decisions in complex environments, with a current research focus on improving its efficiency and robustness. Recent work explores enhancements such as refined credit assignment methods (e.g., VinePPO), incorporation of human feedback and safety mechanisms (e.g., HI-PPO, PRPO), and addressing challenges in high-dimensional spaces and sample efficiency through techniques like diffusion model integration. These advancements are significant for various applications, including robotics, autonomous systems, and large language model alignment, where PPO's ability to learn effective policies from interactions with the environment is crucial.

Papers

May 1, 2024

No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO
Skander Moalla, Andrea Miele, Razvan Pascanu, Caglar Gulcehre
Reinforcement Learning Proximal Policy Optimization Individual Representation Appropriate Trust Policy Optimization Event Collapse Dynamic Representation Policy Deep Reinforcement Learning Representation Collapse

April 29, 2024

DPO Meets PPO: Reinforced Token Optimization for RLHF
Han Zhong, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, Liwei Wang
Proximal Policy Optimization Direct Preference Optimization Policy Optimization Reinforcement Learning From Human Feedback Bandit Model Token Optimization

April 26, 2024

An Explainable Deep Reinforcement Learning Model for Warfarin Maintenance Dosing Using Policy Distillation and Action Forging
Sadjad Anzabi Zadeh, W. Nick Street, Barrett W. Thomas
Deep Reinforcement Learning Proximal Policy Optimization Action Feature Policy Distillation Dosing Model Explainable Deep Reinforcement Learning Warfarin Dosing

April 25, 2024

REBEL: Reinforcement Learning via Regressing Relative Rewards
Zhaolin Gao, Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, J. Andrew Bagnell, Jason D. Lee, Wen Sun
Reinforcement Learning Proximal Policy Optimization Policy OpTimization Natural Policy Gradient

April 20, 2024

On the stability of Lipschitz continuous control problems and its application to reinforcement learning
Namkyeong Cho, Yeoneung Kim
Reinforcement Learning Application Proficiency Proximal Policy Optimization Core Stability Optimal Control Lipschitz Operator Control Problem Hamilton Jacobi Bellman

April 18, 2024

JointPPO: Diving Deeper into the Effectiveness of PPO in Multi-Agent Reinforcement Learning
Chenxing Liu, Guizhong Liu
Multi Agent Reinforcement Learning Proximal Policy Optimization Policy Optimization Multi Agent Challenge Diving Deep Transformer Based Policy

April 16, 2024

Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, Yi Wu
Reinforcement Learning Proximal Policy Optimization Direct Preference Optimization Comprehensive Study Policy Optimization Actor Critic Algorithm LLM Alignment Reward Free

April 14, 2024

Generative AI Agents with Large Language Model for Satellite Networks via a Mixture of Experts Transmission
Ruichen Zhang, Hongyang Du, Yinqiu Liu, Dusit Niyato, Jiawen Kang, Zehui Xiong, Abbas Jamalipour, Dong In Kim
Mixture Component Generative Artificial Intelligence Proximal Policy Optimization Satellite Network

April 12, 2024

Dataset Reset Policy Optimization for RLHF
Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D. Lee, Wen Sun
Reinforcement Learning Proximal Policy Optimization Policy OpTimization RLHF V Offline Preference Reset Free

April 5, 2024

April 3, 2024

March 12, 2024

Adaptive Gain Scheduling using Reinforcement Learning for Quadcopter Control
Mike Timmerman, Aryan Patel, Tim Reinhart
Reinforcement Learning Proximal Policy Optimization Continuous Control Quadrotor Control Automatic Gain Control

March 11, 2024

$\mathbf{(N,K)}$-Puzzle: A Cost-Efficient Testbed for Benchmarking Reinforcement Learning Algorithms in Generative Language Model
Yufeng Zhang, Liyu Chen, Boyi Liu, Yingxiang Yang, Qiwen Cui, Yunzhe Tao, Hongxia Yang
Language Model Reinforcement Learning Proximal Policy Optimization Reinforcement Learning Algorithm Generative Language Model Reinforcement Learning Benchmark K Content Test Bed Direct Policy

March 4, 2024

ZSL-RPPO: Zero-Shot Learning for Quadrupedal Locomotion in Challenging Terrains using Recurrent Proximal Policy Optimization
Yao Zhao, Tao Wu, Yijie Zhu, Xiang Lu, Jun Wang, Haitham Bou-Ammar, Xinyu Zhang, Peng Du
LeArning Abstract Proximal Policy Optimization Quadrupedal Robot Quadrupedal Locomotion Real to Sim

February 28, 2024

Is Crowdsourcing Breaking Your Bank? Cost-Effective Fine-Tuning of Pre-trained Language Models with Proximal Policy Optimization
Shuo Yang, Gjergji Kasneci
Language Model Pre Trained Language Model Proximal Policy Optimization Crowdsourcing Context Self Supervised Task Memory Efficient Fine Tuning

February 25, 2024

Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibration
Xin Mao, Feng-Lin Li, Huimin Xu, Wei Zhang, Anh Tuan Luu
Reinforcement Learning Human Feedback Proximal Policy Optimization Language Model Alignment Reward Value

February 22, 2024

Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker
Reinforcement Learning LeArning Abstract Medical LLM Human Feedback Proximal Policy Optimization Style Representation Online Reinforcement Learning AI Alignment Basic Concept

Proximal Policy Optimization

Papers

No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO

DPO Meets PPO: Reinforced Token Optimization for RLHF

An Explainable Deep Reinforcement Learning Model for Warfarin Maintenance Dosing Using Policy Distillation and Action Forging

REBEL: Reinforcement Learning via Regressing Relative Rewards

On the stability of Lipschitz continuous control problems and its application to reinforcement learning

JointPPO: Diving Deeper into the Effectiveness of PPO in Multi-Agent Reinforcement Learning

Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study

Generative AI Agents with Large Language Model for Satellite Networks via a Mixture of Experts Transmission

Dataset Reset Policy Optimization for RLHF

Enhancing IoT Intelligence: A Transformer-based Reinforcement Learning Methodology

A proximal policy optimization based intelligent home solar management

Heterogeneous Multi-Agent Reinforcement Learning for Zero-Shot Scalable Collaboration

Learning Quadrupedal Locomotion via Differentiable Simulation

Solving a Real-World Optimization Problem Using Proximal Policy Optimization with Curriculum Learning and Reward Engineering

Adaptive Gain Scheduling using Reinforcement Learning for Quadcopter Control

$\mathbf{(N,K)}$-Puzzle: A Cost-Efficient Testbed for Benchmarking Reinforcement Learning Algorithms in Generative Language Model

ZSL-RPPO: Zero-Shot Learning for Quadrupedal Locomotion in Challenging Terrains using Recurrent Proximal Policy Optimization

Is Crowdsourcing Breaking Your Bank? Cost-Effective Fine-Tuning of Pre-trained Language Models with Proximal Policy Optimization

Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibration

Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs