PPO Algorithm

Proximal Policy Optimization (PPO) is a reinforcement learning algorithm used to train agents to make optimal decisions in complex environments by iteratively improving a policy while constraining policy updates to prevent drastic changes. Current research focuses on enhancing PPO's efficiency and robustness, particularly through modifications like appraisal-guided PPO for modeling cognitive processes and variants designed for multi-agent systems and handling shared resources. These advancements are significant for improving the performance and applicability of reinforcement learning in diverse fields, including robotics, natural language processing (via alignment with human preferences), and resource optimization problems.

Papers

August 27, 2024

UNA: Unifying Alignments of RLHF/PPO, DPO and KTO by a Generalized Implicit Reward Function
Zhichao Wang, Bin Bi, Can Huang, Shiva Kumar Pentyala, Zixu James Zhu, Sitaram Asur, Na Claire Cheng
Fine Tuning Reward Signal Better Alignment Implicit Reward RLHF V PPO Algorithm

July 29, 2024

Appraisal-Guided Proximal Policy Optimization: Modeling Psychological Disorders in Dynamic Grid World
Hari Prasad, Chinnu Jacob, Imthias Ahamed T. P
Reinforcement Learning Mental Health Proximal Policy Optimization PPO Algorithm Grid World Continuous Emotion

July 23, 2024

A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO and More
Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Zixu, Zhu, Xiang-Bo Mao, Sitaram Asur, Na, Cheng
Large Language Model Self Supervised Learning Training Data LLM Alignment Instruction Fine Tuning Pre Training Corpus RLHF V PPO Algorithm

June 13, 2024

Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback
Hamish Ivison, Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A. Smith, Yejin Choi, Hannaneh Hajishirzi
Preference Feedback Best Practice Modern Language Model Preference Dataset PPO Algorithm

May 25, 2024

Deep Reinforcement Learning with Enhanced PPO for Safe Mobile Robot Navigation
Hamid Taheri, Seyed Rasoul Hosseini, Mohammad Ali Nekoui
Deep Reinforcement Learning Autonomous Navigation Autonomous Robot Safe Robot Navigation PPO Algorithm Obstacle Free

April 5, 2024

Heterogeneous Multi-Agent Reinforcement Learning for Zero-Shot Scalable Collaboration
Xudong Guo, Daming Shi, Junjie Yu, Wenhui Fan
Multi Agent Reinforcement Learning Proximal Policy Optimization Multi Agent Challenge Zero Shot Coordination PPO Algorithm Heterogeneous Multi Agent Reinforcement Learning Real World Multi Agent

March 24, 2024

The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization
Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, Lewis Tunstall
Reinforcement Learning Case Study Human Feedback OpenAI Codex Reinforcement Learning From Human Feedback Model Checkpoint PPO Algorithm

December 6, 2023

Can language agents be alternatives to PPO? A Preliminary Empirical Study On OpenAI Gym
Junjie Sheng, Zixiao Huang, Chuyun Shen, Wenhao Li, Yun Hua, Bo Jin, Hongyuan Zha, Xiangfeng Wang
Language Agent Sequential Decision Making Problem Shot Approach Sequential Decision Making Task PPO Algorithm Previously Generated Alternative OpenAI Gym Language Capability

October 26, 2023

Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks
Ryan Sullivan, Akarsh Kumar, Shengyi Huang, John P. Dickerson, Joseph Suarez
Proximal Policy Optimization DeepMind Control Suite Learning Environment Robust Reward PPO Algorithm

October 8, 2023

FP3O: Enabling Proximal Policy Optimization in Multi-Agent Cooperation with Parameter-Sharing Versatility
Lang Feng, Dong Xing, Junru Zhang, Gang Pan
Multi Agent Reinforcement Learning Proximal Policy Optimization Parameter Sharing Multi Agent Cooperation Multi Agent MuJoCo PPO Algorithm

May 8, 2023

Local Optimization Achieves Global Optimality in Multi-Agent Reinforcement Learning
Yulai Zhao, Zhuoran Yang, Zhaoran Wang, Jason D. Lee
Multi Agent Reinforcement Learning Markov Game Multi Agent Policy PPO Algorithm

April 27, 2023

Adversarial Policy Optimization in Deep Reinforcement Learning
Md Masudur Rahman, Yexiang Xue
Deep Reinforcement Learning Policy OpTimization Based Policy Adversarial Policy PPO Algorithm Deep Reinforcement Learning Algorithm

February 27, 2023

A Reinforcement Learning Approach for Scheduling Problems With Improved Generalization Through Order Swapping
Deepak Vivekanandan, Samuel Wirth, Patrick Karlbauer, Noah Klarmann
Task Scheduling Metaheuristic Algorithm Order Matter Improved Generalization Scheduling Problem Job Shop Scheduling Problem PPO Algorithm

December 15, 2022

Multi-Agent Reinforcement Learning with Shared Resources for Inventory Management
Yuandong Ding, Mingxiao Feng, Guozi Liu, Wei Jiang, Chuheng Zhang, Li Zhao, Lei Song, Houqiang Li, Yan Jin, Jiang Bian
Multi Agent Reinforcement Learning Stochastic Game Inventory Management PPO Algorithm Replenishment Decision

September 1, 2022

DRL Enabled Coverage and Capacity Optimization in STAR-RIS Assisted Networks
Xinyu Gao, Wenqiang Yi, Yuanwei Liu, Jianhua Zhang, Ping Zhang
Wireless Network Higher Coverage Rate PPO Algorithm Ga DRL STAR Ri Capacity Optimization

January 31, 2022

Trust Region Bounds for Decentralized PPO Under Non-stationarity
Mingfei Sun, Sam Devlin, Jacob Beck, Katja Hofmann, Shimon Whiteson
Actor Critic Cooperative Multi Agent Reinforcement Learning Trust Region Non Stationarity PPO Algorithm

December 7, 2021

PTR-PPO: Proximal Policy Optimization with Prioritized Trajectory Replay
Xingxing Liang, Yang Ma, Yanghe Feng, Zhong Liu
Proximal Policy Optimization PPO Algorithm Trajectory Wise Reward Trajectory Preference Trajectory Replay

October 20, 2021

CIM-PPO:Proximal Policy Optimization with Liu-Correntropy Induced Metric
Yunxiao Guo, Han Long, Xiaojun Duan, Kaiyuan Feng, Maochu Li, Xiaying Ma
Reinforcement Learning Policy Gradient Proximal Policy Optimization KL Divergence PPO Algorithm Correntropy Learning

PPO Algorithm

Papers

UNA: Unifying Alignments of RLHF/PPO, DPO and KTO by a Generalized Implicit Reward Function

Appraisal-Guided Proximal Policy Optimization: Modeling Psychological Disorders in Dynamic Grid World

A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO and More

Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback

Deep Reinforcement Learning with Enhanced PPO for Safe Mobile Robot Navigation

Heterogeneous Multi-Agent Reinforcement Learning for Zero-Shot Scalable Collaboration

The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization

Can language agents be alternatives to PPO? A Preliminary Empirical Study On OpenAI Gym

Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks

FP3O: Enabling Proximal Policy Optimization in Multi-Agent Cooperation with Parameter-Sharing Versatility

Local Optimization Achieves Global Optimality in Multi-Agent Reinforcement Learning

Adversarial Policy Optimization in Deep Reinforcement Learning

A Reinforcement Learning Approach for Scheduling Problems With Improved Generalization Through Order Swapping

Multi-Agent Reinforcement Learning with Shared Resources for Inventory Management

DRL Enabled Coverage and Capacity Optimization in STAR-RIS Assisted Networks

Trust Region Bounds for Decentralized PPO Under Non-stationarity

PTR-PPO: Proximal Policy Optimization with Prioritized Trajectory Replay

CIM-PPO:Proximal Policy Optimization with Liu-Correntropy Induced Metric