Reinforcement Learning From Human Feedback

Reinforcement learning from human feedback (RLHF) aims to align large language models (LLMs) with human preferences by training them to maximize rewards derived from human evaluations. Current research emphasizes improving efficiency and robustness, focusing on techniques like macro-actions, distributional reward models, and novel policy optimization algorithms (e.g., variations of PPO and DPO) to address issues such as reward hacking and covariate shift in multi-turn dialogues. This field is crucial for developing safer and more helpful LLMs, impacting both the advancement of AI alignment research and the practical deployment of trustworthy AI systems across various applications.

Papers

September 25, 2023

Aligning Large Multimodal Models with Factually Augmented RLHF
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, Trevor Darrell
Vision Language Model Cross Modal Large Multimodal Model Multimodal Information Reinforcement Learning From Human Feedback Vision Language Alignment

September 12, 2023

Mitigating the Alignment Tax of RLHF
Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, Tong Zhang
Reinforcement Learning Reinforcement Learning From Human Feedback Model Averaging Multi Objective Reward

September 1, 2023

Efficient RLHF: Reducing the Memory Usage of PPO
Michael Santacroce, Yadong Lu, Han Yu, Yuanzhi Li, Yelong Shen
Reinforcement Learning Proximal Policy Optimization Reward Model Supervised Fine Tuning Policy Optimization Reinforcement Learning From Human Feedback Memory Management

July 11, 2023

Secrets of RLHF in Large Language Models Part I: PPO
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang
Reinforcement Learning Proximal Policy Optimization Reward Model Policy Optimization Reinforcement Learning From Human Feedback Dark Secret

June 25, 2023

Is RLHF More Difficult than Standard RL?
Yuanhao Wang, Qinghua Liu, Chi Jin
Reinforcement Learning Reinforcement Learning From Human Feedback Robust Reinforcement Learning Standard Reinforcement Learning

April 18, 2023

Provably Feedback-Efficient Reinforcement Learning via Active Reward Learning
Dingwen Kong, Lin F. Yang
Reinforcement Learning Reward Function Reinforcement Learning From Human Feedback Semi Supervised Reward

January 26, 2023

Principled Reinforcement Learning with Human Feedback from Pairwise or $K$-wise Comparisons
Banghua Zhu, Jiantao Jiao, Michael I. Jordan
Reinforcement Learning Human Feedback Consistent Comparison Inverse Reinforcement Learning Reinforcement Learning From Human Feedback

November 12, 2022

The Expertise Problem: Learning from Specialized Feedback
Oliver Daniels-Koch, Rachel Freedman
Reinforcement Learning Human Feedback Reinforcement Learning From Human Feedback Human Teacher Expert Feedback Expert Advice Problem