Reward Model Training

Reward model training aims to create models that accurately reflect human preferences, crucial for aligning large language models (LLMs) with desired behaviors. Current research focuses on improving reward model performance through techniques like contrastive learning, goal-conditioned training, and hybrid alignment frameworks that incorporate both token-level and sequence-level supervision, often employing Proximal Policy Optimization (PPO) or simpler ranking-based methods. These advancements address challenges such as reward score scaling, objective mismatches, and vulnerability to data poisoning, ultimately leading to more reliable and effective LLM alignment and improved performance on downstream tasks. The impact extends to safer and more helpful AI systems across various applications.

Papers

November 18, 2024

PSPO*: An Effective Process-supervised Policy Optimization for Reasoning Alignment
Jiawei Li, Xinyue Liang, Yizhe Yang, Chong Feng, Yang Gao
Complex Reasoning Reasoning Task Alignment Problem Policy OpTimization Reward Model Training Process Supervision

October 22, 2024

Optimal Design for Reward Modeling in RLHF
Antoine Scheid, Etienne Boursier, Alain Durmus, Michael I. Jordan, Pierre Ménard, Eric Moulines, Michal Valko
Reinforcement Learning Reinforcement Learning From Human Feedback Optimal Design Pairwise Preference Bandit Learning Reward Model Training

August 29, 2024

Fluent and Accurate Image Captioning with a Self-Trained Reward Model
Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara
Image Captioning Image Caption Captioning Datasets Reward Model Training

July 18, 2024

Learning Goal-Conditioned Representations for Language Reward Models
Vaskar Nath, Dylan Slack, Jeff Da, Yuntao Ma, Hugh Zhang, Spencer Whitehead, Sean Hendryx
Reinforcement Learning Reward Model Self Supervised Objective Traditional Reinforcement Learning Goal Conditioned Language Reward Reward Model Training

July 4, 2024

HAF-RM: A Hybrid Alignment Framework for Reward Model Training
Shujun Liu, Xiaoyu Shen, Yuhang Lai, Siyuan Wang, Shengbin Yue, Zengfeng Huang, Xuanjing Huang, Zhongyu Wei
Reward Model Optimal Alignment Reward Model Training

April 8, 2024

Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data
Tim Baumgärtner, Yang Gao, Dana Alon, Donald Metzler
Reinforcement Learning From Human Feedback Reward Model Training Best of N Poisoned Data Preference Poisoning

April 1, 2024

Prior Constraints-based Reward Model Training for Aligning Large Language Models
Hang Zhou, Chenglong Wang, Yimin Hu, Tong Xiao, Chunliang Zhang, Jingbo Zhu
Large Language Model Reward Model Training

March 28, 2024

Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model
Qi Gou, Cam-Tu Nguyen
Reinforcement Learning Direct Preference Optimization Preference Optimization Data Selection Reference Model Reward Model Training

March 20, 2024

RewardBench: Evaluating Reward Models for Language Modeling
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi
Reward Model Implicit Reward Reward Model Training Reward Value

March 15, 2024

Parameter Efficient Reinforcement Learning from Human Feedback
Hakim Sidahmed, Samrat Phatale, Alex Hutcheson, Zhuonan Lin, Zhang Chen, Zac Yu, Jarvis Jin, Simral Chaudhary, Roman Komarytsia, Christiane Ahlheim, Yonghao Zhu, Bowen Li, Saravanan Ganesh, Bill Byrne, Jessica Hoffmann, Hassan Mansoor, Wei Li, Abhinav Rastogi, Lucas Dixon
Reinforcement Learning Human Feedback Efficient Reinforcement Learning Reward Model Training

January 19, 2024

Reinforcement learning for question answering in programming domain using public community scoring as a human feedback
Alexey Gorbatovski, Sergey Kovalchuk
Reinforcement Learning Human Feedback Domain Specific Yes No Question Evaluation Set Community Question Answering Reward Model Training Preference Reward

October 31, 2023

The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback
Nathan Lambert, Roberto Calandra
Reinforcement Learning Human Feedback Model Based Reinforcement Learning Reward Model Human Alignment Model Mismatch Reward Model Training

May 25, 2023

RewriteLM: An Instruction-Tuned Large Language Model for Text Rewriting
Lei Shu, Liangchen Luo, Jayakumar Hoskere, Yun Zhu, Yinxiao Liu, Simon Tong, Jindong Chen, Lei Meng
Text Modality Natural Language Instruction Chain of Thought Prompting Instruction Generation Rewrite System Reward Model Training

April 11, 2023

RRHF: Rank Responses to Align Language Models with Human Feedback without tears
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, Fei Huang
Large Language Model Language Model Reinforcement Learning Human Feedback Large Language Model Alignment RLHF V Reward Model Training Tear Film

July 18, 2022

MAD for Robust Reinforcement Learning in Machine Translation
Domenic Donato, Lei Yu, Wang Ling, Chris Dyer
Machine Translation Proximal Policy Optimization Robust Reinforcement Learning Machine Translation Model Policy Gradient Algorithm Reward Model Training Mad Icp