Reward Value

Reward value, a crucial element in reinforcement learning, aims to quantify the desirability of different states or actions, guiding agents towards optimal behavior. Current research focuses on improving reward model accuracy and efficiency, exploring methods like uncertainty-aware models and zeroth-order policy gradients to bypass the limitations of traditional reward inference. These advancements are vital for aligning large language models with human preferences and enabling efficient reinforcement learning in complex, real-world scenarios where reward evaluation is costly. The development of robust and reliable reward models is key to advancing the field and unlocking the full potential of reinforcement learning across diverse applications.

Papers

October 18, 2024

How to Evaluate Reward Models for RLHF
Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N. Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, Ion Stoica
Reward Model RLHF V Reward Value

October 1, 2024

Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown
Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, Junge Zhang
Reward Model Reward Prediction Reward Ambiguity Reward Value

September 25, 2024

Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference
Qining Zhang, Lei Ying
Reinforcement Learning Human Feedback Policy Gradient Direct Preference Optimization Reward Value Reward Inference

March 20, 2024

RewardBench: Evaluating Reward Models for Language Modeling
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi
Reward Model Implicit Reward Reward Model Training Reward Value

February 25, 2024

Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibration
Xin Mao, Feng-Lin Li, Huimin Xu, Wei Zhang, Anh Tuan Luu
Reinforcement Learning Human Feedback Proximal Policy Optimization Language Model Alignment Reward Value

October 23, 2023

Policy Gradient with Kernel Quadrature
Satoshi Hayakawa, Tetsuro Morimura
Gaussian Process Policy Gradient Reinforcement Learning Task Reward Value Kernel Quadrature

September 26, 2023

STARC: A General Framework For Quantifying Differences Between Reward Functions
Joar Skalse, Lucy Farnik, Sumeet Ramesh Motwani, Erik Jenner, Adam Gleave, Alessandro Abate
Reinforcement Learning Reward Function General Framework Reward Learning Reward Value Unraveling Learning Difference

May 17, 2023

A proof of imitation of Wasserstein inverse reinforcement learning for multi-objective optimization
Akira Kitaoka, Riki Eto
Multi Objective Optimization Feature Imitation Partial Proof Optimal Solution Iteration Head Reward Value

November 23, 2022

Actively Learning Costly Reward Functions for Reinforcement Learning
André Eberhard, Houssam Metni, Georg Fahland, Alexander Stroh, Pascal Friederich
Reinforcement Learning Deep Reinforcement Learning Reward Function Real World Optimization Problem Reward Value