Reward Estimation Accuracy

Reward estimation accuracy in reinforcement learning focuses on precisely capturing human preferences or objectives to effectively train AI agents. Current research emphasizes improving the efficiency and accuracy of reward models, exploring techniques like variational preference learning, label smoothing, and incorporating reward margins to address issues such as overfitting and the impact of varying human agreement. These advancements aim to reduce reliance on extensive human feedback and enhance the reliability of learned reward functions, ultimately leading to more robust and aligned AI systems across diverse applications.

Papers

August 19, 2024

Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, Natasha Jaques
Reinforcement Learning Human Feedback Reward Model Preference Learning Reward Estimation Accuracy

May 29, 2024

Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation
Fengshuo Bai, Rui Zhao, Hongming Zhang, Sijia Cui, Ying Wen, Yaodong Yang, Bo Xu, Lei Han
Reinforcement Learning Policy Learning Preference Based Reinforcement Learning Policy Regularization Experience Pool Reward Estimation Accuracy

April 7, 2024

Towards Understanding the Influence of Reward Margin on Preference Model Performance
Bowen Qin, Duanyu Feng, Xi Yang
Language Model Reward Model External Influence Reward Estimation Accuracy Reward Maximizing

November 2, 2023

The Impact of Preference Agreement in Reinforcement Learning from Human Feedback: A Case Study in Summarization
Sian Gooding, Hassan Mansoor
Reinforcement Learning Case Study Language Generation Human Feedback Structured Summary Text Summarization Preference Feedback Reward Estimation Accuracy

December 9, 2022

On the Sensitivity of Reward Inference to Misspecified Human Models
Joey Hong, Kush Bhatia, Anca Dragan
Reward Function Reward Learning Human Model Network Sensitivity Reward Inference Reward Estimation Accuracy

October 4, 2022

Reward Estimation Accuracy

Papers

Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning

Efficient Preference-based Reinforcement Learning via Aligned Experience Estimation

Towards Understanding the Influence of Reward Margin on Preference Model Performance

The Impact of Preference Agreement in Reinforcement Learning from Human Feedback: A Case Study in Summarization

On the Sensitivity of Reward Inference to Misspecified Human Models

Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees

Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guarantees