Reward Prediction

Reward prediction focuses on accurately estimating the value of different actions or outcomes in reinforcement learning and related fields, aiming to guide agents towards optimal behavior. Current research emphasizes improving the robustness and efficiency of reward models, exploring techniques like multi-armed bandits for adaptive reward selection, uncertainty quantification to improve reliability, and the use of natural language critiques for more explicit reasoning. These advancements are crucial for aligning large language models and other AI systems with human preferences, enabling more effective training and safer deployment in various applications, including robotics and human-computer interaction.

Papers

November 13, 2024

R3HF: Reward Redistribution for Enhancing Reinforcement Learning from Human Feedback
Jiahui Li, Tai-wei Chang, Fengda Zhang, Kun Kuang, Long Chen
Large Language Model Reinforcement Learning Human Feedback Reward Prediction Reward Redistribution

October 29, 2024

Learning Successor Features the Simple Way
Raymond Chua, Arna Ghosh, Christos Kaplanis, Blake A. Richards, Doina Precup
Deep Reinforcement Learning Successor Representation Reward Prediction Continual Learning Scenario Successor Feature Representation Collapse

October 2, 2024

LASeR: Learning to Adaptively Select Reward Models with Multi-Armed Bandits
Duy Nguyen, Archiki Prasad, Elias Stengel-Eskin, Mohit Bansal
Multi Armed Bandit Reward Model LLM Based Iterative Training Reward Prediction Laser Reliability

October 1, 2024

Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown
Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, Junge Zhang
Reward Model Reward Prediction Reward Ambiguity Reward Value

August 21, 2024

Critique-out-Loud Reward Models
Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D. Chang, Prithviraj Ammanabrolu
Reward Model Reward Prediction Preference Modeling

July 23, 2024

Adapting Image-based RL Policies via Predicted Rewards
Weiyao Wang, Xinyuan Fang, Gregory D. Hager
Reinforcement Learning Fine Tuning Domain Randomization Reinforcement Learning Policy Reward Prediction Reward Fine Tuning

June 12, 2024

PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences
Daiwei Chen, Yi Chen, Aniket Rege, Ramya Korlakai Vinayak
Reward Model Human Preference Reward Prediction Pluralistic Alignment Heterogeneous Preference

June 7, 2024

Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learning
Subhojyoti Mukherjee, Josiah P. Hanna, Qiaomin Xie, Robert Nowak
Multi Task Decision Transformer Near Optimal Policy Reward Prediction Contextual Bandit Setting Structured Bandit

May 24, 2024

$i$REPO: $i$mplicit Reward Pairwise Difference based Empirical Preference Optimization
Long Tan Le, Han Shu, Tung-Anh Nguyen, Choong Seon Hong, Nguyen H. Tran
Large Language Model Language Model Large Language Model Alignment Reward Prediction

May 15, 2024

Restless Bandit Problem with Rewards Generated by a Linear Gaussian Dynamical System
Jonathan Gornet, Bruno Sinopoli
Multi Armed Bandit Reward Report Reward Prediction Cumulative Reward Linear Gaussian

March 18, 2024

The Value of Reward Lookahead in Reinforcement Learning
Nadav Merlis, Dorian Baudry, Vianney Perchet
Reinforcement Learning Net Present Value Worst Case Reward Prediction Reward Free Exploration

February 13, 2024

PRDP: Proximal Reward Difference Prediction for Large-Scale Reward Finetuning of Diffusion Models
Fei Deng, Qifei Wang, Wei Wei, Matthias Grundmann, Tingbo Hou
Diffusion Model Reward Prediction Reward Fine Tuning

December 19, 2023

InstructVideo: Instructing Video Diffusion Models with Human Feedback
Hangjie Yuan, Shiwei Zhang, Xiang Wang, Yujie Wei, Tao Feng, Yining Pan, Yingya Zhang, Ziwei Liu, Samuel Albanie, Dong Ni
Human Feedback Reward Model Video Diffusion Model Text to Video Diffusion Model Reward Prediction Reward Fine Tuning

November 2, 2023

DreamSmooth: Improving Model-based Reinforcement Learning via Reward Smoothing
Vint Lee, Pieter Abbeel, Youngwoon Lee
Model Based Reinforcement Learning Sparse Reward Reward Prediction Differentiable Reward

October 13, 2023

Distance-rank Aware Sequential Reward Learning for Inverse Reinforcement Learning with Sub-optimal Demonstrations
Lu Li, Yuxin Pan, Ruobing Chen, Jie Liu, Zilin Wang, Yu Liu, Zhiheng Li
Inverse Reinforcement Learning Reward Learning Reward Maximization Suboptimal Demonstration Reward Prediction Reward Ambiguity

May 25, 2023

Lucy-SKG: Learning to Play Rocket League Efficiently Using Deep Reinforcement Learning
Vasileios Moschopoulos, Pantelis Kyriakidis, Aristotelis Lazaridis, Ioannis Vlahavas
Deep Reinforcement Learning Sample Efficient Reinforcement Learning Game Playing Agent Reward Prediction

May 23, 2023

Video Prediction Models as Rewards for Reinforcement Learning
Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, Pieter Abbeel
Reinforcement Learning Reward Report Video Prediction Reward Signal Reward Prediction

May 14, 2023

Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling
Yuta Saito, Qingyang Ren, Thorsten Joachims
Action Space Policy Evaluation Linear Estimator Joint Modeling Reward Prediction

November 15, 2022

Automatic Evaluation of Excavator Operators using Learned Reward Functions
Pranav Agarwal, Marek Teichmann, Sheldon Andrews, Samira Ebrahimi Kahou
Reward Function Automatic Evaluation Task Specific Reward Reward Prediction

May 28, 2022

Federated Neural Bandits
Zhongxiang Dai, Yao Shu, Arun Verma, Flint Xiaofeng Fan, Bryan Kian Hsiang Low, Patrick Jaillet
Reward Function Contextual Bandit Reward Prediction Federated Bandit