Reward Report

Reward report research centers on efficiently learning reward functions to guide reinforcement learning (RL) agents, particularly in complex domains like large language models (LLMs) and robotics. Current efforts focus on improving reward model accuracy and efficiency through techniques like active learning, parameter insertion within existing model architectures, and leveraging vision-language models (VLMs) to generate dense reward functions. This research is crucial for advancing RL's capabilities in safety-critical applications and for aligning AI systems more effectively with human preferences, ultimately leading to more robust and beneficial AI systems.

Papers

November 17, 2024

Improving User Experience in Preference-Based Optimization of Reward Functions for Assistive Robots
Nathaniel Dennler, Zhonghao Shi, Stefanos Nikolaidis, Maja Matarić
Social Robot Simple Function Preference Learning Robot Behavior Assistive Robot Reward Report Robot Trajectory User Experience Covariance Matrix Adaptation Evolution Strategy Preference Based Optimization

November 15, 2024

Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation
Yihong Guo, Yixuan Wang, Yuanyuan Shi, Pan Xu, Anqi Liu
Domain Adaptation Imitation Learning Policy OpTimization Reward Report Learning Augmented Generative Adversarial Imitation

November 2, 2024

Rule Based Rewards for Language Model Safety
Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, Lilian Weng
Language Model Reinforcement Learning Reward Report Simple RULE AI Feedback Safety Fine Tuning Preference Model

October 30, 2024

Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewards
Irmak Guzey, Yinlong Dai, Georgy Savva, Raunaq Bhirangi, Lerrel Pinto
Robotics Domain Human Generated Robotic Hand Reward Report Robot Hand Training Robot

October 24, 2024

Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model
Wenhong Zhu, Zhiwei He, Xiaofeng Wang, Pengfei Liu, Rui Wang
Language Model Preference Optimization Reward Report Model Alignment Weak to Strong Weak Model Aligned Model

October 23, 2024

Adaptive Dense Reward: Understanding the Gap Between Action and Reward Space in Alignment
Yanshi Li, Shaopan Xiong, Gengru Chen, Xiaoyang Li, Yijia Luo, Xingyao Zhang, Yanhui Huang, Xingyuan Bu, Yingshui Tan, Chun Yuan, Jiamang Wang, Wenbo Su, Bo Zheng
Reinforcement Learning Alignment Problem Deep Space Action Feature Reinforcement Learning From Human Feedback Reward Report Level Supervision Dense Reward

October 8, 2024

T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Design
Jiachen Li, Qian Long, Jian Zheng, Xiaofeng Gao, Robinson Piramuthu, Wenhu Chen, William Yang Wang
Raw Data Reward Report Video Generation Model Text to Video Diffusion Model Consistency Distillation T2V Generation

October 3, 2024

Dual Active Learning for Reinforcement Learning from Human Feedback
Pangpang Liu, Chengchun Shi, Will Wei Sun
Large Language Model Reinforcement Learning Offline Reinforcement Learning Human Feedback Reward Report Dual Learning Reward Estimator

August 20, 2024

Predicting Rewards Alongside Tokens: Non-disruptive Parameter Insertion for Efficient Inference Intervention in Large Language Model
Chenhan Yuan, Fei Huang, Ru Peng, Keming Lu, Bowen Yu, Chang Zhou, Jingren Zhou
Large Language Model Transformer Architecture K TOKEN Reward Report Insertion Task Inference Engine

July 18, 2024

Analyzing and Bridging the Gap between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learning
Shuyu Yin, Fei Wen, Peilin Liu, Tao Luo
Deep Reinforcement Learning Optimal Policy Reward Report Reward Maximization Reward Structure Monetary Reward

July 3, 2024

Efficient Imitation Without Demonstrations via Value-Penalized Auxiliary Control from Examples
Trevor Ablett, Bryan Chan, Jayce Haoran Wang, Jonathan Kelly
Reinforcement Learning LeArning Abstract Reward Function Noisy Demonstration Reward Report Fewer Example Auxiliary Loss Auxiliary Feature Gradient Penalty

June 21, 2024

Evolution of Rewards for Food and Motor Action by Simulating Birth and Death
Yuji Kanagawa, Kenji Doya
Reward Function Specie Evolution Reward Report Reward Structure Efficient Engine

June 12, 2024

It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF
Taiming Lu, Lingfeng Shen, Xinyu Yang, Weiting Tan, Beidi Chen, Huaxiu Yao
Reinforcement Learning Human Feedback Reward Model Reinforcement Learning From Human Feedback Reward Report Training Model Model Augmentation Policy Model

June 4, 2024

Rectifying Reinforcement Learning for Reward Matching
Haoran He, Emmanuel Bengio, Qingpeng Cai, Ling Pan
Reinforcement Learning Reward Report Generative Flow Network Energy Based GFlowNets

June 3, 2024

Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning
Andreas Schlaginhaufen, Maryam Kamgarpour
Optimal Policy Inverse Reinforcement Learning Task Transferability Reward Report Reward Learning Transition Dynamic Regularized Reinforcement Learning

May 28, 2024

Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignment
Keming Lu, Bowen Yu, Fei Huang, Yang Fan, Runji Lin, Chang Zhou
Reinforcement Learning Alignment Problem Supervised Fine Tuning Reinforcement Learning From Human Feedback Reward Report Online Merging Local Income Tax

May 16, 2024

Reward Centering
Abhishek Naik, Yi Wan, Manan Tomar, Richard S. Sutton
Reward Report Average Reward Discount Factor

May 15, 2024

Restless Bandit Problem with Rewards Generated by a Linear Gaussian Dynamical System
Jonathan Gornet, Bruno Sinopoli
Multi Armed Bandit Reward Report Reward Prediction Cumulative Reward Linear Gaussian

May 6, 2024

Policy Learning for Balancing Short-Term and Long-Term Rewards
Peng Wu, Ziyu Shen, Feng Xie, Zhongyao Wang, Chunchen Liu, Yan Zeng
Optimal Policy Long Term Reward Report Short Term

May 2, 2024

Continuously evolving rewards in an open-ended environment
Richard M. Bailey
Reward Function Open Ended Reward Report Global Endogenous Variate