Reward Report

Reward report research centers on efficiently learning reward functions to guide reinforcement learning (RL) agents, particularly in complex domains like large language models (LLMs) and robotics. Current efforts focus on improving reward model accuracy and efficiency through techniques like active learning, parameter insertion within existing model architectures, and leveraging vision-language models (VLMs) to generate dense reward functions. This research is crucial for advancing RL's capabilities in safety-critical applications and for aligning AI systems more effectively with human preferences, ultimately leading to more robust and beneficial AI systems.

Papers

January 10, 2025

On the Partial Identifiability in Reward Learning: Choosing the Best Reward
Filippo Lazzati, Alberto Maria Metelli
Reward Report Reward Learning Reward Structure Partial Identifiability

January 8, 2025

Constraints as Rewards: Reinforcement Learning for Robots without Reward Functions
Yu Ishihara, Noriaki Takasugi, Kotaro Kawakami, Masaya Kinoshita, Kazumi Aoyama
Reinforcement Learning Non Humanoid Robot Reward Function Participation Constraint Reward Report Reward Engineering

January 6, 2025

Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model
Yueqin Yin, Shentao Yang, Yujia Xie, Ziyi Yang, Yuting Sun, Hany Awadalla, Weizhu Chen, Mingyuan Zhou
Language Model Full Model Human Language Reward Model Reinforcement Learning From Human Feedback Reward Report Reward Learning Text Segmentation Bandit Learning

December 19, 2024

December 18, 2024

Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes
Katarzyna Kobalczyk, Claudio Fanconi, Hao Sun, Mihaela van der Schaar
Large Language Model Reward Model Human Preference LLM Fine Tuning Reward Report Neural Process LLM Policy Heterogeneous Preference

December 6, 2024

Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards for Visuomotor Robot Policy Alignment
Ran Tian, Yilin Wu, Chenfeng Xu, Masayoshi Tomizuka, Jitendra Malik, Andrea Bajcsy
Reward Function Reward Report Visuomotor Policy Optimal Alignment Partial Feedback

December 5, 2024

Beyond the Binary: Capturing Diverse Preferences With Reward Regularization
Vishakh Padmakumar, Chuanyang Jin, Hannah Rose Kirk, He He
Large Language Model Reward Report Binary Code Preference Datasets Synthetic Preference Preference Fine Tuning Preference Aggregation Interest Preference

November 27, 2024

AToM: Aligning Text-to-Motion Model at Event-Level with GPT-4Vision Reward
Haonan Han, Xiangzuo Wu, Huan Liao, Zunnan Xu, Zhongyuan Hu, Ronghui Li, Yachao Zhang, Xiu Li
Motion Generation Reward Report Text to Motion GPT 4 Vision Single Atom Event Type Motion Annotation

November 26, 2024

Accelerating Proximal Policy Optimization Learning Using Task Prediction for Solving Games with Delayed Rewards
Ahmad Ahmad, Mehdi Kermanshah, Kevin Leahy, Zachary Serlin, Ho Chit Siu, Makai Mann, Cristian-Ioan Vasile, Roberto Tron, Calin Belta
Reinforcement Learning Policy Gradient Optimal Policy Proximal Policy Optimization Reward Report Trust Region Policy Optimization Game Solving Task Prediction

November 24, 2024

From Laws to Motivation: Guiding Exploration through Law-Based Reasoning and Rewards
Ziyu Chen, Zhiqing Xiao, Xinbei Jiang, Junbo Zhao
Reinforcement Learning Complex Reasoning Autonomous Agent Reward Report Legal Reasoning Self Motivation Guided Exploration Game Environment

November 22, 2024

Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward
Zhiwei Jia, Yuesong Nan, Huixi Zhao, Gengdai Liu
Reinforcement Learning Diffusion Model Reward Model Reward Report Reward Maximization Value Based Reinforcement Learning

November 17, 2024

Improving User Experience in Preference-Based Optimization of Reward Functions for Assistive Robots
Nathaniel Dennler, Zhonghao Shi, Stefanos Nikolaidis, Maja Matarić
Social Robot Simple Function Preference Learning Robot Behavior Assistive Robot Reward Report Robot Trajectory User Experience Covariance Matrix Adaptation Evolution Strategy Preference Based Optimization

November 15, 2024

Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation
Yihong Guo, Yixuan Wang, Yuanyuan Shi, Pan Xu, Anqi Liu
Domain Adaptation Imitation Learning Policy OpTimization Reward Report Learning Augmented Generative Adversarial Imitation

November 2, 2024

Rule Based Rewards for Language Model Safety
Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, Lilian Weng
Language Model Reinforcement Learning Reward Report Simple RULE AI Feedback Safety Fine Tuning Preference Model

October 30, 2024

Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewards
Irmak Guzey, Yinlong Dai, Georgy Savva, Raunaq Bhirangi, Lerrel Pinto
Robotics Domain Human Generated Robotic Hand Reward Report Robot Hand Training Robot

October 24, 2024

Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model
Wenhong Zhu, Zhiwei He, Xiaofeng Wang, Pengfei Liu, Rui Wang
Language Model Preference Optimization Reward Report Model Alignment Weak to Strong Weak Model Aligned Model

October 23, 2024

Adaptive Dense Reward: Understanding the Gap Between Action and Reward Space in Alignment
Yanshi Li, Shaopan Xiong, Gengru Chen, Xiaoyang Li, Yijia Luo, Xingyao Zhang, Yanhui Huang, Xingyuan Bu, Yingshui Tan, Chun Yuan, Jiamang Wang, Wenbo Su, Bo Zheng
Reinforcement Learning Alignment Problem Deep Space Action Feature Reinforcement Learning From Human Feedback Reward Report Level Supervision Dense Reward

October 8, 2024

T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Design
Jiachen Li, Qian Long, Jian Zheng, Xiaofeng Gao, Robinson Piramuthu, Wenhu Chen, William Yang Wang
Raw Data Reward Report Video Generation Model Text to Video Diffusion Model Consistency Distillation T2V Generation

October 3, 2024

Dual Active Learning for Reinforcement Learning from Human Feedback
Pangpang Liu, Chengchun Shi, Will Wei Sun
Large Language Model Reinforcement Learning Offline Reinforcement Learning Human Feedback Reward Report Dual Learning Reward Estimator

Reward Report

Papers

On the Partial Identifiability in Reward Learning: Choosing the Best Reward

Constraints as Rewards: Reinforcement Learning for Robots without Reward Functions

Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model

AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling

Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigation

Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes

Maximizing Alignment with Minimal Feedback: Efficiently Learning Rewards for Visuomotor Robot Policy Alignment

Beyond the Binary: Capturing Diverse Preferences With Reward Regularization

AToM: Aligning Text-to-Motion Model at Event-Level with GPT-4Vision Reward

Accelerating Proximal Policy Optimization Learning Using Task Prediction for Solving Games with Delayed Rewards

From Laws to Motivation: Guiding Exploration through Law-Based Reasoning and Rewards

Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Reward

Improving User Experience in Preference-Based Optimization of Reward Functions for Assistive Robots

Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation

Rule Based Rewards for Language Model Safety

Bridging the Human to Robot Dexterity Gap through Object-Oriented Rewards

Weak-to-Strong Preference Optimization: Stealing Reward from Weak Aligned Model

Adaptive Dense Reward: Understanding the Gap Between Action and Reward Space in Alignment

T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Design

Dual Active Learning for Reinforcement Learning from Human Feedback