Reward Function

Reward functions, crucial for guiding reinforcement learning agents towards desired behaviors, are the focus of intense research. Current efforts center on automatically learning reward functions from diverse sources like human preferences, demonstrations (including imperfect ones), and natural language descriptions, often employing techniques like inverse reinforcement learning, large language models, and Bayesian optimization within various architectures including transformers and generative models. This research is vital for improving the efficiency and robustness of reinforcement learning, enabling its application to complex real-world problems where manually designing reward functions is impractical or impossible. The ultimate goal is to create more adaptable and human-aligned AI systems.

Papers

August 22, 2024

Balancing Act: Prioritization Strategies for LLM-Designed Restless Bandit Rewards
Shresth Verma, Niclas Boehmer, Lingkai Kong, Milind Tambe
Reinforcement Learning Multi Armed Bandit Reward Function Balancing Strategy Restless Bandit Priority Based Gini Index

August 21, 2024

Advances in Preference-based Reinforcement Learning: A Review
Youssef Abdelkareem, Shady Shehata, Fakhri Karray
Reinforcement Learning Narrative Review Reward Function Human Preference Preference Based Reinforcement Learning Real World Task

August 11, 2024

A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals
Grace Liu, Michael Tang, Benjamin Eysenbach
Environment Exploration Reward Function Manipulation Task Noisy Demonstration Active Exploration Pseudo Goal Robust Skill Contrastive Reinforcement Learning RL Algorithm

August 10, 2024

Representation Alignment from Human Feedback for Cross-Embodiment Reward Learning from Mixed-Quality Demonstrations
Connor Mattson, Anurag Aribandi, Daniel S. Brown
Human Feedback Reward Function Representation Alignment Suboptimal Demonstration Cross Embodiment Cross Domain Reinforcement Learning

August 8, 2024

August 6, 2024

Empathy Level Alignment via Reinforcement Learning for Empathetic Response Generation
Hui Ma, Bo Zhang, Bo Xu, Jian Wang, Hongfei Lin, Xiao Sun
Reinforcement Learning Reward Function Empathetic Response Generation Response Generation Model Empathy Detection

July 22, 2024

WayEx: Waypoint Exploration using a Single Demonstration
Mara Levy, Nirat Saini, Abhinav Shrivastava
Reward Function Video Demonstration Traditional Reinforcement Learning Waypoint Generation

July 20, 2024

Adapt2Reward: Adapting Video-Language Models to Generalizable Robotic Rewards via Failure Prompts
Yanting Yang, Minghao Chen, Qibo Qiu, Jiahao Wu, Wenxiao Wang, Binbin Lin, Ziyu Guan, Xiaofei He
Reward Function Video Language Model Language Reward Future Failure

July 13, 2024

Preserving the Privacy of Reward Functions in MDPs through Deception
Shashank Reddy Chirra, Pradeep Varakantham, Praveen Paruchuri
Differential Privacy Reward Function Privacy Policy Inverse Reinforcement Learning Multi Agent Sequential Decision

July 3, 2024

June 28, 2024

Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs
Zichao Shen, Tianchen Zhu, Qingyun Sun, Shiqi Gao, Jianxin Li
Large Language Model Reinforcement Learning Reward Function Large Relevance Improvement Human Preference Reward Engineering

June 26, 2024

Combining Automated Optimisation of Hyperparameters and Reward Shape
Julian Dierkes, Emma Cramer, Holger H. Hoos, Sebastian Trimpe
Deep Reinforcement Learning Reward Function Proximal Policy Optimization Related Hyperparameters Reward Shaping Optimization Process Simultaneous Optimization

June 24, 2024

OCALM: Object-Centric Assessment with Language Models
Timo Kaufmann, Jannis Blüml, Antonia Wüst, Quentin Delfosse, Kristian Kersting, Eyke Hüllermeier
Language Model Reinforcement Learning Reward Function Interpretable Reward

June 22, 2024

The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret
Lukas Fluri, Leon Lang, Alessandro Abate, Patrick Forré, David Krueger, Joar Skalse
Reinforcement Learning Reward Function Reward Model Efficient Optimization Low Regret Regret Matching Low Training Error

June 21, 2024

Evolution of Rewards for Food and Motor Action by Simulating Birth and Death
Yuji Kanagawa, Kenji Doya
Reward Function Specie Evolution Reward Report Reward Structure Efficient Engine

June 15, 2024

Generating and Evolving Reward Functions for Highway Driving with Large Language Models
Xu Han, Qiannan Yang, Xianda Chen, Xiaowen Chu, Meixin Zhu
Reinforcement Learning Autonomous Driving Reward Function Autonomous Driving Technology Highway Driving Reward Function Design Simulated Driving

June 14, 2024

Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, Ethan Perez, Evan Hubinger
Reinforcement Learning Reward Function Design Specification Game Environment Reward Hacking Visual Sycophancy

June 7, 2024

Robust Reward Design for Markov Decision Processes
Shuo Wu, Haoxiang Ma, Jie Fu, Shuo Han
Native Robustness Markov Decision Process Reward Function Mixed Integer Linear Reward Design Robust Reward Robust Solution

Reward Function

Papers

Balancing Act: Prioritization Strategies for LLM-Designed Restless Bandit Rewards

Advances in Preference-based Reinforcement Learning: A Review

A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals

Representation Alignment from Human Feedback for Cross-Embodiment Reward Learning from Mixed-Quality Demonstrations

Non-maximizing policies that fulfill multi-criterion aspirations in expectation

Listwise Reward Estimation for Offline Preference-based Reinforcement Learning

Empathy Level Alignment via Reinforcement Learning for Empathetic Response Generation

WayEx: Waypoint Exploration using a Single Demonstration

Adapt2Reward: Adapting Video-Language Models to Generalizable Robotic Rewards via Failure Prompts

Preserving the Privacy of Reward Functions in MDPs through Deception

Efficient Imitation Without Demonstrations via Value-Penalized Auxiliary Control from Examples

On Generalization for Generative Flow Networks

Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs

Combining Automated Optimisation of Hyperparameters and Reward Shape

OCALM: Object-Centric Assessment with Language Models

The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret

Evolution of Rewards for Food and Motor Action by Simulating Birth and Death

Generating and Evolving Reward Functions for Highway Driving with Large Language Models

Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models

Robust Reward Design for Markov Decision Processes