Reward Inference

Reward inference aims to decipher human preferences or reward functions from observed behavior, a crucial step in aligning artificial intelligence with human values and enabling effective reinforcement learning from human feedback (RLHF). Current research focuses on developing model-free RLHF algorithms that bypass explicit reward model inference, addressing challenges like overfitting and distribution shifts inherent in traditional approaches. These methods, often employing techniques like direct preference optimization or graph-based transductive inference, aim to improve the efficiency and robustness of RLHF, particularly in complex scenarios with limited or noisy data. The ultimate goal is to create more reliable and human-aligned AI systems across various applications, from robotics to large language models.

Papers

November 4, 2024

Learning to Assist Humans without Inferring Rewards
Vivek Myers, Evan Ellis, Sergey Levine, Benjamin Eysenbach, Anca Dragan
Reinforcement Learning LeArning Abstract Real Human Inverse Reinforcement Learning Novel Predecessor and Successor Reward Inference

September 25, 2024

Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference
Qining Zhang, Lei Ying
Reinforcement Learning Human Feedback Policy Gradient Direct Preference Optimization Reward Value Reward Inference

June 11, 2024

Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis
Qining Zhang, Honghao Wei, Lei Ying
Reinforcement Learning Human Feedback Episodic Markov Decision Process Model Free Algorithm Trajectory Wise Reward Policy Identification Reward Inference

February 6, 2024

Transductive Reward Inference on Graph
Bohao Qu, Xiaofeng Cao, Qing Guo, Yi Chang, Ivor W. Tsang, Chengqi Zhang
Graph Drawing Interpretable Reward Reward Inference

August 24, 2023

Conditional Kernel Imitation Learning for Continuous State Environments
Rishabh Agrawal, Nathan Dahlin, Rahul Jain, Ashutosh Nayyar
Reinforcement Learning Imitation Learning Inverse Reinforcement Learning Based Imitation Continuous Environment Reward Inference Conditional Imitation

December 9, 2022

On the Sensitivity of Reward Inference to Misspecified Human Models
Joey Hong, Kush Bhatia, Anca Dragan
Reward Function Reward Learning Human Model Network Sensitivity Reward Inference Reward Estimation Accuracy

June 6, 2022

Robust Pareto Set Identification with Contaminated Bandit Feedback
Kerem Bozgan, Cem Tekin
Bandit Feedback Pareto Optimal Gaussian Reward Reward Inference Multi Objective Multi Armed Bandit Pareto Set Identification

November 12, 2021

Human irrationality: both bad and good for reward inference
Lawrence Chan, Andrew Critch, Anca Dragan
Reward Inference