Reward Ambiguity

Reward ambiguity, the uncertainty in defining optimal reward functions for reinforcement learning (RL) agents, hinders effective training and reliable performance. Current research focuses on developing methods to quantify and manage this uncertainty, employing techniques like ensemble learning, uncertainty-aware reward models, and optimal transport theory to better represent and learn from human preferences or expert demonstrations. These advancements aim to improve the sample efficiency and robustness of RL algorithms, particularly in complex tasks where specifying precise reward functions is challenging, with applications ranging from robotics to language model training. Ultimately, addressing reward ambiguity is crucial for building more reliable and adaptable AI systems.

Papers

October 15, 2024

Ctrl-U: Robust Conditional Image Generation via Uncertainty-aware Reward Modeling
Guiyu Zhang, Huan-ang Gao, Zijian Jiang, Hao Zhao, Zhedong Zheng
Reward Model Conditional Image Generation Reward Ambiguity

October 1, 2024

Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown
Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, Junge Zhang
Reward Model Reward Prediction Reward Ambiguity Reward Value

July 22, 2024

Exploring and Addressing Reward Confusion in Offline Preference Learning
Xin Chen, Sam Toyer, Florian Shkurti
Reinforcement Learning Active Learning Spurious Correlation User Preference Offline Preference Reward Ambiguity

June 6, 2024

Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning
Abdullah Akgül, Manuel Haußmann, Melih Kandemir
Large Relevance Improvement Model Based Offline Reinforcement Learning Uncertainty Propagation Reliable Uncertainty Uncertainty Sampling Reward Ambiguity Model Based Policy

May 10, 2024

Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimation
JoonHo Lee, Jae Oh Woo, Juree Seok, Parisa Hassanzadeh, Wooseok Jang, JuYoun Son, Sima Didari, Baruch Gutow, Heng Hao, Hankyu Moon, Wenjun Hu, Yeong-Dae Kwon, Taehee Lee, Seungjai Min
Language Model Instruction Quality Reward Ambiguity Response Quality

February 5, 2024

Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Models
Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng, Anh Tuan Luu, Junxian He, Pang Wei Koh, Bryan Hooi
High Uncertainty Anticipation Uncertainty Aware Thought Reasoning Reward Ambiguity

January 23, 2024

On the Stochastic (Variance-Reduced) Proximal Gradient Method for Regularized Expected Reward Optimization
Ling Liang, Haizhao Yang
Stochastic Gradient Stochastic Way Proximal Gradient Reward Ambiguity Stochastic Proximal Gradient ProbAbilistic Gradient

December 30, 2023

Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles
Yuanzhao Zhai, Han Zhang, Yu Lei, Yue Yu, Kele Xu, Dawei Feng, Bo Ding, Huaimin Wang
Reinforcement Learning Human Feedback Reward Model Low Rank Adaptation Reinforcement Learning From Human Feedback Reward Ambiguity

December 22, 2023

REBEL: A Regularization-Based Solution for Reward Overoptimization in Robotic Reinforcement Learning from Human Feedback
Souradip Chakraborty, Anukriti Singh, Amisha Bhaskar, Pratap Tokekar, Dinesh Manocha, Amrit Singh Bedi
Reinforcement Learning Human Feedback Reward Function Regularization Model Robot Policy Learning Reward Overoptimization Reward Ambiguity

November 5, 2023

Uncertainty Estimation for Safety-critical Scene Segmentation via Fine-grained Reward Maximization
Hongzheng Yang, Cheng Chen, Yueyao Chen, Markus Scheppach, Hon Chi Yip, Qi Dou
Fine Grained Uncertainty Estimation Vision Based Model Uncertainty Scene Segmentation Reward Ambiguity Surgical Scene Segmentation

October 18, 2023

Understanding Reward Ambiguity Through Optimal Transport Theory in Inverse Reinforcement Learning
Ali Baheri
Optimal Transport Reward Function Inverse Reinforcement Learning Task Specific Reward Reward Ambiguity

October 13, 2023

Distance-rank Aware Sequential Reward Learning for Inverse Reinforcement Learning with Sub-optimal Demonstrations
Lu Li, Yuxin Pan, Ruobing Chen, Jie Liu, Zilin Wang, Yu Liu, Zhiheng Li
Inverse Reinforcement Learning Reward Learning Reward Maximization Suboptimal Demonstration Reward Prediction Reward Ambiguity

February 8, 2023

Fairness in Matching under Uncertainty
Siddartha Devic, David Kempe, Vatsal Sharan, Aleksandra Korolova
Procedural Fairness High Uncertainty Anticipation Matching Accuracy Individual Fairness Two Sided Algorithmic Decision Reward Ambiguity Fair Utility

January 3, 2023

Risk-Averse MDPs under Reward Ambiguity
Haolin Ruan, Zhi Chen, Chin Pang Ho
Markov Decision Process Reward Ambiguity Robust Markov Decision Process Robust Risk

May 24, 2022

Reward Uncertainty for Exploration in Preference-based Reinforcement Learning
Xinran Liang, Katherine Shu, Kimin Lee, Pieter Abbeel
Reinforcement Learning Environment Exploration Reward Model Preference Based Reinforcement Learning Reward Ambiguity

Reward Ambiguity

Papers

Ctrl-U: Robust Conditional Image Generation via Uncertainty-aware Reward Modeling

Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown

Exploring and Addressing Reward Confusion in Offline Preference Learning

Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning

Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimation

Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Models

On the Stochastic (Variance-Reduced) Proximal Gradient Method for Regularized Expected Reward Optimization

Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles

REBEL: A Regularization-Based Solution for Reward Overoptimization in Robotic Reinforcement Learning from Human Feedback

Uncertainty Estimation for Safety-critical Scene Segmentation via Fine-grained Reward Maximization

Understanding Reward Ambiguity Through Optimal Transport Theory in Inverse Reinforcement Learning

Distance-rank Aware Sequential Reward Learning for Inverse Reinforcement Learning with Sub-optimal Demonstrations

Fairness in Matching under Uncertainty

Risk-Averse MDPs under Reward Ambiguity

Reward Uncertainty for Exploration in Preference-based Reinforcement Learning