Robust Reward

Robust reward design in reinforcement learning aims to create reward functions that reliably guide agents towards desired behaviors, even in the presence of noise, ambiguity, or unexpected situations. Current research focuses on improving reward model robustness through techniques like noise-resilient reward functions (e.g., using binary mutual information), causal frameworks to filter out irrelevant artifacts, and methods that learn from noisy or suboptimal demonstrations. These advancements are crucial for deploying reinforcement learning agents in real-world settings where perfect reward signals are unavailable, impacting fields like robotics, natural language processing, and autonomous systems.

Papers

November 13, 2024

Robot See, Robot Do: Imitation Reward for Noisy Financial Environments
Sven Goluža, Tomislav Kovačević, Stjepan Begušić, Zvonko Kostanjčar
Reinforcement Learning Robot Person Imitation Learning Positive Reinforcement Reward Function Design Robust Reward Robot Gaze Conditioned Imitation

September 24, 2024

The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewards
Sukai Huang, Nir Lipovetzky, Trevor Cohn
Reward Model Sparse Reward Environment Robust Reward Auxiliary Reward

September 20, 2024

RRM: Robust Reward Model Training Mitigates Reward Hacking
Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, Daniel Sohn, Anastasiia Makarova, Jeremiah Liu, Yuan Liu, Bilal Piot, Abe Ittycheriah, Aviral Kumar, Mohammad Saleh
Large Language Model Reward Model Robust Reward Reward Hacking

August 27, 2024

Learning Robust Reward Machines from Noisy Labels
Roko Parac, Lorenzo Nodari, Leo Ardon, Daniel Furelos-Blanco, Federico Cerutti, Alessandra Russo
Reinforcement Learning Noisy Label Policy Learning Inductive Logic Programming Robust Reward

August 22, 2024

RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data
Chenglong Wang, Yang Gan, Yifu Huo, Yongyu Mu, Murun Yang, Qiaozhi He, Tong Xiao, Chunliang Zhang, Tongran Liu, Quan Du, Di Yang, Jingbo Zhu
Robust Reward

July 24, 2024

Exploring Domain Robust Lightweight Reward Models based on Router Mechanism
Hyuk Namgoong, Jeesu Jung, Sangkeun Jung, Yoonhyung Roh
Language Model Reward Model Dynamic Routing Robust Reward

June 21, 2024

Robust Reinforcement Learning from Corrupted Human Feedback
Alexander Bukharin, Ilgee Hong, Haoming Jiang, Zichong Li, Qingru Zhang, Zixuan Zhang, Tuo Zhao
Reinforcement Learning Direct Preference Optimization Preference Optimization Reinforcement Learning From Human Feedback Robust Reinforcement Learning Robust Reward

June 10, 2024

Boosting Robustness in Preference-Based Reinforcement Learning with Dynamic Sparsity
Calarina Muslimani, Bram Grooten, Deepak Ranganatha Sastry Mamillapalli, Mykola Pechenizkiy, Decebal Constantin Mocanu, Matthew E. Taylor
Reinforcement Learning Native Robustness Preference Based Reinforcement Learning Robust Reward Dynamic Sparsity

June 7, 2024

Robust Reward Design for Markov Decision Processes
Shuo Wu, Haoxiang Ma, Jie Fu, Shuo Han
Native Robustness Markov Decision Process Reward Function Mixed Integer Linear Reward Design Robust Reward Robust Solution

May 8, 2024

Robust Reward Placement under Uncertainty
Petros Petsinis, Kaichen Zhang, Andreas Pavlogiannis, Jingbo Zhou, Panagiotis Karras
High Uncertainty Anticipation Dynamic Programming Robust Reward Feasible Reward Mobility Model

April 10, 2024

Reward Learning from Suboptimal Demonstrations with Applications in Surgical Electrocautery
Zohre Karimi, Shing-Hei Ho, Bao Thach, Alan Kuntz, Daniel S. Brown
Reinforcement Learning Financial Application Reward Function Robotic Surgery Reward Learning Suboptimal Demonstration Robust Reward

November 22, 2023

Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai Li, Weihan Shen, Xiaolong Zhu, Xiu Li
Diffusion Model Human Feedback Reward Model Diffusion Policy Robust Reward

October 26, 2023

Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks
Ryan Sullivan, Akarsh Kumar, Shengyi Huang, John P. Dickerson, Joseph Suarez
Proximal Policy Optimization DeepMind Control Suite Learning Environment Robust Reward PPO Algorithm

October 23, 2023

Active teacher selection for reinforcement learning from human feedback
Rachel Freedman, Justin Svegliato, Kyle Wray, Stuart Russell
Reinforcement Learning Human Feedback Reward Model Robust Reward Teacher Selection

October 11, 2023

RoboCLIP: One Demonstration is Enough to Learn Robot Policies
Sumedh A Sontakke, Jesse Zhang, Sébastien M. R. Arnold, Karl Pertsch, Erdem Bıyık, Dorsa Sadigh, Chelsea Finn, Laurent Itti
Imitation Learning Textual Demonstration Robot Policy Reward Structure Reward Function Design Robust Reward

June 6, 2023

PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation
Runze Liu, Yali Du, Fengshuo Bai, Jiafei Lyu, Xiu Li
Robotic Manipulation Preference Alignment Preference Based Reinforcement Learning Robust Reward

May 17, 2023

Model-Free Robust Average-Reward Reinforcement Learning
Yue Wang, Alvaro Velasquez, George Atia, Ashley Prater-Bennette, Shaofeng Zou
Q Learning Model Free Value Iteration Robust Reward

January 2, 2023

Robust Average-Reward Markov Decision Processes
Yue Wang, Alvaro Velasquez, George Atia, Ashley Prater-Bennette, Shaofeng Zou
Average Reward Markov Decision Process Robust Reward Optimal Robust Policy

July 28, 2022

Graph Inverse Reinforcement Learning from Diverse Videos
Sateesh Kumar, Jonathan Zamora, Nicklas Hansen, Rishabh Jangir, Xiaolong Wang
Inverse Reinforcement Learning Reward Function Design Cross Embodiment Robust Reward Third Person Diverse Video

November 18, 2021

Assisted Robust Reward Design
Jerry Zhi-Yang He, Anca D. Dragan
Reward Design Reward Function Design Robust Reward Complex Reward Function Design Flexibility