Conservative Reward

Conservative reward methods in reinforcement learning aim to improve the robustness and safety of learned policies, particularly in offline settings or when reward models are uncertain. Current research focuses on developing algorithms that incorporate conservatism into reward estimation, often through penalization techniques or by leveraging model-based approaches to constrain policy optimization within safe regions of the state space. This focus addresses critical challenges in deploying reinforcement learning agents in real-world scenarios where data scarcity, model inaccuracies, and potential for catastrophic failures are significant concerns, leading to more reliable and trustworthy AI systems.

Papers

May 30, 2024

Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Models
Masatoshi Uehara, Yulai Zhao, Ehsan Hajiramezanali, Gabriele Scalia, Gökcen Eraslan, Avantika Lal, Sergey Levine, Tommaso Biancalani
Diffusion Model Generative Modeling Reward Model Bridging Text Model Based Optimization Conservative Reward

April 1, 2024

GOV-REK: Governed Reward Engineering Kernels for Designing Robust Multi-Agent Reinforcement Learning Systems
Ashish Rana, Michael Oesterle, Jannik Brinkmann
Multi Agent Reinforcement Learning Product Design Reward Engineering Agent Learning Conservative Reward

October 26, 2023

CROP: Conservative Reward for Model-based Offline Policy Optimization
Hao Li, Xiao-Hu Zhou, Xiao-Liang Xie, Shi-Qi Liu, Zhen-Qiu Feng, Xiao-Yin Liu, Mei-Jiang Gui, Tian-Yu Xiang, De-Xing Huang, Bo-Xian Yao, Zeng-Guang Hou
Offline Reinforcement Learning Model Based Offline Reinforcement Learning Offline Policy Conservative Reward

July 2, 2023

Defending Against Poisoning Attacks in Federated Learning with Blockchain
Nanqing Dong, Zhipeng Wang, Jiahao Sun, Michael Kampffmeyer, William Knottenbelt, Eric Xing
Poisoning Attack Blockchain Based Platform Smart Contract Ledger Technology Data Owner Conservative Reward

April 11, 2023

A Data-Driven State Aggregation Approach for Dynamic Discrete Choice Models
Sinong Geng, Houssam Nassif, Carlos A. Manzanares
Inverse Reinforcement Learning Dynamic Programming Conservative Reward State Sequence Dynamic Discrete Choice

February 9, 2023

CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning
Sheng Yue, Guanbo Wang, Wei Shao, Zhaofeng Zhang, Sen Lin, Ju Ren, Junshan Zhang
Reward Function Inverse Reinforcement Learning Exploration Exploitation Reward Misspecification Conservative Reward

January 27, 2023

A Memory Efficient Deep Reinforcement Learning Approach For Snake Game Autonomous Agents
Md. Rafat Rahman Tushar, Shahnewaz Siddique
Reinforcement Learning Deep Reinforcement Learning Memory Replay Snake Robot Lightweight Convolutional Neural Network Conservative Reward

October 28, 2022

Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferences
Lin Guan, Karthik Valmeekam, Subbarao Kambhampati
Human Preference Reward Learning Agent Behavior Goal Generation Behavioral Feature Conservative Reward

Conservative Reward

Papers

Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Models

GOV-REK: Governed Reward Engineering Kernels for Designing Robust Multi-Agent Reinforcement Learning Systems

CROP: Conservative Reward for Model-based Offline Policy Optimization

Defending Against Poisoning Attacks in Federated Learning with Blockchain

A Data-Driven State Aggregation Approach for Dynamic Discrete Choice Models

CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning

A Memory Efficient Deep Reinforcement Learning Approach For Snake Game Autonomous Agents

Relative Behavioral Attributes: Filling the Gap between Symbolic Goal Specification and Reward Learning from Human Preferences