Reward Engineering

Reward engineering, the process of designing reward functions for reinforcement learning (RL) agents, aims to efficiently guide agents towards desired behaviors. Current research focuses on mitigating the need for extensive manual reward design through methods like preference-based RL, which leverages human feedback or large language models (LLMs) to implicitly define rewards, and techniques that learn reusable or adaptable reward functions across multiple tasks. These advancements are significant because they reduce the substantial human effort and domain expertise traditionally required for effective RL, enabling broader application in robotics, game AI, and other complex domains.

Papers

August 22, 2024

S-EPOA: Overcoming the Indivisibility of Annotations with Skill-Driven Preference-Based Reinforcement Learning
Ni Mu, Yao Luan, Yiqin Yang, Qing-shan Jia
Reinforcement Learning Preference Optimization Preference Learning Skill Learning Annotated Chapter Information Preference Based Reinforcement Learning Reward Engineering

July 11, 2024

PrefCLM: Enhancing Preference-based Reinforcement Learning with Crowdsourced Large Language Models
Ruiqi Wang, Dezhong Zhao, Ziqin Yuan, Ike Obi, Byung-Cheol Min
Reinforcement Learning Large Scale Language Model Reward Engineering External Feedback Synthetic Feedback

June 28, 2024

Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs
Zichao Shen, Tianchen Zhu, Qingyun Sun, Shiqi Gao, Jianxin Li
Large Language Model Reinforcement Learning Reward Function Large Relevance Improvement Human Preference Reward Engineering

June 12, 2024

RILe: Reinforced Imitation Learning
Mert Albaba, Sammy Christen, Thomas Langarek, Christoph Gebhardt, Otmar Hilliges, Michael J. Black
Reinforcement Learning Imitation Learning Inverse Reinforcement Learning Expert Model Reward Engineering

April 30, 2024

Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning
Calarina Muslimani, Matthew E. Taylor
Reward Function Preference Based Reinforcement Learning Reward Engineering Human in the Loop Reinforcement Optimal Data

April 25, 2024

DrS: Learning Reusable Dense Rewards for Multi-Stage Tasks
Tongzhou Mu, Minghua Liu, Hao Su
Sparse Reward Reward Engineering Dense Reward Multi Stage Task

April 12, 2024

Hindsight PRIORs for Reward Learning from Human Preferences
Mudit Verma, Katherine Metcalf
Reinforcement Learning Policy Learning Human Preference Reward Learning Hindsight Instruction Reward Engineering

April 3, 2024

Solving a Real-World Optimization Problem Using Proximal Policy Optimization with Curriculum Learning and Reward Engineering
Abhijeet Pendyala, Asma Atamna, Tobias Glasmachers
Reinforcement Learning Optimal Policy Proximal Policy Optimization Curriculum Learning Efficient Waste Real World Optimization Problem Reward Engineering

April 1, 2024

GOV-REK: Governed Reward Engineering Kernels for Designing Robust Multi-Agent Reinforcement Learning Systems
Ashish Rana, Michael Oesterle, Jannik Brinkmann
Multi Agent Reinforcement Learning Product Design Reward Engineering Agent Learning Conservative Reward

March 19, 2024

AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents
Jieming Cui, Tengyu Liu, Nian Liu, Yaodong Yang, Yixin Zhu, Siyuan Huang
Imitation Learning Reward Shaping Reward Engineering Interactive Agent Humanoid Agent

February 6, 2024

RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback
Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, Zackory Erickson
Reinforcement Learning Reward Function Vision Language Foundation Model Reward Structure Reward Engineering

November 7, 2023

A Novel Variational Lower Bound for Inverse Reinforcement Learning
Yikang Gui, Prashant Doshi
Reward Function Inverse Reinforcement Learning Reward Engineering Variational Bound Optimal Node

September 15, 2023

Reward Engineering for Generating Semi-structured Explanation
Jiuzhou Han, Wray Buntine, Ehsan Shareghi
Language Model Structured Explanation Reward Engineering

August 24, 2023

Not Only Rewards But Also Constraints: Applications on Legged Robot Locomotion
Yunho Kim, Hyunsik Oh, Jeonghyun Lee, Jinhyeok Choi, Gwanghyeon Ji, Moonkyu Jung, Donghoon Youm, Jemin Hwangbo
Financial Application Reward Report Neural Network Controller Legged Robot Locomotion Complex Robotic System Reward Engineering

August 15, 2023

Generating Personas for Games with Multimodal Adversarial Imitation Learning
William Ahlberg, Alessandro Sestini, Konrad Tollmar, Linus Gisslén
Reinforcement Learning Reward Model Video Game Reward Engineering Generative Adversarial Imitation Learning Persona Knowledge

July 11, 2023

Incentive Engineering for Concurrent Games
David Hyland, Julian Gutierrez, Michael Wooldridge
Multi Agent System Temporal Logic Game Theoretic Reward Engineering Concurrent Game Behavioral Game

March 24, 2023

Optimal Transport for Offline Imitation Learning
Yicheng Luo, Zhengyao Jiang, Samuel Cohen, Edward Grefenstette, Marc Peter Deisenroth
Offline Reinforcement Learning Optimal Transport Offline Imitation Learning Reward Engineering

March 16, 2023

Learning Rewards to Optimize Global Performance Metrics in Deep Reinforcement Learning
Junqi Qian, Paul Weng, Chenmien Tan
Reinforcement Learning Deep Reinforcement Learning Performance Metric Reward Engineering Learning Reward

October 18, 2022

Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexity
Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham M. Kakade, Sergey Levine
Reinforcement Learning Sample Complexity Reward Function Complementary Benefit Reward Shaping Reward Engineering

September 16, 2022

Optimizing Industrial HVAC Systems with Hierarchical Reinforcement Learning
William Wong, Praneet Dutta, Octavian Voicu, Yuri Chervonyi, Cosmin Paduraru, Jerry Luo
Reinforcement Learning Hierarchical Reinforcement Learning Reward Engineering Heating, Ventilation, and Air Conditioning Industrial Cooling System