Reward Structure

Reward structure, the design and optimization of reward functions in reinforcement learning (RL) and related fields, is crucial for training agents to achieve desired behaviors. Current research focuses on addressing challenges like reward sparsity, objective mismatches between reward maximization and other objectives (e.g., safety), and the automated design of reward functions using techniques such as large language models (LLMs) and evolutionary algorithms. These advancements are improving the efficiency and robustness of RL agents across diverse applications, from autonomous driving and robotics to resource allocation in decentralized systems and even musical improvisation. The ultimate goal is to create more effective and reliable RL systems by developing principled methods for defining and learning appropriate reward structures.

Papers

November 1, 2024

Unified theory of upper confidence bound policies for bandit problems targeting total reward, maximal reward, and more
Nobuaki Kikkawa, Hiroshi Ohno
Bandit Algorithm High Confidence Prior Policy Bandit Problem Reward Structure Upper Confidence Bound Reward Maximizing

October 25, 2024

Toward Finding Strong Pareto Optimal Policies in Multi-Agent Reinforcement Learning
Bang Giang Le, Viet Cuong Ta
Reinforcement Learning Multi Agent Reinforcement Learning Optimal Policy Reward Structure Pareto Optimality Pareto Optimal Policy

July 18, 2024

Analyzing and Bridging the Gap between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learning
Shuyu Yin, Fei Wen, Peilin Liu, Tao Luo
Deep Reinforcement Learning Optimal Policy Reward Report Reward Maximization Reward Structure Monetary Reward

June 21, 2024

Evolution of Rewards for Food and Motor Action by Simulating Birth and Death
Yuji Kanagawa, Kenji Doya
Reward Function Specie Evolution Reward Report Reward Structure Efficient Engine

June 20, 2024

Beyond Optimism: Exploration With Partially Observable Rewards
Simone Parisi, Alireza Kazemipour, Michael Bowling
Reinforcement Learning Environment Exploration Reinforcement Learning Algorithm Exploration Method Reward Structure

June 12, 2024

Adaptive Swarm Mesh Refinement using Deep Reinforcement Learning with Local Rewards
Niklas Freymuth, Philipp Dahlinger, Tobias Würth, Simon Reisch, Luise Kärger, Gerhard Neumann
Deep Reinforcement Learning Reward Structure Adaptive Mesh Refinement Adaptive Meshing Volumetric Mesh

June 3, 2024

REvolve: Reward Evolution with Large Language Models using Human Feedback
Rishi Hazra, Alkis Sygkounas, Andreas Persson, Amy Loutfi, Pedro Zuidberg Dos Martires
Autonomous Driving Reward Function Reward Structure Reward Design

May 2, 2024

Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation
Shangding Gu, Bilgehan Sel, Yuhao Ding, Lu Wang, Qingwei Lin, Ming Jin, Alois Knoll
Reinforcement Learning Visual Perspective Safe Reinforcement Learning Reward Structure Probabilistic Safety Safe Reinforcement Learning Algorithm Gradient Manipulation

April 1, 2024

Bounds of Block Rewards in Honest PinFi Systems
Qi He, Yunwei Mao, Ju Li
Game Theoretic General Bound Reward Structure Decentralized Finance Equilibrium Model

February 25, 2024

Reinforcement Learning Jazz Improvisation: When Music Meets Game Theory
Vedant Tapiavala, Joshua Piesner, Sourjyamoy Barman, Feng Fu
Music Industry Game Theory Reward Structure Jazz Improvisation

February 15, 2024

Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment
Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, Jianshu Chen
Reinforcement Learning Foundation Model Multi Objective Reinforcement Learning Multiple Objective Reward Structure Multi Objective Alignment

February 7, 2024

Optimistic Thompson Sampling for No-Regret Learning in Unknown Games
Yingru Li, Liangqi Liu, Wenqiang Pu, Hao Liang, Zhi-Quan Luo
Bandit Feedback Thompson Sampling Reward Structure

February 6, 2024

RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback
Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, Zackory Erickson
Reinforcement Learning Reward Function Vision Language Foundation Model Reward Structure Reward Engineering

January 25, 2024

Sample Efficient Reinforcement Learning by Automatically Learning to Compose Subtasks
Shuai Han, Mehdi Dastani, Shihan Wang
Reinforcement Learning Machine Learning Reward Function Sparse Reward Environment Sample Efficient Reinforcement Learning Reward Structure

January 22, 2024

Experience-Learning Inspired Two-Step Reward Method for Efficient Legged Locomotion Learning Towards Natural and Robust Gaits
Yinghui Li, Jinze Wu, Xin Liu, Weizhong Guo, Yufei Xue
High Efficiency Legged Robot Adversarial Imitation Learning Reward Structure Locomotion Policy Biological Learning Hexapod Robot Experiential Learning Robust Gait Motion Robust

January 16, 2024

CNN-DRL with Shuffled Features in Finance
Sina Montazeri, Akram Mirzaeinia, Amir Mirzaeinia
Deep Reinforcement Learning CNN Network Reward Report Reward Structure Neural Agent

December 16, 2023

Deriving Rewards for Reinforcement Learning from Symbolic Behaviour Descriptions of Bipedal Walking
Daniel Harnack, Christoph Lüth, Lukas Gross, Shivesh Kumar, Frank Kirchner
Reinforcement Learning Reward Function Efficient Reinforcement Learning Reward Structure Symbolic AI Hybrid Automaton

December 14, 2023

Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft
Hao Li, Xue Yang, Zhaokai Wang, Xizhou Zhu, Jie Zhou, Yu Qiao, Xiaogang Wang, Hongsheng Li, Lewei Lu, Jifeng Dai
Sparse Reward Reward Learning Reward Structure Dense Reward

November 15, 2023

Assessing the Robustness of Intelligence-Driven Reinforcement Learning
Lorenzo Nodari, Federico Cerutti
Native Robustness Reward Structure Reinforcement Learning System

November 10, 2023

Optimal Cooperative Multiplayer Learning Bandits with Noisy Rewards and No Communication
William Chang, Yuanhao Lu
Timely Communication Reward Report Confidence Bound Optimal Action Reward Structure Cooperative Bandit

Reward Structure

Papers

Unified theory of upper confidence bound policies for bandit problems targeting total reward, maximal reward, and more

Toward Finding Strong Pareto Optimal Policies in Multi-Agent Reinforcement Learning

Analyzing and Bridging the Gap between Maximizing Total Reward and Discounted Reward in Deep Reinforcement Learning

Evolution of Rewards for Food and Motor Action by Simulating Birth and Death

Beyond Optimism: Exploration With Partially Observable Rewards

Adaptive Swarm Mesh Refinement using Deep Reinforcement Learning with Local Rewards

REvolve: Reward Evolution with Large Language Models using Human Feedback

Balance Reward and Safety Optimization for Safe Reinforcement Learning: A Perspective of Gradient Manipulation

Bounds of Block Rewards in Honest PinFi Systems

Reinforcement Learning Jazz Improvisation: When Music Meets Game Theory

Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment

Optimistic Thompson Sampling for No-Regret Learning in Unknown Games

RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback

Sample Efficient Reinforcement Learning by Automatically Learning to Compose Subtasks

Experience-Learning Inspired Two-Step Reward Method for Efficient Legged Locomotion Learning Towards Natural and Robust Gaits

CNN-DRL with Shuffled Features in Finance

Deriving Rewards for Reinforcement Learning from Symbolic Behaviour Descriptions of Bipedal Walking

Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft

Assessing the Robustness of Intelligence-Driven Reinforcement Learning

Optimal Cooperative Multiplayer Learning Bandits with Noisy Rewards and No Communication