Reward Machine

Reward machines are automata-based formalisms used to represent complex, non-Markovian reward functions in reinforcement learning, aiming to improve learning efficiency and enable the specification of intricate tasks. Current research focuses on synthesizing reward machines from various sources (e.g., planning, expert demonstrations, large language models), developing algorithms for learning with noisy or uncertain interpretations of the reward structure, and extending their application to multi-agent settings and partially observable environments. This work holds significant promise for advancing reinforcement learning's capabilities in complex real-world scenarios by providing structured representations of reward functions and facilitating knowledge transfer across tasks.

Papers

November 4, 2024

Show, Don't Tell: Learning Reward Machines from Demonstrations for Reinforcement Learning-Based Cardiac Pacemaker Synthesis
John Komp, Dananjay Srinivas, Maria Pacheco, Ashutosh Trivedi
Reinforcement Learning Reward Function Noisy Demonstration Positive Reinforcement Design Specification Reward Machine Uncertain Episode Length

August 19, 2024

Synthesis of Reward Machines for Multi-Agent Equilibrium Design (Full Version)
Muhammad Najib, Giuseppe Perelli
Critical Synthesis Game Theoretic Mechanism Design Reward Machine Adaptive Incentive Equilibrium Design

August 15, 2024

Maximally Permissive Reward Machines
Giovanni Varricchione, Natasha Alechina, Mehdi Dastani, Brian Logan
Learning Agent Planning Model AI Planning Reward Machine

June 20, 2024

Bayesian Inverse Reinforcement Learning for Non-Markovian Rewards
Noah Topper, Alvaro Velasquez, George Atia
Inverse Reinforcement Learning Reward Machine Non Markovian Reward Markovian Reward

May 31, 2024

Reward Machines for Deep RL in Noisy and Uncertain Environments
Andrew C. Li, Zizhao Chen, Toryn Q. Klassen, Pashootan Vaezipoor, Rodrigo Toro Icarte, Sheila A. McIlraith
Deep Reinforcement Learning Industrial Disturbing Noise Deep RL Uncertain Environment Reward Machine Complex Reward Function

May 24, 2024

Knowledge-Informed Auto-Penetration Testing Based on Reinforcement Learning with Reward Machine
Yuanliang Li, Hanzheng Dai, Jun Yan
Reinforcement Learning Domain Knowledge Deep Q Learning Penetration Testing Reward Machine

April 30, 2024

Numeric Reward Machines
Kristina Levina, Nikolaos Pappas, Athanasios Karapantelakis, Aneta Vulgarakis Feljan, Jendrik Seipp
Q Learning Reward Learning Reward Machine Feasible Reward

March 8, 2024

Multi-Agent Reinforcement Learning with a Hierarchy of Reward Machines
Xuejing Zheng, Chao Yu
Multi Agent Reinforcement Learning Policy Learning Part Whole Hierarchy Cooperative Multi Agent Reinforcement Learning Reward Machine

February 11, 2024

Using Large Language Models to Automate and Expedite Reinforcement Learning with Reward Machine
Shayan Meshkat Alsadat, Jean-Raphael Gaglione, Daniel Neider, Ufuk Topcu, Zhe Xu
Reinforcement Learning Domain Knowledge Reinforcement Learning Algorithm Closed Loop Decomposed Automation Correction Reward Machine

November 15, 2023

Adversarial Attacks to Reward Machine-based Reinforcement Learning
Lorenzo Nodari
Reinforcement Learning Adversarial Attack Adversarial Scenario Reward Machine

August 18, 2023

Learning Reward Machines through Preference Queries over Sequences
Eric Hsiung, Joydeep Biswas, Swarat Chaudhuri
Preference Feedback Sequence of Sequence Partial Feedback Reward Machine Non Markovian Reward

August 14, 2023

Omega-Regular Reward Machines
Ernst Moritz Hahn, Mateo Perez, Sven Schewe, Fabio Somenzi, Ashutosh Trivedi, Dominik Wojtczak
Reinforcement Learning Reward Machine Non Markovian Reward Omega Regular

July 11, 2023

Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learning
Guy Azran, Mohamad H. Danesh, Stefano V. Albrecht, Sarah Keren
Deep Reinforcement Learning Context Aware Unseen Task Symbolic Representation Reward Machine Efficient Transfer

May 27, 2023

Reinforcement Learning With Reward Machines in Stochastic Games
Jueming Hu, Jean-Raphael Gaglione, Yanze Wang, Zhe Xu, Ufuk Topcu, Yongming Liu
Reinforcement Learning Multi Agent Reinforcement Learning Q Learning Stochastic Game Reward Machine

December 27, 2022

Measuring an artificial intelligence agent's trust in humans using machine incentives
Tim Johnson, Nick Obradovich
Artificial Intelligence Real Human Appropriate Trust Artificial Intelligence Agent Reward Machine Trust Game

November 20, 2022

Noisy Symbolic Abstractions for Deep RL: A case study with Reward Machines
Andrew C. Li, Zizhao Chen, Pashootan Vaezipoor, Toryn Q. Klassen, Rodrigo Toro Icarte, Sheila A. McIlraith
Reinforcement Learning Case Study Policy Learning Deep RL Reward Machine Symbolic Language

May 31, 2022

Hierarchies of Reward Machines
Daniel Furelos-Blanco, Mark Law, Anders Jonsson, Krysia Broda, Alessandra Russo
Sparse Reward Part Whole Hierarchy Finite State Machine Reward Machine Subtask Assignment

April 20, 2022

December 17, 2021

Learning Reward Machines: A Study in Partially Observable Reinforcement Learning
Rodrigo Toro Icarte, Ethan Waldie, Toryn Q. Klassen, Richard Valenzano, Margarita P. Castro, Sheila A. McIlraith
Reinforcement Learning Study Feature Optimal Policy Partially Observable Markov Decision Process Optimal Behavior Reward Machine