Non Markovian Reward

Non-Markovian reward modeling in reinforcement learning addresses the challenge of learning optimal policies when rewards depend on the entire history of states and actions, rather than just the current state. Current research focuses on developing algorithms and model architectures, such as reward machines, transformers, and multiple instance learning methods, to effectively capture these temporal dependencies and learn from various feedback types, including trajectory preferences and bagged rewards. This research is significant because it enables the application of reinforcement learning to complex real-world scenarios with inherently non-Markovian reward structures, improving the performance and robustness of AI agents in diverse domains. The development of efficient algorithms with theoretical guarantees is a key focus, alongside the exploration of interpretable models for better understanding and verification of learned policies.

Papers

October 26, 2024

Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learning
Yuting Tang, Xin-Qiang Cai, Jing-Cheng Pang, Qiyu Wu, Yao-Xiang Ding, Masashi Sugiyama
Reinforcement Learning Reward Function Partial Sum Non Markovian Reward Reward Delay Markovian Reward

September 20, 2024

PrefMMT: Modeling Human Preferences in Preference-based Reinforcement Learning with Multimodal Transformers
Dezhong Zhao, Ruiqi Wang, Dayoon Suh, Taehyeon Kim, Ziqin Yuan, Byung-Cheol Min, Guohua Chen
Reward Model Human Preference Multimodal Transformer Robot Trajectory Cross Modal Interaction Preference Based Reinforcement Learning Non Markovian Reward Preference Modeling

August 19, 2024

Efficient Reinforcement Learning in Probabilistic Reward Machines
Xiaofeng Lin, Xuezhou Zhang
Markov Decision Process Efficient Reinforcement Learning Non Markovian Reward

June 20, 2024

Bayesian Inverse Reinforcement Learning for Non-Markovian Rewards
Noah Topper, Alvaro Velasquez, George Atia
Inverse Reinforcement Learning Reward Machine Non Markovian Reward Markovian Reward

March 22, 2024

Planning with a Learned Policy Basis to Optimally Solve Complex Tasks
Guillermo Infante, David Kuric, Anders Jonsson, Vicenç Gómez, Herke van Hoof
Reinforcement Learning Task Planning Policy Learning Sequential Decision Making Problem Complex Task Stochastic Environment Global Optimality Non Markovian Reward

February 6, 2024

Reinforcement Learning from Bagged Reward
Yuting Tang, Xin-Qiang Cai, Yao-Xiang Ding, Qiyu Wu, Guoqing Liu, Masashi Sugiyama
Reinforcement Learning Reward Function Non Markovian Reward

January 20, 2024

Detecting Hidden Triggers: Mapping Non-Markov Reward Functions to Markov
Gregory Hyde, Eugene Santos Jr
Reinforcement Learning Algorithm Markov Model Non Markovian Reward Markovian Reward Imperceptible Trigger

December 8, 2023

Remembering to Be Fair: Non-Markovian Fairness in Sequential Decision Making
Parand A. Alamdari, Toryn Q. Klassen, Elliot Creager, Sheila A. McIlraith
Sequential Decision Sequential Decision Making Personal Memory Fair Decision Non Markovian Reward

October 18, 2023

September 30, 2023

Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards
Silviu Pitis
Multiple Objective Non Markovian Reward Consistent Aggregation Temporal Preference Markovian Reward Q Aggregation

August 18, 2023

Learning Reward Machines through Preference Queries over Sequences
Eric Hsiung, Joydeep Biswas, Swarat Chaudhuri
Preference Feedback Sequence of Sequence Partial Feedback Reward Machine Non Markovian Reward

August 14, 2023

Omega-Regular Reward Machines
Ernst Moritz Hahn, Mateo Perez, Sven Schewe, Fabio Somenzi, Ashutosh Trivedi, Dominik Wojtczak
Reinforcement Learning Reward Machine Non Markovian Reward Omega Regular

June 22, 2023

Quantum Pufferfish Privacy: A Flexible Privacy Framework for Quantum Systems
Theshani Nuradha, Ziv Goldfeld, Mark M. Wilde
Differential Privacy Quantum Algorithm Quantum System Non Markovian Reward Privacy Framework Pufferfish Privacy

January 8, 2023

Learning Symbolic Representations for Reinforcement Learning of Non-Markovian Behavior
Phillip J. K. Christoffersen, Andrew C. Li, Rodrigo Toro Icarte, Sheila A. McIlraith
Reinforcement Learning Non Markovian Real World Reinforcement Learning Non Markovian Reward

October 18, 2022

Deep Black-Box Reinforcement Learning with Movement Primitives
Fabian Otto, Onur Celik, Hongyi Zhou, Hanna Ziesche, Ngo Anh Vien, Gerhard Neumann
Reinforcement Learning Deep Reinforcement Learning Motion Primitive Non Markovian Reward Step Reinforcement Learning

August 25, 2022

Learning Task Automata for Reinforcement Learning using Hidden Markov Models
Alessandro Abate, Yousif Almulla, James Fox, David Hyland, Michael Wooldridge
Reinforcement Learning Hidden Markov Model Non Markovian Reward Learning Automaton

May 30, 2022

Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learning
Joseph Early, Tom Bewley, Christine Evers, Sarvapali Ramchurn
Reinforcement Learning Reward Function Non Markovian Reward Trajectory Label

November 8, 2021

Dueling RL: Reinforcement Learning with Trajectory Preferences
Aldo Pacchiano, Aadirupa Saha, Jonathan Lee
Reinforcement Learning Regret Guarantee Traditional Reinforcement Learning Human Like RL Non Markovian Reward Trajectory Preference

November 5, 2021

Regular Decision Processes for Grid Worlds
Nicky Lenaers, Martijn van Otterlo
Reinforcement Learning Markov Decision Process Sequential Decision Non Markovian Reward Grid World