Markovian Reward

Markovian reward functions, commonly used in reinforcement learning, assign rewards based solely on the current state, simplifying the learning process. However, recent research highlights limitations of this approach, particularly in handling multi-objective tasks, risk sensitivity, and scenarios with delayed or non-Markovian rewards. Current efforts focus on developing alternative reward structures, such as reward machines and multi-dimensional rewards, and on methods for mapping non-Markovian rewards into equivalent Markovian representations to leverage existing algorithms. These advancements are crucial for improving the robustness and applicability of reinforcement learning in complex real-world scenarios.

Papers

October 26, 2024

Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learning
Yuting Tang, Xin-Qiang Cai, Jing-Cheng Pang, Qiyu Wu, Yao-Xiang Ding, Masashi Sugiyama
Reinforcement Learning Reward Function Partial Sum Non Markovian Reward Reward Delay Markovian Reward

June 20, 2024

Bayesian Inverse Reinforcement Learning for Non-Markovian Rewards
Noah Topper, Alvaro Velasquez, George Atia
Inverse Reinforcement Learning Reward Machine Non Markovian Reward Markovian Reward

January 26, 2024

On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Tasks
Joar Skalse, Alessandro Abate
Reinforcement Learning Fundamental Limitation Risk Sensitive Multi Objective Reinforcement Learning Procedural Activity Markovian Reward

January 20, 2024

Detecting Hidden Triggers: Mapping Non-Markov Reward Functions to Markov
Gregory Hyde, Eugene Santos Jr
Reinforcement Learning Algorithm Markov Model Non Markovian Reward Imperceptible Trigger Markovian Reward

December 15, 2023

Robustness Verification of Deep Reinforcement Learning Based Control Systems using Reward Martingales
Dapeng Zhi, Peixin Wang, Cheng Chen, Min Zhang
Deep Reinforcement Learning Robustness Verification Deep Reinforcement Learning Based Markovian Reward

October 18, 2023

On The Expressivity of Objective-Specification Formalisms in Reinforcement Learning
Rohan Subramani, Marcus Williams, Max Heitmann, Halfdan Holm, Charlie Griffin, Joar Skalse
Reinforcement Learning Multi Objective Reinforcement Learning Behavior Expressivity Style Reinforcement Learning Objective Markovian Reward

September 30, 2023

Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards
Silviu Pitis
Multiple Objective Non Markovian Reward Consistent Aggregation Markovian Reward Temporal Preference Q Aggregation

July 22, 2023

On the Expressivity of Multidimensional Markov Reward
Shuwa Miura
Markov Decision Process Reward Function Sequential Decision Behavior Expressivity Style Deterministic Policy Markovian Reward

May 28, 2023

Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approach
Yudi Zhang, Yali Du, Biwei Huang, Ziyan Wang, Jun Wang, Meng Fang, Mykola Pechenizkiy
Reinforcement Learning Causal Approach Markovian Reward Reward Redistribution

February 4, 2022

Model-Free Reinforcement Learning for Symbolic Automata-encoded Objectives
Anand Balakrishnan, Stefan Jakšić, Edgar A. Aguilar, Dejan Ničković, Jyotirmoy V. Deshmukh
Reinforcement Learning Model Free Reinforcement Learning Symbolic Expression State Reward Markovian Reward

Markovian Reward

Papers

Beyond Simple Sum of Delayed Rewards: Non-Markovian Reward Modeling for Reinforcement Learning

Bayesian Inverse Reinforcement Learning for Non-Markovian Rewards

On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Tasks

Detecting Hidden Triggers: Mapping Non-Markov Reward Functions to Markov

Robustness Verification of Deep Reinforcement Learning Based Control Systems using Reward Martingales

On The Expressivity of Objective-Specification Formalisms in Reinforcement Learning

Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards

On the Expressivity of Multidimensional Markov Reward

Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approach

Model-Free Reinforcement Learning for Symbolic Automata-encoded Objectives