Average Reward

Average reward reinforcement learning focuses on maximizing the long-run average reward per time step in sequential decision-making problems, offering an alternative to the discounted reward framework commonly used in reinforcement learning. Current research emphasizes developing efficient algorithms, such as Q-learning variants (including RVI and optimistic Q-learning), actor-critic methods (like RVI-SAC and ARO-DDPG), and model-based approaches, often addressing challenges posed by partially observable environments and large state spaces. These advancements are crucial for tackling real-world problems where the long-term average performance is paramount, improving the applicability of reinforcement learning to continuous control tasks and other domains with ongoing interactions.

Papers

October 16, 2024

Reinforcement Learning with LTL and $ω$-Regular Objectives via Optimality-Preserving Translation to Average Rewards
Xuan-Bach Le, Dominik Wagner, Leon Witzman, Alexander Rabinovich, Luke Ong
Reinforcement Learning Optimal Policy Temporal Logic Average Reward Omega Regular

October 2, 2024

Efficient Learning of POMDPs with Known Observation Model in Average-Reward Setting
Alessio Russo, Alberto Maria Metelli, Marcello Restelli
Markov Decision Process Efficient Learning C POMDPs Stochastic Optimal Control POMDP Policy Average Reward Bayes Optimal Policy

August 29, 2024

On Convergence of Average-Reward Q-Learning in Weakly Communicating Markov Decision Processes
Yi Wan, Huizhen Yu, Richard S. Sutton
Markov Decision Process Early Stage Convergence Q Learning Average Reward Average Reward Reinforcement Learning

August 4, 2024

RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learning
Yukinari Hisaki, Isao Ono
Soft Actor Critic Policy Deep Reinforcement Learning Average Reward Average Reward Reinforcement Learning

July 18, 2024

Optimistic Q-learning for average reward and episodic reinforcement learning
Priyank Agrawal, Shipra Agrawal
Q Learning Regret Minimization Episodic Reinforcement Learning Model Free Algorithm Average Reward Average Reward Reinforcement Learning

June 17, 2024

Constrained Reinforcement Learning with Average Reward Objective: Model-Based and Model-Free Algorithms
Vaneet Aggarwal, Washim Uddin Mondal, Qinbo Bai
Reinforcement Learning Primal Dual Constrained Reinforcement Learning Constrained Markov Decision Process Average Reward Markov Decision Process Model Free Algorithm Average Reward

June 13, 2024

On Value Iteration Convergence in Connected MDPs
Arsenii Mustafin, Alex Olshevsky, Ioannis Ch. Paschalidis
Optimal Policy Value Iteration Average Reward

May 16, 2024

Reward Centering
Abhishek Naik, Yi Wan, Manan Tomar, Richard S. Sutton
Reward Report Average Reward Discount Factor

April 19, 2024

Sample-efficient Learning of Infinite-horizon Average-reward MDPs with General Function Approximation
Jianliang He, Han Zhong, Zhuoran Yang
Sample Efficient Average Reward Markov Decision Process Average Reward Optimal State

May 31, 2023

Restless Bandits with Average Reward: Breaking the Uniform Global Attractor Assumption
Yige Hong, Qiaomin Xie, Yudong Chen, Weina Wang
Discrete Time Restless Bandit Global Attractor Asymptotic Optimality Average Reward

May 24, 2023

Inverse Reinforcement Learning with the Average Reward Criterion
Feiyang Wu, Jingyang Ke, Anqi Wu
Inverse Reinforcement Learning Policy Mirror Descent Average Reward Markov Decision Process Average Reward

May 20, 2023

Off-Policy Average Reward Actor-Critic with Deterministic Policy Search
Naman Saxena, Subhojyoti Khastigir, Shishir Kolathaya, Shalabh Bhatnagar
Policy Gradient Low Rank Actor Critic Algorithm Average Reward Policy Policy Gradient

April 7, 2023

Full Gradient Deep Reinforcement Learning for Average-Reward Criterion
Tejas Pagare, Vivek Borkar, Konstantin Avrachenkov
DQN Agent Restless Multi Armed Bandit Average Reward Markov Reward Process

January 13, 2023

Decentralized model-free reinforcement learning in stochastic games with average-reward objective
Romain Cravic, Nicolas Gast, Bruno Gaujal
Decentralized Learning Decentralized FL Stochastic Game Zero Sum Markov Game Low Regret Average Reward

September 30, 2022

On Convergence of Average-Reward Off-Policy Control Algorithms in Weakly Communicating MDPs
Yi Wan, Richard S. Sutton
Early Stage Convergence Q Learning Policy Algorithm Average Reward

February 7, 2022

Reward-Respecting Subtasks for Model-Based Reinforcement Learning
Richard S. Sutton, Marlos C. Machado, G. Zacharias Holland, David Szepesvari, Finbarr Timbers, Brian Tanner, Adam White
Reinforcement Learning Model Based Reinforcement Learning Temporal Abstraction Average Reward Option Discovery

Average Reward

Papers

Reinforcement Learning with LTL and $ω$-Regular Objectives via Optimality-Preserving Translation to Average Rewards

Efficient Learning of POMDPs with Known Observation Model in Average-Reward Setting

On Convergence of Average-Reward Q-Learning in Weakly Communicating Markov Decision Processes

RVI-SAC: Average Reward Off-Policy Deep Reinforcement Learning

Optimistic Q-learning for average reward and episodic reinforcement learning

Constrained Reinforcement Learning with Average Reward Objective: Model-Based and Model-Free Algorithms

On Value Iteration Convergence in Connected MDPs

Reward Centering

Sample-efficient Learning of Infinite-horizon Average-reward MDPs with General Function Approximation

Restless Bandits with Average Reward: Breaking the Uniform Global Attractor Assumption

Inverse Reinforcement Learning with the Average Reward Criterion

Off-Policy Average Reward Actor-Critic with Deterministic Policy Search

Full Gradient Deep Reinforcement Learning for Average-Reward Criterion

Decentralized model-free reinforcement learning in stochastic games with average-reward objective

On Convergence of Average-Reward Off-Policy Control Algorithms in Weakly Communicating MDPs

Reward-Respecting Subtasks for Model-Based Reinforcement Learning