Tabular Markov Decision Process

Tabular Markov Decision Processes (MDPs) are a fundamental framework for sequential decision-making problems with a finite number of states and actions, aiming to find optimal policies that maximize cumulative rewards. Current research emphasizes developing efficient algorithms for policy evaluation and optimization, focusing on areas like safe data collection, principled policy gradient methods, and federated learning approaches for collaborative multi-agent settings. These advancements improve sample efficiency, address safety constraints, and enable scalable solutions for complex real-world applications, impacting fields such as robotics, personalized medicine, and resource management.

Papers

November 7, 2024

Structure Matters: Dynamic Policy Gradient
Sara Klein, Xiangyuan Zhang, Tamer Başar, Simon Weissmann, Leif Döring
Policy Gradient Inner Structure Infinite Horizon Tabular Markov Decision Process

June 4, 2024

SaVeR: Optimal Data Collection Strategy for Safe Policy Evaluation in Tabular MDP
Subhojyoti Mukherjee, Josiah P. Hanna, Robert Nowak
Markov Decision Process Policy Evaluation Data Collection Tabular Markov Decision Process

May 21, 2024

Towards Principled, Practical Policy Gradient for Bandits and Tabular MDPs
Michael Lu, Matin Aghaei, Anant Raj, Sharan Vaswani
Markov Decision Process Multi Armed Bandit Policy Gradient General Principle Tabular Markov Decision Process Softmax Policy

March 15, 2024

Horizon-Free Regret for Linear Markov Decision Processes
Zihan Zhang, Jason D. Lee, Yuxin Chen, Simon S. Du
Reinforcement Learning Markov Decision Process Regret Bound Tabular Markov Decision Process Free Regret

March 11, 2024

On the Global Convergence of Policy Gradient in Average Reward Markov Decision Processes
Navdeep Kumar, Yashaswini Murthy, Itai Shufaro, Kfir Y. Levy, R. Srikant, Shie Mannor
Policy Gradient Global Convergence Tabular Markov Decision Process Average Reward Markov Decision Process Hybrid Reward Average Reward Reinforcement Learning

February 8, 2024

Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices
Jiin Woo, Laixi Shi, Gauri Joshi, Yuejie Chi
Offline Reinforcement Learning Single Agent Tabular Markov Decision Process Federated Offline Reinforcement Learning

November 1, 2023

Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning
Tong Yang, Shicong Cen, Yuting Wei, Yuxin Chen, Yuejie Chi
Multi Task Policy OpTimization Federated Reinforcement Learning Multi Task Reinforcement Learning Tabular Markov Decision Process

October 31, 2023

Towards Instance-Optimality in Online PAC Reinforcement Learning
Aymen Al-Marjani, Andrea Tirinzoni, Emilie Kaufmann
Sample Complexity Tabular Markov Decision Process Instance Dependent Instance Optimal Sub Optimality Gap

October 27, 2023

Model-free Posterior Sampling via Learning Rate Randomization
Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Remi Munos, Alexey Naumov, Pierre Perrault, Michal Valko, Pierre Menard
Q Learning Learning Rate Regret Minimization Posterior Sampling Episodic Markov Decision Process Tabular Markov Decision Process

June 24, 2023

Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data
Sunil Madhow, Dan Qiao, Ming Yin, Yu-Xiang Wang
Reinforcement Learning Data Efficient Tabular Markov Decision Process Offline Policy Evaluation

June 12, 2023

Cancellation-Free Regret Bounds for Lagrangian Approaches in Constrained Markov Decision Processes
Adrian Müller, Pragnya Alatur, Giorgia Ramponi, Niao He
Regret Bound Primal Dual Constrained Markov Decision Process Tabular Markov Decision Process Lagrangian Method

June 9, 2023

Near-optimal Conservative Exploration in Reinforcement Learning under Episode-wise Constraints
Donghao Li, Ruiquan Huang, Cong Shen, Jing Yang
Reinforcement Learning Tabular Markov Decision Process Conservative Constraint Conservative Exploration

May 18, 2023

The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyond
Jiin Woo, Gauri Joshi, Yuejie Chi
Reinforcement Learning Q Learning Linear Speedup Tabular Markov Decision Process Blessing Generation

February 18, 2023

Best of Both Worlds Policy Optimization
Christoph Dann, Chen-Yu Wei, Julian Zimmert
Reinforcement Learning Policy OpTimization Tabular Markov Decision Process Gap Dependent

February 13, 2023

Breaking the Curse of Multiagency: Provably Efficient Decentralized Multi-Agent RL with Function Approximation
Yuanhao Wang, Qinghua Liu, Yu Bai, Chi Jin
Multi Agent Reinforcement Learning Markov Game Function Approximation Reversal Curse Tabular Markov Decision Process Decentralized Stochastic Gradient Descent Linear Quadratic

February 7, 2023

Breaking the Curse of Multiagents in a Large State Space: RL in Markov Games with Independent Linear Function Approximation
Qiwen Cui, Kaiqing Zhang, Simon S. Du
Multi Agent Reinforcement Learning State Space Markov Game Linear Function Approximation Human Like RL Tabular Markov Decision Process

January 30, 2023

Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalents
Wenhao Xu, Xuefeng Gao, Xuedong He
Markov Decision Process Regret Bound Recursive Algorithm Risk Sensitive Reinforcement Learning Tabular Markov Decision Process Optimal Dependence Entropic Risk Certainty Equivalent

October 11, 2022

Multi-User Reinforcement Learning with Low Rank Rewards
Naman Agarwal, Prateek Jain, Suhas Kowshik, Dheeraj Nagaraj, Praneeth Netrapalli
Reinforcement Learning Multi Agent Reinforcement Learning Low Rank Constrained Markov Decision Process Tabular Markov Decision Process

August 11, 2022

Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity
Laixi Shi, Yuejie Chi
Offline Reinforcement Learning Robust Policy Value Iteration Distributionally Robust Model Based Offline Reinforcement Learning Tabular Markov Decision Process Robust Offline Reinforcement Learning

July 18, 2022

A Deep Reinforcement Learning Approach for Finding Non-Exploitable Strategies in Two-Player Atari Games
Zihan Ding, Dijia Su, Qinghua Liu, Chi Jin
Markov Game Deep Reinforcement Learning Approach Tabular Markov Decision Process ATARI Game Nash Equilibrium Policy Double DQN

Tabular Markov Decision Process

Papers

Structure Matters: Dynamic Policy Gradient

SaVeR: Optimal Data Collection Strategy for Safe Policy Evaluation in Tabular MDP

Towards Principled, Practical Policy Gradient for Bandits and Tabular MDPs

Horizon-Free Regret for Linear Markov Decision Processes

On the Global Convergence of Policy Gradient in Average Reward Markov Decision Processes

Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices

Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning

Towards Instance-Optimality in Online PAC Reinforcement Learning

Model-free Posterior Sampling via Learning Rate Randomization

Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data

Cancellation-Free Regret Bounds for Lagrangian Approaches in Constrained Markov Decision Processes

Near-optimal Conservative Exploration in Reinforcement Learning under Episode-wise Constraints

The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyond

Best of Both Worlds Policy Optimization

Breaking the Curse of Multiagency: Provably Efficient Decentralized Multi-Agent RL with Function Approximation

Breaking the Curse of Multiagents in a Large State Space: RL in Markov Games with Independent Linear Function Approximation

Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalents

Multi-User Reinforcement Learning with Low Rank Rewards

Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity

A Deep Reinforcement Learning Approach for Finding Non-Exploitable Strategies in Two-Player Atari Games