Tabular Markov Decision Process

Tabular Markov Decision Processes (MDPs) are a fundamental framework for sequential decision-making problems with a finite number of states and actions, aiming to find optimal policies that maximize cumulative rewards. Current research emphasizes developing efficient algorithms for policy evaluation and optimization, focusing on areas like safe data collection, principled policy gradient methods, and federated learning approaches for collaborative multi-agent settings. These advancements improve sample efficiency, address safety constraints, and enable scalable solutions for complex real-world applications, impacting fields such as robotics, personalized medicine, and resource management.

27papers

Papers

February 5, 2025

Gap-Dependent Bounds for Federated Q-learning
Haochen Zhang, Zhong Zheng, Lingzhou Xue
Regret Bound Gap Dependent Q$ Learning Tabular Markov Decision Process Optimal Communication Federated Prompt Cooperation

November 7, 2024

Structure Matters: Dynamic Policy Gradient
Sara Klein, Xiangyuan Zhang, Tamer Başar, Simon Weissmann, Leif Döring
Infinite Horizon Policy Gradient Tabular Markov Decision Process Inner Structure

June 4, 2024

SaVeR: Optimal Data Collection Strategy for Safe Policy Evaluation in Tabular MDP
Subhojyoti Mukherjee, Josiah P. Hanna, Robert Nowak
Policy Evaluation Markov Decision Process Tabular Markov Decision Process Data Collection

May 21, 2024

Towards Principled, Practical Policy Gradient for Bandits and Tabular MDPs
Michael Lu, Matin Aghaei, Anant Raj, Sharan Vaswani
Markov Decision Process Policy Gradient Softmax Policy Tabular Markov Decision Process General Principle Multi Armed Bandit

March 15, 2024

Horizon-Free Regret for Linear Markov Decision Processes
Zihan Zhang, Jason D. Lee, Yuxin Chen, Simon S. Du
Free Regret Markov Decision Process Tabular Markov Decision Process Reinforcement Learning Regret Bound

March 11, 2024

On the Global Convergence of Policy Gradient in Average Reward Markov Decision Processes
Navdeep Kumar, Yashaswini Murthy, Itai Shufaro, Kfir Y. Levy, R. Srikant, Shie Mannor
Average Reward Markov Decision Process Policy Gradient Hybrid Reward Average Reward Reinforcement Learning Global Convergence Tabular Markov Decision Process

February 8, 2024

Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices
Jiin Woo, Laixi Shi, Gauri Joshi, Yuejie Chi
Offline Reinforcement Learning Single Agent Federated Offline Reinforcement Learning Tabular Markov Decision Process

November 1, 2023

Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning
Tong Yang, Shicong Cen, Yuting Wei, Yuxin Chen, Yuejie Chi
Multi Task Policy OpTimization Multi Task Reinforcement Learning Tabular Markov Decision Process Federated Reinforcement Learning

October 31, 2023

Towards Instance-Optimality in Online PAC Reinforcement Learning
Aymen Al-Marjani, Andrea Tirinzoni, Emilie Kaufmann
Instance Dependent Sample Complexity Sub Optimality Gap Tabular Markov Decision Process Instance Optimal

October 27, 2023

Model-free Posterior Sampling via Learning Rate Randomization
Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Remi Munos, Alexey Naumov, Pierre Perrault, Michal Valko, Pierre Menard
Regret Minimization Learning Rate Episodic Markov Decision Process Posterior Sampling Tabular Markov Decision Process Q Learning

June 24, 2023

Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data
Sunil Madhow, Dan Qiao, Ming Yin, Yu-Xiang Wang
Tabular Markov Decision Process Reinforcement Learning Offline Policy Evaluation Data Efficient

June 12, 2023

Cancellation-Free Regret Bounds for Lagrangian Approaches in Constrained Markov Decision Processes
Adrian Müller, Pragnya Alatur, Giorgia Ramponi, Niao He
Lagrangian Method Regret Bound Tabular Markov Decision Process Primal Dual Constrained Markov Decision Process

June 9, 2023

Near-optimal Conservative Exploration in Reinforcement Learning under Episode-wise Constraints
Donghao Li, Ruiquan Huang, Cong Shen, Jing Yang
Conservative Exploration Conservative Constraint Reinforcement Learning Tabular Markov Decision Process

May 18, 2023

The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyond
Jiin Woo, Gauri Joshi, Yuejie Chi
Linear Speedup Q Learning Blessing Generation Reinforcement Learning Tabular Markov Decision Process

February 18, 2023

Best of Both Worlds Policy Optimization
Christoph Dann, Chen-Yu Wei, Julian Zimmert
Reinforcement Learning Tabular Markov Decision Process Gap Dependent Policy OpTimization

February 13, 2023

Breaking the Curse of Multiagency: Provably Efficient Decentralized Multi-Agent RL with Function Approximation
Yuanhao Wang, Qinghua Liu, Yu Bai, Chi Jin
Decentralized Stochastic Gradient Descent Markov Game Reversal Curse Tabular Markov Decision Process Multi Agent Reinforcement Learning Function Approximation Linear Quadratic

February 7, 2023

Breaking the Curse of Multiagents in a Large State Space: RL in Markov Games with Independent Linear Function Approximation
Qiwen Cui, Kaiqing Zhang, Simon S. Du
Markov Game Multi Agent Reinforcement Learning Tabular Markov Decision Process Human Like RL State Space Linear Function Approximation

January 30, 2023

Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalents
Wenhao Xu, Xuefeng Gao, Xuedong He
Certainty Equivalent Risk Sensitive Reinforcement Learning Tabular Markov Decision Process Optimal Dependence Recursive Algorithm Entropic Risk Markov Decision Process Regret Bound

October 11, 2022

Multi-User Reinforcement Learning with Low Rank Rewards
Naman Agarwal, Prateek Jain, Suhas Kowshik, Dheeraj Nagaraj, Praneeth Netrapalli
Low Rank Reinforcement Learning Multi Agent Reinforcement Learning Constrained Markov Decision Process Tabular Markov Decision Process

August 11, 2022

Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity
Laixi Shi, Yuejie Chi
Offline Reinforcement Learning Robust Offline Reinforcement Learning Robust Policy Tabular Markov Decision Process Model Based Offline Reinforcement Learning Value Iteration Distributionally Robust

Tabular Markov Decision Process

Papers

Gap-Dependent Bounds for Federated Q-learning

Structure Matters: Dynamic Policy Gradient

SaVeR: Optimal Data Collection Strategy for Safe Policy Evaluation in Tabular MDP

Towards Principled, Practical Policy Gradient for Bandits and Tabular MDPs

Horizon-Free Regret for Linear Markov Decision Processes

On the Global Convergence of Policy Gradient in Average Reward Markov Decision Processes

Federated Offline Reinforcement Learning: Collaborative Single-Policy Coverage Suffices

Federated Natural Policy Gradient and Actor Critic Methods for Multi-task Reinforcement Learning

Towards Instance-Optimality in Online PAC Reinforcement Learning

Model-free Posterior Sampling via Learning Rate Randomization

Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data

Cancellation-Free Regret Bounds for Lagrangian Approaches in Constrained Markov Decision Processes

Near-optimal Conservative Exploration in Reinforcement Learning under Episode-wise Constraints

The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyond

Best of Both Worlds Policy Optimization

Breaking the Curse of Multiagency: Provably Efficient Decentralized Multi-Agent RL with Function Approximation

Breaking the Curse of Multiagents in a Large State Space: RL in Markov Games with Independent Linear Function Approximation

Regret Bounds for Markov Decision Processes with Recursive Optimized Certainty Equivalents

Multi-User Reinforcement Learning with Low Rank Rewards

Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity