Episodic Markov Decision Process

Episodic Markov Decision Processes (EMDPs) model sequential decision-making problems where interactions conclude after a fixed number of steps, focusing on learning optimal policies to maximize cumulative rewards. Current research emphasizes developing provably efficient algorithms, particularly for model-free approaches and settings with function approximation, often employing techniques like upper confidence bounds, posterior sampling, and reference-advantage decomposition to handle stochasticity and improve sample efficiency. These advancements are significant for both theoretical understanding of reinforcement learning and practical applications, enabling faster and more robust learning in complex environments with limited data.

Papers

October 3, 2024

Learning a Fast Mixing Exogenous Block MDP using a Single Trajectory
Alexander Levine, Peter Stone, Amy Zhang
Representation Learning Latent Space Latent State Episodic Markov Decision Process

July 31, 2024

Tractable and Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation
Taehyun Cho, Seungyub Han, Kyungjae Lee, Seokhun Ju, Dohyeong Kim, Jungwoo Lee
Value Function Distributional Reinforcement Learning Regret Analysis Episodic Markov Decision Process Approximate Dynamic Programming

June 19, 2024

Infinite-Horizon Reinforcement Learning with Multinomial Logistic Function Approximation
Jaehyun Park, Junyeop Kwon, Dabeen Lee
Reinforcement Learning Markov Decision Process Model Based Reinforcement Learning Episodic Markov Decision Process Multinomial Logistic

June 11, 2024

Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis
Qining Zhang, Honghao Wei, Lei Ying
Reinforcement Learning Human Feedback Episodic Markov Decision Process Model Free Algorithm Trajectory Wise Reward Policy Identification Reward Inference

May 31, 2024

Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation
Fengdi Che, Chenjun Xiao, Jincheng Mei, Bo Dai, Ramki Gummadi, Oscar A Ramirez, Christopher K Harris, A. Rupam Mahmood, Dale Schuurmans
Function Approximation Episodic Markov Decision Process Bootstrapping End to End Goal Node Trajectory Truncation

May 30, 2024

MetaCURL: Non-stationary Concave Utility Reinforcement Learning
Bianca Marin Moreno, Margaux Brégère, Pierre Gaillard, Nadia Oudjane
Markov Decision Process Episodic Markov Decision Process State Action Pair Non Stationary Reinforcement Learning Optimal Dynamic Regret

May 29, 2024

Federated Q-Learning with Reference-Advantage Decomposition: Almost Optimal Regret and Logarithmic Communication Cost
Zhong Zheng, Haochen Zhang, Lingzhou Xue
Q Learning Federated Prompt Cooperation Federated Reinforcement Learning Episodic Markov Decision Process

April 23, 2024

The Power of Resets in Online Reinforcement Learning
Zakaria Mhammedi, Dylan J. Foster, Alexander Rakhlin
Reinforcement Learning Real Power Online Reinforcement Learning Episodic Markov Decision Process Optimal Value Function Global Reset Feature

April 11, 2024

Differentially Private Reinforcement Learning with Self-Play
Dan Qiao, Yu-Xiang Wang
Multi Agent Reinforcement Learning Differential Privacy Self Play Episodic Markov Decision Process Multi Agent RL Single Agent RL Private Reinforcement Learning

April 4, 2024

Exploration is Harder than Prediction: Cryptographically Separating Reinforcement Learning from Supervised Learning
Noah Golowich, Ankur Moitra, Dhruv Rohatgi
Reinforcement Learning Human Prediction Supervised Learning Environment Exploration Episodic Markov Decision Process

December 22, 2023

Federated Q-Learning: Linear Regret Speedup with Low Communication Cost
Zhong Zheng, Fengyu Gao, Lingzhou Xue, Jing Yang
Reinforcement Learning Optimal Policy Single Agent Episodic Markov Decision Process Communication Cost

October 27, 2023

Model-free Posterior Sampling via Learning Rate Randomization
Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Remi Munos, Alexey Naumov, Pierre Perrault, Michal Valko, Pierre Menard
Q Learning Learning Rate Regret Minimization Posterior Sampling Episodic Markov Decision Process Tabular Markov Decision Process

October 23, 2023

A Doubly Robust Approach to Sparse Reinforcement Learning
Wonyoung Kim, Garud Iyengar, Assaf Zeevi
Reinforcement Learning Episodic Markov Decision Process Robust Framework Transition Probability Low Regret Parameter Efficient Sparsity

October 16, 2023

Posterior Sampling-based Online Learning for Episodic POMDPs
Dengwang Tang, Dongze Ye, Rahul Jain, Ashutosh Nayyar, Pierluigi Nuzzo
Multi Agent C POMDPs Episodic Markov Decision Process Posterior Mode Bayesian Regret

October 11, 2023

Online RL in Linearly $q^\pi$-Realizable MDPs Is as Easy as in Linear MDPs If You Learn What to Ignore
Gellért Weisz, András György, Csaba Szepesvári
Online Reinforcement Learning Episodic Markov Decision Process Linear MDPs Linear Markov Decision Process

August 28, 2023

Rate-Optimal Policy Optimization for Linear Markov Decision Processes
Uri Sherman, Alon Cohen, Tomer Koren, Yishay Mansour
Optimal Policy Policy OpTimization Bandit Feedback Regret Minimization Episodic Markov Decision Process Regret Rate

August 17, 2023

Improving Sample Efficiency of Model-Free Algorithms for Zero-Sum Markov Games
Songtao Feng, Ming Yin, Yu-Xiang Wang, Jing Yang, Yingbin Liang
Multi Agent Reinforcement Learning Model Free Reinforcement Learning Markov Game Sample Efficiency Episodic Markov Decision Process Single Agent Reinforcement Learning Zero Sum Markov Game Model Free Algorithm

June 23, 2023

Active Coverage for PAC Reinforcement Learning
Aymen Al-Marjani, Andrea Tirinzoni, Emilie Kaufmann
Reinforcement Learning Active Exploration Episodic Markov Decision Process Reward Free Exploration Entity Coverage

June 4, 2023

Regret Bounds for Risk-sensitive Reinforcement Learning with Lipschitz Dynamic Risk Measures
Hao Liang, Zhi-quan Luo
Regret Bound Risk Sensitive Episodic Markov Decision Process Risk Sensitive Reinforcement Learning Lipschitz Operator Risk Measure Spectral Risk

May 18, 2023

Online Resource Allocation in Episodic Markov Decision Processes
Duksang Lee, William Overman, Dabeen Lee
Markov Decision Process Resource Allocation Optimal Regret Episodic Markov Decision Process Stochastic Reward Online Resource Allocation