Linear Markov Decision Process

Linear Markov Decision Processes (MDPs) are a fundamental framework for sequential decision-making under uncertainty, aiming to find optimal policies that maximize cumulative rewards. Current research heavily focuses on developing computationally efficient algorithms for solving linear MDPs, particularly those employing linear function approximation, and addressing challenges in large or infinite state spaces through techniques like hybrid RL and contrastive representation learning. These advancements are significant because they improve the scalability and sample efficiency of reinforcement learning algorithms, leading to more practical applications in diverse fields such as robotics and personalized medicine.

Papers

October 31, 2024

Local Linearity: the Key for No-regret Reinforcement Learning in Continuous MDPs
Davide Maran, Alberto Maria Metelli, Matteo Papini, Marcello Restelli
Regret Algorithm Linear MDPs Linear Markov Decision Process Local Linearity Continuous Time Markov Decision Process

September 16, 2024

Provably Efficient Infinite-Horizon Average-Reward Reinforcement Learning with Linear Function Approximation
Woojin Chae, Dabeen Lee
Linear Function Approximation Bellman Equation Linear Markov Decision Process Linear Mixture Markov Decision Process

September 7, 2024

Sample and Oracle Efficient Reinforcement Learning for MDPs with Linearly-Realizable Value Functions
Zakaria Mhammedi
Markov Decision Process Near Optimal Policy Plausible Learning Linear Markov Decision Process

August 8, 2024

Hybrid Reinforcement Learning Breaks Sample Size Barriers in Linear MDPs
Kevin Tan, Wei Fan, Yuting Wei
Linear Markov Decision Process Hybrid Reinforcement Learning

June 6, 2024

How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach
Filippo Lazzati, Mirco Mutti, Alberto Maria Metelli
Inverse Reinforcement Learning State Space Efficient Approach Linear Markov Decision Process Reward Free Exploration Feasible Reward

May 27, 2024

Trajectory Data Suffices for Statistically Efficient Learning in Offline RL with Linear $q^\pi$-Realizability and Concentrability
Volodymyr Tkachuk, Gellért Weisz, Csaba Szepesvári
Offline Reinforcement Learning Efficient Learning Trajectory Data Finite Horizon Linear Ordered Data Linear MDPs Linear Markov Decision Process

February 6, 2024

No-Regret Reinforcement Learning in Smooth MDPs
Davide Maran, Alberto Maria Metelli, Matteo Papini, Marcello Restell
Reinforcement Learning Markov Decision Process Regret Minimization Regret Guarantee Linear Markov Decision Process Regret Learning

October 17, 2023

Value-Biased Maximum Likelihood Estimation for Model-based Reinforcement Learning in Discounted Linear MDPs
Yu-Heng Hung, Ping-Chun Hsieh, Akshay Mete, P. R. Kumar
Model Based Reinforcement Learning Optimal Regret O$ Regret Infinite Horizon Linear Markov Decision Process

October 11, 2023

Online RL in Linearly $q^\pi$-Realizable MDPs Is as Easy as in Linear MDPs If You Learn What to Ignore
Gellért Weisz, András György, Csaba Szepesvári
Online Reinforcement Learning Episodic Markov Decision Process Linear MDPs Linear Markov Decision Process

September 18, 2023

Exploring and Learning in Sparse Linear MDPs without Computationally Intractable Oracles
Noah Golowich, Ankur Moitra, Dhruv Rohatgi
LeArning Abstract Computational Approach Near Optimal Policy Linear Markov Decision Process Computational Learning

September 5, 2023

Regret Analysis of Policy Gradient Algorithm for Infinite Horizon Average Reward Markov Decision Processes
Qinbo Bai, Washim Uddin Mondal, Vaneet Aggarwal
Regret Analysis O$ Regret Policy Gradient Algorithm Average Reward Markov Decision Process Linear Markov Decision Process

May 22, 2023

Regularization and Variance-Weighted Regression Achieves Minimax Optimality in Linear MDPs: Theory and Practice
Toshinori Kitamura, Tadashi Kozuno, Yunhao Tang, Nino Vieillard, Michal Valko, Wenhao Yang, Jincheng Mei, Pierre Ménard, Mohammad Gheshlaghi Azar, Rémi Munos, Olivier Pietquin, Matthieu Geist, Csaba Szepesvári, Wataru Kumagai, Yutaka Matsuo
Theoretical Understanding Practice Mode Mirror Descent Infinite Horizon Minimax Optimal Linear Markov Decision Process Value Based Deep Reinforcement Learning

May 15, 2023

A Theoretical Analysis of Optimistic Proximal Policy Optimization in Linear Markov Decision Processes
Han Zhong, Tong Zhang
Proximal Policy Optimization Theoretical Analysis Adversarial Markov Decision Process Linear Markov Decision Process

February 24, 2023

Logarithmic Switching Cost in Reinforcement Learning beyond Linear MDPs
Dan Qiao, Ming Yin, Yu-Xiang Wang
Reinforcement Learning Optimal Regret Sublinear Regret Linear Markov Decision Process

February 14, 2023

Improved Regret Bounds for Linear Adversarial MDPs via Linear Optimization
Fang Kong, Xiangcheng Zhang, Baoxiang Wang, Shuai Li
Markov Decision Process Regret Bound Linear Programming Adversarial Environment Adversarial Markov Decision Process Linear Markov Decision Process

December 27, 2022

Model-Based Reinforcement Learning with Multinomial Logistic Function Approximation
Taehyun Hwang, Min-hwan Oh
Model Based Reinforcement Learning Efficient Reinforcement Learning Episodic Markov Decision Process Multinomial Logistic Linear Markov Decision Process

July 14, 2022

Making Linear MDPs Practical via Contrastive Representation Learning
Tianjun Zhang, Tongzheng Ren, Mengjiao Yang, Joseph E. Gonzalez, Dale Schuurmans, Bo Dai
Representation Learning Markov Decision Process Contrastive Representation Learning LOw Rank Linear Markov Decision Process

December 2, 2021

Differentially Private Exploration in Reinforcement Learning with Linear Representation
Paul Luyo, Evrard Garcelon, Alessandro Lazaric, Matteo Pirotta
Reinforcement Learning Markov Decision Process Linear Representation Linear Markov Decision Process Linear Mixture Markov Decision Process

Linear Markov Decision Process

Papers

Local Linearity: the Key for No-regret Reinforcement Learning in Continuous MDPs

Provably Efficient Infinite-Horizon Average-Reward Reinforcement Learning with Linear Function Approximation

Sample and Oracle Efficient Reinforcement Learning for MDPs with Linearly-Realizable Value Functions

Hybrid Reinforcement Learning Breaks Sample Size Barriers in Linear MDPs

How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach

Trajectory Data Suffices for Statistically Efficient Learning in Offline RL with Linear $q^\pi$-Realizability and Concentrability

No-Regret Reinforcement Learning in Smooth MDPs

Value-Biased Maximum Likelihood Estimation for Model-based Reinforcement Learning in Discounted Linear MDPs

Online RL in Linearly $q^\pi$-Realizable MDPs Is as Easy as in Linear MDPs If You Learn What to Ignore

Exploring and Learning in Sparse Linear MDPs without Computationally Intractable Oracles

Regret Analysis of Policy Gradient Algorithm for Infinite Horizon Average Reward Markov Decision Processes

Regularization and Variance-Weighted Regression Achieves Minimax Optimality in Linear MDPs: Theory and Practice

A Theoretical Analysis of Optimistic Proximal Policy Optimization in Linear Markov Decision Processes

Logarithmic Switching Cost in Reinforcement Learning beyond Linear MDPs

Improved Regret Bounds for Linear Adversarial MDPs via Linear Optimization

Model-Based Reinforcement Learning with Multinomial Logistic Function Approximation

Making Linear MDPs Practical via Contrastive Representation Learning

Differentially Private Exploration in Reinforcement Learning with Linear Representation