Linear Mixture Markov Decision Process

Linear Mixture Markov Decision Processes (LMDPs) are a framework for reinforcement learning where the environment's dynamics are modeled as a linear combination of known basis functions, simplifying learning in complex systems. Current research focuses on developing computationally efficient algorithms with improved regret bounds, particularly those that are horizon-free and adapt to the variance of the reward or transition probabilities. These advancements are significant because they enable efficient and effective learning in high-dimensional settings, with applications ranging from personalized services to robotics, where traditional methods struggle. The development of variance-adaptive and horizon-free algorithms represents a key step towards more robust and practical reinforcement learning solutions.

Papers

November 5, 2024

Near-Optimal Dynamic Regret for Adversarial Linear Mixture MDPs
Long-Fei Li, Peng Zhao, Zhi-Hua Zhou
Global Optimization Adversarial Markov Decision Process Optimal Dynamic Regret Linear Mixture Markov Decision Process

October 19, 2024

Learning Infinite-Horizon Average-Reward Linear Mixture MDPs of Bounded Span
Woojin Chae, Kihyuk Hong, Yufan Zhang, Ambuj Tewari, Dabeen Lee
LeArning Abstract Regret Guarantee Norm Bounded Minimax Optimal Regret Linear Mixture Markov Decision Process Substitute Span

September 16, 2024

Provably Efficient Infinite-Horizon Average-Reward Reinforcement Learning with Linear Function Approximation
Woojin Chae, Dabeen Lee
Linear Function Approximation Bellman Equation Linear Markov Decision Process Linear Mixture Markov Decision Process

March 17, 2024

Prior-dependent analysis of posterior sampling reinforcement learning with function approximation
Yingru Li, Zhi-Quan Luo
Reinforcement Learning Function Approximation Model Based Prior Bayesian Regret Linear Mixture Markov Decision Process Prior Dependent

March 7, 2024

Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transition
Long-Fei Li, Peng Zhao, Zhi-Hua Zhou
Reinforcement Learning Practical Algorithm Bandit Feedback Adversarial Markov Decision Process Linear Mixture Markov Decision Process Transition Uncertainty

December 7, 2023

Horizon-Free and Instance-Dependent Regret Bounds for Reinforcement Learning with General Function Approximation
Jiayi Huang, Han Zhong, Liwei Wang, Lin F. Yang
Reinforcement Learning New Horizon Instance Dependent Regret Planning Horizon Instance Dependent Linear Mixture Markov Decision Process

March 17, 2023

Optimal Horizon-Free Reward-Free Exploration for Linear Mixture MDPs
Junkai Zhang, Weitong Zhang, Quanquan Gu
Reward Free Reward Free Reinforcement Learning Reward Free Exploration Linear Mixture Markov Decision Process Reward Aware

February 27, 2023

Optimistic Planning by Regularized Dynamic Programming
Antoine Moulin, Gergely Neu
Markov Decision Process Near Optimal Policy Approximate Dynamic Programming Regularized Reinforcement Learning Linear Mixture Markov Decision Process

February 21, 2023

Variance-Dependent Regret Bounds for Linear Bandits and Reinforcement Learning: Adaptivity and Computational Efficiency
Heyang Zhao, Jiafan He, Dongruo Zhou, Tong Zhang, Quanquan Gu
Reinforcement Learning Adaptive Importance Linear Bandit Linear Contextual Bandit Computational Efficiency Variance Dependent Regret Linear Mixture Markov Decision Process Free Regret

August 10, 2022

Learning Two-Player Mixture Markov Games: Kernel Function Approximation and Correlated Equilibrium
Chris Junchi Li, Dongruo Zhou, Quanquan Gu, Michael I. Jordan
Nash Equilibrium Markov Game Nonlinear Function Approximation Correlated Equilibrium Kernel Approximation Linear Mixture Markov Decision Process

May 23, 2022

Computationally Efficient Horizon-Free Reinforcement Learning for Linear Mixture MDPs
Dongruo Zhou, Quanquan Gu
Contextual Bandit Episodic Reinforcement Learning Linear Mixture Markov Decision Process Heterogeneous Bagging

January 18, 2022

Differentially Private Reinforcement Learning with Linear Function Approximation
Xingyu Zhou
Positive Reinforcement Linear Function Approximation Finite Horizon Linear Mixture Markov Decision Process Private Reinforcement Learning

December 2, 2021

Differentially Private Exploration in Reinforcement Learning with Linear Representation
Paul Luyo, Evrard Garcelon, Alessandro Lazaric, Matteo Pirotta
Reinforcement Learning Markov Decision Process Linear Representation Linear Markov Decision Process Linear Mixture Markov Decision Process

November 5, 2021

Improved Regret Analysis for Variance-Adaptive Linear Bandits and Horizon-Free Linear Mixture MDPs
Yeoneung Kim, Insoon Yang, Kwang-Sung Jun
Linear Bandit Regret Analysis Linear Mixture Markov Decision Process Variance Dependent Regret Free Regret

Linear Mixture Markov Decision Process

Papers

Near-Optimal Dynamic Regret for Adversarial Linear Mixture MDPs

Learning Infinite-Horizon Average-Reward Linear Mixture MDPs of Bounded Span

Provably Efficient Infinite-Horizon Average-Reward Reinforcement Learning with Linear Function Approximation

Prior-dependent analysis of posterior sampling reinforcement learning with function approximation

Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transition

Horizon-Free and Instance-Dependent Regret Bounds for Reinforcement Learning with General Function Approximation

Optimal Horizon-Free Reward-Free Exploration for Linear Mixture MDPs

Optimistic Planning by Regularized Dynamic Programming

Variance-Dependent Regret Bounds for Linear Bandits and Reinforcement Learning: Adaptivity and Computational Efficiency

Learning Two-Player Mixture Markov Games: Kernel Function Approximation and Correlated Equilibrium

Computationally Efficient Horizon-Free Reinforcement Learning for Linear Mixture MDPs

Differentially Private Reinforcement Learning with Linear Function Approximation

Differentially Private Exploration in Reinforcement Learning with Linear Representation

Improved Regret Analysis for Variance-Adaptive Linear Bandits and Horizon-Free Linear Mixture MDPs