Contextual Markov Decision Process

Contextual Markov Decision Processes (CMDPs) extend standard reinforcement learning by incorporating contextual information that influences transition dynamics and rewards, enabling more realistic modeling of time-varying environments. Current research focuses on developing efficient algorithms, particularly model-based approaches and those leveraging function approximation (like linear models), to address the increased complexity of CMDPs, often incorporating techniques like meta-learning and multi-agent learning. These advancements aim to improve sample efficiency and generalization capabilities, leading to more robust and adaptable AI agents for applications ranging from power grid management to personalized recommendations. The theoretical analysis of sample complexity and regret bounds remains a key focus, driving progress towards provably efficient CMDP solutions.

Papers

October 24, 2024

Learning to Look: Seeking Information for Decision Making via Policy Factorization
Shivin Dass, Jiaheng Hu, Ben Abbatematteo, Peter Stone, Roberto Martín-Martín
Decision Making Manipulation Task Full Information Health Policy Recommendation Prime Factorization Contextual Markov Decision Process

June 3, 2024

Stochastic Bilevel Optimization with Lower-Level Contextual Markov Decision Processes
Vinzenz Thoma, Barna Pasztor, Andreas Krause, Giorgia Ramponi, Yifan Hu
Bilevel Optimization Stochastic Bilevel Optimization Contextual Markov Decision Process

May 28, 2024

Offline Oracle-Efficient Learning for Contextual MDPs via Layerwise Exploration-Exploitation Tradeoff
Jian Qian, Haichen Hu, David Simchi-Levi
Contextual Bandit Action Free Offline Exploration Exploitation Trade Near Optimal Algorithm Reward Free Reinforcement Learning Contextual Markov Decision Process Contextual MDPs

February 5, 2024

Sample Complexity Characterization for Linear Contextual MDPs
Junze Deng, Yuan Cheng, Shaofeng Zou, Yingbin Liang
Sample Complexity Linear Ordered Data MDP Model Contextual Markov Decision Process Contextual MDPs Time Varying Environment

January 19, 2024

Stochastic Dynamic Power Dispatch with High Generalization and Few-Shot Adaption via Contextual Meta Graph Reinforcement Learning
Bairong Deng, Tao Yu, Zhenning Pan, Xuehan Zhang, Yufeng Wu, Qiaoyi Ding
Strong Generalization Adaptation Concern Contextual Markov Decision Process Power Dispatch

July 11, 2023

Scaling Distributed Multi-task Reinforcement Learning with Experience Sharing
Sanae Amani, Khushbu Pahwa, Vladimir Braverman, Lin F. Yang
Optimal Policy Multiplicative Size Scaling Multi Task Reinforcement Learning Contextual Markov Decision Process Experience Sharing

June 13, 2023

Stepsize Learning for Policy Gradient Methods in Contextual Markov Decision Processes
Luca Sabbioni, Francesco Corda, Marcello Restelli
Policy Gradient Action Space Model Free Meta Reinforcement Learning Batch Reinforcement Learning Contextual Markov Decision Process Policy Based Algorithm

February 4, 2023

Reinforcement Learning with History-Dependent Dynamic Contexts
Guy Tennenholtz, Nadav Merlis, Lior Shani, Martin Mladenov, Craig Boutilier
Reinforcement Learning Non Markovian Contextual Markov Decision Process Contextual MDPs

November 27, 2022

Eluder-based Regret for Stochastic Contextual MDPs
Orin Levy, Asaf Cassel, Alon Cohen, Yishay Mansour
Optimal Regret Stochastic Way Regret Minimization Contextual Markov Decision Process Contextual MDPs RL Algorithm

October 13, 2022

Multi-agent Dynamic Algorithm Configuration
Ke Xue, Jiacheng Xu, Lei Yuan, Miqing Li, Chao Qian, Zongzhang Zhang, Yang Yu
Reinforcement Learning Single Agent Reinforcement Learning Contextual Markov Decision Process

August 3, 2022

AACC: Asymmetric Actor-Critic in Contextual Reinforcement Learning
Wangyang Yue, Yuan Zhou, Xiaochuan Zhang, Yuchen Hua, Zhiyuan Wang, Guang Kou
Reinforcement Learning Actor Critic Contextual Markov Decision Process Asymmetric Actor Critic

June 1, 2022

Provably Efficient Lifelong Reinforcement Learning with Linear Function Approximation
Sanae Amani, Lin F. Yang, Ching-An Cheng
Sublinear Regret Linear Function Approximation Lifelong Reinforcement Learning Contextual Markov Decision Process

Contextual Markov Decision Process

Papers

Learning to Look: Seeking Information for Decision Making via Policy Factorization

Stochastic Bilevel Optimization with Lower-Level Contextual Markov Decision Processes

Offline Oracle-Efficient Learning for Contextual MDPs via Layerwise Exploration-Exploitation Tradeoff

Sample Complexity Characterization for Linear Contextual MDPs

Stochastic Dynamic Power Dispatch with High Generalization and Few-Shot Adaption via Contextual Meta Graph Reinforcement Learning

Scaling Distributed Multi-task Reinforcement Learning with Experience Sharing

Stepsize Learning for Policy Gradient Methods in Contextual Markov Decision Processes

Reinforcement Learning with History-Dependent Dynamic Contexts

Eluder-based Regret for Stochastic Contextual MDPs

Multi-agent Dynamic Algorithm Configuration

AACC: Asymmetric Actor-Critic in Contextual Reinforcement Learning

Provably Efficient Lifelong Reinforcement Learning with Linear Function Approximation