Single Policy Concentrability

Single policy concentrability is a key assumption in offline reinforcement learning (RL), focusing on ensuring that the offline dataset sufficiently covers the state-action space visited by the target policy. Current research emphasizes developing provably efficient algorithms under this assumption, often employing primal-dual methods, marginalized importance sampling, or pessimism-based approaches within actor-critic frameworks. This focus stems from the need to address the challenge of partial data coverage in offline RL, enabling reliable learning of optimal policies from limited historical data. Advances in this area are crucial for making offline RL more practical and applicable to real-world scenarios where extensive online data collection is infeasible or costly.

Papers

October 27, 2023

Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage
Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, Mohammad Ghavamzadeh
Offline Reinforcement Learning Constructive Approach Distribution Shift Limited Field Offline Reinforcement Learning Algorithm Distributionally Robust Distribution Shift Problem Single Policy Concentrability

May 24, 2023

Provable Offline Preference-Based Reinforcement Learning
Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D. Lee, Wen Sun
Reinforcement Learning Offline Data Implicit Reward State Reward Single Policy Concentrability Offline Preference Based Reinforcement Learning

February 7, 2023

Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability
Hanlin Zhu, Amy Zhang
Policy Learning Goal Conditioned Reinforcement Learning Offline Goal Conditioned Reinforcement Learning Single Policy Concentrability

January 30, 2023

Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning
Hanlin Zhu, Paria Rashidinejad, Jiantao Jiao
Offline Reinforcement Learning Bellman Error Single Policy Concentrability

November 1, 2022

Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
Paria Rashidinejad, Hanlin Zhu, Kunhe Yang, Stuart Russell, Jiantao Jiao
Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Conservative Policy Augmented Lagrangian Single Policy Concentrability

July 13, 2022

A Near-Optimal Primal-Dual Method for Off-Policy Learning in CMDP
Fan Chen, Junyu Zhang, Zaiwen Wen
Safe Reinforcement Learning Policy Learning Primal Dual Constrained Markov Decision Process Near Optimal Single Policy Concentrability

February 9, 2022

Offline Reinforcement Learning with Realizability and Single-policy Concentrability
Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, Jason D. Lee
Offline Reinforcement Learning Primal Dual Multiple Realizability Sample Efficient Offline Reinforcement Learning Single Policy Concentrability

Single Policy Concentrability

Papers

Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage

Provable Offline Preference-Based Reinforcement Learning

Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability

Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning

Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian

A Near-Optimal Primal-Dual Method for Off-Policy Learning in CMDP

Offline Reinforcement Learning with Realizability and Single-policy Concentrability