Conservative Q Learning

Conservative Q-learning (CQL) is an offline reinforcement learning algorithm designed to mitigate the risk of overestimating value functions, a common problem when learning from static datasets. Current research focuses on improving CQL's performance and robustness through techniques like incorporating novel neural network architectures (e.g., Kolmogorov-Arnold Networks), addressing data imbalances, and developing more nuanced approaches to pessimism in value estimation. These advancements are significant because they enhance the reliability and applicability of offline RL in various domains, including robotics, healthcare, and resource management, where online learning is impractical or unsafe.

Papers

October 30, 2024

Offline Reinforcement Learning and Sequence Modeling for Downlink Link Adaptation
Samuele Peri, Alessio Russo, Gabor Fodor, Pablo Soldati
Reinforcement Learning Offline Reinforcement Learning Adaptation Concern Sequence Modeling Radio Access Network Conservative Q Learning Downlink Communication

September 15, 2024

KAN v.s. MLP for Offline Reinforcement Learning
Haihong Guo, Fengxin Li, Jiao Li, Hongyan Liu
Offline Reinforcement Learning Kolmogorov Arnold Network Conservative Q Learning

June 20, 2024

Equivariant Offline Reinforcement Learning
Arsh Tangri, Ondrej Biza, Dian Wang, David Klee, Owen Howell, Robert Platt
Online Reinforcement Learning Conservative Q Learning Robot Policy Learning Implicit Q Learning

June 6, 2024

Strategically Conservative Q-Learning
Yutaka Shimizu, Joey Hong, Sergey Levine, Masayoshi Tomizuka
Offline Reinforcement Learning Distribution Data Conservative Q Learning Conservative Value Estimation

February 13, 2024

Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning for Digital Twins
Eslam Eldeeb, Houssem Sifaou, Osvaldo Simeone, Mohammad Shehab, Hirley Alves
Multi Agent Reinforcement Learning Multi Agent System Digital Twin Quantile Regression Distributional Reinforcement Learning Multi Agent Reinforcement Conservative Q Learning Offline Safe Reinforcement Learning

February 3, 2024

MinMaxMin $Q$-learning
Nitsan Soffair, Shie Mannor
Mini Batch Q$ Learning Conservative Q Learning

December 24, 2023

A Conservative Approach for Few-Shot Transfer in Off-Dynamics Reinforcement Learning
Paul Daoudi, Christophe Prieur, Bogdan Robu, Merwan Barlier, Ludovic Dos Santos
Imitation Learning Formality Transfer Trained Source Model Conservative Q Learning Conservative Prediction

September 22, 2023

Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning
Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang, Xiangyang Ji
Multi Agent Reinforcement Learning Offline Multi Agent Reinforcement Learning Conservative Q Learning Conservative Value Estimation

July 6, 2023

Offline Reinforcement Learning with Imbalanced Datasets
Li Jiang, Sijie Cheng, Jielin Qiu, Haoran Xu, Wai Kin Chan, Zhao Ding
Offline Reinforcement Learning Imbalanced Datasets Conservative Q Learning

June 13, 2023

Pruning the Way to Reliable Policies: A Multi-Objective Deep Q-Learning Approach to Critical Care
Ali Shirali, Alexander Schubert, Ahmed Alaa
Reinforcement Learning Constructive Approach Reward Function Edge Pruning Deep Q Learning Conservative Q Learning Intensive Care Multi Objective Deep Reinforcement Learning

May 23, 2023

When should we prefer Decision Transformers for Offline Reinforcement Learning?
Prajjwal Bhargava, Rohan Chitnis, Alborz Geramifard, Shagun Sodhani, Amy Zhang
Imitation Learning Offline Reinforcement Learning Q Learning Decision Transformer Conservative Q Learning Robotics Benchmark

March 9, 2023

Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, Sergey Levine
Offline Reinforcement Learning Q Learning Offline Reinforcement Learning Algorithm Conservative Q Learning Online Fine Tuning

March 7, 2023

Environment Transformer and Policy Optimization for Model-Based Offline Reinforcement Learning
Pengqin Wang, Meixin Zhu, Shaojie Shen
Policy OpTimization Model Based Offline Reinforcement Learning Conservative Q Learning

February 15, 2023

Deep Offline Reinforcement Learning for Real-world Treatment Optimization Applications
Milashini Nambiar, Supriyo Ghosh, Priscilla Ong, Yu En Chan, Yong Mong Bee, Pavitra Krishnaswamy
Reinforcement Learning Policy Reinforcement Learning Conservative Q Learning Double Deep Q Network Optimal Treatment

February 14, 2023

Conservative State Value Estimation for Offline Reinforcement Learning
Liting Chen, Jie Yan, Zhengdao Shao, Lu Wang, Qingwei Lin, Saravan Rajmohan, Thomas Moscibroda, Dongmei Zhang
Offline Reinforcement Learning Conservative Q Learning Conservative Value Estimation Function Estimation

January 3, 2023

Contextual Conservative Q-Learning for Offline Reinforcement Learning
Ke Jiang, Jiayu Yao, Xiaoyang Tan
Offline Reinforcement Learning Conservative Q Learning Inverse Dynamic

November 28, 2022

State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learning
Chen Chen, Hongyao Tang, Yi Ma, Chao Wang, Qianli Shen, Dong Li, Jianye Hao
Offline Reinforcement Learning Conservative Q Learning Behavior Regularization

November 2, 2022

Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints
Anikait Singh, Aviral Kumar, Quan Vuong, Yevgen Chebotar, Sergey Levine
Offline Reinforcement Learning Based Policy Real World Dataset Conservative Q Learning

October 14, 2022

Mutual Information Regularized Offline Reinforcement Learning
Xiao Ma, Bingyi Kang, Zhongwen Xu, Min Lin, Shuicheng Yan
Offline Reinforcement Learning Mutual Information Conservative Q Learning Mutual Information Regularization Behavior Regularization

October 13, 2022

Sustainable Online Reinforcement Learning for Auto-bidding
Zhiyu Mou, Yusen Huo, Rongquan Bai, Mingzhou Xie, Chuan Yu, Jian Xu, Bo Zheng
Reinforcement Learning Conservative Q Learning Auto Bidding