Value Based Reinforcement Learning

Value-based reinforcement learning (VBRL) aims to train agents to make optimal decisions by learning a value function that estimates the expected cumulative reward for each state-action pair. Current research focuses on improving the scalability and efficiency of VBRL algorithms, particularly for large action spaces and complex environments, exploring techniques like stochastic Q-learning, classification-based value function training, and incorporating large language models for improved sample efficiency. These advancements address limitations in existing methods, leading to improved performance in various applications, including game playing, robotics, and resource allocation problems. The resulting improvements in sample efficiency and robustness are significant for both theoretical understanding and practical deployment of reinforcement learning agents.

Papers

November 10, 2024

A Variance Minimization Approach to Temporal-Difference Learning
Xingguo Chen, Yu Gong, Shangdong Yang, Wenhao Wang
LeArning Abstract Value Based Reinforcement Learning Error Minimizing Noise

November 7, 2024

Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning
Joey Hong, Anca Dragan, Sergey Levine
Language Model Q Learning Supervised Fine Tuning Value Based Reinforcement Learning Value Based

October 14, 2024

Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning
Harley Wiltzer, Marc G. Bellemare, David Meger, Patrick Shafto, Yash Jhaveri
Reinforcement Learning Distributional Reinforcement Learning Advantage Feedback Traditional Reinforcement Learning Value Based Reinforcement Learning Action Gap

June 8, 2024

Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL
Qi Lv, Xiang Deng, Gongwei Chen, Michael Yu Wang, Liqiang Nie
Offline Reinforcement Learning Mamba in Mamba Value Based Reinforcement Learning Evolutionary Policy Search Evolving Policy

May 16, 2024

Stochastic Q-learning for Large Discrete Action Spaces
Fares Fourati, Vaneet Aggarwal, Mohamed-Slim Alouini
Reinforcement Learning Q Learning Value Based Reinforcement Learning

March 6, 2024

Stop Regressing: Training Value Functions via Classification for Scalable Deep RL
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taïga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, Rishabh Agarwal
Reinforcement Learning Deep Reinforcement Learning Classification Code Value Function Value Based Reinforcement Learning

February 25, 2024

How Can LLM Guide RL? A Value-Based Approach
Shenao Zhang, Sirui Zheng, Shuqi Ke, Zhihan Liu, Wanxin Jin, Jianbo Yuan, Yingxiang Yang, Hongxia Yang, Zhaoran Wang
Large Language Model Reinforcement Learning Reinforcement Learning Algorithm Value Based Reinforcement Learning Value Based

January 21, 2024

Constrained Reinforcement Learning for Adaptive Controller Synchronization in Distributed SDN
Ioannis Panitsas, Akrit Mudvari, Leandros Tassiulas
Constrained Reinforcement Learning Software Defined Adaptive Controller Value Based Reinforcement Learning

December 28, 2023

Rethinking Model-based, Policy-based, and Value-based Reinforcement Learning via the Lens of Representation Complexity
Guhao Feng, Han Zhong
Reinforcement Learning Camera Lens Model Free Reinforcement Learning Value Based Reinforcement Learning Representation Complexity

October 4, 2023

$\mathcal{B}$-Coder: Value-Based Deep Reinforcement Learning for Program Synthesis
Zishun Yu, Yunzhe Tao, Liyu Chen, Tao Sun, Hongxia Yang
Reinforcement Learning Program Synthesis Value Based Reinforcement Learning Value Based Value Based Deep Reinforcement Learning Policy Based Reinforcement Learning

September 6, 2023

Marketing Budget Allocation with Offline Constrained Deep Reinforcement Learning
Tianchi Cai, Jiyan Jiang, Wenpeng Zhang, Shiji Zhou, Xierui Song, Li Yu, Lihong Gu, Xiaodong Zeng, Jinjie Gu, Guannan Zhang
Reinforcement Learning Optimal Policy Value Based Reinforcement Learning Marketing Budget Allocation

July 10, 2023

Measuring and Mitigating Interference in Reinforcement Learning
Vincent Liu, Han Wang, Ruo Yu Tao, Khurram Javed, Adam White, Martha White
Reinforcement Learning Measurement System Catastrophic Interference DQN Agent Value Based Reinforcement Learning Interference Mitigation

May 30, 2023

Bigger, Better, Faster: Human-level Atari with human-level efficiency
Max Schwarzer, Johan Obando-Ceron, Aaron Courville, Marc Bellemare, Rishabh Agarwal, Pablo Samuel Castro
Sample Efficient ATARI Game Value Based Reinforcement Learning Value Estimation Superhuman Performance

November 22, 2022

A Deep Reinforcement Learning Approach to Rare Event Estimation
Anthony Corso, Kyu-Young Kim, Shubh Gupta, Grace Gao, Mykel J. Kochenderfer
Adaptive Importance Deep Reinforcement Learning Approach Rare Event Value Based Reinforcement Learning Monte Carlo Sampling Proposal Distribution

September 6, 2022

Finite-Time Error Bounds for Greedy-GQ
Yue Wang, Yi Zhou, Shaofeng Zou
Reinforcement Learning Finite Time Policy Algorithm Value Based Reinforcement Learning

June 17, 2022

Generalised Policy Improvement with Geometric Policy Composition
Shantanu Thakoor, Mark Rowland, Diana Borsa, Will Dabney, Rémi Munos, André Barreto
Policy Iteration Value Based Reinforcement Learning Generalized Policy Improvement Markovian Policy

June 3, 2022

Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, Marc G. Bellemare
Reinforcement Learning Deep Reinforcement Learning Much Progress Real World Reinforcement Learning Value Based Reinforcement Learning Tabular Reinforcement Learning

June 1, 2022

The Phenomenon of Policy Churn
Tom Schaul, André Barreto, John Quan, Georg Ostrovski
Rare Event Greedy SLIM Value Based Reinforcement Learning Customer Churn Greedy Policy Implicit Exploration

May 26, 2022

Does DQN Learn?
Aditya Gopalan, Gugan Thoppe
Reinforcement Learning Q Learning Environment Exploration Average Approximation Value Based Reinforcement Learning Greedy Exploration Policy Switching

March 20, 2022

Smoothing Advantage Learning
Yaozhong Gan, Zhe Zhang, Xiaoyang Tan
Sub Optimal Value Based Reinforcement Learning Bellman Operator Advantage Learning