Q Function

The Q-function, central to reinforcement learning, estimates the expected cumulative reward for taking a specific action in a given state. Current research focuses on improving Q-function estimation accuracy and efficiency, particularly through variance reduction techniques, and exploring its application in diverse settings such as multi-agent systems, continuous action spaces, and large language model alignment. These advancements are driving progress in offline reinforcement learning, enabling more efficient and robust decision-making in complex environments and leading to improved performance in various applications, including robotics and healthcare.

Papers

September 18, 2023

Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions
Yevgen Chebotar, Quan Vuong, Alex Irpan, Karol Hausman, Fei Xia, Yao Lu, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, Keerthana Gopalakrishnan, Julian Ibarz, Ofir Nachum, Sumedh Sontakke, Grecia Salazar, Huong T Tran, Jodilyn Peralta, Clayton Tan, Deeksha Manjunath, Jaspiar Singht, Brianna Zitkovich, Tomas Jackson, Kanishka Rao, Chelsea Finn, Sergey Levine
Reinforcement Learning Offline Reinforcement Learning Q Learning Q Function Task Specific Policy Q Transformer

July 30, 2023

Variance Control for Distributional Reinforcement Learning
Qi Kuang, Zhoufan Zhu, Liwen Zhang, Fan Zhou
Variance Reduction Distributional Reinforcement Learning Distributional Assumption Q Function Function Estimation

July 25, 2023

Offline Reinforcement Learning with On-Policy Q-Function Regularization
Laixi Shi, Robert Dadashi, Yuejie Chi, Pablo Samuel Castro, Matthieu Geist
Offline Reinforcement Learning Estimation Task Q Function Policy Regularization Poor Extrapolation Performance

June 28, 2023

Continuous-time q-learning for mean-field control problems
Xiaoli Wei, Xiang Yu
Reinforcement Learning Q Learning Mean Field Q Function Entropy Regularized McKean Vlasov

June 5, 2023

Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic
Tianying Ji, Yu Luo, Fuchun Sun, Xianyuan Zhan, Jianwei Zhang, Huazhe Xu
Policy Learning Net Present Value Financial Success Q Function Policy Deep Reinforcement Learning Action Sampling

May 29, 2023

VA-learning as a more efficient alternative to Q-learning
Yunhao Tang, Rémi Munos, Mark Rowland, Michal Valko
Reinforcement Learning Q Learning Q Function Efficient Alternative

May 2, 2023

Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare
Shengpu Tang, Maggie Makar, Michael W. Sjoding, Finale Doshi-Velez, Jenna Wiens
Reinforcement Learning Action Space Healthcare System State Action Space Q Function

March 15, 2023

On the Benefits of Leveraging Structural Information in Planning Over the Learned Model
Jiajun Shen, Kananart Kuwaranancharoen, Raid Ayoub, Pietro Mercati, Shreyas Sundaram
Task Planning Sample Complexity Model Based Reinforcement Learning Complementary Benefit Q Function State Action Pair Structural Information Learned Model Grid World

February 15, 2023

Prioritized offline Goal-swapping Experience Replay
Wenyan Yang, Joni Pajarinen, Dinging Cai, Joni Kämäräinen
Experience Replay Q Function Offline Goal Conditioned Reinforcement Learning

February 7, 2023

Catch Me If You Can: Improving Adversaries in Cyber-Security With Q-Learning Algorithms
Arti Bandhana, Ondřej Lukáš, Sebastian Garcia, Tomáš Kroupa
Q Learning Q Function Adversary Agent Pseudo Victim AttackER Cyber Security Double Q Learning

February 1, 2023

Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders
David Bruns-Smith, Angela Zhou
Offline Reinforcement Learning Unobserved Confounders Q Function Fitted Q Iteration Q Evaluation Robust Policy Evaluation

January 15, 2023

Efficiently Computing Sparse Fourier Transforms of $q$-ary Functions
Yigit Efe Erginbas, Justin Singh Kang, Amirali Aghazadeh, Kannan Ramchandran
Q Function Fourier Transform Sparse Fourier

January 5, 2023

Extreme Q-Learning: MaxEnt RL without Entropy
Divyansh Garg, Joey Hejna, Matthieu Geist, Stefano Ermon
Offline Reinforcement Learning Q Learning Q Function Stable Entropy Maximum Entropy Reinforcement Learning MaxEnt RL Modern Deep Reinforcement Learning

November 22, 2022

Greedy based Value Representation for Optimal Coordination in Multi-agent Reinforcement Learning
Lipeng Wan, Zeyang Liu, Xingyu Chen, Xuguang Lan, Nanning Zheng
Multi Agent Reinforcement Learning Q Function Greedy SLIM Coordination Strategy Value Function Decomposition

October 14, 2022

Model-Free Characterizations of the Hamilton-Jacobi-Bellman Equation and Convex Q-Learning in Continuous Time
Fan Lu, Joel Mathias, Sean Meyn, Karanjit Kalsi
Reinforcement Learning Model Free Continuous Time Finite Horizon Q Function Hamilton Jacobi Bellman Convex Q Learning

September 7, 2022

On the Convergence of Monte Carlo UCB for Random-Length Episodic MDPs
Zixuan Dong, Che Wang, Keith Ross
Reinforcement Learning Early Stage Convergence Q Learning Monte Carlo Finite Horizon Episodic Markov Decision Process Q Function

July 22, 2022

Lagrangian Method for Q-Function Learning (with Applications to Machine Translation)
Huang Bojun
Financial Application Machine Translation Lagrangian System Q Function Lagrangian Method Bellman Equation

June 15, 2022

Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning
Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, Yi Wu
Policy Gradient Practice Mode Cooperative Multi Agent Reinforcement Learning Multi Agent Challenge Q Function Multi Agent Policy

June 1, 2022

CoNSoLe: Convex Neural Symbolic Learning
Haoran Li, Yang Weng, Hanghang Tong
Video Game Deep Q Learning Q Function Lower Level Convexity Simplification Equation Learner Network Neural Symbolic Learning

April 28, 2022

Bilinear value networks
Zhang-Wei Hong, Ge Yang, Pulkit Agrawal
Q Function Q Value Bilinear Matrix Inequality Multi Goal Reinforcement Learning Value Network

Q Function

Papers

Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions

Variance Control for Distributional Reinforcement Learning

Offline Reinforcement Learning with On-Policy Q-Function Regularization

Continuous-time q-learning for mean-field control problems

Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic

VA-learning as a more efficient alternative to Q-learning

Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare

On the Benefits of Leveraging Structural Information in Planning Over the Learned Model

Prioritized offline Goal-swapping Experience Replay

Catch Me If You Can: Improving Adversaries in Cyber-Security With Q-Learning Algorithms

Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders

Efficiently Computing Sparse Fourier Transforms of $q$-ary Functions

Extreme Q-Learning: MaxEnt RL without Entropy

Greedy based Value Representation for Optimal Coordination in Multi-agent Reinforcement Learning

Model-Free Characterizations of the Hamilton-Jacobi-Bellman Equation and Convex Q-Learning in Continuous Time

On the Convergence of Monte Carlo UCB for Random-Length Episodic MDPs

Lagrangian Method for Q-Function Learning (with Applications to Machine Translation)

Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning

CoNSoLe: Convex Neural Symbolic Learning

Bilinear value networks