Function Estimation

Function estimation, specifically of Q-functions in reinforcement learning (RL), aims to accurately approximate the value of taking specific actions in different states to guide optimal decision-making. Current research focuses on improving the accuracy and efficiency of Q-function estimation, addressing issues like overestimation bias through techniques such as double Q-learning and ensemble methods, and mitigating divergence in offline RL settings by employing conservative estimation strategies and architectural improvements like LayerNorm. These advancements are crucial for enhancing the performance and stability of RL algorithms across various applications, from robotics and game playing to personalized medicine and resource management.

Papers

September 6, 2024

Gaussian-Mixture-Model Q-Functions for Reinforcement Learning by Riemannian Optimization
Minh Vu, Konstantinos Slavakis
Reinforcement Learning Loss Function Deep Q Network Riemannian Optimization Function Estimation

June 14, 2024

Finite-Time Analysis of Simultaneous Double Q-learning
Hyunjun Na, Donghwan Lee
Reinforcement Learning Finite Time Q$ Learning Double Q Learning Function Estimation

October 6, 2023

Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL
Yang Yue, Rui Lu, Bingyi Kang, Shiji Song, Gao Huang
Human Prediction Human Understanding Good Better Q Value Offline RL Algorithm Function Estimation

August 3, 2023

Minimax Optimal Q Learning with Nearest Neighbors
Puning Zhao, Lifeng Lai
Markov Decision Process Nearest Neighbor State Space Minimax Q Learning Function Estimation Q$ Learning Algorithm

July 30, 2023

Variance Control for Distributional Reinforcement Learning
Qi Kuang, Zhoufan Zhu, Liwen Zhang, Fan Zhou
Variance Reduction Distributional Reinforcement Learning Distributional Assumption Q Function Function Estimation

June 20, 2023

Adaptive Ensemble Q-learning: Minimizing Estimation Bias via Error Feedback
Hang Wang, Sen Lin, Junshan Zhang
Q Learning Error Feedback Estimation Bias Adaptive Ensemble Function Estimation Ensemble Q Learning

May 4, 2023

Correcting for Interference in Experiments: A Case Study at Douyin
Vivek F. Farias, Hao Li, Tianyi Peng, Xinyuyang Ren, Huawei Zhang, Andrew Zheng
Case Study Treatment Effect Optical Experiment Language Correction Monte Carlo Catastrophic Interference Two Sided Function Estimation

February 14, 2023

Conservative State Value Estimation for Offline Reinforcement Learning
Liting Chen, Jie Yan, Zhengdao Shao, Lu Wang, Qingwei Lin, Saravan Rajmohan, Thomas Moscibroda, Dongmei Zhang
Offline Reinforcement Learning Conservative Q Learning Conservative Value Estimation Function Estimation

June 6, 2022

April 7, 2022

Q-learning with online random forests
Joosung Min, Lloyd T. Elliott
Random Forest Q Learning Model Free Reinforcement Learning Q$ Learning Function Estimation

February 26, 2022

Statistically Efficient Advantage Learning for Offline Reinforcement Learning in Infinite Horizons
Chengchun Shi, Shikai Luo, Yuan Le, Hongtu Zhu, Rui Song
Reinforcement Learning Offline Reinforcement Learning Policy OpTimization Policy Optimization New Horizon Function Estimation Advantage Learning

February 9, 2022

Transferred Q-learning
Elynn Y. Chen, Michael I. Jordan, Sai Li
Transfer Learning Q$ Learning Transfer Reinforcement Learning Function Estimation Q$ Learning Algorithm Target Policy

January 17, 2022

On Well-posedness and Minimax Optimal Rates of Nonparametric Q-function Estimation in Off-policy Evaluation
Xiaohong Chen, Zhengling Qi
Policy Evaluation Infinite Horizon Function Estimation Smoothness Constraint Nonparametric Instrumental Variable Rate Maximization