Fitted Q Iteration

Fitted Q-Iteration (FQI) is an offline reinforcement learning algorithm aiming to estimate optimal action-value functions (Q-functions) iteratively from a fixed dataset, thereby learning optimal policies without further interaction with the environment. Current research focuses on improving FQI's sample efficiency and robustness through various techniques, including employing max-plus linear approximators, alternative loss functions (like log-loss), and incorporating structural assumptions like reward-relevance filtering or action impact regularity to reduce the complexity of the problem. These advancements enhance FQI's applicability to real-world problems where online data collection is expensive or impossible, particularly in domains like healthcare and economics, by providing theoretical guarantees and improved empirical performance.

Papers

September 12, 2024

Fitted Q-Iteration via Max-Plus-Linear Approximation
Y. Liu, M. A. S. Kolarijani
Markov Decision Process Offline Reinforcement Learning Linear Function Approximation Fitted Q Iteration Max Affine

March 8, 2024

Switching the Loss Reduces the Cost in Batch Reinforcement Learning
Alex Ayoub, Kaiwen Wang, Vincent Liu, Samuel Robertson, James McInerney, Dawen Liang, Nathan Kallus, Csaba Szepesvári
Hidden CoST Interest Loss Near Optimal Policy Batch Reinforcement Learning Fitted Q Iteration Small Loss

January 23, 2024

Reward-Relevance-Filtered Linear Offline Reinforcement Learning
Angela Zhou
Reinforcement Learning Policy Evaluation Sparse Regression Fitted Q Iteration Decision Sparsity

June 22, 2023

Fitted Value Iteration Methods for Bicausal Optimal Transport
Erhan Bayraktar, Bingyan Han
Optimal Transport Dynamic Programming Linear Programming Numerical Experiment Value Iteration Fitted Q Iteration

February 1, 2023

Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounders
David Bruns-Smith, Angela Zhou
Offline Reinforcement Learning Unobserved Confounders Q Function Fitted Q Iteration Q Evaluation Robust Policy Evaluation

November 14, 2022

On the Global Convergence of Fitted Q-Iteration with Two-layer Neural Network Parametrization
Mudit Gaur, Vaneet Aggarwal, Mridul Agarwal
Neural Network Sample Complexity Deep Q Learning Global Convergence Two Layer ReLU Fitted Q Iteration

March 31, 2022

Neural Q-learning for solving PDEs
Samuel N. Cohen, Deqing Jiang, Justin Sirignano
Q Learning Ordinary Differential Equation High Dimensional Partial Differential Equation Elliptic Partial Differential Equation Monotone Neural Network Fitted Q Iteration

March 8, 2022

A Complete Characterization of Linear Estimators for Offline Policy Evaluation
Juan C. Perdomo, Akshay Krishnamurthy, Peter Bartlett, Sham Kakade
Reinforcement Learning Linear Function Approximation Function Approximation Linear Estimator Offline Policy Evaluation Complete Characterization Fitted Q Iteration

January 31, 2022

Optimal Estimation of Off-Policy Policy Gradient via Double Fitted Iteration
Chengzhuo Ni, Ruiqi Zhang, Xiang Ji, Xuezhou Zhang, Mengdi Wang
Policy Gradient Optimal Estimation Fitted Q Iteration Policy Policy Gradient

November 15, 2021

Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning
Vincent Liu, James R. Wright, Martha White
Offline Reinforcement Learning Exogenous Variable Fitted Q Iteration

November 5, 2021

Perturbational Complexity by Distribution Mismatch: A Systematic Analysis of Reinforcement Learning in Reproducing Kernel Hilbert Space
Jihao Long, Jiequn Han
Reinforcement Learning Markov Decision Process Reproducing Kernel Hilbert Space State Action Space Fitted Q Iteration Distribution Mismatch Asymptotic Complexity