Bellman Error

Bellman error, the discrepancy between the estimated and true value functions in reinforcement learning, is a central focus in improving the accuracy and efficiency of learning algorithms. Current research investigates minimizing Bellman error through various approaches, including modifications to existing algorithms like Q-learning and the development of novel methods such as those employing logistic likelihood functions or anchoring mechanisms to accelerate convergence. These efforts aim to enhance the performance of reinforcement learning agents, particularly in offline settings with limited data or in the presence of adversarial attacks, ultimately leading to more robust and efficient AI systems. Reducing Bellman error is crucial for improving both the theoretical understanding and practical application of reinforcement learning.

Papers

October 31, 2024

Deterministic Exploration via Stationary Bellman Error Maximization
Sebastian Griesbach, Carlo D'Eramo
Reinforcement Learning Environment Exploration Intrinsic Reward Exploration Policy Bellman Error Random Exploration

June 22, 2024

To Err is Robotic: Rapid Value-Based Trial-and-Error during Deployment
Maximilian Du, Alexander Khazatsky, Tobias Gerstenberg, Chelsea Finn
Non Humanoid Robot Robot Person Software Deployment Robot Policy Bellman Error Trial and Error

June 17, 2024

The Role of Inherent Bellman Error in Offline Reinforcement Learning with Linear Function Approximation
Noah Golowich, Ankur Moitra
Reinforcement Learning Offline Reinforcement Learning Integral Role Linear Function Approximation Bellman Error Bellman Completeness

March 12, 2024

Symmetric Q-learning: Reducing Skewness of Bellman Error in Online Reinforcement Learning
Motoki Omura, Takayuki Osa, Yusuke Mukuta, Tatsuya Harada
Deep Reinforcement Learning Online Reinforcement Learning State of the Art Reinforcement Error Distribution Skewed Non IID Bellman Error

February 3, 2024

Towards Optimal Adversarial Robust Q-learning with Bellman Infinity-error
Haoran Li, Zicheng Zhang, Wang Luo, Congying Han, Yudong Hu, Tiande Guo, Shichen Liao
Deep Reinforcement Learning Q Learning Robust Policy Bellman Error Optimal Robust Policy State Adversarial

October 10, 2023

Bi-Level Offline Policy Optimization with Limited Exploration
Wenzhuo Zhou
Offline Reinforcement Learning Policy Optimization Regret Guarantee Offline Policy Pure Exploration Bellman Error Adversarial Estimator

July 5, 2023

LLQL: Logistic Likelihood Q-Learning for Reinforcement Learning
Outongyi Lv, Bingxin Zhou
Reinforcement Learning Bellman Equation Modern Reinforcement Learning Bellman Error Logistic Model

May 26, 2023

Accelerating Value Iteration with Anchoring
Jongmin Lee, Ernest K. Ryu
Value Iteration Modern Reinforcement Learning Bellman Error Optimality Criterion Independent Rate

January 31, 2023

January 30, 2023

Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning
Hanlin Zhu, Paria Rashidinejad, Jiantao Jiao
Offline Reinforcement Learning Bellman Error Single Policy Concentrability

January 28, 2022

Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error
Scott Fujimoto, David Meger, Doina Precup, Ofir Nachum, Shixiang Shane Gu
Value Function State Action Pair Bellman Equation Bellman Operator Bellman Error Value Prediction

January 19, 2022

Critic Algorithms using Cooperative Networks
Debangshu Banerjee, Kavita Wagh
Markov Decision Process DQN Agent Residual Model Critic Model Bellman Error Bellman Update