Bellman Update

The Bellman update is a fundamental iterative process in reinforcement learning used to estimate optimal value functions, guiding an agent to make optimal decisions in a given environment. Current research focuses on improving the efficiency and accuracy of Bellman updates, exploring methods like iterated Q-networks for multiple updates and parameterized Bellman operators to learn approximations directly, rather than relying solely on sample-based estimations. These advancements aim to address computational challenges and improve sample efficiency in various reinforcement learning algorithms, impacting applications ranging from game playing to robotics and network optimization.

Papers

November 14, 2024

To bootstrap or to rollout? An optimal and adaptive interpolation
Wenlong Mou, Jian Qian
Reinforcement Learning Bellman Operator Sample Efficient Adaptation Bellman Update

April 3, 2024

Reinforcement Learning in Categorical Cybernetics
Jules Hedges, Riu Rodríguez Sakamoto
Reinforcement Learning Value Iteration Category Theory Action Value Function Bellman Update

March 4, 2024

Iterated $Q$-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning
Théo Vincent, Daniel Palenicek, Boris Belousov, Jan Peters, Carlo D'Eramo
Reinforcement Learning Deep Reinforcement Learning Actor Critic Q Network Action Value Function Bellman Operator Bellman Update

December 20, 2023

Parameterized Projected Bellman Operator
Théo Vincent, Alberto Maria Metelli, Boris Belousov, Jan Peters, Marcello Restelli, Carlo D'Eramo
Reinforcement Learning Value Iteration Bellman Operator Bellman Update Transition Operator

July 11, 2023

Reinforcement Learning with Non-Cumulative Objective
Wei Cui, Wei Yu
Reinforcement Learning Bellman Equation Differentiable Objective Bellman Update

January 26, 2022

Combining Optimal Path Search With Task-Dependent Learning in a Neural Network
Tomas Kulvicius, Minija Tamosiunaite, Florentin Wörgötter
Neural Network Task Specific Optimal Path Path Finding Network Learning Bellman Update Path Augmentation

January 19, 2022

Critic Algorithms using Cooperative Networks
Debangshu Banerjee, Kavita Wagh
Markov Decision Process DQN Agent Residual Model Critic Model Bellman Error Bellman Update