Bellman Operator

The Bellman operator is a fundamental concept in reinforcement learning, used to iteratively improve estimates of optimal value functions by recursively applying a dynamic programming principle. Current research focuses on enhancing the robustness and efficiency of Bellman-based algorithms, particularly in the face of corrupted data, high-dimensional state spaces, and safety constraints, often employing techniques like robust empirical Bellman operators, knowledge distillation, and iterated Q-networks. These advancements are crucial for improving the reliability and applicability of reinforcement learning in real-world scenarios, ranging from finance and robotics to safety-critical systems.

Papers

November 14, 2024

To bootstrap or to rollout? An optimal and adaptive interpolation
Wenlong Mou, Jian Qian
Reinforcement Learning Bellman Operator Sample Efficient Adaptation Bellman Update

September 5, 2024

Robust Q-Learning under Corrupted Rewards
Sreejeet Maity, Aritra Mitra
Q Learning Bellman Operator Vanilla Reinforcement Learning

May 10, 2024

Projection by Convolution: Optimal Sample Complexity for Reinforcement Learning in Continuous-Space MDPs
Davide Maran, Alberto Maria Metelli, Matteo Papini, Marcello Restelli
Reinforcement Learning Sample Complexity Optimal Policy Low Rank Direct Convolution Projection Bias Bellman Operator

May 8, 2024

Markowitz Meets Bellman: Knowledge-distilled Reinforcement Learning for Portfolio Management
Gang Hu, Ming Gu
Knowledge Distillation Portfolio Management Policy Distillation Bellman Operator Portfolio Performance

March 4, 2024

Iterated $Q$-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning
Théo Vincent, Daniel Palenicek, Boris Belousov, Jan Peters, Carlo D'Eramo
Reinforcement Learning Deep Reinforcement Learning Actor Critic Q Network Action Value Function Bellman Operator Bellman Update

January 23, 2024

Learning safety critics via a non-contractive binary bellman operator
Agustin Castellano, Hancheng Min, Juan Andrés Bazerque, Enrique Mallada
Reinforcement Learning Bellman Equation Bellman Operator Safety Critic

December 20, 2023

Parameterized Projected Bellman Operator
Théo Vincent, Alberto Maria Metelli, Boris Belousov, Jan Peters, Marcello Restelli, Carlo D'Eramo
Reinforcement Learning Value Iteration Bellman Operator Bellman Update Transition Operator

December 7, 2023

MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator
Xiao-Yin Liu, Xiao-Hu Zhou, Guotao Li, Hao Li, Mei-Jiang Gui, Tian-Yu Xiang, De-Xing Huang, Zeng-Guang Hou
Offline Reinforcement Learning Model Based Offline Reinforcement Learning Bellman Operator Conservative Policy

September 28, 2023

Multi-Bellman operator for convergence of $Q$-learning with linear function approximation
Diogo S. Carvalho, Pedro A. Santos, Francisco S. Melo
Early Stage Convergence Linear Function Approximation Q$ Learning Bellman Operator Q$ Learning Algorithm

February 25, 2023

On Bellman's principle of optimality and Reinforcement learning for safety-constrained Markov decision process
Rahul Misra, Rafał Wisniewski, Carsten Skovmose Kallesøe
Reinforcement Learning Safe Reinforcement Learning General Principle Near Optimality Constrained Markov Decision Process Reinforcement Learning Problem Bellman Operator

February 2, 2023

Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Function
Ruijie Zheng, Xiyao Wang, Huazhe Xu, Furong Huang
Model Based Reinforcement Learning Model Ensemble Single Model Bellman Operator Lipschitz Regularization

January 31, 2023

An Efficient Solution to s-Rectangular Robust Markov Decision Processes
Navdeep Kumar, Kfir Levy, Kaixin Wang, Shie Mannor
Optimal Policy Value Iteration Efficient Approach Bellman Operator Rectangular Robust Markov Decision Process

November 21, 2022

Simultaneously Updating All Persistence Values in Reinforcement Learning
Luca Sabbioni, Luca Al Daire, Lorenzo Bisi, Alberto Maria Metelli, Marcello Restelli
Reinforcement Learning Q Learning Action Repetition Bellman Operator Persistence Point Importance

July 15, 2022

Set-based value operators for non-stationary Markovian environments
Sarah H. Q. Li, Assalé Adjé, Pierre-Loïc Garoche, Behçet Açıkmeşe
Markov Decision Process Non Stationary Environment Bellman Operator Set Based

March 20, 2022

Smoothing Advantage Learning
Yaozhong Gan, Zhe Zhang, Xiaoyang Tan
Sub Optimal Value Based Reinforcement Learning Bellman Operator Advantage Learning

January 29, 2022

Bellman Meets Hawkes: Model-Based Reinforcement Learning via Temporal Point Processes
Chao Qu, Xiaoyu Tan, Siqiao Xue, Xiaoming Shi, James Zhang, Hongyuan Mei
Reinforcement Learning Model Based Reinforcement Learning Sequential Decision Making Problem Temporal Point Process Hawkes Process Bellman Operator Asynchronous Stochastic

January 28, 2022

Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error
Scott Fujimoto, David Meger, Doina Precup, Ofir Nachum, Shixiang Shane Gu
Value Function State Action Pair Bellman Equation Bellman Operator Bellman Error Value Prediction

December 9, 2021

Reinforcement Learning with Almost Sure Constraints
Agustin Castellano, Hancheng Min, Juan Bazerque, Enrique Mallada
Reinforcement Learning Markov Decision Process Constrained Markov Decision Process Near Optimal Policy Bellman Operator Stationary Policy

Bellman Operator

Papers

To bootstrap or to rollout? An optimal and adaptive interpolation

Robust Q-Learning under Corrupted Rewards

Projection by Convolution: Optimal Sample Complexity for Reinforcement Learning in Continuous-Space MDPs

Markowitz Meets Bellman: Knowledge-distilled Reinforcement Learning for Portfolio Management

Iterated $Q$-Network: Beyond One-Step Bellman Updates in Deep Reinforcement Learning

Learning safety critics via a non-contractive binary bellman operator

Parameterized Projected Bellman Operator

MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator

Multi-Bellman operator for convergence of $Q$-learning with linear function approximation

On Bellman's principle of optimality and Reinforcement learning for safety-constrained Markov decision process

Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Function

An Efficient Solution to s-Rectangular Robust Markov Decision Processes

Simultaneously Updating All Persistence Values in Reinforcement Learning

Set-based value operators for non-stationary Markovian environments

Smoothing Advantage Learning

Bellman Meets Hawkes: Model-Based Reinforcement Learning via Temporal Point Processes

Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error

Reinforcement Learning with Almost Sure Constraints