Optimal Value Function

Optimal value function research in reinforcement learning centers on efficiently estimating the optimal value function, which dictates the best actions an agent should take to maximize its cumulative reward. Current research focuses on improving the accuracy and efficiency of value function approximation, addressing issues like overestimation bias using novel Bellman operators and exploring the use of tailored neural networks and other function approximators to better capture the structure of the optimal value function in various settings, including those with costly actions or budgetary constraints. These advancements are crucial for improving the sample efficiency and scalability of reinforcement learning algorithms, enabling their application to more complex and high-dimensional problems in diverse fields like robotics, finance, and operations research.

Papers

June 26, 2024

Boosting Soft Q-Learning by Bounding
Jacob Adamczyk, Volodymyr Makarenko, Stas Tiomkin, Rahul V. Kulkarni
Zero Shot Value Function General Bound Optimal Value Function Soft Q Learning

June 5, 2024

UDQL: Bridging The Gap between MSE Loss and The Optimal Value Function in Offline Reinforcement Learning
Yu Zhang, Rui Yu, Zhipeng Yao, Wenyuan Zhang, Jun Wang, Liming Zhang
Offline Reinforcement Learning Offline Reinforcement Learning Algorithm Mean Squared Error Optimal Value Function Operator Prediction

May 29, 2024

$\epsilon$-Optimally Solving Zero-Sum POSGs
Erwan Escudie, Matthia Sabatelli, Jilles Dibangoye
Value Iteration Observable Stochastic Game Optimal Value Function State Occupancy Matching Zero Sum

April 23, 2024

The Power of Resets in Online Reinforcement Learning
Zakaria Mhammedi, Dylan J. Foster, Alexander Rakhlin
Reinforcement Learning Real Power Online Reinforcement Learning Episodic Markov Decision Process Optimal Value Function Global Reset Feature

February 13, 2024

Model approximation in MDPs with unbounded per-step cost
Berk Bozkurt, Aditya Mahajan, Ashutosh Nayyar, Yi Ouyang
Optimal Policy Optimal Value Function Approximate Model Per Iteration

November 15, 2023

On Convex Optimal Value Functions For POSGs
Rafael F. Cunha, Jacopo Castellini, Johan Peralez, Jilles S. Dibangoye
Reinforcement Learning Markov Game Multi Agent Planning Observable Stochastic Game Optimal Value Function

June 30, 2023

Resetting the Optimizer in Deep RL: An Empirical Study
Kavosh Asadi, Rasool Fakoor, Shoham Sabach
Deep Reinforcement Learning Stochastic Gradient Descent Empirical Study Deep RL Superior Optimizer Optimization Landscape Optimal Value Function

May 24, 2023

Neural Lyapunov and Optimal Control
Daniel Layeghi, Steve Tonneau, Michael Mistry
Reinforcement Learning Proximal Policy Optimization Optimal Control Reinforcement Learning Algorithm Lyapunov Function Neural Lyapunov Optimal Value Function

April 3, 2023

Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning
Tongzhou Wang, Antonio Torralba, Phillip Isola, Amy Zhang
Optimal Value Function Goal Oriented Reinforcement Learning Quasimetric Learning

March 5, 2023

Bounding the Optimal Value Function in Compositional Reinforcement Learning
Jacob Adamczyk, Volodymyr Makarenko, Argenis Arriojas, Stas Tiomkin, Rahul V. Kulkarni
Reinforcement Learning Reward Function General Bound Entropy Regularized Optimal Value Function Compositional Reinforcement Learning Function Composition

February 25, 2023

Exponential Hardness of Reinforcement Learning with Linear Function Approximation
Daniel Kane, Sihan Liu, Shachar Lovett, Gaurav Mahajan, Csaba Szepesvári, Gellért Weisz
Reinforcement Learning Linear Function Approximation Exponential Mechanism Optimal Value Function

February 19, 2023

Leveraging Prior Knowledge in Reinforcement Learning via Double-Sided Bounds on the Value Function
Jacob Adamczyk, Stas Tiomkin, Rahul Kulkarni
Reinforcement Learning Zero Shot Transfer Learning Value Function Prior Knowledge General Bound Optimal Value Function

May 31, 2022

Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraints
David Mguni, Aivar Sootla, Juliusz Ziomek, Oliver Slumbers, Zipeng Dai, Kun Shao, Jun Wang
Reinforcement Learning Reinforcement Learning Benchmark Timing Analysis Selective Engagement Budget Constraint Optimal Value Function Portfolio Optimization Problem Action Cost

March 14, 2022

Optimal Admission Control for Multiclass Queues with Time-Varying Arrival Rates via State Abstraction
Marc Rigter, Danial Dervovic, Parisa Hassanzadeh, Jason Long, Parisa Zehtabi, Daniele Magazzeni
Markov Decision Process Discrete Time State Abstraction Optimal Value Function Admission Control

February 11, 2022

Computational-Statistical Gaps in Reinforcement Learning
Daniel Kane, Sihan Liu, Shachar Lovett, Gaurav Mahajan
Reinforcement Learning Linear Function Approximation Efficient Reinforcement Learning Optimal Value Function Computational Gap

December 7, 2021

Tailored neural networks for learning optimal value functions in MPC
Dieter Teichrib, Moritz Schulze Darup
Neural Network Optimal Control Model Predictive Control Optimal Value Function

November 4, 2020

Optimal Control-Based Baseline for Guided Exploration in Policy Gradient Methods
Xubo Lyu, Site Li, Seth Siriya, Ye Pu, Mo Chen
Deep Reinforcement Learning NCD Method Policy Gradient Guided Exploration Optimal Value Function Optimal Baseline