Distributional Reinforcement Learning

Distributional reinforcement learning (DRL) aims to learn the entire distribution of future rewards, rather than just the expected value, offering a more nuanced understanding of uncertainty in decision-making. Current research focuses on developing efficient algorithms, often employing quantile regression, generative models (like energy-based models and diffusion models), and various distributional Bellman operators, to accurately estimate and utilize these reward distributions. This approach enhances robustness and allows for risk-sensitive decision-making, finding applications in diverse fields such as finance, robotics, and wireless network management, where handling uncertainty is crucial for optimal and safe performance.

Papers

October 14, 2024

Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning
Harley Wiltzer, Marc G. Bellemare, David Meger, Patrick Shafto, Yash Jhaveri
Reinforcement Learning Distributional Reinforcement Learning Advantage Feedback Traditional Reinforcement Learning Value Based Reinforcement Learning Action Gap

October 2, 2024

Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space
Yangming Li, Chieh-Hsin Lai, Carola-Bibiane Schönlieb, Yuki Mitsufuji, Stefano Ermon
Generative Modeling Score Based Generative Distributional Reinforcement Learning Non Linear Operator

September 25, 2024

Offline and Distributional Reinforcement Learning for Radio Resource Management
Eslam Eldeeb, Hirley Alves
Reinforcement Learning Distributional Reinforcement Learning Action Free Offline Stochastic Environment Radio Resource Management Future Network

August 31, 2024

Foundations of Multivariate Distributional Reinforcement Learning
Harley Wiltzer, Jesse Farebrother, Arthur Gretton, Mark Rowland
Reinforcement Learning Flawed Foundation Distributional Reinforcement Learning Reward Signal

August 22, 2024

EX-DRL: Hedging Against Heavy Losses with EXtreme Distributional Reinforcement Learning
Parvin Malekzadeh, Zissis Poulos, Jacky Chen, Zeyu Wang, Konstantinos N. Plataniotis
Quantile Regression Distributional Reinforcement Learning Hedging Strategy Quantile Temporal Difference Large Loss

August 14, 2024

Off-Policy Reinforcement Learning with High Dimensional Reward
Dong Neuck Lee, Michael R. Kosorok
Policy Reinforcement Learning Distributional Reinforcement Learning Distributional Bellman

July 31, 2024

Tractable and Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation
Taehyun Cho, Seungyub Han, Kyungjae Lee, Seokhun Ju, Dohyeong Kim, Jungwoo Lee
Value Function Distributional Reinforcement Learning Regret Analysis Episodic Markov Decision Process Approximate Dynamic Programming

July 19, 2024

On Policy Evaluation Algorithms in Distributional Reinforcement Learning
Julian Gerstenberg, Ralph Neininger, Denis Spiegel
Markov Decision Process Policy Evaluation Distributional Reinforcement Learning Reward Distribution

July 18, 2024

PG-Rainbow: Using Distributional Reinforcement Learning in Policy Gradient Methods
WooJae Jeon, KangJun Lee, Jeewoo Lee
Policy Gradient Proximal Policy Optimization Distributional Reinforcement Learning Reinforcement Learning Task Policy Gradient Algorithm

May 23, 2024

Policy Gradient Methods for Risk-Sensitive Distributional Reinforcement Learning with Provable Convergence
Minheng Xiao, Xian Yu, Lei Ying
Policy Gradient Distributional Reinforcement Learning Risk Sensitive Reinforcement Learning Provable Convergence Distributional Policy Gradient

May 4, 2024

CTD4 -- A Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Critics
David Valencia, Henry Williams, Trevor Gee, Bruce A MacDonald, Minas Liarokapis
Reinforcement Learning Kalman Filter Distributional Reinforcement Learning Multiple Critic Distributional Policy Gradient

March 16, 2024

Neural-Kernel Conditional Mean Embeddings
Eiki Shimizu, Kenji Fukumizu, Dino Sejdinovic
Distributional Reinforcement Learning Hyper Parameter Kernel Based Conditional Density Kernel Mean Embeddings Mean Embeddings

March 9, 2024

Statistical Efficiency of Distributional Temporal Difference Learning
Yang Peng, Liangyu Zhang, Zhihua Zhang
Policy Evaluation Distributional Reinforcement Learning Hilbert Space Quantile Temporal Difference

February 28, 2024

Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation
Yu Chen, Xiangcheng Zhang, Siwei Wang, Longbo Huang
Reinforcement Learning Distributional Reinforcement Learning Risk Sensitive Reinforcement Learning Risk Measure Risk Averse Reinforcement Learning

February 19, 2024

Decentralized Multi-Robot Navigation for Autonomous Surface Vehicles with Distributional Reinforcement Learning
Xi Lin, Yewei Huang, Fanfei Chen, Brendan Englot
Collision Avoidance Distributional Reinforcement Learning Autonomous Surface Vehicle Multi Robot Navigation

February 13, 2024

February 12, 2024

Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model
Mark Rowland, Li Kevin Wenliang, Rémi Munos, Clare Lyle, Yunhao Tang, Will Dabney
Generative Modeling Distributional Reinforcement Learning Distributional Bellman

February 11, 2024

Distributional Reinforcement Learning

Papers

Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learning

Bellman Diffusion: Generative Modeling as Learning a Linear Operator in the Distribution Space

Offline and Distributional Reinforcement Learning for Radio Resource Management

Foundations of Multivariate Distributional Reinforcement Learning

EX-DRL: Hedging Against Heavy Losses with EXtreme Distributional Reinforcement Learning

Off-Policy Reinforcement Learning with High Dimensional Reward

Tractable and Provably Efficient Distributional Reinforcement Learning with General Value Function Approximation

On Policy Evaluation Algorithms in Distributional Reinforcement Learning

PG-Rainbow: Using Distributional Reinforcement Learning in Policy Gradient Methods

Policy Gradient Methods for Risk-Sensitive Distributional Reinforcement Learning with Provable Convergence

CTD4 -- A Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Critics

Neural-Kernel Conditional Mean Embeddings

Statistical Efficiency of Distributional Temporal Difference Learning

Provable Risk-Sensitive Distributional Reinforcement Learning with General Function Approximation

Decentralized Multi-Robot Navigation for Autonomous Surface Vehicles with Distributional Reinforcement Learning

A Distributional Analogue to the Successor Representation

Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learning for Digital Twins

Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model

More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning

Echoes of Socratic Doubt: Embracing Uncertainty in Calibrated Evidential Reinforcement Learning