Distributional Reinforcement Learning

Distributional reinforcement learning (DRL) aims to learn the entire distribution of future rewards, rather than just the expected value, offering a more nuanced understanding of uncertainty in decision-making. Current research focuses on developing efficient algorithms, often employing quantile regression, generative models (like energy-based models and diffusion models), and various distributional Bellman operators, to accurately estimate and utilize these reward distributions. This approach enhances robustness and allows for risk-sensitive decision-making, finding applications in diverse fields such as finance, robotics, and wireless network management, where handling uncertainty is crucial for optimal and safe performance.

Papers

February 12, 2024

Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model
Mark Rowland, Li Kevin Wenliang, Rémi Munos, Clare Lyle, Yunhao Tang, Will Dabney
Generative Modeling Distributional Reinforcement Learning Distributional Bellman

February 11, 2024

February 8, 2024

Off-policy Distributional Q($\lambda$): Distributional RL without Importance Sampling
Yunhao Tang, Mark Rowland, Rémi Munos, Bernardo Ávila Pires, Will Dabney
Policy Evaluation Importance Sampling Distributional Reinforcement Learning Distributional Reduction

February 2, 2024

Distributional Off-policy Evaluation with Bellman Residual Minimization
Sungee Hong, Zhengling Qi, Raymond K. W. Wong
Policy Evaluation Wasserstein Distance Distributional Reinforcement Learning Bellman Residual

January 4, 2024

A Robust Quantile Huber Loss With Interpretable Parameter Adjustment In Distributional Reinforcement Learning
Parvin Malekzadeh, Konstantinos N. Plataniotis, Zissis Poulos, Zeyu Wang
Distributional Reinforcement Learning Parameter Update Huber Loss Huber Distribution

December 23, 2023

Distributional Reinforcement Learning-based Energy Arbitrage Strategies in Imbalance Settlement Mechanism
Seyed Soroush Karimi Madahi, Bert Claessens, Chris Develder
Distributional Reinforcement Learning Data Imbalance Battery Management System Energy Arbitrage

December 12, 2023

Noise Distribution Decomposition based Multi-Agent Distributional Reinforcement Learning
Wei Geng, Baidi Xiao, Rongpeng Li, Ning Wei, Dong Wang, Zhifeng Zhao
Reinforcement Learning Multi Agent Reinforcement Learning Distributional Reinforcement Learning Noise Decomposition

December 9, 2023

Distributional Bellman Operators over Mean Embeddings
Li Kevin Wenliang, Grégoire Delétang, Matthew Aitchison, Marcus Hutter, Anian Ruoss, Arthur Gretton, Mark Rowland
Deep Reinforcement Learning Distributional Reinforcement Learning Temporal Difference Learning Learning Environment Distributional Bellman

November 13, 2023

An introduction to reinforcement learning for neuroscience
Kristopher T. Jensen
Reinforcement Learning Gentle Introduction Distributional Reinforcement Learning Neuroscience Research System Neuroscience

October 31, 2023

Beyond Average Return in Markov Decision Processes
Alexandre Marthe, Aurélien Garivier, Claire Vernade
Markov Decision Process Dynamic Programming Distributional Reinforcement Learning Return Trajectory

October 25, 2023

Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion
Taehyun Cho, Seungyub Han, Heesoo Lee, Kyungjae Lee, Jungwoo Lee
Common Pitfall Distributional Reinforcement Learning Optimistic Exploration Distributional Bellman Exploration Bias

October 8, 2023

Distributional Reinforcement Learning with Online Risk-awareness Adaption
Yupeng Wu, Wenjie Huang
Reinforcement Learning Distributional Reinforcement Learning Adaptive Risk

September 29, 2023

Estimation and Inference in Distributional Reinforcement Learning
Liangyu Zhang, Yang Peng, Jiadong Liang, Wenhao Yang, Zhihua Zhang
Scientific Inference Estimation Task Policy Evaluation Distributional Reinforcement Learning Sample Efficiency Statistical Functionals

September 25, 2023

Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learning
Lukas Schneider, Jonas Frey, Takahiro Miki, Marco Hutter
Proximal Policy Optimization Distributional Reinforcement Learning Quadrupedal Locomotion

August 12, 2023

Value-Distributional Model-Based Reinforcement Learning
Carlos E. Luis, Alessandro G. Bottero, Julia Vinogradska, Felix Berkenkamp, Jan Peters
Markov Decision Process Policy OpTimization Quantile Regression Distributional Reinforcement Learning

August 3, 2023

Bag of Policies for Distributional Deep Exploration
Asen Nachkov, Luchen Li, Giulia Luise, Filippo Valdettaro, Aldo Faisal
Reinforcement Learning Efficient Exploration Distributional Reinforcement Learning Prior Policy Bag Prototype Distributional Policy Gradient

July 30, 2023

July 13, 2023

Cramer Type Distances for Learning Gaussian Mixture Models by Gradient Descent
Ruichong Zhang
Gradient Descent Gaussian Mixture Model Deep Q Network Distributional Reinforcement Learning Cramer Wold Distance

Distributional Reinforcement Learning

Papers

Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model

More Benefits of Being Distributional: Second-Order Bounds for Reinforcement Learning

Echoes of Socratic Doubt: Embracing Uncertainty in Calibrated Evidential Reinforcement Learning

Off-policy Distributional Q($\lambda$): Distributional RL without Importance Sampling

Distributional Off-policy Evaluation with Bellman Residual Minimization

A Robust Quantile Huber Loss With Interpretable Parameter Adjustment In Distributional Reinforcement Learning

Distributional Reinforcement Learning-based Energy Arbitrage Strategies in Imbalance Settlement Mechanism

Noise Distribution Decomposition based Multi-Agent Distributional Reinforcement Learning

Distributional Bellman Operators over Mean Embeddings

An introduction to reinforcement learning for neuroscience

Beyond Average Return in Markov Decision Processes

Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion

Distributional Reinforcement Learning with Online Risk-awareness Adaption

Estimation and Inference in Distributional Reinforcement Learning

Learning Risk-Aware Quadrupedal Locomotion using Distributional Reinforcement Learning

Value-Distributional Model-Based Reinforcement Learning

Bag of Policies for Distributional Deep Exploration

Robust Unmanned Surface Vehicle Navigation with Distributional Reinforcement Learning

Variance Control for Distributional Reinforcement Learning

Cramer Type Distances for Learning Gaussian Mixture Models by Gradient Descent