Distributional Reinforcement Learning

Distributional reinforcement learning (DRL) aims to learn the entire distribution of future rewards, rather than just the expected value, offering a more nuanced understanding of uncertainty in decision-making. Current research focuses on developing efficient algorithms, often employing quantile regression, generative models (like energy-based models and diffusion models), and various distributional Bellman operators, to accurately estimate and utilize these reward distributions. This approach enhances robustness and allows for risk-sensitive decision-making, finding applications in diverse fields such as finance, robotics, and wireless network management, where handling uncertainty is crucial for optimal and safe performance.

Papers

June 11, 2023

PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithm
Wensong Bai, Chao Zhang, Yichao Fu, Peilin Zhao, Hui Qian, Bin Dai
Distributional Reinforcement Learning Control Benchmark Push Forward

June 4, 2023

A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learning
Wei-Fang Sun, Cheng-Kuang Lee, Simon See, Chun-Yi Lee
Multi Agent Reinforcement Learning Unified Framework Distributional Reinforcement Learning Multi Agent Challenge Stochastic Reward Value Function Factorization

June 2, 2023

Improving the generalizability and robustness of large-scale traffic signal control
Tianyu Shi, Francois-Xavier Devailly, Denis Larocque, Laurent Charlin
Native Robustness Multi Agent Reinforcement Learning Stronger Generalizability Distributional Reinforcement Learning Traffic Signal Control Traffic Pattern

May 28, 2023

The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation
Mark Rowland, Yunhao Tang, Clare Lyle, Rémi Munos, Marc G. Bellemare, Will Dabney
Reinforcement Learning Distributional Reinforcement Learning Provable Benefit TD Learning Value Estimation Quantile Temporal Difference

May 26, 2023

Distributional Reinforcement Learning with Dual Expectile-Quantile Regression
Sami Jullien, Romain Deffayet, Jean-Michel Renders, Paul Groth, Maarten de Rijke
Distributional Reinforcement Learning Distribution Free

May 25, 2023

The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learning
Kaiwen Wang, Kevin Zhou, Runzhe Wu, Nathan Kallus, Wen Sun
Reinforcement Learning Regret Bound Complementary Benefit Distributional Reinforcement Learning Small Loss

April 27, 2023

One-Step Distributional Reinforcement Learning
Mastane Achab, Reda Alami, Yasser Abdelaziz Dahou Djilali, Kirill Fedyanin, Eric Moulines
Reinforcement Learning Distributional Reinforcement Learning Step to Step Dynamic

April 19, 2023

Robust Route Planning with Distributional Reinforcement Learning in a Stochastic Road Network Environment
Xi Lin, Paul Szenher, John D. Martin, Brendan Englot
Deep Reinforcement Learning Path Planning Distributional Reinforcement Learning Stochastic Environment Optimal Planning

March 23, 2023

Policy Evaluation in Distributional LQR
Zifan Wang, Yulong Gao, Siyi Wang, Michael M. Zavlanos, Alessandro Abate, Karl H. Johansson
Policy Gradient Policy Evaluation Risk Sensitive Distributional Reinforcement Learning

March 4, 2023

Wasserstein Actor-Critic: Directed Exploration via Optimism for Continuous-Actions Control
Amarildo Likmeta, Matteo Sacco, Alberto Maria Metelli, Marcello Restelli
Reinforcement Learning Q Learning Epistemic Uncertainty Distributional Reinforcement Learning Wasserstein Dictionary Directed Exploration Continuous Action

March 3, 2023

Toward Risk-based Optimistic Exploration for Cooperative Multi-Agent Reinforcement Learning
Jihwan Oh, Joonkee Kim, Minchan Jeong, Se-Young Yun
Multi Agent Reinforcement Learning Cooperative Multi Agent Reinforcement Learning Distributional Reinforcement Learning Collaborative Exploration Optimistic Exploration

February 3, 2023

Distributional constrained reinforcement learning for supply chain optimization
Jaime Sabal Bermúdez, Antonio del Rio Chanona, Calvin Tsay
Policy Gradient Distributional Reinforcement Learning Reinforcement Learning Research Constrained Policy Optimization Supply Chain Optimization

January 11, 2023

An Analysis of Quantile Temporal-Difference Learning
Mark Rowland, Rémi Munos, Mohammad Gheshlaghi Azar, Yunhao Tang, Georg Ostrovski, Anna Harutyunyan, Karl Tuyls, Marc G. Bellemare, Will Dabney
Reinforcement Learning General Analysis Stochastic Approximation Distributional Reinforcement Learning TD Learning Quantile Temporal Difference

December 30, 2022

Risk-Sensitive Policy with Distributional Reinforcement Learning
Thibaut Théate, Damien Ernst
Reinforcement Learning Distributional Reinforcement Learning Sequential Decision Making Policy Risk Aware Policy

December 29, 2022

Invariance to Quantile Selection in Distributional Continuous Control
Felix Grün, Muhammad Saif-ur-Rehman, Tobias Glasmachers, Ioannis Iossifidis
Action Space Alpha Invariance Actor Critic Algorithm Quantile Regression Distributional Reinforcement Learning Distributional Reduction

November 28, 2022

Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probability
Whiyoung Jung, Myungsik Cho, Jongeui Park, Youngchul Sung
Distributional Reinforcement Learning Constrained Reinforcement Learning Outage Probability

October 25, 2022

Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Bounds
Hao Liang, Zhi-Quan Luo
Regret Bound Distributional Reinforcement Learning Regret Analysis Risk Sensitive Reinforcement Learning

September 29, 2022

How Does Return Distribution in Distributional Reinforcement Learning Help Optimization?
Ke Sun, Bei Jiang, Linglong Kong
Distributional Reinforcement Learning

August 28, 2022

Normality-Guided Distributional Reinforcement Learning for Continuous Control
Ju-Seung Byun, Andrew Perrault
Distributional Reinforcement Learning Continuous Control Learning Probability Distribution

July 15, 2022

QuaDUE-CCM: Interpretable Distributional Reinforcement Learning using Uncertain Contraction Metrics for Precise Quadrotor Trajectory Tracking
Yanran Wang, James O'Keeffe, Qiuchen Qian, David Boyle
Distributional Reinforcement Learning Contraction Theory Td MPC Stable Control Quadrotor Trajectory

Distributional Reinforcement Learning

Papers

PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithm

A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learning

Improving the generalizability and robustness of large-scale traffic signal control

The Statistical Benefits of Quantile Temporal-Difference Learning for Value Estimation

Distributional Reinforcement Learning with Dual Expectile-Quantile Regression

The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learning

One-Step Distributional Reinforcement Learning

Robust Route Planning with Distributional Reinforcement Learning in a Stochastic Road Network Environment

Policy Evaluation in Distributional LQR

Wasserstein Actor-Critic: Directed Exploration via Optimism for Continuous-Actions Control

Toward Risk-based Optimistic Exploration for Cooperative Multi-Agent Reinforcement Learning

Distributional constrained reinforcement learning for supply chain optimization

An Analysis of Quantile Temporal-Difference Learning

Risk-Sensitive Policy with Distributional Reinforcement Learning

Invariance to Quantile Selection in Distributional Continuous Control

Quantile Constrained Reinforcement Learning: A Reinforcement Learning Framework Constraining Outage Probability

Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Bounds

How Does Return Distribution in Distributional Reinforcement Learning Help Optimization?

Normality-Guided Distributional Reinforcement Learning for Continuous Control

QuaDUE-CCM: Interpretable Distributional Reinforcement Learning using Uncertain Contraction Metrics for Precise Quadrotor Trajectory Tracking