Distributional Bellman

Distributional Bellman methods in reinforcement learning aim to learn not just the expected reward, but the entire distribution of possible future rewards, enabling more nuanced decision-making, especially in risk-sensitive scenarios. Current research focuses on developing theoretically sound algorithms, including model-based approaches and those utilizing mean embeddings or categorical representations of distributions, addressing challenges like high-dimensional rewards and biased exploration stemming from optimism-based methods. These advancements offer improved performance in various applications, particularly where understanding the uncertainty of future outcomes is crucial, such as robotics and financial modeling.

Papers

August 14, 2024

Off-Policy Reinforcement Learning with High Dimensional Reward
Dong Neuck Lee, Michael R. Kosorok
Policy Reinforcement Learning Distributional Reinforcement Learning Distributional Bellman

February 12, 2024

Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model
Mark Rowland, Li Kevin Wenliang, Rémi Munos, Clare Lyle, Yunhao Tang, Will Dabney
Generative Modeling Distributional Reinforcement Learning Distributional Bellman

December 9, 2023

Distributional Bellman Operators over Mean Embeddings
Li Kevin Wenliang, Grégoire Delétang, Matthew Aitchison, Marcus Hutter, Anian Ruoss, Arthur Gretton, Mark Rowland
Deep Reinforcement Learning Distributional Reinforcement Learning Temporal Difference Learning Learning Environment Distributional Bellman

October 25, 2023

Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion
Taehyun Cho, Seungyub Han, Heesoo Lee, Kyungjae Lee, Jungwoo Lee
Common Pitfall Distributional Reinforcement Learning Optimistic Exploration Distributional Bellman Exploration Bias

July 2, 2023

Is Risk-Sensitive Reinforcement Learning Properly Resolved?
Ruiwen Zhou, Minghuan Liu, Kan Ren, Xufang Luo, Weinan Zhang, Dongsheng Li
Q Learning Risk Sensitive Distributional Reinforcement Learning Risk Sensitive Reinforcement Learning Distributional Bellman

October 26, 2022

Categorical SDEs with Simplex Diffusion
Pierre H. Richemond, Sander Dieleman, Arnaud Doucet
Diffusion Model Hawkes Process Diffusion SDE Distributional Bellman

May 24, 2022

Distributional Hamilton-Jacobi-Bellman Equations for Continuous-Time Reinforcement Learning
Harley Wiltzer, David Meger, Marc G. Bellemare
Reinforcement Learning Optimal Policy Risk Aware Control Distributional Bellman Distributional Reduction

January 31, 2022

On solutions of the distributional Bellman equation
Julian Gerstenberg, Ralph Neininger, Denis Spiegel
Solution Path Distributional Reinforcement Learning Distributional Bellman