Distributional Policy Gradient

Distributional policy gradient methods in reinforcement learning aim to learn not just the expected value of rewards, but the entire distribution of possible outcomes, enabling more robust and risk-sensitive decision-making. Current research focuses on developing efficient algorithms, such as those employing continuous distributions and Kalman filtering for improved stability and sample efficiency, and applying these methods to diverse domains including robotics, natural language processing, and multi-agent systems. This approach offers significant advantages over traditional methods by providing a richer understanding of uncertainty and allowing for the incorporation of risk preferences, leading to improved performance and reliability in complex environments.

Papers

May 23, 2024

Policy Gradient Methods for Risk-Sensitive Distributional Reinforcement Learning with Provable Convergence
Minheng Xiao, Xian Yu, Lei Ying
Policy Gradient Distributional Reinforcement Learning Risk Sensitive Reinforcement Learning Provable Convergence Distributional Policy Gradient

May 4, 2024

CTD4 -- A Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Critics
David Valencia, Henry Williams, Trevor Gee, Bruce A MacDonald, Minas Liarokapis
Reinforcement Learning Kalman Filter Distributional Reinforcement Learning Multiple Critic Distributional Policy Gradient

February 4, 2024

BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback
Gaurav Pandey, Yatin Nandwani, Tahira Naseem, Mayank Mishra, Guangxuan Xu, Dinesh Raghu, Sachindra Joshi, Asim Munawar, Ramón Fernandez Astudillo
Reinforcement Learning Language Generation Human Feedback Bayesian Perspective Brain Function Amortized Inference Distributional Policy Gradient Amortised Inference Reward Conditioned

January 11, 2024

The Distributional Reward Critic Architecture for Perturbed-Reward Reinforcement Learning
Xi Chen, Zhihui Zhu, Andrew Perrault
Reinforcement Learning Distributional Policy Gradient Perturbation Artifact Targeted Perturbation

September 1, 2023

Parallel Distributional Prioritized Deep Reinforcement Learning for Unmanned Aerial Vehicles
Alisson Henrique Kolling, Victor Augusto Kich, Junior Costa de Jesus, Andressa Cavalcante da Silva, Ricardo Bedin Grando, Paulo Lilles Jorge Drews-Jr, Daniel F. T. Gamarra
Deep Reinforcement Learning Unmanned Aerial Vehicle Soft Actor Critic Autonomous Aerial Navigation Distributional Policy Gradient

August 3, 2023

Bag of Policies for Distributional Deep Exploration
Asen Nachkov, Luchen Li, Giulia Luise, Filippo Valdettaro, Aldo Faisal
Reinforcement Learning Efficient Exploration Distributional Reinforcement Learning Prior Policy Bag Prototype Distributional Policy Gradient

February 16, 2023

Aligning Language Models with Preferences through f-divergence Minimization
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, Marc Dymetman
Preference Feedback Target Distribution Divergence Regularization Distributional Policy Gradient Divergence Optimization Generative Control

January 18, 2023

Multi-compartment Neuron and Population Encoding improved Spiking Neural Network for Deep Distributional Reinforcement Learning
Yinqian Sun, Yi Zeng, Feifei Zhao, Zhuoya Zhao
Spiking Neural Network Distributional Policy Gradient Compartmental Neuron Population Encoding

November 22, 2022

Decision-making with Speculative Opponent Models
Jing Sun, Shuo Chen, Cong Zhang, Yining Ma, Jie Zhang
Decision Making Multi Agent Challenge Opponent Modeling Distributional Policy Gradient

August 19, 2022

A Risk-Sensitive Approach to Policy Optimization
Jared Markowitz, Ryan W. Gardner, Ashley Llorens, Raman Arora, I-Jeng Wang
Deep Reinforcement Learning Policy Gradient Policy OpTimization Risk Sensitive Distributional Policy Gradient

December 1, 2021

Controlling Conditional Language Models without Catastrophic Forgetting
Tomasz Korbak, Hady Elsahar, German Kruszewski, Marc Dymetman
Generative Model Catastrophic Forgetting Energy Based Model Pre Trained Generative Model Training Method Conditional Language Model Distributional Policy Gradient

Distributional Policy Gradient

Papers

Policy Gradient Methods for Risk-Sensitive Distributional Reinforcement Learning with Provable Convergence

CTD4 -- A Deep Continuous Distributional Actor-Critic Agent with a Kalman Fusion of Multiple Critics

BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback

The Distributional Reward Critic Architecture for Perturbed-Reward Reinforcement Learning

Parallel Distributional Prioritized Deep Reinforcement Learning for Unmanned Aerial Vehicles

Bag of Policies for Distributional Deep Exploration

Aligning Language Models with Preferences through f-divergence Minimization

Multi-compartment Neuron and Population Encoding improved Spiking Neural Network for Deep Distributional Reinforcement Learning

Decision-making with Speculative Opponent Models

A Risk-Sensitive Approach to Policy Optimization

Controlling Conditional Language Models without Catastrophic Forgetting