Maximum Entropy Reinforcement Learning

Maximum entropy reinforcement learning (MaxEnt RL) aims to find optimal policies that balance maximizing expected rewards with maximizing policy entropy, promoting exploration and robustness. Current research focuses on improving the efficiency and expressiveness of MaxEnt RL algorithms, particularly in continuous action spaces, through methods like diffusion models, normalizing flows, and novel actor-critic architectures such as Soft Actor-Critic (SAC) and its variants. These advancements address limitations in existing approaches, leading to improved performance in diverse applications, including robotics, multi-agent systems, and congestion control in networks. The resulting more stable and efficient algorithms are significant for advancing the field and enabling the deployment of RL in complex real-world scenarios.

Papers

November 1, 2024

Hierarchical Preference Optimization: Learning to achieve goals via feasible subgoals prediction
Utsav Singh, Souradip Chakraborty, Wesley A. Suttle, Brian M. Sadler, Anit Kumar Sahu, Mubarak Shah, Vinay P. Namboodiri, Amrit Singh Bedi
Robot Navigation Direct Preference Optimization Hierarchical Reinforcement Learning Pseudo Goal Maximum Entropy Reinforcement Learning Goal Model

October 2, 2024

Sampling from Energy-based Policies using Diffusion
Vineet Jain, Tara Akhound-Sadegh, Siamak Ravanbakhsh
Reinforcement Learning Diffusion Explainer Diffusion Sampling Maximum Entropy Reinforcement Learning

September 13, 2024

Average-Reward Maximum Entropy Reinforcement Learning for Underactuated Double Pendulum Tasks
Jean Seong Bjorn Choe, Bumkyu Choi, Jong-kook Kim
Underactuated System Entropy Regularized Maximum Entropy Reinforcement Learning AI City Challenge

July 25, 2024

Maximum Entropy On-Policy Actor-Critic via Entropy Advantage Estimation
Jean Seong Bjorn Choe, Jong-Kook Kim
Proximal Policy Optimization Actor Critic Maximum Entropy Reinforcement Learning Trust Region Policy Optimization Efficient Policy Learning

June 20, 2024

Soft-QMIX: Integrating Maximum Entropy For Monotonic Value Function Factorization
Wentse Chen, Shiyu Huang, Jeff Schneider
Multi Agent Reinforcement Learning Maximum Entropy Deterministic Policy Maximum Entropy Reinforcement Learning Value Function Factorization Soft QMIX

May 22, 2024

Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow
Chen-Hao Chao, Chien Feng, Wei-Fang Sun, Cheng-Kuang Lee, Simon See, Chun-Yi Lee
Reinforcement Learning Normalizing Flow Action Space Maximum Entropy Reinforcement Learning Actor Critic Framework MaxEnt RL

May 2, 2024

S$^2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic
Safa Messaoud, Billel Mokeddem, Zhenghai Xue, Linsey Pang, Bo An, Haipeng Chen, Sanjay Chawla
Soft Actor Critic Energy Based Stein Variational Gradient Descent Maximum Entropy Reinforcement Learning Expressive Policy MaxEnt RL

March 4, 2024

Tsallis Entropy Regularization for Linearly Solvable MDP and Linear Quadratic Regulator
Yota Hashizume, Koshi Oishi, Kenji Kashima
Regularization Model Linear Quadratic Regulator Shannon Entropy Linear MDPs Maximum Entropy Reinforcement Learning Tsallis Entropy

February 15, 2024

Discrete Probabilistic Inference as Control in Multi-path Environments
Tristan Deleu, Padideh Nouri, Nikolay Malkin, Doina Precup, Yoshua Bengio
Markov Decision Process External Control Inference Problem Stochastic Policy Maximum Entropy Reinforcement Learning MaxEnt RL

December 21, 2023

Maximum entropy GFlowNets with soft Q-learning
Sobhan Mohammadpour, Emmanuel Bengio, Emma Frejinger, Pierre-Luc Bacon
Generative Flow Network Energy Based GFlowNets Maximum Entropy Maximum Entropy Reinforcement Learning Soft Q Learning

December 15, 2023

Communication-Efficient Soft Actor-Critic Policy Collaboration via Regulated Segment Mixture
Xiaoxue Yu, Rongpeng Li, Chengchao Liang, Zhifeng Zhao
Multi Agent Reinforcement Learning Vehicle Plan Multi Agent Reinforcement Mixed Autonomy Traffic Maximum Entropy Reinforcement Learning Domain Mixture

October 17, 2023

Sim-to-Real Transfer of Adaptive Control Parameters for AUV Stabilization under Current Disturbance
Thomas Chaffre, Jonathan Wheare, Andrew Lammas, Paulo Santos, Gilles Le Chenadec, Karl Sammut, Benoit Clement
Deep Reinforcement Learning Autonomous Underwater Vehicle Adaptive Control Unknown Disturbance Sim to Real Transfer Maximum Entropy Reinforcement Learning Adaptive Controller AUV Velocity

May 1, 2023

BCEdge: SLO-Aware DNN Inference Services with Adaptive Batching on Edge Platforms
Ziyang Zhang, Huan Li, Yang Zhao, Changyao Lin, Jie Liu
Deep Neural Network Learning Based Edge Inference DNN Inference Maximum Entropy Reinforcement Learning Edge Platform

March 22, 2023

Matryoshka Policy Gradient for Entropy-Regularized RL: Convergence and Global Optimality
François Ged, Maria Han Veiga
Policy Gradient Early Stage Convergence Maximum Entropy Global Optimality Entropy Regularized Maximum Entropy Reinforcement Learning Softmax Policy

March 20, 2023

Multimodal Shannon Game with Images
Vilém Zouhar, Sunit Bhattacharya, Ondřej Bojar
Language Model Language Understanding Multimodal Information Maximum Entropy Reinforcement Learning Modality Specific Information

March 1, 2023

A Variational Approach to Mutual Information-Based Coordination for Multi-Agent Reinforcement Learning
Woojun Kim, Whiyoung Jung, Myungsik Cho, Youngchul Sung
Multi Agent Reinforcement Learning Variational Method Maximum Entropy Reinforcement Learning Multi Agent Actor Critic

February 21, 2023

Learning to Play Text-based Adventure Games with Maximum Entropy Reinforcement Learning
Weichen Li, Rati Devidze, Sophie Fellenz
Reinforcement Learning LeArning Abstract Q Learning Deep Q Learning Text Based Game Maximum Entropy Reinforcement Learning

February 20, 2023

Multiagent Inverse Reinforcement Learning via Theory of Mind Reasoning
Haochen Wu, Pedro Sequeira, David V. Pynadath
Theoretical Understanding Reward Function Maximum Entropy Reinforcement Learning Unknown Teammate Mind Reasoning

February 2, 2023

MARLIN: Soft Actor-Critic based Reinforcement Learning for Congestion Control in Real Networks
Raffaele Galliera, Alessandro Morelli, Roberto Fronteddu, Niranjan Suri
Reinforcement Learning Real World Novel Reinforcement Learning Maximum Entropy Reinforcement Learning Congestion Control Soft Actor Critic Algorithm Efficient Protocol

January 5, 2023

Extreme Q-Learning: MaxEnt RL without Entropy
Divyansh Garg, Joey Hejna, Matthieu Geist, Stefano Ermon
Offline Reinforcement Learning Q Learning Q Function Stable Entropy Maximum Entropy Reinforcement Learning MaxEnt RL Modern Deep Reinforcement Learning

Maximum Entropy Reinforcement Learning

Papers

Hierarchical Preference Optimization: Learning to achieve goals via feasible subgoals prediction

Sampling from Energy-based Policies using Diffusion

Average-Reward Maximum Entropy Reinforcement Learning for Underactuated Double Pendulum Tasks

Maximum Entropy On-Policy Actor-Critic via Entropy Advantage Estimation

Soft-QMIX: Integrating Maximum Entropy For Monotonic Value Function Factorization

Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow

S$^2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic

Tsallis Entropy Regularization for Linearly Solvable MDP and Linear Quadratic Regulator

Discrete Probabilistic Inference as Control in Multi-path Environments

Maximum entropy GFlowNets with soft Q-learning

Communication-Efficient Soft Actor-Critic Policy Collaboration via Regulated Segment Mixture

Sim-to-Real Transfer of Adaptive Control Parameters for AUV Stabilization under Current Disturbance

BCEdge: SLO-Aware DNN Inference Services with Adaptive Batching on Edge Platforms

Matryoshka Policy Gradient for Entropy-Regularized RL: Convergence and Global Optimality

Multimodal Shannon Game with Images

A Variational Approach to Mutual Information-Based Coordination for Multi-Agent Reinforcement Learning

Learning to Play Text-based Adventure Games with Maximum Entropy Reinforcement Learning

Multiagent Inverse Reinforcement Learning via Theory of Mind Reasoning

MARLIN: Soft Actor-Critic based Reinforcement Learning for Congestion Control in Real Networks

Extreme Q-Learning: MaxEnt RL without Entropy