Entropy Exploration

Entropy exploration in reinforcement learning aims to develop agents that efficiently explore their environment by maximizing the entropy of their state visitation distribution, thereby encouraging diverse and comprehensive state coverage. Current research focuses on improving exploration efficiency through various techniques, including Bayesian approaches, variational methods, and the use of predecessor and successor representations to guide exploration strategies, often incorporating intrinsic rewards based on state entropy or value-conditional state entropy. These advancements are significant for improving sample efficiency in reinforcement learning, particularly in challenging scenarios with sparse or no rewards, and have implications for both offline and online learning paradigms.

Papers

May 25, 2024

Constrained Ensemble Exploration for Unsupervised Skill Discovery
Chenjia Bai, Rushuai Yang, Qiaosheng Zhang, Kang Xu, Yi Chen, Ting Xiao, Xuelong Li
Skill Discovery Unsupervised Reinforcement Learning Ensemble Based Reward Free Entropy Exploration

November 22, 2023

Probabilistic Inference in Reinforcement Learning Done Right
Jean Tarbouriech, Tor Lattimore, Brendan O'Donoghue
Reinforcement Learning Markov Decision Process Action Value Function Entropy Exploration

October 7, 2023

Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Exploration
Ziqi Zhang, Xiao Xiong, Zifeng Zhuang, Jinxin Liu, Donglin Wang
Reinforcement Learning Offline Reinforcement Learning Reinforcement Learning Algorithm Online Fine Tuning Entropy Exploration

June 26, 2023

Maximum State Entropy Exploration using Predecessor and Successor Representations
Arnav Kumar Jain, Lucas Lehnert, Irina Rish, Glen Berseth
Successor Representation Exploration Policy Reliable Learning Exploration Algorithm Random Exploration Entropy Exploration Novel Predecessor and Successor

May 31, 2023

Accelerating Reinforcement Learning with Value-Conditional State Entropy Exploration
Dongyoung Kim, Jinwoo Shin, Pieter Abbeel, Younggyo Seo
Reinforcement Learning Exploration Method Dense Environment Entanglement Entropy Conditional Entropy Entropy Function Entropy Exploration

April 21, 2023

On the Importance of Exploration for Real Life Learned Algorithms
Steffen Gracla, Carsten Bockelmann, Armin Dekorsy
Practical Algorithm Real World Environment Exploration Importance Aware Deep Q Network Adaptive Exploration Greedy Exploration Entropy Exploration

March 14, 2023

Fast Rates for Maximum Entropy Exploration
Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Remi Munos, Alexey Naumov, Pierre Perrault, Yunhao Tang, Michal Valko, Pierre Menard
Fast Rate Reward Free Exploration Entropy Maximization Entropy Exploration Entropy Regularized Markov Decision Process

October 12, 2022

Maximum entropy exploration in contextual bandits with neural networks and energy based models
Adam Elwood, Marco Leonardi, Ashraf Mohamed, Alessandro Rozza
Neural Network Full Model Contextual Bandit Energy Policy Research Exploration Exploitation Trade Entropy Exploration Reward Estimator

May 31, 2022

k-Means Maximum Entropy Exploration
Alexander Nedergaard, Matthew Cook
Reinforcement Learning Algorithm Entropy Exploration Artificial Curiosity

February 7, 2022

The Importance of Non-Markovianity in Maximum State Entropy Exploration
Mirco Mutti, Riccardo De Santi, Marcello Restelli
Importance Aware Non Markovian Entropy Exploration