Optimistic Exploration

Optimistic exploration in reinforcement learning aims to efficiently discover high-reward actions in complex environments by prioritizing uncertain, potentially rewarding areas. Current research focuses on improving sample efficiency through techniques like Thompson sampling, scaling model capacity with regularization, and decoupling exploration and exploitation using optimistic and pessimistic actors. These advancements are significantly impacting the field by enabling faster learning in challenging scenarios, such as continuous control tasks with sparse rewards and safety constraints, and leading to improved performance in robotics and other applications.

Papers

October 7, 2024

Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling
Jasmine Bayrooti, Carl Henrik Ek, Amanda Prorok
High Efficiency Model Based Reinforcement Learning Sparse Reward Thompson Sampling State Action Space Sample Efficient Reinforcement Learning Optimistic Exploration

May 25, 2024

Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoś, Marek Cygan
Reinforcement Learning Near Optimal Policy Optimistic Exploration Positive Scaling

December 26, 2023

Efficient Reinforcement Learning via Decoupling Exploration and Utilization
Jingpu Yang, Helin Wang, Qirui Zhao, Zhecheng Shi, Zirui Song, Miao Fang
Reinforcement Learning Efficient Reinforcement Learning Multi Agent Decoupling Coefficient Single Agent Reinforcement Learning Resource Utilization Optimistic Exploration Reinforcement Learning Paradigm

December 19, 2023

OVD-Explorer: Optimism Should Not Be the Sole Pursuit of Exploration in Noisy Environments
Jinyi Liu, Zhi Wang, Yan Zheng, Jianye Hao, Chenjia Bai, Junjie Ye, Zhen Wang, Haiyin Piao, Yang Sun
Reinforcement Learning Environment Exploration Noisy Environment Exploration Efficiency Optimistic Exploration Optimistic Algorithm Pursuit Strategy

October 25, 2023

Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion
Taehyun Cho, Seungyub Han, Heesoo Lee, Kyungjae Lee, Jungwoo Lee
Common Pitfall Distributional Reinforcement Learning Optimistic Exploration Distributional Bellman Exploration Bias

October 11, 2023

COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RL
Xiyao Wang, Ruijie Zheng, Yanchao Sun, Ruonan Jia, Wichayaporn Wongkamjan, Huazhe Xu, Furong Huang
Reinforcement Learning Model Based Reinforcement Learning Model Based High Level Plan Learned Dynamic Optimistic Exploration

June 13, 2023

A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning
Siyuan Guo, Yanchao Sun, Jifeng Hu, Sili Huang, Hechang Chen, Haiyin Piao, Lichao Sun, Yi Chang
Offline Reinforcement Learning Unified Alignment Efficient Exploration Uncertainty Aware State Action Distribution Optimistic Exploration Offline to Online Reinforcement Learning

March 16, 2023

Conditionally Optimistic Exploration for Cooperative Deep Multi-Agent Reinforcement Learning
Xutong Zhao, Yangchen Pan, Chenjun Xiao, Sarath Chandar, Janarthanan Rajendran
Multi Agent Reinforcement Learning Efficient Exploration Multi Agent Deep Reinforcement Learning Collaborative Exploration Optimistic Exploration

March 3, 2023

Toward Risk-based Optimistic Exploration for Cooperative Multi-Agent Reinforcement Learning
Jihwan Oh, Joonkee Kim, Minchan Jeong, Se-Young Yun
Multi Agent Reinforcement Learning Cooperative Multi Agent Reinforcement Learning Distributional Reinforcement Learning Collaborative Exploration Optimistic Exploration

November 16, 2022

Global Optimization with Parametric Function Approximation
Chong Liu, Yu-Xiang Wang
Bayesian Optimization Global Optimization UCB Algorithm Optimistic Exploration Parametric Approximation

July 29, 2022

Sample-efficient Safe Learning for Online Nonlinear Control with Control Barrier Functions
Wenhao Luo, Wen Sun, Ashish Kapoor
Reinforcement Learning Control Barrier Function Safe Exploration Safe Learning Online Control Algorithm Optimistic Exploration

May 16, 2022

From Dirichlet to Rubin: Optimistic Exploration in RL without Bonuses
Daniil Tiapkin, Denis Belomestny, Eric Moulines, Alexey Naumov, Sergey Samsonov, Yunhao Tang, Michal Valko, Pierre Menard
Reinforcement Learning Multi Armed Bandit Reward Report Value Iteration UCB Algorithm Human Like RL Optimistic Exploration Ordered Matrix Dirichlet

April 5, 2022

Jump-Start Reinforcement Learning
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, Sergey Levine, Karol Hausman
Reinforcement Learning Optimistic Exploration

December 1, 2021

DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learning
Archana Bura, Aria HasanzadeZonuzy, Dileep Kalathil, Srinivas Shakkottai, Jean-Francois Chamberland
Safe Reinforcement Learning Finite Horizon Simple Regret Optimistic Exploration Model Based Safe Reinforcement Learning Reward Aware

Optimistic Exploration

Papers

Efficient Model-Based Reinforcement Learning Through Optimistic Thompson Sampling

Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control

Efficient Reinforcement Learning via Decoupling Exploration and Utilization

OVD-Explorer: Optimism Should Not Be the Sole Pursuit of Exploration in Noisy Environments

Pitfall of Optimism: Distributional Reinforcement Learning by Randomizing Risk Criterion

COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RL

A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning

Conditionally Optimistic Exploration for Cooperative Deep Multi-Agent Reinforcement Learning

Toward Risk-based Optimistic Exploration for Cooperative Multi-Agent Reinforcement Learning

Global Optimization with Parametric Function Approximation

Sample-efficient Safe Learning for Online Nonlinear Control with Control Barrier Functions

From Dirichlet to Rubin: Optimistic Exploration in RL without Bonuses

Jump-Start Reinforcement Learning

DOPE: Doubly Optimistic and Pessimistic Exploration for Safe Reinforcement Learning