Reward Free Exploration

Reward-free exploration (RFE) in reinforcement learning focuses on efficiently learning a model of the environment's dynamics without using reward information, enabling subsequent efficient policy optimization for any reward function. Current research emphasizes developing sample-efficient algorithms, often employing model-based approaches and techniques like uncertainty-weighted learning or entropy maximization, to improve exploration in various settings including linear and low-rank Markov Decision Processes. These advancements are significant because they reduce the reliance on reward signals, making reinforcement learning more adaptable to diverse tasks and potentially more robust to reward specification challenges in real-world applications.

Papers

June 24, 2024

Uncertainty-Aware Reward-Free Exploration with General Function Approximation
Junkai Zhang, Weitong Zhang, Dongruo Zhou, Quanquan Gu
Reinforcement Learning Unsupervised Reinforcement Learning Reward Free Reinforcement Learning Reward Free Exploration

June 6, 2024

How does Inverse RL Scale to Large State Spaces? A Provably Efficient Approach
Filippo Lazzati, Mirco Mutti, Alberto Maria Metelli
Inverse Reinforcement Learning State Space Efficient Approach Linear Markov Decision Process Reward Free Exploration Feasible Reward

March 18, 2024

The Value of Reward Lookahead in Reinforcement Learning
Nadav Merlis, Dorian Baudry, Vianney Perchet
Reinforcement Learning Net Present Value Worst Case Reward Prediction Reward Free Exploration

June 23, 2023

Active Coverage for PAC Reinforcement Learning
Aymen Al-Marjani, Andrea Tirinzoni, Emilie Kaufmann
Reinforcement Learning Active Exploration Episodic Markov Decision Process Reward Free Exploration Entity Coverage

June 15, 2023

Reward-Free Curricula for Training Robust World Models
Marc Rigter, Minqi Jiang, Ingmar Posner
Native Robustness World Model Minimax Regret Reward Free Reward Free Exploration

April 14, 2023

Minimax-Optimal Reward-Agnostic Exploration in Reinforcement Learning
Gen Li, Yuling Yan, Yuxin Chen, Jianqing Fan
Reinforcement Learning Reward Free Exploration Agnostic Exploration Sample Efficient Offline Reinforcement Learning

March 20, 2023

Improved Sample Complexity for Reward-free Reinforcement Learning under Low-rank MDPs
Yuan Cheng, Ruiquan Huang, Jing Yang, Yingbin Liang
Sample Complexity Low Rank Reward Free Reinforcement Learning Reward Free Exploration

March 17, 2023

Optimal Horizon-Free Reward-Free Exploration for Linear Mixture MDPs
Junkai Zhang, Weitong Zhang, Quanquan Gu
Reward Free Reward Free Reinforcement Learning Reward Free Exploration Linear Mixture Markov Decision Process Reward Aware

March 14, 2023

Fast Rates for Maximum Entropy Exploration
Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Remi Munos, Alexey Naumov, Pierre Perrault, Yunhao Tang, Michal Valko, Pierre Menard
Fast Rate Reward Free Exploration Entropy Maximization Entropy Exploration Entropy Regularized Markov Decision Process

February 13, 2023

Provably Safe Reinforcement Learning with Step-wise Violation Constraints
Nuoya Xiong, Yihan Du, Longbo Huang
Reward Free Exploration Step Wise Violation

October 11, 2022

Exploration via Elliptical Episodic Bonuses
Mikael Henaff, Roberta Raileanu, Minqi Jiang, Tim Rocktäschel
Reinforcement Learning Environment Exploration Sparse Reward Reward Free Exploration Episodic Bonus

June 28, 2022

Safe Exploration Incurs Nearly No Additional Sample Complexity for Reward-free RL
Ruiquan Huang, Jing Yang, Yingbin Liang
Sample Complexity Near Optimal Policy Safe Exploration Reward Free Reinforcement Learning Reward Free Exploration Reinforcement Learning Paradigm

June 21, 2022

On the Statistical Efficiency of Reward-Free Exploration in Non-Linear RL
Jinglin Chen, Aditya Modi, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal
Low Rank Sample Efficiency Reward Free Reinforcement Learning Reward Free Exploration Reward Aware

February 16, 2022

Branching Reinforcement Learning
Yihan Du, Wei Chen
Reinforcement Learning Regret Minimization Reward Free Exploration Complex Trajectory Trajectory Set

February 13, 2022

Sample-Efficient Reinforcement Learning with loglog(T) Switching Cost
Dan Qiao, Ming Yin, Ming Min, Yu-Xiang Wang
Reinforcement Learning Hidden CoST Sample Efficient Reinforcement Learning Regret Algorithm Adaptive Policy Reward Free Exploration Regret Optimal Algorithm

December 2, 2021

Reward-Free Attacks in Multi-Agent Reinforcement Learning
Ted Fujimoto, Timothy Doster, Adam Attarian, Jill Brandenberger, Nathan Hodas
Multi Agent Reinforcement Learning Reward Free Exploration Male Victim High Entropy