Hindsight Goal Relabeling

Hindsight goal relabeling is a technique in reinforcement learning that retroactively assigns goals to past experiences, effectively turning any trajectory into a successful demonstration for reaching its final state. Current research focuses on improving the efficiency and robustness of this approach, particularly in sparse-reward environments, through methods like hierarchical reinforcement learning and algorithms that leverage preference-based learning or minimize divergence between learned policies and demonstrations. This technique offers a powerful way to learn from limited or delayed feedback, significantly advancing the capabilities of reinforcement learning agents in complex and challenging real-world scenarios, such as robotics and language model instruction following.

Papers

October 16, 2024

SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabeling
Loris Gaven, Clement Romac, Thomas Carta, Sylvain Lamprier, Olivier Sigaud, Pierre-Yves Oudeyer
LLM Agent Online Reinforcement Learning Autonomous LLM Multi Goal Reinforcement Learning Hindsight Goal Relabeling

October 9, 2024

Identifying and Addressing Delusions for Target-Directed Decision-Making
Mingde Zhao, Tristan Sylvain, Doina Precup, Yoshua Bengio
Better Generalization RL Agent False Premise Hallucination Agent State Goal Directed Decision Hindsight Goal Relabeling

April 20, 2024

PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabeling
Utsav Singh, Wesley A. Suttle, Brian M. Sadler, Vinay P. Namboodiri, Amrit Singh Bedi
Sparse Reward Hierarchical Reinforcement Learning Sparse Reward Environment Hindsight Goal Relabeling Pose Invariant Face Recognition

April 14, 2024

Provable Interactive Learning with Hindsight Instruction Feedback
Dipendra Misra, Aldo Pacchiano, Robert E. Schapire
Interactive Learning Hindsight Instruction Best Response Instruction Response Pair Hindsight Goal Relabeling

February 10, 2023

The Wisdom of Hindsight Makes Language Models Better Instruction Followers
Tianjun Zhang, Fangchen Liu, Justin Wong, Pieter Abbeel, Joseph E. Gonzalez
Language Model Reinforcement Learning Hindsight Instruction Deriving Wisdom Reward Free Supervised Finetuning Hindsight Goal Relabeling

September 26, 2022

Understanding Hindsight Goal Relabeling from a Divergence Minimization Perspective
Lunjun Zhang, Bradly C. Stadie
Imitation Learning Suboptimal Demonstration Hindsight Instruction Multi Goal Reinforcement Learning Divergence Optimization Hindsight Goal Relabeling

December 2, 2021

Hindsight Task Relabelling: Experience Replay for Sparse Reward Meta-RL
Charles Packer, Pieter Abbeel, Joseph E. Gonzalez
Reinforcement Learning Sparse Reward Meta Reinforcement Learning Experience Replay Sparse Reward Environment Proxy Reward Hindsight Goal Relabeling