ETHER: Aligning Emergent Communication for Hindsight Experience Replay

Kevin Yandoka Denamganaï; Daniel Hernandez; Ozan Vardal; Sondess Missaoui; James Alfred Walker

cs.CLJul 28, 2023

ETHER: Aligning Emergent Communication for Hindsight Experience Replay

Authors: Kevin Yandoka Denamganaï, Daniel Hernandez, Ozan Vardal, Sondess Missaoui, James Alfred Walker

Organizations: Department of Computer Science University of York York, UK

Abstract

Hindsight Experience Replay (HER) enhances sample efficiency in goal-conditioned reinforcement learning (RL) by relabelling failed trajectories with goals that were actually achieved. However, HER assumes access to a goal relabelling function and a predicate function that determines whether a goal has been satisfied. These assumptions break down in instruction-following tasks, where goals are expressed in natural language and differ from the state space. We formalize this as the Hindsight Reinforcement Learning problem, which shows the need to jointly learn these functions alongside the RL policy. To address it, we propose ETHER (Emergent Textual Hindsight Experience Replay), an agent that leverages Emergent Communication. ETHER uses a referential game (RG) to train a speaker and a listener to develop a grounded, artificial language describing environment states. It partially aligns this emergent language with instruction language using co-occurrence patterns between task instructions and RL observations. Experiments on BabyAI's PickupDist task show that ETHER's learned RG speaker and listener can function as the goal relabelling and predicate functions of HER, improving sample efficiency despite imperfect language alignment. Our work bridges Emergent Communication and goal-conditioned RL, opening the door to wider applications of HER.

Figures & tables

Panel of Figure 1 — Figure 1 : (a): Schema of a discriminative object-centric/2-player/ $L$ -signal/ $N=0$ -round/ $K=64$ -distractor visual referential game [ 22 ] using a Straight-Through Gumbel-Softmax (STGS) communication channel [ 27 ] , with a modified STGS loss adapted to LazImpa [ 65 ] , termed STGS-LazImpa (see Appendix F.1 ). Stimuli undergo data augmentation from [ 23 ] to enforce object-centricism [ 17 , 22 ] , differing between speaker and listener. (b, c): Histograms of semantic counts from goal-conditioned trajectories by BabyAI’s expert agent and a Random agent, showing that goal semantics align with the most frequently observed features.

Figure 2 : ETHER uses three agents, an RL agent and a speaker and a listener from a RG. Failed trajectories, where the RL agent did not follow instructions $g$ , are relabelled by the RG speaker with an alternative goal $\hat{g}=m_{RG}(s_{T})$ . The RG listener , within the predicate function $f_{RG}$ , adjusts the rewards throughout the trajectory to reflect the extent to which each state match the alternative goal $\hat{g}$ . The relabelled trajectory is then added to the replay buffer, from which samples can be used to update all agents with losses $J_{HRL}$ and $J_{RG}$ .

Agent	Success %	Any-Colour	Any-Shape
R2D2	$58.01\pm 4.07$	–	–
Pickup+Descr HER	$61.72\pm 3.91$	$100.00\pm 0.0$	$100.0\pm 0.0$
Pickup HER	$74.41\pm 6.83$	$6.24\pm 1.48$	$6.24\pm 1.48$
ETHER	$67.68\pm 4.56$	$24.76\pm 5.30$	$15.95\pm 22.56$

Table 1 : Performances (mean

\pm

std.err.) of different agents after

1

M observations at the instruction-following task, and language alignment’s accuracies in terms of the Any-Shape and Any-Colour metrics.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

R2D2
Number of actors	32
Actor update interval	1 env. step
Sequence unroll length	20
Sequence length overlap	10
Sequence burn-in length	10
N-steps return	3

Appendix

Table 2 : Hyper-parameter values relevant to R2D2 in the different architectures presented. Missing parameters follow Ape-X [ 33 ] .

Agent	Mean
R2D2 (no Burn-In)	13.02 $\pm$ 1.26
HIGhER+ (no Burn-In)	16.02 $\pm$ 1.79
HIGhER++ (n=1) (no Burn-In)	14.97 $\pm$ 1.19
HIGhER++ (n=2) (no Burn-In)	15.89 $\pm$ 0.60
HIGhER++ (n=4) (no Burn-In)	13.93 $\pm$ 2.29

Appendix

Table 3 : Success ratios (mean and standard deviation) for agents without the burn-in feature of R2D2 after 200k steps in a modified version of the BabyAI PickUpDist-v0 task. 3 random seeds for each agent.

Panel of Figure 3 — Figure 3 : Left: Trajectories for the blue colour goal from BabyAI’s built-in expert agent which always reaches the goal. Right: Random agent trajectories. In both cases the semantics of the goal are among the most observed semantic features for any given trajectory. This effect is less pronounced in the random agent.

Panel of Figure 4 — Figure 4 : Left: Trajectories for colorless objects from BabyAI’s built-in expert agent which always reaches the goal. Right: Random agent trajectories. In both cases the semantics of the goal are among the most observed semantic features for any given trajectory. This effect is less pronounced in the random agent.

Panel of Figure 5 — Figure 5 : Left: Trajectories for the green color goal from BabyAI’s built-in expert agent which always reaches the goal. Right: Random agent trajectories. In both cases the semantics of the goal are among the most observed semantic features for any given trajectory. This effect is less pronounced in the random agent.

Panel of Figure 6 — Figure 6 : Left: Trajectories for the grey color goal from BabyAI’s built-in expert agent which always reaches the goal. Right: Random agent trajectories. In both cases the semantics of the goal are among the most observed semantic features for any given trajectory. This effect is less pronounced in the random agent.

Panel of Figure 7 — Figure 7 : Left: Trajectories for the purple color goal from BabyAI’s built-in expert agent which always reaches the goal. Right: Random agent trajectories. In both cases the semantics of the goal are among the most observed semantic features for any given trajectory. This effect is less pronounced in the random agent.

Panel of Figure 8 — Figure 8 : Left: Trajectories for the red color goal from BabyAI’s built-in expert agent which always reaches the goal. Right: Random agent trajectories. In both cases the semantics of the goal are among the most observed semantic features for any given trajectory. This effect is less pronounced in the random agent.

Panel of Figure 9 — Figure 9 : Left: Trajectories for the yellow color goal from BabyAI’s built-in expert agent which always reaches the goal. Right: Random agent trajectories. In both cases the semantics of the goal are among the most observed semantic features for any given trajectory. This effect is less pronounced in the random agent.

Explore similar work

Select-to-Act: Hierarchical Reinforcement Learning via Adaptive Language Guidance
Jun 21, 2026Hanping Zhang, Adam Koziak, Yuhong GuoHierarchical Reinforcement Learning English
School of Computer Science, Carleton University, Ottawa, Canada · Canada CIFAR AI Chair, Amii, Canada
Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrations
Jul 5, 2026Zichao Li, Gang Wu, Zichao Wang +5Large Language Model Agents Hindsight
Mila, McGill University · Adobe Research
Learning More from Less: Reinforcement Learning from Hindsight
Jul 10, 2026Iris Xu, Sunshine Jiang, John Marangola +8Reinforcement Learning Post-Training Hindsight
1Massachusetts Institute of Technology · 2MIT-IBM Computing Research Lab · 3Stanford University +1

Jun 21, 2026, cs.LGJ/K move · Enter open · S save

Select-to-Act: Hierarchical Reinforcement Learning via Adaptive Language Guidance

Hanping Zhang, Adam Koziak, Yuhong Guo

School of Computer Science, Carleton University, Ottawa, Canada · Canada CIFAR AI Chair, Amii, Canada

Hierarchical Reinforcement Learning English

Open paper AskPDF arXiv

Reinforcement Learning (RL) has been widely applied to sequential decision-making, yet it often suffers from poor sample efficiency due to costly interactions with the environment. A limited line of recent work has started exploring improving RL efficiency by leveraging external knowledge expressed in natural-language instructions. However, the few existing approaches typically treat the entire instruction as a single conditioning input, failing to account for the stage-dependent nature of language guidance, especially in complex environments. In this paper, we propose \emph{Hierarchical Reinforcement Learning with Language Instructions (HRLLI)}, a hierarchical RL framework that explicitly models natural-language instructions as dynamically selectable semantic guidance during decision-making. HRLLI decomposes instructions into a set of piecewise guidance elements, where each instruction piece may become relevant at different stages of interaction with the environment. A novel hierarchical RL policy structure is then formulated in a \emph{Select-to-Act} paradigm: a high-level semantic policy acts as a guidance selector that selects the most relevant instruction piece to the current state to guide the low-level agent's decision, while a low-level policy executes environment actions conditioned on the selected guidance. The two-level policies are learned simultaneously to maximize augmented expected returns from interactions with the environment. This design enables the agent to adaptively ground language instructions into stage-specific decisions during interaction. Experiments on the instruction-intensive RTFM benchmark show that HRLLI consistently outperforms strong instruction-conditioned RL baselines, demonstrating that explicitly modeling adaptive instruction selection significantly improves the effectiveness of RL.

Abstract

Explore similar work

Select-to-Act: Hierarchical Reinforcement Learning via Adaptive Language Guidance

Spinning Straw into Gold: Relabeling LLM Agent Trajectories in Hindsight for Successful Demonstrations

Learning More from Less: Reinforcement Learning from Hindsight