Maximum Entropy Behavior Exploration for Sim2Real Zero-Shot Reinforcement Learning

Jiajun Hu; Nuria Armengol Urpi; Jin Cheng; Stelian Coros

cs.LGMar 26, 2026

Maximum Entropy Behavior Exploration for Sim2Real Zero-Shot Reinforcement Learning

Authors: Jiajun Hu, Nuria Armengol Urpi, Jin Cheng, Stelian Coros

Organizations: Department of Mechanical Engineering, EPFL · Department of Computer Science, ETH Zurich · Max Planck Institute for Intelligent Systems

Abstract

Zero-shot reinforcement learning (RL) algorithms aim to learn a family of policies from a reward-free dataset, and recover optimal policies for any reward function directly at test time. Naturally, the quality of the pretraining dataset determines the performance of the recovered policies across tasks. However, pre-collecting a relevant, diverse dataset without prior knowledge of the downstream tasks of interest remains a challenge. In this work, we study $\textit{online}$ zero-shot RL for quadrupedal control on real robotic systems, building upon the Forward-Backward (FB) algorithm. We observe that undirected exploration yields low-diversity data, leading to poor downstream performance and rendering policies impractical for direct hardware deployment. Therefore, we introduce FB-MEBE, an online zero-shot RL algorithm that combines an unsupervised behavior exploration strategy with a regularization critic. FB-MEBE promotes exploration by maximizing the entropy of the achieved behavior distribution. Additionally, a regularization critic shapes the recovered policies toward more natural and physically plausible behaviors. We empirically demonstrate that FB-MEBE achieves and improved performance compared to other exploration strategies in a range of simulated downstream tasks, and that it renders natural policies that can be seamlessly deployed to hardware without further finetuning. Videos and code available on our website.

Figures & tables

Figure 1: FB-MEBE improves behavior exploration by maximizing the behavior entropy during training. With the regularizer critic, FB-MEBE achieves zero-shot hardware deployment by optimizing reward at inference time.

Figure 2: Limitations of undirected exploration: Standard FB algorithm with undirected exploration (FB) suffers from performance degradation and entropy stagnation (left, higher is better) and yields unnatural behaviors (right, lower is better). While introducing a behavior regularizer (FB-Reg) leads to more plausible behaviors (right), it leads to severe performance degradation and further reduces behavior diversity (left).

Figure 3: Maximum Entropy Behavior Forward-Backward Exploration: FB-MEBE is an online zero-shot RL algorithm that collects data by acting with behaviors $z_{exp}$ that maximize the entropy of the achieved behavior distribution. Collected data is used to train policies with a regularized loss combining policy improvement objective based on the FB action value function ( $Q_{FB}$ ) and a critic ( $Q_{reg}$ ) trained on a behavior regularizer reward to induce meaningful locomotion patterns.

Figure 4: Left: Zero-shot performance averaged over 17 downstream velocity tracking tasks (top) and orientation tasks (bottom). We plot normalized return wrt the mean return of Fast-TD3 ( Seo et al., 2025 ) . Results are averaged across 5 random seeds, with shaded regions indicating

\pm

1-standard deviation. Middle: Policy entropy on

\left[v_{x},v_{y},w_{z}\right]

(top) and

\left[g_{x},g_{y},g_{z}\right]

(bottom) measured at 300K training steps. Right: Feet slippage measured at 300K training steps (lower is better).

Figure 5: Zero-shot scores for locomotion tasks. Results are averaged over 5 random seeds, and error bars denote

\pm

1-standard deviation.

Figure 6: Zero-shot scores for orientation tasks. Results are averaged over 5 random seeds, and error bars denote

\pm

1-standard deviation.

Panel of Figure 7 — Figure 7: FB-MEBE allows for zero-shot sim2real transfer to the Unitree Go2 quadrupedal robot.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Comparison between different betas on FB-MEBE on locomotion tasks

Figure 9: Comparison on velocity-tracking tasks between FB-MEBE and an ablation FB-MEBE - abl leveraging different sampling strategies for training the FB algorithm

Figure 10: Velocity $\left[v_{x},v_{y}\right]$ distribution of FB replay buffer at different training step. The color represents the density of data points.

Figure 11: Velocity $\left[v_{x},v_{y}\right]$ distribution of FB-MEBE replay buffer at different training step. The color represents the density of data points.

Parameter	Value
Action Scale	$0.5$
Proportional Gain $K_{p}$	$25.0$
Derivative Gain $K_{d}$	$0.5$
Control Frequency [Hz]	$50\text{Hz}$

Appendix

Table 1: Joint Control Parameters

Table 13

Network	Obs Dimension Breakdown
Critic $(F)$	$\left[\mathbf{v}_{3},\boldsymbol{\omega}_{3},\mathbf{g}_{3},h_{\text{base},1},\mathbf{q}_{12},\dot{\mathbf{q}}_{12}\right]$
$B$	$\left[\mathbf{v}_{3},\boldsymbol{\omega}_{3},\mathbf{g}_{3},h_{\text{base},1}\right]$
Actor	$\left[\mathbf{v}_{3},\boldsymbol{\omega}_{3},\mathbf{g}_{3},\mathbf{q}_{12},\dot{\mathbf{q}}_{12},\mathbf{a}_{12}\right]$
Critic (Reg)	$\left[\mathbf{v}_{3},\boldsymbol{\omega}_{3},\mathbf{g}_{3},h_{\text{base},1},\mathbf{q}_{12},\dot{\mathbf{q}}_{12},\mathbf{a}_{12},h_{\text{feet},4},F_{\text{feet},4}\right]$

Appendix

Table 4: Inputs for different networks

Hyperparameter	Value
Number of training steps	150k
Number of parallel environments	2048
Number of rollout steps between each agent update	10
Number of gradient steps per agent update	10
Policy delay (TD3 actor update interval)	2
Number of initial steps with random actions	1000

Appendix

Table 5: Training hyperparameters

Hyperparameter	Critic $(F)$	B	Critic $(\text{Reg})$	Actor
Learning Rate	$5\cdot 10^{-5}$	$5\cdot 10^{-5}$	$5\cdot 10^{-5}$	$5\cdot 10^{-5}$
Soft update coefficient	$0.01$	$0.01$	$0.005$	$0.01$
Input Variables	$(s,a,z)$	$(s)$	$(s,a)$	$(s,z)$
Output Dim	$d$	$d$	$1$	$12$
Embedding Hidden Units	$1024$	–	–	$1024$
Feed Forward Hidden Layers	$2$	$2$	$4$	$2$

Appendix

Table 6: Network hyperparameters

Hyperparameter	Value
Flow architecture	RealNVP-style normalizing flow
Number of coupling layers	10
Hidden dimension	64
Activation function	ReLU
Scale output activation	Tanh
Masking strategy	Alternating checkerboard / channel mask

Appendix

Table 7: Hyperparameters of the Normalizing Flow.

Hyperparameter	Value
normalizing flow update frequency	1000 steps
sample buffer size	100k
$\epsilon$ (for stability)	0.1
$\beta$ (reverse sampling strength)	2.0

Appendix

Table 8: Hyperparameters of reverse sampling.

Zero-Shot RL

See all 17 papers

Improving Zero-Shot Offline RL via Behavioral Task Sampling
Apr 28, 2026Nazim Bendib, Nicolas Perrin-Gilbert, Olivier SigaudZero-Shot RL Offline RL
Institut des Systèmes Intelligents et de Robotique, ISIR, Sorbonne Université, Paris, France.
Exploration and Online Transfer with Behavioral Foundation Models
Jun 29, 2026Louis Bagot, Mathieu Lefort, Laëtitia MatignonReinforcement Learning RL Exploration
SyCoSMA · Université Lyon 1, CNRS, INSA Lyon LIRIS - UMR5205, 69622 Villeurbanne, France · LIRIS, SyCoSMA, IRISA, MALT, UR +1
BFM-Zero: A Promptable Behavioral Foundation Model for Humanoid Control Using Unsupervised Reinforcement Learning
Nov 6, 2025Yitang Li, Zhengyi Luo, Tonghe Zhang +10Humanoid Robot Control Behavioral Foundation Models
1Carnegie Mellon University · 2Meta

Apr 28, 2026J/K move · Enter open · S save

Improving Zero-Shot Offline RL via Behavioral Task Sampling

Nazim Bendib, Nicolas Perrin-Gilbert, Olivier Sigaud

Institut des Systèmes Intelligents et de Robotique, ISIR, Sorbonne Université, Paris, France.

Zero-Shot RL Offline RL Multi-Task RL Reinforcement Learning Generalization in RL Goal-Conditioned RL+1

Open paper AskPDF arXiv

Offline zero-shot reinforcement learning (RL) aims to learn agents that optimize unseen reward functions without additional environment interaction. The standard approach to this problem trains task-conditioned policies by sampling task vectors that define linear reward functions over learned state representations. In most existing algorithms, these task vectors are randomly sampled, implicitly assuming this adequately captures the structure of the task space. We argue that doing so leads to suboptimal zero-shot generalization. To address this limitation, we propose extracting task vectors directly from the offline dataset and using them to define the task distribution used for policy training. We introduce a simple and general reward function extraction procedure that integrates into existing offline zero-shot RL algorithms. Across multiple benchmark environments and baselines, our approach improves zero-shot performance by an average of 20%, highlighting the importance of principled task sampling in offline zero-shot RL.

Abstract

Zero-Shot RL

Explore similar work

Improving Zero-Shot Offline RL via Behavioral Task Sampling

Exploration and Online Transfer with Behavioral Foundation Models

BFM-Zero: A Promptable Behavioral Foundation Model for Humanoid Control Using Unsupervised Reinforcement Learning