Measuring Empowerment from Offline Data for Intrinsically Motivated Exploration

Ishir Garg; Vivek Myers; Qiyang Li; Catherine Ji; Benjamin Eysenbach; Sergey Levine

cs.LGOct 17, 2026

Measuring Empowerment from Offline Data for Intrinsically Motivated Exploration

Authors: Ishir Garg, Vivek Myers, Qiyang Li, Catherine Ji, Benjamin Eysenbach, Sergey Levine

Organizations: UC Berkeley · Princeton University

Abstract

Large, unlabeled datasets of prior experience have become increasingly available for agentic tasks in domains such as robotics and coding. A common recipe for these tasks is to train a policy using the prior data, and then further improve it with online rollouts. But while much work has studied how to use both the offline and online data for policy extraction, far less work has been done to understand how the prior data can be used to efficiently explore during the online learning phase. Previous work has attempted to formalize this idea by defining the empowerment, a measure of how many future states an agent can realize; states with higher empowerment require more exploration as there are more futures to explore. However, existing methods for estimating empowerment require online policy rollouts in the environment. We propose the first method for estimating empowerment from unlabeled prior data, without any environment interaction. By jointly learning an empowerment-maximizing policy together with its discounted state-occupancy measure, our estimator can be trained on unlabeled trajectories generated from any unknown policy. We demonstrate that our empowerment estimator qualitatively gives intuitive empowerment measures and show that the pretrained estimator can be used to improve exploration in the online goal-conditioned reinforcement learning setting. Code is available at https://github.com/ishirgarg/offline-empowerment.

Figures & tables

Panel of Figure 1 — Algorithm 1 Offline Empowerment Estimation

Panel of Figure 2 — Figure 2: Empowerment (middle panel) recovered by our offline estimator along real dataset trajectories, alongside the underlying state (bottom). (a) antsoccer : the ant walks up to the ball, kicks it away, walks back, dribbles it, and kicks it again. (b) cube-single : one cycle in which the arm grabs the block, picks it up, and places it down in a new spot. Rendered frames (top) show the scene at the timestep marked by each hollow circle. The $x$ -axis has timesteps at the phase boundaries as tick labels.

Panel of Figure 3 — Figure 3: Empowerment maps for our method and the four baselines of Section 5.1 , on antmaze-medium-navigate (top) and antsoccer-arena-navigate with the ball (circled in white) fixed off-center (bottom). DADS is skill-empowerment; CF, CF-FlowBC, and DV are action-empowerment.

Panel of Figure 4 — Figure 4: Online success rate on each task (top line of each panel title) with each offline dataset (bottom line), for CRL, RND, and SUPE with and without our empowerment bonus. We use 5 seeds.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Number of skills $K$	50
$Q$ / $V$ network width / depth	512 / 3
Policy network width / depth	512 / 3
Value latent dimension $d$	256
Layer normalization	yes
Learning rate ( $Q$ / $V$ and policy)	$3\times 10^{-4}$

Appendix

Table 1: Hyperparameters for offline empowerment pretraining (ours).

Environment	Dataset	$\sigma_{a}$
antmaze-medium	navigate	0.01
antmaze-medium	stitch	0.01
antmaze-medium	explore	0.01
antsoccer-arena	navigate	0.01
antsoccer-arena	stitch	0.01
pointmaze-teleport	navigate	0.01

Appendix

Table 2: Action noise std

\sigma_{a}

of the empowerment estimator trained on each offline dataset.

Figure 5: Validation empowerment over training for four action noise scales

\sigma_{a}

, on antmaze-medium-navigate-v0 . The converged empowerment is robust to

\sigma_{a}

; all other hyperparameters are fixed at the values in Table 1 .

Figure 6: Validation empowerment over training for four behavioral constraint coefficients

\alpha_{\text{bc}}

, on antsoccer-arena-stitch-v0 . Empowerment is roughly flat until

\alpha_{\text{bc}}=10^{-2}

, where it collapses.

Figure 7: Empowerment maps on antsoccer-arena-stitch-v0 at matched ball position, for $\alpha_{\text{bc}}=10^{-3}$ (left) vs. $\alpha_{\text{bc}}=10^{-2}$ (right). Heavy BC regularization produces a hole in empowerment exactly at the ball, where the offline data is dominated by a single (dribbling) behavior.

Figure 8: Online success rate of CRL + Empowerment for different weights

\beta

of the empowerment bonus, on each task (top line of each panel title) with each offline dataset (bottom line);

\beta=0

corresponds to CRL. Lines show the mean over 5 seeds and shaded regions show

\pm 1

standard error of the mean, smoothed as in Fig. 4 .

Task	Goals	Horizon	Offline datasets
antmaze-medium	25	1000	navigate, stitch, explore
pointmaze-teleport	41	1000	navigate, stitch
antsoccer-arena	8	500	navigate, stitch
cube-single	30	200	play, noisy

Appendix

Table 3: Online tasks and the offline datasets

\mathcal{D}

paired with them.

Hyperparameter	Value
Actor network width / depth	512 / 3
Critic network width / depth	512 / 3
Critic latent dimension	512
Layer normalization	yes
Learning rate	$3\times 10^{-4}$
Batch size	1024

Appendix

Table 4: Hyperparameters shared by all online agents (CRL + RLPD).

Hyperparameter	Value
Empowerment bonus (ours)
$B_{\omega}$ network width / depth	512 / 3 (two heads)
Bonus weight $\beta$	per task ( Table 6 )
Future-state samples per skill for $\hat{\mathcal{E}}$	64
RND
Target / predictor network width / depth	512 / 2

Appendix

Table 5: Hyperparameters of the exploration bonuses.

Task	Dataset	CRL + Emp.	CRL + RND	CRL + RND + Emp.	SUPE + Emp.
		$\beta$	$\beta_{\text{RND}}$	$\beta$ / $\beta_{\text{RND}}$	$\beta$
cube-single	play	10	1	10 / 1	30
cube-single	noisy	30	1	30 / 1	10
antmaze-medium	navigate	30	3	30 / 3	10
antmaze-medium	stitch	10	3	10 / 3	10
antmaze-medium	explore	10	3	10 / 3	10

Appendix

Table 6: Bonus weights used for each task and dataset in Fig. 4 .

Empowerment

See all 10 papers

Scaling Empowered Agents
Oct 9, 2026Catherine Ji, Vivek Myers, Joshua Lin +3Empowerment
Department of Physics Princeton University · Department of Computer Science UC Berkeley · Department of Mathematics Princeton University +1
JumpStart Your Policy Learning with Lessons from 160,000 Training Runs
Sep 12, 2026Nabil Omi, Eric Bae, Chung Yik Edward Yeung +2Offline RL Algorithm Selection
University of Washington · Microsoft
Flexible Empowerment at Reasoning with Extended Best-of-N Sampling
Apr 17, 2026Taisuke KobayashiReinforcement Learning Best-of-N Sampling
National Institute of Informatics and The Graduate University for Advanced Studies, SOKENDAI Japan

Oct 9, 2026J/K move · Enter open · S save

Scaling Empowered Agents

Catherine Ji, Vivek Myers, Joshua Lin, Alexandra Wingate+2

Department of Physics Princeton University · Department of Computer Science UC Berkeley · Department of Mathematics Princeton University+1

Empowerment

Open paper AskPDF arXiv

Empowerment measures the capacity of an agent to actively control its environment. In this work, we explain the connection between skill learning and empowerment. While skill learning is not empowerment, we introduce a simple scalable recipe for empowerment maximization: empowerment corresponds to the number of executable skills from a state, so maximizing empowerment corresponds to reaching states that are good starting points for many skills. Our recipe applies to arbitrary underlying skill learning methods. Whereas most prior work on empowerment looks at tabular settings, this recipe enables us to study empowerment at a scale where we can realize theoretically-predicted but empirically-elusive properties of empowerment (e.g., gathering resources, environment modification, survival). Experiments show that empowerment maximization unlocks interesting emergent properties, such as building a house in Crafter. Website and code at this https url.

Abstract

Empowerment

Explore similar work

Scaling Empowered Agents

JumpStart Your Policy Learning with Lessons from 160,000 Training Runs

Flexible Empowerment at Reasoning with Extended Best-of-N Sampling