Authors: Catherine Ji, Vivek Myers, Joshua Lin, Alexandra Wingate, Sergey Levine, Benjamin Eysenbach
Organizations: Department of Physics Princeton University · Department of Computer Science UC Berkeley · Department of Mathematics Princeton University · Department of Computer Science Princeton University
Empowerment measures the capacity of an agent to actively control its environment. In this work, we explain the connection between skill learning and empowerment. While skill learning is not empowerment, we introduce a simple scalable recipe for empowerment maximization: empowerment corresponds to the number of executable skills from a state, so maximizing empowerment corresponds to reaching states that are good starting points for many skills. Our recipe applies to arbitrary underlying skill learning methods. Whereas most prior work on empowerment looks at tabular settings, this recipe enables us to study empowerment at a scale where we can realize theoretically-predicted but empirically-elusive properties of empowerment (e.g., gathering resources, environment modification, survival). Experiments show that empowerment maximization unlocks interesting emergent properties, such as building a house in Crafter. Website and code at this https url.
Figures & tables
Figure 1: Empowerment-maximizing behavior across environments. (a) In the BuilderBench task, a gripper interacts with blocks in an open-ended setting. The potential empowerment policy is visualized with the white rollout, navigating to the blocks and staying there. Then, the effective empowerment policies, or skills, knock the blocks in different directions (colorful lines and blocks). (b) In the first version of the AntSoccer task, an agent can be empowered without interacting with the ball. The potential empowerment policy (white) navigates towards the central area of the arena, while the skills branch out from this point. (c) In the second version of the AntSoccer task, an agent can only be empowered if it controls the ball. The trained potential empowerment policy reaches the soccer ball, while skills kick the soccer ball in different directions. (d) In Craftax-Classic, the potential empowerment agent gathers materials, builds a crafting table, then builds tools using gathered materials.
Figure 2: The potential empowerment estimate increases with control in Crafter. (a) The learned empowerment estimate increases consistently during training. 5 seeds, p.m. standard error after per-seed EMA ( α=0.05 ). (b) Simultaneously, the agent intrinsically learns to complete achievements in the environment without access to external rewards . 5 seeds, evaluated over 128 random environments, p.m. standard error.
Figure 3: Potential empowerment and control increases over training (continuous settings). We compare SEE (Black) and baselines VIM (Orange) and LatentGCE (Green). (a) SEE visits diverse states across all environments, which is necessary to learn correct models and decoders of dynamics and skills. (b) SEE consistently learns potential empowerment policies that navigate towards central regions of the arena in Ant, to the ball in AntSoccer, and to the blocks in BuilderBench. VIM is unstable and largely stationary in Ant and AntSoccer. LatentGCE is stationary in Ant. Five seeds for SEE and LatentGCE and three for VIM, p.m. standard error. Eval metrics averaged over 128 randomized environments.
Figure 4: Potential empowerment policies accrue empowerment, while skills spend it. (Top row) Over the course of the trajectory, the potential empowerment policies (white lines) “gather” empowerment, reaching empowered states. Correspondingly, the empowerment estimates increase over the course of the trajectory. (Bottom row) At the gray skill switch mark, executing a skill (colorful lines) from the reached state “spends” the empowerment, reflecting the intuition that potential empowerment policies collect power and effective empowerment spends power. Traces use one seed and four skill branches per environment.
Figure 5: Empowerment maximization has interesting emergent behaviors. (a) In Crafter, the empowered agent quickly accumulates many resources and uses a tool (crafting table, wooden icon with saw) to craft other tools in the game. Resources boxed in white. (b) Early in training, the agent learns to build a “house” consisting of plants and crafting tables. Built tiles boxed in white. (c) The agent also naturally learns to survive, avoiding and killing enemies like zombies and skeletons. One seed per example; further qualitative examples across seeds in Appendix E .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Algorithm 1 SEE
Parameter
Ant
AntSoccer
BuilderBench
Encoder observations
Full state (42)
Ball x,y (2)
Block positions (9)
Observation noise std
0.05
0.05
0.01
Skill dimension
2
2
9
Online env steps per seed
500M
500M
100M
Replay capacity (transitions)
10M
10M
6M
Batch size
1,024
1,024
256
Appendix
Table 1: Hyperparameters for SEE with joint MI in continuous environments.
Figure 6: Achievements in a single Craftax-Classic two-phase episode. Icons indicate when achievements first occur. (a) The black line shows the achievements of the shared potential phase up to the skill switch (vertical gray line). In this rollout, these achievements include collecting wood, building a crafting a table, then crafting a wooden sword. The colorful skill rows show the achievements rolled out from the switch point sharing the same starting state, including sleeping/waking up, building a pickaxe, planting a tree, collecting stone, building a furnace, and defeating an enemy. (b) The lines indicate the cumulative achievement counts over the rollout, achieved without the use of any extrinsic reward. One seed.
Parameters
Value
Encoder observations
inventory (12 features)
Encoder data
CMI branches from copied states
Observation noise std (classifier bottleneck)
0
Future-sampling discount
0.95
Logit temperature
0.1 (fixed)
Encoder learning rate
1e-3
Appendix
Table 2: Hyperparameters in Craftax-Classic (SEE, CMI (Ours); quantitative curves).
Figure 7: Empowerment fields on low-dimensional benchmarks. For the canonical environments of a pendulum and a ball in a box with hard walls, various empowerment maximization methods predict (normalized) empowerment fields over state space. (a) A linearized reference reveals that the potential empowerment is maximized when the pendulum points straight up. For the Ball in Box, the potential empowerment is maximal at the center of the box. Thus, a potential empowerment policy over a long effective horizon should seek a straight-up state in the pendulum and the central state for Ball in Box. (b) The empowerment field predicted by LatentGCE method correctly identifies these potential empowerment states. (c) VIM does not recover these high empowerment regions. One seed per learned field.
Parameter
Value
Training budget (Ant / AntSoccer / BuilderBench)
500M / 500M / 100M
Episode length / probe horizon H
400 / 30 steps
Parallel environments / rollout length
256 / 80 steps
Probe starts per update
256, one uniformly sampled rollout time per environment
PPO controller
Actor / critic hidden layers
[256, 256], tanh
Appendix
Table 3: VIM hyperparameters for Ant, AntSoccer, and BuilderBench. Each environment uses 3 training seeds.
Parameter
Value
Training budget (Ant / AntSoccer / BuilderBench)
500M / 500M / 100M
Episode length / probe horizon H
400 / 100 steps
Parallel environments / rollout length
256 / 80 steps
Probe starts per update
256, one uniformly sampled rollout time per environment
PPO controller
Actor / critic hidden layers
[256, 256], tanh
Appendix
Table 4: LatentGCE hyperparameters for Ant, AntSoccer, and BuilderBench. Each environment uses 5 training seeds.
Figure 8: Learned empowerment in continuous settings. SEE (Black) empowerment values gradually increase over time. Meanwhile, VIM (Orange) empowerment values are unstable and are very large at the start of training (Orange arrows point beyond the displayed y-axis). The LatentGCE (Green) empowerment values do not monotonically increase in time. *As a note, learned estimates use different objectives and regularization, so their magnitudes are not directly comparable across methods. Five seeds for SEE and LatentGCE and three for VIM, p.m. standard error.
Figure 9: We compare the full method with variants that remove the two-phase rollout (no switching, orange), replace joint MI with conditional MI (green), or replace the bilinear logit with a general MLP (pair decoder, purple). The solid black curve shows SEE, and the colored dashed curves show the ablations. (a) For all methods, average empowerment increases over training. ∗ Learned empowerment estimates use different objectives and regularization, so their magnitudes are not directly comparable across methods. (b) The skill visitation of SEE quickly saturates for these environments. Meanwhile, the ablations struggle to get visitation in a subset of the different environments. (c) SEE consistently learns potential policies that reach the center of the arena (Ant), approach the ball (AntSoccer), and reach the blocks (BuilderBench). In BuilderBench, conditional MI and the pair decoder potential empowerment rollouts reach the blocks, while no switching ends far away. In Ant, all ablations end far from the center of the arena. In AntSoccer, the ablations end up further from the ball than SEE. (d) SEE consistently learns effective policies with large displacements. Five seeds for SEE; three per ablation.
Figure 10: Craftax-Classic ablations (CMI SEE: 5 seeds; each ablation: 3). The black curve is the SEE with the CMI classifier (Ours). The ablations are SEE with Joint MI (Ours), no switching (25% potential-policy episodes during training), and pair decoder. (a) Learned empowerment estimates rise over training for the base method. (b) Total achievements per rollout. (c) Achievements in the first 100 potential-policy actions. (d) New achievements in the subsequent skill phase of up to 900 actions. All methods use 128 fresh worlds and skills per seed and evaluation checkpoint. Curves show mean ± standard error across training seeds, with per-seed EMA ( α=0.05 ) only in (a). Achievement panels are unsmoothed and display every tenth evaluation checkpoint plus the final checkpoint.
Figure 11: Reward normalization in continuous environments. We compare the annealed schedule with normalized ( λ=1 ) and unnormalized ( λ=0 ) rewards. Columns show learned empowerment, training skill visitation, target distance after 100 potential-policy actions, and displacement during the subsequent 300 skill actions. Learned estimates use different objectives and are not directly comparable across settings. Five seeds for SEE; three per ablation. Curves show mean ± standard error across seeds. Training curves use per-seed EMA ( α=0.05 ); evaluation curves are unsmoothed.
Figure 12: Reward normalization in Craftax-Classic. We compare CMI SEE with the annealed schedule against fixed λ=0 and λ=1 . Five seeds for SEE; three per ablation. Curves show mean ± standard error across seeds. The learned-estimate panel uses per-seed EMA ( α=0.05 ). Achievements use 128 fresh worlds and skills per seed and checkpoint, with 100 potential-policy actions followed by up to 900 skill actions. Achievement evaluations are unsmoothed; every tenth checkpoint and the final checkpoint are shown.
Figure 13: Additional environment-modification behaviors. Stills of environment modifications over the remaining 4 seeds in Craftax-Classic. Each panel shows a selected rollout. White outlines box placed tiles.
Empowerment captures the capacity for an agent to actively control its environment. While conceptually appealing as an information-theoretic quantity, the connection between empowerment and structurally central states that provide broad access to future outcomes has remained an open question. In this work, we link empowerment maximization and skill-learning methods to provide new geometries for interpreting and analyzing empowerment. Our analyses answer longstanding open questions on the connections between empowerment and structural centrality. Our analyses also reveal distinctions between information and reward geometries, highlighting important theoretical implications to build scalable empowerment-maximization methods. Website and code can be found at https://empowerment-geometry.github.io/.
Catherine Ji, Vivek Myers, Sergey Levine +1
Department of Computer Science UC Berkeley · Department of Computer Science Princeton University
Recently, the practice of augmenting LLM agent capability with skills has gained prevalence. We explore the cost effective adaptation of agents to novel domains by means of learning skills. Existing works focus on performance gain over cost effectiveness. As a result, little is known about what skill learning strategies save cost. We argue that among all the different skill learning methods, those that view skills as programs can achieve the best cost reduction. By executing sequences of actions deterministically, a program-augmented agent can reliably and cheaply achieve goals that would otherwise require trial and error and risk degenerate behavior over long horizons. An agent can learn at inference time by incrementally discovering these programs and equipping them for future tasks. We hypothesize that past trajectories contain enough signal to guide skill learning, even without replay or validation, provided the agent can learn to analyze them. To test our claims, we propose SpeedRunner, a coding agent that analyzes trajectories and refactors skills for better performance on future tasks. Across three different embodied environments, we show that SpeedRunner consistently achieves the frontier in learning and cost reduction while remaining robust against distribution shifts and environmental randomness.
Large language model (LLM) agents can continually improve without parameter updates by converting historical experience into reusable procedural knowledge. However, existing methods often consolidate experience based on semantic similarity or LLM judgments, which may merge superficially related but behaviorally incompatible strategies and thereby degrade performance. To address the issue, we propose SkillCommit, an online skill evolution framework that continuously transforms experience into a hierarchical library of reusable skills. Rather than generalizing from individual experiences immediately, SkillCommit first preserves each experience as an instance-specific patch, retaining its locally validated behavior. As patches accumulate, SkillCommit abstracts those sharing a common behavioral mechanism into higher-level skills. Specifically, for each incoming skill, embedding-based retrieval first identifies candidate related skills. Cross-instance replay and an LLM-based mechanism check determine whether these skills transfer across cases and share a common underlying mechanism. Candidates that pass both checks are abstracted into a higher-level skill and committed only if it preserves the validated behavior of all constituent skills. Experiments on RuleArena, OpenExempt and KOR-Bench demonstrate that SkillCommit consistently improves agent performance across diverse domains. Moreover, the learned skills transfer across model scales and families, enabling cross-model experience transfer.
Yu He, Weikai Yang
Nanjing University · The Hong Kong University of Science and Technology (Guangzhou)