cs.AIOct 9, 2026
SaveScaling Empowered Agents
Organizations: Department of Physics Princeton University · Department of Computer Science UC Berkeley · Department of Mathematics Princeton University · Department of Computer Science Princeton University
Abstract
Empowerment measures the capacity of an agent to actively control its environment. In this work, we explain the connection between skill learning and empowerment. While skill learning is not empowerment, we introduce a simple scalable recipe for empowerment maximization: empowerment corresponds to the number of executable skills from a state, so maximizing empowerment corresponds to reaching states that are good starting points for many skills. Our recipe applies to arbitrary underlying skill learning methods. Whereas most prior work on empowerment looks at tabular settings, this recipe enables us to study empowerment at a scale where we can realize theoretically-predicted but empirically-elusive properties of empowerment (e.g., gathering resources, environment modification, survival). Experiments show that empowerment maximization unlocks interesting emergent properties, such as building a house in Crafter. Website and code at this https url.
Figures & tables
Figure 1: Empowerment-maximizing behavior across environments. (a) In the BuilderBench task, a gripper interacts with blocks in an open-ended setting. The potential empowerment policy is visualized with the white rollout, navigating to the blocks and staying there. Then, the effective empowerment policies, or skills, knock the blocks in different directions (colorful lines and blocks). (b) In the first version of the AntSoccer task, an agent can be empowered without interacting with the ball. The potential empowerment policy (white) navigates towards the central area of the arena, while the skills branch out from this point. (c) In the second version of the AntSoccer task, an agent can only be empowered if it controls the ball. The trained potential empowerment policy reaches the soccer ball, while skills kick the soccer ball in different directions. (d) In Craftax-Classic, the potential empowerment agent gathers materials, builds a crafting table, then builds tools using gathered materials.
Figure 2: The potential empowerment estimate increases with control in Crafter. (a) The learned empowerment estimate increases consistently during training. seeds, p.m. standard error after per-seed EMA ( ). (b) Simultaneously, the agent intrinsically learns to complete achievements in the environment without access to external rewards . seeds, evaluated over random environments, p.m. standard error.
Figure 3: Potential empowerment and control increases over training (continuous settings). We compare SEE (Black) and baselines VIM (Orange) and LatentGCE (Green). (a) SEE visits diverse states across all environments, which is necessary to learn correct models and decoders of dynamics and skills. (b) SEE consistently learns potential empowerment policies that navigate towards central regions of the arena in Ant, to the ball in AntSoccer, and to the blocks in BuilderBench. VIM is unstable and largely stationary in Ant and AntSoccer. LatentGCE is stationary in Ant. Five seeds for SEE and LatentGCE and three for VIM, p.m. standard error. Eval metrics averaged over randomized environments.
Figure 4: Potential empowerment policies accrue empowerment, while skills spend it. (Top row) Over the course of the trajectory, the potential empowerment policies (white lines) “gather” empowerment, reaching empowered states. Correspondingly, the empowerment estimates increase over the course of the trajectory. (Bottom row) At the gray skill switch mark, executing a skill (colorful lines) from the reached state “spends” the empowerment, reflecting the intuition that potential empowerment policies collect power and effective empowerment spends power. Traces use one seed and four skill branches per environment.
Figure 5: Empowerment maximization has interesting emergent behaviors. (a) In Crafter, the empowered agent quickly accumulates many resources and uses a tool (crafting table, wooden icon with saw) to craft other tools in the game. Resources boxed in white. (b) Early in training, the agent learns to build a “house” consisting of plants and crafting tables. Built tiles boxed in white. (c) The agent also naturally learns to survive, avoiding and killing enemies like zombies and skeletons. One seed per example; further qualitative examples across seeds in Appendix E .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Algorithm 1 SEE
| Parameter | Ant | AntSoccer | BuilderBench |
|---|---|---|---|
| Encoder observations | Full state (42) | Ball (2) | Block positions (9) |
| Observation noise std | 0.05 | 0.05 | 0.01 |
| Skill dimension | 2 | 2 | 9 |
| Online env steps per seed | 500M | 500M | 100M |
| Replay capacity (transitions) | 10M | 10M | 6M |
| Batch size | 1,024 | 1,024 | 256 |
Table 1: Hyperparameters for SEE with joint MI in continuous environments.
Figure 6: Achievements in a single Craftax-Classic two-phase episode. Icons indicate when achievements first occur. (a) The black line shows the achievements of the shared potential phase up to the skill switch (vertical gray line). In this rollout, these achievements include collecting wood, building a crafting a table, then crafting a wooden sword. The colorful skill rows show the achievements rolled out from the switch point sharing the same starting state, including sleeping/waking up, building a pickaxe, planting a tree, collecting stone, building a furnace, and defeating an enemy. (b) The lines indicate the cumulative achievement counts over the rollout, achieved without the use of any extrinsic reward. One seed.
| Parameters | Value |
|---|---|
| Encoder observations | inventory (12 features) |
| Encoder data | CMI branches from copied states |
| Observation noise std (classifier bottleneck) | 0 |
| Future-sampling discount | |
| Logit temperature | 0.1 (fixed) |
| Encoder learning rate | 1e-3 |
Table 2: Hyperparameters in Craftax-Classic (SEE, CMI (Ours); quantitative curves).
Figure 7: Empowerment fields on low-dimensional benchmarks. For the canonical environments of a pendulum and a ball in a box with hard walls, various empowerment maximization methods predict (normalized) empowerment fields over state space. (a) A linearized reference reveals that the potential empowerment is maximized when the pendulum points straight up. For the Ball in Box, the potential empowerment is maximal at the center of the box. Thus, a potential empowerment policy over a long effective horizon should seek a straight-up state in the pendulum and the central state for Ball in Box. (b) The empowerment field predicted by LatentGCE method correctly identifies these potential empowerment states. (c) VIM does not recover these high empowerment regions. One seed per learned field.
| Parameter | Value |
| Training budget (Ant / AntSoccer / BuilderBench) | 500M / 500M / 100M |
| Episode length / probe horizon | 400 / 30 steps |
| Parallel environments / rollout length | 256 / 80 steps |
| Probe starts per update | 256, one uniformly sampled rollout time per environment |
| PPO controller | |
| Actor / critic hidden layers | [256, 256], tanh |
Table 3: VIM hyperparameters for Ant, AntSoccer, and BuilderBench. Each environment uses 3 training seeds.
| Parameter | Value |
| Training budget (Ant / AntSoccer / BuilderBench) | 500M / 500M / 100M |
| Episode length / probe horizon | 400 / 100 steps |
| Parallel environments / rollout length | 256 / 80 steps |
| Probe starts per update | 256, one uniformly sampled rollout time per environment |
| PPO controller | |
| Actor / critic hidden layers | [256, 256], tanh |
Table 4: LatentGCE hyperparameters for Ant, AntSoccer, and BuilderBench. Each environment uses 5 training seeds.
Figure 8: Learned empowerment in continuous settings. SEE (Black) empowerment values gradually increase over time. Meanwhile, VIM (Orange) empowerment values are unstable and are very large at the start of training (Orange arrows point beyond the displayed y-axis). The LatentGCE (Green) empowerment values do not monotonically increase in time. *As a note, learned estimates use different objectives and regularization, so their magnitudes are not directly comparable across methods. Five seeds for SEE and LatentGCE and three for VIM, p.m. standard error.
Figure 9: We compare the full method with variants that remove the two-phase rollout (no switching, orange), replace joint MI with conditional MI (green), or replace the bilinear logit with a general MLP (pair decoder, purple). The solid black curve shows SEE, and the colored dashed curves show the ablations. (a) For all methods, average empowerment increases over training. ∗ Learned empowerment estimates use different objectives and regularization, so their magnitudes are not directly comparable across methods. (b) The skill visitation of SEE quickly saturates for these environments. Meanwhile, the ablations struggle to get visitation in a subset of the different environments. (c) SEE consistently learns potential policies that reach the center of the arena (Ant), approach the ball (AntSoccer), and reach the blocks (BuilderBench). In BuilderBench, conditional MI and the pair decoder potential empowerment rollouts reach the blocks, while no switching ends far away. In Ant, all ablations end far from the center of the arena. In AntSoccer, the ablations end up further from the ball than SEE. (d) SEE consistently learns effective policies with large displacements. Five seeds for SEE; three per ablation.
Figure 10: Craftax-Classic ablations (CMI SEE: 5 seeds; each ablation: 3). The black curve is the SEE with the CMI classifier (Ours). The ablations are SEE with Joint MI (Ours), no switching (25% potential-policy episodes during training), and pair decoder. (a) Learned empowerment estimates rise over training for the base method. (b) Total achievements per rollout. (c) Achievements in the first 100 potential-policy actions. (d) New achievements in the subsequent skill phase of up to 900 actions. All methods use 128 fresh worlds and skills per seed and evaluation checkpoint. Curves show mean standard error across training seeds, with per-seed EMA ( ) only in (a). Achievement panels are unsmoothed and display every tenth evaluation checkpoint plus the final checkpoint.
Figure 11: Reward normalization in continuous environments. We compare the annealed schedule with normalized ( ) and unnormalized ( ) rewards. Columns show learned empowerment, training skill visitation, target distance after 100 potential-policy actions, and displacement during the subsequent 300 skill actions. Learned estimates use different objectives and are not directly comparable across settings. Five seeds for SEE; three per ablation. Curves show mean standard error across seeds. Training curves use per-seed EMA ( ); evaluation curves are unsmoothed.
Figure 12: Reward normalization in Craftax-Classic. We compare CMI SEE with the annealed schedule against fixed and . Five seeds for SEE; three per ablation. Curves show mean standard error across seeds. The learned-estimate panel uses per-seed EMA ( ). Achievements use 128 fresh worlds and skills per seed and checkpoint, with 100 potential-policy actions followed by up to 900 skill actions. Achievement evaluations are unsmoothed; every tenth checkpoint and the final checkpoint are shown.
Figure 13: Additional environment-modification behaviors. Stills of environment modifications over the remaining seeds in Craftax-Classic. Each panel shows a selected rollout. White outlines box placed tiles.