Building World Models from Intrinsic Motivation
Organizations: Neuroscience Institute, Carnegie Mellon University · Robotics Institute, Carnegie Mellon University · Department of Computer Science, University of Maryland, College Park · Machine Learning Department, Carnegie Mellon University
Abstract
In the absence of extrinsic rewards or task supervision, human behavior is guided by intrinsic motivation---neural signals that drive curiosity, creativity, play, and other essential facets of intelligence. Intrinsic motivation plays an important developmental role in building robust world models that enable generalization and adaptation. Yet existing studies leave open how a broad range of intrinsic motivations interacts with world-model representation learning to support downstream control. This limits our understanding of which intrinsically collected experiences build reusable models. To address these gaps, we integrate 24 intrinsic objectives, including established methods and explicit variants spanning curiosity, entropy, and competency, into a common model-based reinforcement learning framework. We evaluate the world models they build through zero-shot, few-shot, and multi-task control in locomotion, manipulation, and open-ended gaming environments, using three representation-learning strategies: observation reconstruction, latent prediction, and redundancy reduction. Curiosity-driven exploration with observation reconstruction yields the strongest aggregate family profile, although the best individual pairing depends on the experiment. In compositional vision-based tasks, intrinsic motivation derived from both episodic and lifelong novelty---providing a complementary memory to the world model's generic recurrent state---achieves the best zero-shot transfer and few-shot adaptation. These findings motivate designing intrinsic motivation and representation learning jointly. Our results demonstrate world models as a powerful tool for measuring the quality of intrinsically motivated experience, and positions intrinsic motivation as a key ingredient for building world models autonomously.
Figures & tables
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Setting | Small (12M preset) | Large (200M preset) |
|---|---|---|
| Deterministic RSSM state | 2,048 | 8,192 |
| RSSM hidden width | 256 | 1,024 |
| Stochastic variables | 32 | 32 |
| Categories per variable | 16 | 64 |
| MLP width | 256 | 1,024 |
| CNN base depth | 16 | 64 |
| Method | Objective and defining implementation choices |
|---|---|
| Vanilla Curiosity ( Pathak et al., 2017 ) | Local prediction-error baseline using the Euclidean error of an action-conditioned next-RSSM-feature predictor. Unlike ICM, it has no learned intrinsic feature projector; its auxiliary inverse head does not train the shared world-model representation. |
| ICM ( Pathak et al., 2017 ) | Forward prediction error in features learned jointly with inverse dynamics. A latent MLP replaces the original visual encoder and operates above stopped RSSM features; forward-loss weight 0.2 and reward coefficient 0.01. |
| Disagreement ( Pathak et al., 2019 ; Sekar et al., 2020 ) | Mean ensemble variance across five bootstrapped action-conditioned next-feature predictors. We retain the Disagreement/Plan2Explore novelty principle in RSSM space; ReLU predictors concatenate actions at input, with reward scale 10,000 and no log or internal reward normalization. |
| CIG ( Joseph et al., 2026 ) | Rollout-prefix-conditioned information-gain surrogate: per-step rewards are log-determinant increments of a regularized ensemble-disagreement Gram matrix. We use RSSM targets and five independently initialized predictors trained on the same replay transitions without bootstrapping; four 400-unit ELU layers, ridge multiplier 1, and 0.99 aleatoric/reward moving averages. |
| MAX ( Shyam et al., 2019 ) | Jensen–Rényi divergence among probabilistic next-feature predictions, using MAX’s ensemble novelty principle within Dreamer imagination. The plotted fast preset uses five predictors with two 256-unit ReLU layers and log-variance bounds , replacing the larger source ensemble. |
| RND ( Burda et al., 2019 ) | Prediction error against a fixed random 512-dimensional target mapping. MLP target/predictor networks operate on RSSM features, and rewards are divided by a running mean; the source visual encoder and PPO normalization pipeline are replaced. |
| Method | Objective and defining implementation choices |
|---|---|
| APT ( Liu and Abbeel, 2021 ) | APT-style particle-entropy reward from 16 nearest neighbors in a normalized 256-dimensional projection, with clipping 0.0005. This local variant trains its projector to increase neighbor novelty instead of using the source contrastive loss; persistent feature memory and running-mean normalization are additional adaptations. |
| RE3 ( Seo et al., 2021 ) | Novelty from the distance to the 53rd-nearest neighbor in a fixed random embedding. Our RSSM-based recipe uses a 50-dimensional MLP, coefficient 0.1, and RMS-normalized unlogged Euclidean distances against persistent replay-feature memory. |
| Proto-RL ( Yarats et al., 2021 ) | Prototype-assignment prediction with novelty measured by distance to the third-nearest candidate in a feature queue. MLPs above RSSM states replace the source visual representation learner; 128-dimensional features, 512 prototypes, four candidates per prototype, temperature 0.1, three Sinkhorn iterations, and target momentum 0.95. |
| SMM ( Lee et al., 2019 ) | Single-policy SMM-inspired novelty: negative reconstruction log likelihood plus 0.5 times the posterior KL, with the decoder evaluated at the posterior mean. The VAE uses latent width 128 and two 150-unit ReLU layers. This is a novelty proxy, not an exact negative log-density or the source mixture-policy SM4 agent. |
| Method | Objective and defining implementation choices |
|---|---|
| DIAYN ( Eysenbach et al., 2019 ) | State-based skill discrimination, rewarding the sampled skill’s discriminator log likelihood minus its log prior probability. We use 50 uniform categorical skills held for an episode, two 300-unit discriminator layers, and batch size 128. Dreamer’s imagined actor–critic and entropy regularizer replace the source SAC learner. |
| VIC ( Gregor et al., 2017 ) | Initial/final-state option discrimination, with terminal reward equal to the selected option’s decoder log likelihood minus its log prior probability. We instantiate VIC’s option-information objective with 12 uniform categorical options and fixed 15-step duration matched to imagination. |
| VALOR ( Achiam et al., 2018 ) | Trajectory-based option decoding from 10 ordered differences between 11 equally spaced observations. We retain the trajectory-information objective but use an MLP decoder and 12 categorical 15-step options, replacing the source bidirectional recurrent decoder and growing option curriculum. |
| DADS ( Sharma et al., 2020 ) | Skill-conditioned next-feature log likelihood minus its log-mean likelihood under 100 prior-sampled alternatives. Skills are uniform on with collection horizon 200 and two 512-unit predictor layers; a diagonal Gaussian over RSSM features replaces the source mixture-density model. |
| rDADS ( Sharma et al., 2020 ) | Local reverse surrogate: skill log likelihood conditioned on the transition minus that conditioned on the current state, using two learned Gaussian skill densities. It shares DADS’s two-dimensional box prior and collection horizon 200, but changes the prediction direction and density estimators. |
| CIC ( Laskin et al., 2022 ) | Contrastive skill–transition representation learning with particle-neighbor novelty. The local module uses RSSM-transition MLPs and persistent feature memory; 64-dimensional skills on , horizon 50, 64-dimensional features, two 1,024-unit layers, temperature 0.5, 16 nearest neighbors, and batch size 1,024. |
Intrinsic Motivation
Intrinsic motivation studies internal objectives that drive agents to explore, learn skills, and seek informative experiences without explicit task rewards.
See all 42 papers