cs.LGOct 27, 2026
SaveUnderstanding the limits of latent world models
Organizations: Helmholtz Munich Munich Center for Machine Learning (MCML)
Abstract
World models are widely proposed as core components for enabling agents to reason over long temporal and spatial horizons. Yet current evaluations often emphasize task performance while providing limited insight into whether learned world models acquire predictive latent dynamics that reflect the underlying structure of the environment. Here we introduce the concept of faithful latent world models as an equivariance condition between the environment's transition map and the learned latent dynamics. We propose latent rollout consistency (LRC) as a metric to evaluate action-conditioned latent prediction. Strikingly, we find that across the studied state of the art latent world models, none solves the task by actually faithfully modeling the dynamics of the environment.
Figures & tables
Figure 1: Faithful latent dynamics and their evaluation. (a) In a faithful latent world model, applying the true transition and then encoding ( ) gives the same result as encoding first and applying the learned latent dynamics under the same action. (b) Latent rollout consistency (LRC) evaluates this by comparing autoregressive latent predictions with representations inferred from the corresponding realized observations.
Figure 2: A. identity encoding, noisy dynamics; B. nonlinear encoding, noisy dynamics; C. noisy encoder, exact dynamics. LRC at , linear , nonlinear .
Figure 3: Experimental setup. (a) Three world model families: TD-MPC2 and DreamerV3 (task-specific) and LeWM (dynamics learning + MPC), grouped by reconstruction and training objective. (b) Three DeepMind Control Suite environments—Pendulum, CartPole, and Reacher—evaluated with state and pixel observations.
Figure 4: Latent rollout consistency (LRC) vs. Reward. (a) LRC vs. reward averaged across environments. The 95% CI is estimated by bootstrap over model seeds. (b) LRC over rollout horizons. (c) Position-decoding errors for state- and pixel-based models across environments, normalized by a shuffled-target baseline. (d) Velocity-decoding errors, shown as in (c).
Figure 5: CartPole rollouts and state-dependent prediction errors. (a) Predicted latent rollouts are mapped to physical state using the frozen nonlinear probes. (b) Example pole-angle trajectories for DreamerV3, TD-MPC2, and LeWM, with the recorded actions shown above each trajectory. (c) Evaluation-set occupancy over pole angle and angular velocity (top) and cart position and velocity (bottom). (d) Prediction errors over the same state spaces for models trained from states or pixels. Color denotes the error within each bin relative to the median prediction error, and the annotation reports the median error in physical units.
Figure 6: Velocity observability and world-model prediction. (a) Ground-truth velocity is fused with the visual representation. (b–c) Example CartPole rollouts with and without velocity for TD-MPC2 and LeWM. (d–f) LRC over the rollout horizon for DreamerV3, TD-MPC2, and LeWM. Full results are reported in Table 1 .
Table 1: LRC at and , reward ( ), and normalized velocity error ( ); add ground-truth velocity. LRC and errors are in percentage units. Bold: or LRC .
Figure 7: World modeling under nonlinear observation mixing on CartPole. (a–b) Pixel rendering is contrasted with approximately invertible mixing of the underlying state. (c–f) LRC at , task reward, normalized position error, and normalized velocity error for LeWM versus mixing depth, with pixels shown as a reference.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Env. | State variables | Task | ||
|---|---|---|---|---|
| Pendulum (pendulum-swingup) | 3 | 1 | , , pole angular velocity | Swing the pole upright from below and balance it. |
| CartPole (cartpole-swingup) | 5 | 1 | cart position, / of pole angle, cart and pole velocities | Swing the pole upright from below and balance it by moving the cart. |
| Reacher (reacher-hard) | 6 | 2 | two joint angles, two joint velocities, 2D target offset | Bring the fingertip of a two-link arm to a small randomized target. |
Table 2: Evaluation environments and the underlying physical state used as probing targets.
Figure 8: Pendulum data distributions. Marginal distributions of the orientation coordinates, angular velocity, action, and native dm_control reward in the random OU and expert datasets.
Figure 9: CartPole data distributions. Marginal distributions of cart position, pole angle, cart and pole velocities, action, and native dm_control reward in the random OU and expert datasets.
Figure 10: Reacher data distributions. Marginal distributions of the joint positions, target-offset coordinates, joint velocities, two action coordinates, and native dm_control reward in the random OU and expert datasets.
Figure 11: Latent rollout consistency across environments and model families. LRC over the rollout horizon for TD-MPC2, DreamerV3, and LeWM (columns) on Pendulum, CartPole, and Reacher (rows). Curves compare models trained from ground-truth states and pixels with the persistence baseline; shaded regions show variation across model seeds.
Figure 12: Task reward and latent rollout consistency by environment. Episode reward versus LRC at horizon for (a) Pendulum, (b) CartPole, and (c) Reacher. Marker shape identifies the model family and color identifies whether the model receives ground-truth states or pixels; error bars show variation across model seeds.
Figure 13: Marginalizing DreamerV3’s stochastic transitions. LRC over the rollout horizon on Pendulum, CartPole, and Reacher using the deterministic RSSM rollout, the coordinate-wise mean of 500 stochastic rollouts generated from a shared posterior anchor, and the best-matching sampled rollout. The latter uses the realized trajectory to select a sample and therefore represents an oracle upper bound rather than a deployable estimator. Shaded regions show variation across model seeds.
| Model | Obs. type | Seeds | Notes | |
|---|---|---|---|---|
| TD-MPC2 | state | 4 | 512 | Single-task, per-environment |
| pixels | 4 | 512 | Single-task, per-environment | |
| DreamerV3 | state | 4 | 1536 | RSSM posterior: stoch (1024) + deter (512) |
| pixels | 4 | 5120 | Same architecture, visual encoder | |
| LeWM | state | 4 | 192 | Our state-input implementation |
| pixels | 4 | 192 | Original LeWM, frames |
Table 3: World models evaluated in this work. Latent dimension refers to the size of the activation vector used for all downstream analyses.
Figure 14: Training dynamics of the evaluated world models. The top two rows show episode reward over environment steps for TD-MPC2 and DreamerV3 trained from state and RGB observations on CartPole and Reacher. The bottom row shows LeWM’s prediction and SigReg training losses over epochs on the same environments. Curves show means across model seeds and shaded regions show variation across seeds.
Figure 15: Reproduction of LeWM performance on Reacher. The original study reports an 86% success rate for pixel-based LeWM. Our reproduced checkpoints achieve 81.5%, compared with 11.5% for a random policy. Bars show means across four seeds, dots show individual seed means, and whiskers show 95% hierarchical bootstrap confidence intervals over seeds and episodes. The published result is included as a fixed reference without uncertainty.
Figure 16: LeWM cost and planner comparison. Terminal latent cost and decoded state cost for pixel-input (pink) and state-input (gray) LeWM. Planners are random shooting with 50 or 200 candidates and CEM with 4 or 10 iterations of 200 candidates. Reacher compares two CEM settings. Reported LeWM returns use the state cost and the planner with the highest reward. Bars are means over model seeds; error bars are the standard error. Each bar uses 50 evaluation episodes.
Figure 17: State-decoding errors in physical units. Held-out nonlinear-probe errors for models trained from ground-truth states or pixels, reported separately for Pendulum, CartPole, and Reacher. Panels separate the positional and velocity coordinates of each environment and report errors in degrees, centimeters, meters per second, or degrees per second as appropriate. Colored markers show individual model seeds; black markers and error bars summarize performance across seeds.
Figure 18: Linear and nonlinear state decoding. Normalized decoding error for positional variables (left) and velocities (right), averaged across environments for each model family. Bars compare linear probes with nonlinear MLP probes; error bars show variation across environments. The dashed line marks an error equal to 50% of the shuffled-target control.
Figure 19: Nonlinear state mixing across world-model families. LRC at , task reward, normalized position error, and normalized velocity error on CartPole as the mixing depth increases from zero to three layers. Results are shown for TD-MPC2 (blue circles), LeWM (orange squares), and DreamerV3 (green diamonds); Pixels denotes each model’s standard pixel-input condition. Error bars show variation across model seeds.
Figure 20: LeWM controllers on the CartPole interventions. Top: pixel models with one or three frames, with and without ground-truth velocity. Bottom: state-input models after zero to three nonlinear mixing layers. Columns compare the terminal latent cost with the decoded state cost across the planners in Figure 16 . Bars are means over model seeds; error bars are the standard error. Each bar uses 50 evaluation episodes.
More in this topic
World Models
World models represent and predict how environments change, helping agents plan, act, and reason about possible futures.
See all 366 papers