Existing world models typically perceive a scene through current visual frame or a narrow history window. However, these observations may not reveal the physics properties that determine transition dynamics, such as an object's mass distribution, joint damping, or surface friction. Without calibrating these physics factors for world models, predictions tend to regress toward an empirical prior and induce severe drift in novel test scenes. To this end, we propose PhysiCalWorld, which calibrates action-conditioned world models to the instance-specific dynamics via in-context prompting, without fine-tuning or analytical physics modeling. Our method encodes an exploratory interaction clip collected within the test scene into a compact physics descriptor token, which grounds subsequent predictions in the scene's actual physical behavior. PhysiCalWorld is trained end-to-end with the latent prediction loss alone, without physics labels. At inference time, the descriptor token is extracted once and can be reused across rollouts with negligible compute overhead. We evaluate PhysiCalWorld across four environments covering diverse physics variations and show that it substantially improves rollout prediction accuracy and visual planning success rate. The learned descriptors also form a physically structured latent space and generalize beyond the physics ranges seen during training.
Figures & tables
Figure 1: In-Context Physics Calibration for World Models. In a test environment with unknown transition dynamics, an embodied agent first executes exploratory actions to collect an in-context interaction sequence. This sequence is then provided as prompt to calibrate the world model such that future predictions adhere to the actual dynamics of the test environment.
Figure 2: Method Overview. PhysiCalWorld calibrates action-conditioned world models to the actual dynamics of a test scene via prompting with in-context interaction. In (a) , the current visual observation alone cannot identify the physical dynamics of the test scene T , forcing the world model to guess among multiple potential future states. In (b) , starting from the top left, exploratory actions a1:Tcc are first executed in the test scene T , yielding observations o1:Tcc . The context encoder gϕ then extracts physics information from this obs-action sequence and compresses it into a physics descriptor token zphy . Conditioned on zphy , the world model is able to generate future predictions o^t+1 consistent with the test scene’s dynamics.
Figure 3: Environment Demonstrations. We evaluate our method on four simulation environments featuring diverse variations of physics properties. For each environment, we show four images: the top-left image is a random initial observation, while the remaining three images demonstrate how executing the same action sequence yields different future states due to different physics parameters of the scenes. Specifically, in (a) Push-T, the center of mass of the T-block is varied; in (b) Reacher, the joint damping of the two-link arm is varied; in (c) BeerPong, the energy dissipation upon table contact is varied; in (d) FetchSlide, three physics properties vary concurrently: puck mass, surface friction, and aerodynamic drag.
Figure 4: Qualitative Open-Loop Rollout Results in Push-T and BeerPong. We show a subset of the full rollouts due to limited space and crop the synthesized images in BeerPong for clearer view. We use masks to indicate the ground-truth location of the target objects. Note that the physics configuration of these test scenes is unknown to the models and PhysiCal calibrates the predictions solely based on a given in-context interaction clip ( Figure A2 and Figure A3 ), yielding results that closely match the ground truth.
Figure 5Figure 6
Figure 8: Extrapolation to Out-of-Distribution Physics Parameters in Reacher. We report results of open-loop rollouts. For each circle, we sample 15 damping values within a bin centered at its x -coordinate evaluated on 5 episodes per value, 75 total.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
ResNet GroupNorm Groups
8
Spatial Projection Channels
64
Number of Spatial Keypoints
64
Visual Feature Dimension
128
Action Embedding Dimension
10
Proprioceptive Embedding Dimension
10
Appendix
Table A2: Hyperparameters of the Context Encoder gϕ . We use this set of hyperparameters for all our experiments. Note that similar to DINO-WM ( Zhou et al., 2025 ) , we apply a frameskip for the input context clip, such that the duration of the 12-frame context spans several seconds in the environments. See Table A5 for per-environment setup.
Hyperparameter
Value
Projection Intermediate Channels
192
Projection Output Channels
8
Projected Feature Map Shape
16×16×8
Curvature Loss Weight λcurv
0.1
Projection Head Learning Rate
1×10−5
Predictor Learning Rate
5×10−4
Appendix
Table A3: Hyperparameters of the Temporal Straightening Baseline (TS-WM). We use the DINOv2 (patch) + projection variant and follow the hyperparameters used in the original paper ( Wang et al., 2026d ) .
Hyperparameter
Push-T & Reacher
FetchSlide & BeerPong
Population size ( N )
300
64
Elite count ( Melite )
30
8
CEM iterations ( J )
30
8
Planned action length ( T )
5 model steps (25 actions)
1 initial strike / throw command
Appendix
Table A4: Hyperparameters for Visual Latent Planning with CEM. FetchSlide and BeerPong use a smaller optimization budget since they only optimize significantly shorter action sequences.
Push-T
Reacher
Method
25
50
75
25
50
75
TS-WM
53.3
54.7
56.0
41.3
65.3
68.0
DINO-WM
53.3
54.7
56.0
52.0
76.0
82.7
+ Long History
54.7
56.0
58.7
56.0
92.0
96.0
+ Context Prefix
58.7
60.0
60.0
61.3
84.0
92.0
+ Estimated Physics †
57.3
62.7
65.3
69.3
85.3
90.7
Appendix
Table A7: MPC success rates on Push-T and Reacher under different maximum numbers of control steps. Max. control steps denotes the maximum number of control steps allowed across all replanning iterations. Incorporating PhysiCal substantially improves both DINO-WM- and LeWM-based models across all allowed numbers of replans.
Figure A1: Success-threshold sensitivity. Evaluation of planning success rates across varying error tolerances for DINO-WM variants (top) and LeWM variants (bottom). (a) Reacher (DINO-WM-based models) using maximum absolute joint-angle error across joints. (b) FetchSlide (DINO-WM-based models) using terminal puck-XY position error. (c) Reacher (LeWM-based models) across joint-angle tolerances. (d) FetchSlide (LeWM-based models) across puck-XY position tolerances.
Figure A2: Qualitative Example of an In-Context Interaction Clip Ceval used in the Push-T Environment. We provide the in-context interaction clip used by +PhysiCal (Ours) to generate the main-text rollout results in Figure 4 (a). This clip is collected via the same policy (i.e., a CoM-agnostic PPO policy) that is used to generate the training dataset. This clip is encoded by the context encoder gϕ into a physics descriptor token , which is used to calibrate the world model’s predictions.
Figure A3: Qualitative Example of an In-Context Interaction Clip Ceval used in the BeerPong Environment. We show the in-context interaction clip used by +PhysiCal (Ours) to produce the main-text rollout results in Figure 4 (b). This clip is collected by a randomly sampled throwing action. The context encoder gϕ subsequently takes this clip as input and encodes it into a physics descriptor token zphy . As zphy carries the test scene’s transition dynamics inferred from the interaction clip, it can effectively ground the world model’s future predictions in the actual physical dynamics.
Figure A4: Qualitative Example of an In-Context Interaction Clip Ceval in the FetchSlide Environment. We present the in-context interaction clip used by PhysiCalWorld to generate the main-text prediction results in Figure 6 . This clip is collected via a random strike action and is encoded by the context encoder gϕ into a physics descriptor token , which is used to calibrate the world model’s predictions.
Figure A5: Uncropped Open-Loop Rollouts in the BeerPong Environment. We present the full-scene view of the BeerPong rollouts shown in Figure 4 (b), displaying the predicted ball trajectories relative to the table and cup. DINO-WM + Ours accurately predicts the ball’s bounce, while DINO-WM and DINO-WM + Context Prefix overestimate its rebound height. The semi-transparent cyan overlay marks the ground-truth ball position.
Figure A6: Qualitative Open-Loop Rollouts in the Reacher Environment under Out-of-Distribution Physics Configurations. We present an extrapolation example corresponding to the quantitative results in Figure 8 , with the first joint’s damping outside the training range and the second joint’s damping held fixed. Both models predict future observations autoregressively without additional ground-truth observations (i.e., feedback from the environment). DINO-WM + Privileged Physics predicts accurately from t=0 to t=6 , but background blurring emerges at t=7 and progressively intensifies through t=10 . From t=11 to t=19 , its predictions collapse into a nearly uniform blue field, with no discernible arm or scene structure. In contrast, DINO-WM + Ours preserves the scene structure and continues to predict the arm’s motion throughout the rollout. The corresponding interaction clip is shown in Figure A7 .
Figure A7: In-Context Interaction Clip Ceval used in Figure A6 . This clip is collected using randomly sampled actions (the action space is the torques applied at the two joints of the robot arm).
Figure A8: Qualitative Open-Loop Rollouts in the Push-T Environment. We present open-loop rollouts, where models are only given the initial observation and autoregressively predict future observations. All context-conditioned models use the interaction clip shown in Figure A9 . DINO-WM + Ours, LeWM + Ours, DINO-WM + Privileged Physics, and DINO-WM + Context Prefix closely match the ground-truth motion, while the remaining baselines exhibit substantial prediction errors. Though Context Prefix predicts accurately, it directly attends to the full context sequence, making its inference substantially slower.
Figure A9: Qualitative Example of an Exploratory In-Context Interaction Clip Ceval in the Push-T Environment. This in-context interaction clip is used by PhysiCalWorld to generate the open-loop rollout results in Figure A8 . The clip is collected using a CoM-agnostic PPO policy and encoded by the context encoder gϕ into a physics descriptor token to calibrate the world model’s predictions.
Figure A10: Qualitative Open-Loop Rollouts in the Reacher Environment. We show autoregressive predictions of the arm’s motion without providing further ground-truth observations during the rollout. All the models that ingest a context clip use the same one as presented in Figure A11 . DINO-WM + Ours, LeWM + Ours, and DINO-WM + Privileged Physics accurately predict the arm’s motion. Context Prefix overestimates the motion, whereas the remaining baselines underestimate it, suggesting that it uses the context to adjust its predictions but remains less accurate than our models.
Figure A11: Qualitative Example of an In-Context Interaction Clip Ceval in the Reacher Environment. We provide the in-context interaction clip used by PhysiCalWorld to produce the open-loop rollout results in Figure A10 . Generated by randomly sampled joint-torque commands, this clip is encoded by the context encoder gϕ into a physics descriptor token , which is used to calibrate the world model’s predictions.
Figure A12: Qualitative Open-Loop Rollouts in the BeerPong Environment. We provide open-loop predictions of the ball’s trajectory, generated autoregressively without subsequent ground-truth observations. All models that use context receive the interaction clip in Figure A13 . DINO-WM + Ours, LeWM ∗ + Ours, DINO-WM + Privileged Physics, and DINO-WM + Context Prefix accurately reproduce the ball’s bounce, while the remaining baselines overestimate its rebound. The accuracy of Context Prefix, however, comes at the cost of substantially slower inference due to direct attention over the full context sequence. LeWM ∗ denotes TC-LeWM in BeerPong, as we find that this variant can significantly boost the performance of LeWM in BeerPong.
Figure A13: Qualitative Example of an Exploratory In-Context Interaction Clip Ceval in the BeerPong Environment. We present the in-context interaction clip used by PhysiCalWorld to generate the open-loop rollout results in Figure A12 . The context encoder gϕ encodes this clip, collected using a randomly sampled throw action, into a physics descriptor token that calibrates the world model’s predictions.
Figure A14: Qualitative Open-Loop Rollouts in the FetchSlide Environment. We present an eight-frame open-loop rollout, in which each model predicts future observations autoregressively without receiving additional ground-truth observations. The interaction clip shown in Figure A15 is shared by all context-conditioned models. DINO-WM + Ours, LeWM + Ours, and DINO-WM + Privileged Physics closely match the ground-truth puck motion. DINO-WM + Context Prefix slightly underestimates its movement, while the remaining baselines without context exhibit more pronounced underestimation.
Figure A15: Qualitative Example of an Exploratory In-Context Interaction Clip Ceval in the FetchSlide Environment. We provide the in-context interaction clip used by the context-conditioned methods to generate the results in Figure A14 . This clip is collected using a randomly sampled strike action and subsequently encoded by the context encoder gϕ into a physics descriptor token , which is used to calibrate the world model’s predictions.
Figure A16: Qualitative Planning Rollouts in the Push-T Environment, DINO-WM-Based Models. We use CEM to optimize action sequences toward a goal observation and execute them in the environment, comparing model predictions with the resulting motion. All context-conditioned models use the interaction clip shown in Figure A17 . DINO-WM + Ours closely matches the executed motion, while the remaining models misrepresent the T-block’s translational and rotational dynamics.
Figure A17: Qualitative Example of an In-Context Interaction Clip Ceval in the Push-T Environment. We provide the in-context interaction clip used by PhysiCalWorld to generate the planning results in Figure A16 . Collected using a CoM-agnostic PPO policy, this clip is encoded by the context encoder gϕ into a physics descriptor token , which is used to calibrate the world model’s predictions during planning.
Figure A18: Qualitative Planning Rollouts in the Push-T Environment, LeWM-Based Models. Action sequences generated by CEM to reach a goal observation are executed in the environment and compared with the world models’ predictions. LeWM + Ours uses the interaction clip presented in Figure A19 and accurately predicts the resulting motion. In contrast, LeWM underestimates the effect of the push at t=3 , with the T-block moving and rotating substantially farther than predicted at t=4 .
Figure A19: Qualitative Example of an Exploratory In-Context Interaction Clip Ceval in the Push-T Environment. We show the in-context interaction clip used by PhysiCalWorld to produce the planning results in Figure A18 . This clip is collected using a CoM-agnostic PPO policy. The proposed method will use its context encoder gϕ to encode this clip into a physics descriptor token , which is used to calibrate the world model’s predictions during planning.
In outdoor environments, varying terrain and contact conditions can cause identical action commands to produce different motion outcomes for quadruped robots. Predicting these outcomes is essential for reliable navigation and informed path selection. Existing video world models can generate visually plausible futures, yet often struggle to capture robot dynamics and terrain interactions, leading to overly optimistic predictions of action outcomes. We present DR-WM, a physically consistent world model that incorporates robot-terrain contact dynamics to predict the consequences of quadruped actions. Starting from an initial RGB image, our method reconstructs scene geometry and integrates dense, dimensionless priors over terrain contact properties, calibrated using real-world action-response data. The resulting terrain model supports the simulation of physics with spatially varying contact properties such as friction. Given the initial state of the robot and candidate action sequences, we simulate its future motion and use the resulting camera trajectories and rendered geometry to guide the training of a video world model. The generated future observations are then used to update the terrain reconstruction, enabling iterative model rollouts without additional real-world images. Extensive experiments demonstrate that DR-WM substantially outperforms geometry-only baselines and baselines with uniform physical parameters on key physical metrics characterizing action outcomes, while achieving the best performance among the evaluated baselines. These physically grounded predictions support more informed action selection: across 90 challenging scenarios, an 8B vision-language model augmented with DR-WM outperforms an unassisted 32B vision-language model on five of six evaluation metrics.
Ruo Chen, Jiahua Pang, Yingjie Niu +4
DEEP Robotics · Zhejiang University · Beijing Institute of Technology +3
A central challenge in visual intelligence is learning the physical structure of scenes from raw videos: how regions form objects and the laws that govern their interactions. Solving these tasks requires world models capable of inferring distributional states of the world from partial observations - capabilities that current architectures do not provide. We introduce a new class of probabilistic world models that support estimation of the probability of any visual variable, such as appearance and dynamics, conditioned on any other variables. Here, we identify that these models can be trained efficiently with autoregressive sequence modeling, yielding world models from which rich object understanding emerges. First, we demonstrate that our model captures the physical laws governing how objects move by generating multiple plausible future states of the world through sequential inference. Then, by analyzing motion correlations across these futures, we extract objects and articulated object subparts. Having discovered these objects, we show that our world model can manipulate them in 3D. Finally, we demonstrate how physical relationships between objects can be computed from the world model, enabling applications such as Visual Jenga.
World models can predict video without learning dynamics that they reliably preserve. We test whether a frozen DreamerV3 trained only on pendulum video learns a scalar that its own latent transition treats as approximately conserved. A label-free search recovers the same energy-like invariant across independently trained conservative models, while the same procedure finds no comparable invariant in matched damped models. During autonomous rollouts, this quantity drifts. Projecting the latent state back toward its initial level set reduces rollout error in all three conservative models, whereas matched random constraints usually increase it. These results distinguish a dynamically meaningful invariant from a merely decodable correlate and reveal a concrete failure mode: a world model can learn a physical constraint from pixels yet violate that constraint when it imagines forward.