PhysiCalWorld: In-Context Physics Calibration for World Models

Yunhao Luo; Wentao He; Yinpei Dai; Hyelin Nam; Hongze Fu; Joyce Chai; Nima Fazeli; Jeong Joon Park

cs.ROOct 21, 2026

PhysiCalWorld: In-Context Physics Calibration for World Models

Authors: Yunhao Luo, Wentao He, Yinpei Dai, Hyelin Nam, Hongze Fu, Joyce Chai, Nima Fazeli, Jeong Joon Park

Organizations: University of Michigan

Abstract

Existing world models typically perceive a scene through current visual frame or a narrow history window. However, these observations may not reveal the physics properties that determine transition dynamics, such as an object's mass distribution, joint damping, or surface friction. Without calibrating these physics factors for world models, predictions tend to regress toward an empirical prior and induce severe drift in novel test scenes. To this end, we propose PhysiCalWorld, which calibrates action-conditioned world models to the instance-specific dynamics via in-context prompting, without fine-tuning or analytical physics modeling. Our method encodes an exploratory interaction clip collected within the test scene into a compact physics descriptor token, which grounds subsequent predictions in the scene's actual physical behavior. PhysiCalWorld is trained end-to-end with the latent prediction loss alone, without physics labels. At inference time, the descriptor token is extracted once and can be reused across rollouts with negligible compute overhead. We evaluate PhysiCalWorld across four environments covering diverse physics variations and show that it substantially improves rollout prediction accuracy and visual planning success rate. The learned descriptors also form a physically structured latent space and generalize beyond the physics ranges seen during training.

Figures & tables

Figure 1: In-Context Physics Calibration for World Models. In a test environment with unknown transition dynamics, an embodied agent first executes exploratory actions to collect an in-context interaction sequence. This sequence is then provided as prompt to calibrate the world model such that future predictions adhere to the actual dynamics of the test environment.

Figure 2: Method Overview. PhysiCalWorld calibrates action-conditioned world models to the actual dynamics of a test scene via prompting with in-context interaction. In (a) , the current visual observation alone cannot identify the physical dynamics of the test scene $\mathcal{T}$ , forcing the world model to guess among multiple potential future states. In (b) , starting from the top left, exploratory actions $a_{1:T_{c}}^{c}$ are first executed in the test scene $\mathcal{T}$ , yielding observations ${o}_{1:T_{c}}^{c}$ . The context encoder $g_{\phi}$ then extracts physics information from this obs-action sequence and compresses it into a physics descriptor token $\mathbf{z}_{\text{phy}}$ . Conditioned on $\textbf{z}_{\text{phy}}$ , the world model is able to generate future predictions $\hat{o}_{t+1}$ consistent with the test scene’s dynamics.

Figure 3: Environment Demonstrations. We evaluate our method on four simulation environments featuring diverse variations of physics properties. For each environment, we show four images: the top-left image is a random initial observation, while the remaining three images demonstrate how executing the same action sequence yields different future states due to different physics parameters of the scenes. Specifically, in (a) Push-T, the center of mass of the T-block is varied; in (b) Reacher, the joint damping of the two-link arm is varied; in (c) BeerPong, the energy dissipation upon table contact is varied; in (d) FetchSlide, three physics properties vary concurrently: puck mass, surface friction, and aerodynamic drag.

Figure 4: Qualitative Open-Loop Rollout Results in Push-T and BeerPong. We show a subset of the full rollouts due to limited space and crop the synthesized images in BeerPong for clearer view. We use masks to indicate the ground-truth location of the target objects. Note that the physics configuration of these test scenes is unknown to the models and PhysiCal calibrates the predictions solely based on a given in-context interaction clip ( Figure A2 and Figure A3 ), yielding results that closely match the ground truth.

Figure 8: Extrapolation to Out-of-Distribution Physics Parameters in Reacher. We report results of open-loop rollouts. For each circle, we sample 15 damping values within a bin centered at its

x

-coordinate evaluated on 5 episodes per value, 75 total.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
ResNet GroupNorm Groups	8
Spatial Projection Channels	64
Number of Spatial Keypoints	64
Visual Feature Dimension	128
Action Embedding Dimension	10
Proprioceptive Embedding Dimension	10

Appendix

Table A2: Hyperparameters of the Context Encoder

g_{\phi}

. We use this set of hyperparameters for all our experiments. Note that similar to DINO-WM ( Zhou et al., 2025 ) , we apply a frameskip for the input context clip, such that the duration of the 12-frame context spans several seconds in the environments. See Table A5 for per-environment setup.

Hyperparameter	Value
Projection Intermediate Channels	192
Projection Output Channels	8
Projected Feature Map Shape	$16\times 16\times 8$
Curvature Loss Weight $\lambda_{\mathrm{curv}}$	$0.1$
Projection Head Learning Rate	$1\times 10^{-5}$
Predictor Learning Rate	$5\times 10^{-4}$

Appendix

Table A3: Hyperparameters of the Temporal Straightening Baseline (TS-WM). We use the DINOv2 (patch) + projection variant and follow the hyperparameters used in the original paper ( Wang et al., 2026d ) .

Hyperparameter	Push-T & Reacher	FetchSlide & BeerPong
Population size ( $N$ )	300	64
Elite count ( $M_{\text{elite}}$ )	30	8
CEM iterations ( $J$ )	30	8
Planned action length ( $T$ )	5 model steps (25 actions)	1 initial strike / throw command

Appendix

Table A4: Hyperparameters for Visual Latent Planning with CEM. FetchSlide and BeerPong use a smaller optimization budget since they only optimize significantly shorter action sequences.

	Push-T			Reacher
Method	25	50	75	25	50	75
TS-WM	53.3	54.7	56.0	41.3	65.3	68.0
DINO-WM	53.3	54.7	56.0	52.0	76.0	82.7
+ Long History	54.7	56.0	58.7	56.0	92.0	96.0
+ Context Prefix	58.7	60.0	60.0	61.3	84.0	92.0
+ Estimated Physics †	57.3	62.7	65.3	69.3	85.3	90.7

Appendix

Table A7: MPC success rates on Push-T and Reacher under different maximum numbers of control steps. Max. control steps denotes the maximum number of control steps allowed across all replanning iterations. Incorporating PhysiCal substantially improves both DINO-WM- and LeWM-based models across all allowed numbers of replans.

Figure A1: Success-threshold sensitivity. Evaluation of planning success rates across varying error tolerances for DINO-WM variants (top) and LeWM variants (bottom). (a) Reacher (DINO-WM-based models) using maximum absolute joint-angle error across joints. (b) FetchSlide (DINO-WM-based models) using terminal puck-XY position error. (c) Reacher (LeWM-based models) across joint-angle tolerances. (d) FetchSlide (LeWM-based models) across puck-XY position tolerances.

Figure A2: Qualitative Example of an In-Context Interaction Clip $\mathcal{C}_{\text{eval}}$ used in the Push-T Environment. We provide the in-context interaction clip used by +PhysiCal (Ours) to generate the main-text rollout results in Figure 4 (a). This clip is collected via the same policy (i.e., a CoM-agnostic PPO policy) that is used to generate the training dataset. This clip is encoded by the context encoder $g_{\phi}$ into a physics descriptor token , which is used to calibrate the world model’s predictions.

Figure A3: Qualitative Example of an In-Context Interaction Clip $\mathcal{C}_{\text{eval}}$ used in the BeerPong Environment. We show the in-context interaction clip used by +PhysiCal (Ours) to produce the main-text rollout results in Figure 4 (b). This clip is collected by a randomly sampled throwing action. The context encoder $g_{\phi}$ subsequently takes this clip as input and encodes it into a physics descriptor token $\mathbf{z}_{\text{phy}}$ . As $\mathbf{z}_{\text{phy}}$ carries the test scene’s transition dynamics inferred from the interaction clip, it can effectively ground the world model’s future predictions in the actual physical dynamics.

Figure A4: Qualitative Example of an In-Context Interaction Clip $\mathcal{C}_{\text{eval}}$ in the FetchSlide Environment. We present the in-context interaction clip used by PhysiCalWorld to generate the main-text prediction results in Figure 6 . This clip is collected via a random strike action and is encoded by the context encoder $g_{\phi}$ into a physics descriptor token , which is used to calibrate the world model’s predictions.

Figure A5: Uncropped Open-Loop Rollouts in the BeerPong Environment. We present the full-scene view of the BeerPong rollouts shown in Figure 4 (b), displaying the predicted ball trajectories relative to the table and cup. DINO-WM + Ours accurately predicts the ball’s bounce, while DINO-WM and DINO-WM + Context Prefix overestimate its rebound height. The semi-transparent cyan overlay marks the ground-truth ball position.

Figure A6: Qualitative Open-Loop Rollouts in the Reacher Environment under Out-of-Distribution Physics Configurations. We present an extrapolation example corresponding to the quantitative results in Figure 8 , with the first joint’s damping outside the training range and the second joint’s damping held fixed. Both models predict future observations autoregressively without additional ground-truth observations (i.e., feedback from the environment). DINO-WM + Privileged Physics predicts accurately from $t=0$ to $t=6$ , but background blurring emerges at $t=7$ and progressively intensifies through $t=10$ . From $t=11$ to $t=19$ , its predictions collapse into a nearly uniform blue field, with no discernible arm or scene structure. In contrast, DINO-WM + Ours preserves the scene structure and continues to predict the arm’s motion throughout the rollout. The corresponding interaction clip is shown in Figure A7 .

Figure A7: In-Context Interaction Clip $\mathcal{C}_{\text{eval}}$ used in Figure A6 . This clip is collected using randomly sampled actions (the action space is the torques applied at the two joints of the robot arm).

Figure A8: Qualitative Open-Loop Rollouts in the Push-T Environment. We present open-loop rollouts, where models are only given the initial observation and autoregressively predict future observations. All context-conditioned models use the interaction clip shown in Figure A9 . DINO-WM + Ours, LeWM + Ours, DINO-WM + Privileged Physics, and DINO-WM + Context Prefix closely match the ground-truth motion, while the remaining baselines exhibit substantial prediction errors. Though Context Prefix predicts accurately, it directly attends to the full context sequence, making its inference substantially slower.

Figure A9: Qualitative Example of an Exploratory In-Context Interaction Clip $\mathcal{C}_{\text{eval}}$ in the Push-T Environment. This in-context interaction clip is used by PhysiCalWorld to generate the open-loop rollout results in Figure A8 . The clip is collected using a CoM-agnostic PPO policy and encoded by the context encoder $g_{\phi}$ into a physics descriptor token to calibrate the world model’s predictions.

Figure A10: Qualitative Open-Loop Rollouts in the Reacher Environment. We show autoregressive predictions of the arm’s motion without providing further ground-truth observations during the rollout. All the models that ingest a context clip use the same one as presented in Figure A11 . DINO-WM + Ours, LeWM + Ours, and DINO-WM + Privileged Physics accurately predict the arm’s motion. Context Prefix overestimates the motion, whereas the remaining baselines underestimate it, suggesting that it uses the context to adjust its predictions but remains less accurate than our models.

Figure A11: Qualitative Example of an In-Context Interaction Clip $\mathcal{C}_{\text{eval}}$ in the Reacher Environment. We provide the in-context interaction clip used by PhysiCalWorld to produce the open-loop rollout results in Figure A10 . Generated by randomly sampled joint-torque commands, this clip is encoded by the context encoder $g_{\phi}$ into a physics descriptor token , which is used to calibrate the world model’s predictions.

Figure A12: Qualitative Open-Loop Rollouts in the BeerPong Environment. We provide open-loop predictions of the ball’s trajectory, generated autoregressively without subsequent ground-truth observations. All models that use context receive the interaction clip in Figure A13 . DINO-WM + Ours, LeWM ∗ + Ours, DINO-WM + Privileged Physics, and DINO-WM + Context Prefix accurately reproduce the ball’s bounce, while the remaining baselines overestimate its rebound. The accuracy of Context Prefix, however, comes at the cost of substantially slower inference due to direct attention over the full context sequence. LeWM ∗ denotes TC-LeWM in BeerPong, as we find that this variant can significantly boost the performance of LeWM in BeerPong.

Figure A13: Qualitative Example of an Exploratory In-Context Interaction Clip $\mathcal{C}_{\text{eval}}$ in the BeerPong Environment. We present the in-context interaction clip used by PhysiCalWorld to generate the open-loop rollout results in Figure A12 . The context encoder $g_{\phi}$ encodes this clip, collected using a randomly sampled throw action, into a physics descriptor token that calibrates the world model’s predictions.

Figure A14: Qualitative Open-Loop Rollouts in the FetchSlide Environment. We present an eight-frame open-loop rollout, in which each model predicts future observations autoregressively without receiving additional ground-truth observations. The interaction clip shown in Figure A15 is shared by all context-conditioned models. DINO-WM + Ours, LeWM + Ours, and DINO-WM + Privileged Physics closely match the ground-truth puck motion. DINO-WM + Context Prefix slightly underestimates its movement, while the remaining baselines without context exhibit more pronounced underestimation.

Figure A15: Qualitative Example of an Exploratory In-Context Interaction Clip $\mathcal{C}_{\text{eval}}$ in the FetchSlide Environment. We provide the in-context interaction clip used by the context-conditioned methods to generate the results in Figure A14 . This clip is collected using a randomly sampled strike action and subsequently encoded by the context encoder $g_{\phi}$ into a physics descriptor token , which is used to calibrate the world model’s predictions.

Figure A16: Qualitative Planning Rollouts in the Push-T Environment, DINO-WM-Based Models. We use CEM to optimize action sequences toward a goal observation and execute them in the environment, comparing model predictions with the resulting motion. All context-conditioned models use the interaction clip shown in Figure A17 . DINO-WM + Ours closely matches the executed motion, while the remaining models misrepresent the T-block’s translational and rotational dynamics.

Figure A17: Qualitative Example of an In-Context Interaction Clip $\mathcal{C}_{\text{eval}}$ in the Push-T Environment. We provide the in-context interaction clip used by PhysiCalWorld to generate the planning results in Figure A16 . Collected using a CoM-agnostic PPO policy, this clip is encoded by the context encoder $g_{\phi}$ into a physics descriptor token , which is used to calibrate the world model’s predictions during planning.

Figure A18: Qualitative Planning Rollouts in the Push-T Environment, LeWM-Based Models. Action sequences generated by CEM to reach a goal observation are executed in the environment and compared with the world models’ predictions. LeWM + Ours uses the interaction clip presented in Figure A19 and accurately predicts the resulting motion. In contrast, LeWM underestimates the effect of the push at $t=3$ , with the T-block moving and rotating substantially farther than predicted at $t=4$ .

Figure A19: Qualitative Example of an Exploratory In-Context Interaction Clip $\mathcal{C}_{\text{eval}}$ in the Push-T Environment. We show the in-context interaction clip used by PhysiCalWorld to produce the planning results in Figure A18 . This clip is collected using a CoM-agnostic PPO policy. The proposed method will use its context encoder $g_{\phi}$ to encode this clip into a physics descriptor token , which is used to calibrate the world model’s predictions during planning.

World Models

World models represent and predict how environments change, helping agents plan, act, and reason about possible futures.

See all 360 papers

From Pixels to Executable Physics: Behavior-Grounded World Models for Embodied Action Consequence Prediction
Oct 13, 2026Ruo Chen, Jiahua Pang, Yingjie Niu +4Action-Conditioned World Models Embodied World Models
DEEP Robotics · Zhejiang University · Beijing Institute of Technology +3
Physical Object Understanding with a Physically Controllable World Model
May 30, 2026Rahul Venkatesh, Klemen Kotar, Lilian Naing Chen +9Physical Reasoning World Model Learning
Stanford University · OpenAI · Noetik Inc. +1
Correcting a learned physical invariant improves world-model rollouts
Aug 24, 2026Richard BaoWorld Models World Model Learning
Independent Researcher

Oct 13, 2026J/K move · Enter open · S save

From Pixels to Executable Physics: Behavior-Grounded World Models for Embodied Action Consequence Prediction

Ruo Chen, Jiahua Pang, Yingjie Niu, Hongyi Lin+3

DEEP Robotics · Zhejiang University · Beijing Institute of Technology+3

Action-Conditioned World Models Embodied World Models Robot Dynamics Learning Quadruped Locomotion

Open paper AskPDF arXiv

In outdoor environments, varying terrain and contact conditions can cause identical action commands to produce different motion outcomes for quadruped robots. Predicting these outcomes is essential for reliable navigation and informed path selection. Existing video world models can generate visually plausible futures, yet often struggle to capture robot dynamics and terrain interactions, leading to overly optimistic predictions of action outcomes. We present DR-WM, a physically consistent world model that incorporates robot-terrain contact dynamics to predict the consequences of quadruped actions. Starting from an initial RGB image, our method reconstructs scene geometry and integrates dense, dimensionless priors over terrain contact properties, calibrated using real-world action-response data. The resulting terrain model supports the simulation of physics with spatially varying contact properties such as friction. Given the initial state of the robot and candidate action sequences, we simulate its future motion and use the resulting camera trajectories and rendered geometry to guide the training of a video world model. The generated future observations are then used to update the terrain reconstruction, enabling iterative model rollouts without additional real-world images. Extensive experiments demonstrate that DR-WM substantially outperforms geometry-only baselines and baselines with uniform physical parameters on key physical metrics characterizing action outcomes, while achieving the best performance among the evaluated baselines. These physically grounded predictions support more informed action selection: across 90 challenging scenarios, an 8B vision-language model augmented with DR-WM outperforms an unassisted 32B vision-language model on five of six evaluation metrics.

Abstract

World Models

Explore similar work

From Pixels to Executable Physics: Behavior-Grounded World Models for Embodied Action Consequence Prediction

Physical Object Understanding with a Physically Controllable World Model

Correcting a learned physical invariant improves world-model rollouts