Inertial Video World Models
Organizations: Tsinghua University · University of Science and Technology of China · City University of Hong Kong · Hong Kong Polytechnic University ·
Abstract
The physical world has inertia: it evolves from what already exists rather than being recreated at every instant. However, many existing video world models lack an explicit mechanism for inertial state evolution, leaving the continuity of motion and structure implicit in video prediction. During autoregressive rollouts, repeated reconstruction can accumulate errors and destabilize the predicted dynamics. To address this limitation, we introduce InertialWorld, an Inertial Video World Model that makes inertia a structural prior for autoregressive prediction: existing scene evidence is carried across time rather than reconstructed at every step. Central to this formulation is Inertial Memory, which turns memory from passive predictive context into an evolving video state with two complementary views: a dense Eulerian grid preserves scene-aligned evidence, while persistent Lagrangian carriers bind local content to learned canonical coordinates. At each step, the model transports the carrier content according to action-conditioned motion, predicts the complete next video state from the transported carriers and dense grid, and uses a learned gate to combine transported and newly predicted content into the memory for the next step. Extensive experiments across multiple domains, including robot manipulation, 3D game simulation, and open-world navigation, demonstrate that InertialWorld achieves competitive autoregressive prediction performance against strong video world model baselines. Codes link: https://github.com/YuanGao-YG/InertialWorld
Figures & tables
| Robot Manipulation: RT-1 | ||||
|---|---|---|---|---|
| Model | PSNR | SSIM | LPIPS | MSE |
| RLVR-World | ||||
| Vid2World | ||||
| WorldGym | ||||
| DriftWorld | ||||
| InertialWorld (Ours) | ||||
| Method | Flow EPE margin (95% CI) | Flow Cosine margin (95% CI) | Dyn. LPIPS margin (95% CI) |
|---|---|---|---|
| RLVR-World | |||
| Vid2World | |||
| WorldGym | |||
| DriftWorld | |||
| InertialWorld (Ours) | Reference method | ||
| Model | Long-lived content | Post-reappearance content | Late-track content |
|---|---|---|---|
| RT-1, seed 0 | |||
| Plain AR | |||
| Recurrent Memory | |||
| Dense-only | |||
| InertialWorld (Full) | |||
| RT-1, seed 1 | |||
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Configuration |
| Frozen tokenizer interface | |
| Context code | Two lanes; 2,560 tokens; fixed within a rollout. |
| Dynamic code | Four lanes; 320 tokens per predicted frame. |
| Token quantization | Five FSQ digits with levels . |
| World-model predictor | |
| Predictor backbone | Width 1,280; 20 attention heads; 15 transition layers; three appearance-refinement layers; two Context blocks; one interaction layer; MLP ratio 4; zero dropout. |
| Objective | Exact definition and coefficient | Staging | Plain AR |
| Token and visual prediction | |||
| Joint-code CE | on the complete Dynamic joint ID | Active from step 0 | Shared |
| Generation-code CE | Active from step 0 | , all cells | |
| Digit CE | on all FSQ digits | Detail-lane factor over 0–4k | Shared |
| Ordered-digit error | on the ordered FSQ value | Same detail-lane factor | Shared |
| Frozen-decoder objective | RGB edge high frequency chroma motion-region temporal residual | Off through 1k, then by 6k | Shared |
| Action | WM params | WM steps | Peak-LR hold | Quick/ full |
|---|---|---|---|---|
| Robot Manipulation: RT-1 | ||||
| raw 13-D | 818.746M | 60k | 2k–32k | 64/1,024 |
| 3D Game Simulation: CSGO | ||||
| raw 51-D | 818.795M | 30k | 2k–16k | 64/128 |
| Embodied Navigation: RECON | ||||
| local 3-D | 818.733M | 30k | 2k–16k | 13/13 |
| Evaluated episodes | Targets | Resolution | Common alignment |
|---|---|---|---|
| Robot Manipulation: RT-1 | |||
| 1,024 | 32 | ||
| 3D Game Simulation: CSGO | |||
| 70 of 372 test | 32 | Same current anchor and raw 51-D action sequence. | |
| Embodied Navigation: RECON | |||
| 128 | 32 | Same current anchor and adjacent local-motion sequence. | |
| Method | Native context | 32-frame rollout |
| Robot Manipulation: RT-1 | ||
| RLVR-World | released H7 state | Four 7-frame recurrent blocks followed by a final 4-frame output chunk. |
| Vid2World | initial H1, then H9 | One 15-frame generation, followed by 17 single-frame sliding-window generations. |
| WorldGym | H1 | Official one-step latent transition applied 32 times. |
| DriftWorld | H4 | Official four-frame sliding window applied autoregressively for 32 outputs. |
| 3D Game Simulation: CSGO | ||
| Baseline | PSNR | SSIM | LPIPS | MSE/MAE |
|---|---|---|---|---|
| Robot Manipulation: RT-1 ( ) | ||||
| RLVR-World | ||||
| Vid2World | ||||
| WorldGym | ||||
| DriftWorld | ||||
| 3D Game Simulation: CSGO ( ) | ||||
| Robot Manipulation: RT-1 (64 Future Frames) | ||||
|---|---|---|---|---|
| Model | PSNR | SSIM | LPIPS | MSE |
| RLVR-World | ||||
| Vid2World | ||||
| WorldGym | ||||
| DriftWorld | ||||
| InertialWorld (Ours) | ||||
| Diagnostic | Steps 1–32 | Steps 33–64 | |
|---|---|---|---|
| Carrier identity relative to Context | |||
| Same-slot retrieval (%) | |||
| Same-slot cosine | |||
| Separation margin | |||
| Canonical content relative to Context | |||
| Same-slot retrieval (%) | |||
| RT-1, seed 0 (matched 30k) | ||
|---|---|---|
| Variant | Recurrent state | Controlled role |
| Plain AR | Latest hard-predicted Dynamic state | Removes Inertial Memory and directly predicts the next Dynamic state from fixed Context, the action, rollout step, and preceding prediction. |
| Recurrent Memory | Latest hard prediction GRU | Tests generic learned recurrence without the structured Eulerian–Lagrangian state or persistent carrier transport. |
| Dense-only | 320-token scene 64 slots four-frame history | Retains the dense state and matched update recipe while removing regional carriers, canonical content, and carrier transport. |
| Full | Dense-only state 16 Lagrangian carriers | Executes the complete carry–predict–update transition and therefore measures the incremental effect of carriers relative to Dense-only. |
| RT-1, seed 0 | |||
|---|---|---|---|
| Model | Long-lived content | Post-reappearance content | Late-track content |
| Plain AR | |||
| Recurrent Memory | |||
| Dense-only | |||
| InertialWorld (Full) | Reference method | ||
| RT-1, seed 1 | |||
Video World Models
Video world models generate and predict action-conditioned visual futures to simulate changing scenes for embodied agents.
See all 296 papers