Object-centric world models forecast future videos by evolving a set of entity slots, but the variables receiving dynamics supervision are often unconstrained visual features. We introduce \method{}, a mask-grounded soft-Hamiltonian world model that makes its position-like state explicitly depend on slot-owned image support. A frozen video-slot encoder produces slots and masks; spatial moments of mask-owned support form a canonical state Q, temporal differences form P, and a learned energy supplies a soft directional bias to a bounded learned increment. Decoder-relevant appearance and identity are stored separately in a causal visual context. A gated composer and bounded residual then combine this context with the propagated phase state to reconstruct decoder-compatible slots. On OBJ3D, given six observed frames and evaluated over the following 30 frames, \method{} reduces LPIPS by 25.0% and spatial MSE by 33.7% relative to the strongest object-centric baseline. On CLEVRER, given six observed frames and evaluated over the following ten frames, the corresponding reductions are 14.5% and 18.7%. Horizon-resolved visual and object-state measurements show that the complete model accumulates error more slowly throughout the 30-frame closed-loop rollout. Project page:https://github.com/moshwm-anon/-moshwm-anon.github.io.
Figures & tables
Figure 1: Mask grounding and long-horizon prediction on OBJ3D. Owned masks expose object support. SlotFormer matches the immediate future but drifts over closed-loop rollout; MOSH-WM better preserves object identity and scene layout.
Figure 2: MOSH-WM pipeline. A frozen SAVi interface supplies visual slots and decoded masks. Mask-owned pooling extracts Q , raw temporal differences provide P , and causal slots provide C . The SoftHamiltonian update propagates (Q,P) ; a gated composer and bounded spatiotemporal residual return to decoder-compatible slot space. Dashed arrows denote direct state propagation and autoregressive context update.
Figure 3: Long-sequence prediction on OBJ3D. Two observation frames and six future frames are shown at a larger scale than a full-frame strip. All rows use the same six-frame context and independently exported 30-frame closed-loop predictions. Under repeated feedback, MOSH-WM better retains object identity, color, and relative layout.
Figure 4: SlotPi-style scene decomposition by the frozen SAVi observer on OBJ3D. Each row is one observation time. The mask uses a black background and mutually exclusive colors; slots 1–6 are independently decoded RGB components on near-white backgrounds. Fixed columns expose temporal slot consistency and the support used to construct Q .
Reconstruction
LPIPS ↓
PSNR ↑
SSIM ↑
MSE ↓
Frozen SAVi
0.0286
41.05
0.9826
0.376
Phase composer
0.0288
40.81
0.9820
0.395
Table 1: OBJ3D reconstruction quality before rollout. The phase-to-slot composer remains close to the frozen-interface upper bound.
Method
Mean@10
Mean@30
Final@30
MOSH-WM
.0381
.0858
.1744
SlotFormer
.0540
.1144
.2063
SAVi-dyn
.0559
.1197
.2065
Table 3: OBJ3D mean LPIPS up to increasing horizons and LPIPS at the final predicted frame.
Figure 5: OBJ3D closed-loop error accumulation. (a) Mean LPIPS over increasing horizons. (b) Per-frame centroid error. (c) Per-frame pairwise-distance error. All curves use the same six observations.
Variant
LPIPS ↓
MSE ↓
Slot@30 ↓
MOSH-WM ( α=0.5 )
0.0858
4.718
0.02608
residual only
0.1665
8.468
0.08405
physics only
0.1037
5.840
0.04212
w/o context
0.1437
8.595
0.04799
re-encode state
0.1122
6.259
0.06985
α=0.1
0.0964
5.154
0.02970
Table 4: OBJ3D component and mixing-coefficient ablation. All entries use the same six-input, 30-prediction evaluator. Bold identifies each column minimum.
Figure 6: Physion transfer through frozen STEVE. Slot MSE remains lowest for the complete model throughout the 35-step closed-loop rollout.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Validation Slot@30 during the main OBJ3D training run. The curve is used for checkpoint selection.
Attention blocks
LPIPS ↓
MSE ↓
Slot@30 ↓
1
0.1863
9.945
0.06165
2 (default)
0.0873
4.789
0.02755
3
0.1251
6.856
0.03984
Appendix
Table 5: Capacity sensitivity under the shared short-budget protocol.
Figure 8: Additional OBJ3D 6-to-30 rollout on test video 125. Columns show future horizons and rows compare ground truth with three predictors under the same closed-loop protocol.
Figure 9: Compact OBJ3D ablation summary. The plot shows relative error increases when the physical, context, or residual pathway is removed.
Figure 10: Physion slot error over the prediction horizon. The complete model remains lowest throughout the rollout.
Symbol
Shape
Meaning
Xt
H×W×3
video frame
St
N×ds
observer slot tokens
Mt
N×H×W
soft slot masks
Qt
N×dq
mask-grounded coordinate
Pt
N×dq
temporal phase difference
Ct
context-dependent
causal visual memory
Appendix
Table 6: Notation and tensor roles used in the implementation.
Method
@1
@3
@5
@10
@20
@30
Residual-only
.0296
.1215
.2500
.6331
1.3689
1.9220
Hamiltonian-only
.0541
.0620
.0674
.0758
.0858
.0932
MOSH (full)
.0126
.0187
.0230
.0290
.0362
.0422
Appendix
Table 7: Physion test slot MSE over the closed-loop rollout.
Figure 11: Physion qualitative transfer. Ground truth, MOSH prediction, and absolute reconstruction error are shown at horizons 1, 10, 20, and 30. Pixel quality is limited by the frozen STEVE decoder; the experiment is primarily a slot-dynamics transfer test.
Scenario
MOSH @30
Collide
0.0546
Contain
0.0341
Dominoes
0.0403
Drape
0.0683
Drop
0.0321
Link
0.0362
Appendix
Table 8: Scenario-level Physion slot MSE at horizon 30.
Learning world models from offline trajectories enables agents to accomplish different tasks through planning. Object-centric (OC) representations, which decompose a scene into a set of slots that bind to its objects, have been proposed as an inductive bias for world models that are more sample-efficient and generalize better. Yet prior object-centric world models (OCWMs) take the slot encoder as given and evaluate only in-distribution, leaving open whether the object-centric bias actually delivers for planning and what within the OCWM drives it. We conduct a controlled study of OCWMs for visual model-predictive control along two axes: object-centric representation quality and generalization under distribution shift relative to scene-centric models. We find that (i) planning success correlates positively with unsupervised slot-quality metrics (FG-ARI, mBO), though the gains saturate at high slot quality; (ii) with well-bound slots, the auxiliary proprioception inputs and masking inductive bias that prior methods relied on become unnecessary; and (iii) under unseen distribution shifts, the OCWM with well-bound slots is more robust overall than the end-to-end trained scene-centric LeWM, while DINO-WM, built on similar frozen pretrained features, remains competitive -- pointing to pretrained features as a key contributor to robustness.
Shukrullo Nazirjonov, Sai Prasanna, Anna Manasyan +1
University of Tuebingen · Max Planck Institute for Intelligent Systems
We introduce Latent Particle World Model (LPWM), a self-supervised object-centric world model scaled to real-world multi-object datasets and applicable in decision-making. LPWM autonomously discovers keypoints, bounding boxes, and object masks directly from video data, enabling it to learn rich scene decompositions without supervision. Our architecture is trained end-to-end purely from videos and supports flexible conditioning on actions, language, and image goals. LPWM models stochastic particle dynamics via a novel latent action module and achieves state-of-the-art results on diverse real-world and synthetic datasets. Beyond stochastic video modeling, LPWM is readily applicable to decision-making, including goal-conditioned imitation learning, as we demonstrate in the paper. Code, data, pre-trained models and video rollouts are available: https://taldatech.github.io/lpwm-web
Tal Daniel, Carl Qi, Dan Haramati +5
Carnegie Mellon University · UT Austin · Brown University +2
World models that predict future states from video remain limited by flat latent representations that entangle objects, ignore causal structure, and collapse temporal dynamics into a single scale. We present HCLSM, a world model architecture that operates on three interconnected principles: object-centric decomposition via slot attention with spatial broadcast decoding, hierarchical temporal dynamics through a three-level engine combining selective state space models for continuous physics, sparse transformers for discrete events, and compressed transformers for abstract goals, and causal structure learning through graph neural network interaction patterns. HCLSM introduces a two-stage training protocol where spatial reconstruction forces slot specialization before dynamics prediction begins. We train a 68M-parameter model on the PushT robotic manipulation benchmark from the Open X-Embodiment dataset, achieving 0.008 MSE next-state prediction loss with emerging spatial decomposition (SBD loss: 0.0075) and learned event boundaries. A custom Triton kernel for the SSM scan delivers 38x speedup over sequential PyTorch. The full system spans 8,478 lines of Python across 51 modules with 171 unit tests. Code: https://github.com/rightnow-ai/hclsm