Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling

Tal Daniel; Carl Qi; Dan Haramati; Amir Zadeh; Chuan Li; Aviv Tamar; Deepak Pathak; David Held

cs.LGMar 4, 2026

Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling

Authors: Tal Daniel, Carl Qi, Dan Haramati, Amir Zadeh, Chuan Li, Aviv Tamar, Deepak Pathak, David Held

Organizations: Carnegie Mellon University · UT Austin · Brown University · Lambda · Technion

Abstract

We introduce Latent Particle World Model (LPWM), a self-supervised object-centric world model scaled to real-world multi-object datasets and applicable in decision-making. LPWM autonomously discovers keypoints, bounding boxes, and object masks directly from video data, enabling it to learn rich scene decompositions without supervision. Our architecture is trained end-to-end purely from videos and supports flexible conditioning on actions, language, and image goals. LPWM models stochastic particle dynamics via a novel latent action module and achieves state-of-the-art results on diverse real-world and synthetic datasets. Beyond stochastic video modeling, LPWM is readily applicable to decision-making, including goal-conditioned imitation learning, as we demonstrate in the paper. Code, data, pre-trained models and video rollouts are available: https://taldatech.github.io/lpwm-web

Figures & tables

Panel of Figure 1 — Figure 1: Self-supervised object-centric world modeling with LPWM. Top: latent particle decomposition. Bottom left: language-conditioned video generation. Bottom right: latent-action-conditioned video prediction.

Figure 2: Representation discrepancy. Text is typically tokenized into semantically meaningful units such as words or subwords, whereas image representations are most often constructed by dividing the image into a fixed grid of patches (“patchifying”) that do not explicitly encode semantic content.

Model	Obj.-Centric Rep.	Latent Actions	Action Cond.	Text Cond.	End-to-End	Dyn. Module
SCALOR ( Jiang et al., 2019 )	Patch	–	–	–	$\checkmark$	RNN
G-SWM ( Lin et al., 2020a )	Patch	–	–	–	$\checkmark$	RNN
STOVE ( Kossen et al., 2019 )	Patch	–	–	–	$\checkmark$	RNN
OCVT ( Wu et al., 2021b )	Patch	–	–	–	–	Transformer
GATSBI ( Min et al., 2021 )	Patch+Keypt	–	$\checkmark$	–	$\checkmark$	RNN
PARTS ( Zoran et al., 2021 )	Slots	–	–	–	$\checkmark$	RNN

Table 1: Comparison of object-centric video prediction and world modeling methods across key dimensions and representation types. Please refer to Table 4 for an extended comparison.

Figure 3: Latent Particle World Model architecture. Left: Input frames are encoded into particle sets by the Encoder and decoded back to images by the Decoder . The Context module then processes the particles to sample latent actions, which are combined with the particles in the Dynamics module to predict next-step particle states. Right: The Context module models the per-particle latent action distribution. During training, we use the latent inverse dynamics head, while at inference, the latent policy is employed for sampling.

Dataset	Sketchy-U				BAIR-U				Mario-U
Setting	$t:20,c:6,p:44$				$t:16,c:1,p:15$				$t:20,c:6,p:34$
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$
DVAE	25.75 $\pm$ 3.85	0.86 $\pm$ 0.08	0.113 $\pm$ 0.06	140.06	26 $\pm$ 2.2	0.90 $\pm$ 0.03	0.063 $\pm$ 0.02	164.41	23.35 $\pm$ 4.28	0.93 $\pm$ 0.04	0.087 $\pm$ 0.05	277.41
PlaySlot	22.63 $\pm$ 3.90	0.80 $\pm$ 0.09	0.275 $\pm$ 0.06	$-$	17.56 $\pm$ 1.50	0.57 $\pm$ 0.05	0.483 $\pm$ 0.03	$-$	16.38 $\pm$ 2.78	0.68 $\pm$ 0.1	0.314 $\pm$ 0.09	$-$
LPWM (Ours)	28.41 $\pm$ 3.8	0.91 $\pm$ 0.06	0.070 $\pm$ 0.04	85.45	25.66 $\pm$ 1.52	0.89 $\pm$ 0.02	0.062 $\pm$ 0.02	163.91	27.50 $\pm$ 5.52	0.95 $\pm$ 0.04	0.035 $\pm$ 0.02	195.95

Table 2: Quantitative results on latent-action-conditioned ( U ), action-conditioned ( A ), and language-conditioned ( L ) video prediction. FVD is reported for stochastic generation by sampling from the latent policy.

t

is the training horizon,

c

is the conditional frames at inference, and

p

is the predicted frames at inference.

Task	EC Diffusion Policy	EC Diffuser	LPWM (Ours)
1 Cube	$88.7\pm 3$	94.8 $\pm$ 1.5	92.7 $\pm$ 4.5
2 Cubes	$38.8\pm 10.6$	91.7 $\pm$ 3	$74\pm 4$
3 Cubes	$66.8\pm 17$	89.4 $\pm$ 2.5	$62.1\pm 4.4$

Table 3: Imitation learning results (success rates) on PandaPush (left) and OGBench-Scene (right).

Figure 4: LPWM generated goal-conditioned imagined trajectories (top) and actual environment executions (bottom) through a learned mapping to actions on OGBench-Scene . The imagined trajectories closely match the actual executions, demonstrating LPWM’s predictive accuracy.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Spatial-softmax. Given a heatmap $\tilde{\mathcal{H}}\in\mathbb{R}^{H\times W}$ , the softmax function is applied over the spatial dimensions to normalize $\tilde{\mathcal{H}}$ into a probability distribution $\mathcal{H}$ . These values are then used to compute the expected coordinate values for each axis, and their covariance.

Figure 8: A Deep Latent Particle. Each component of a latent particle corresponds to a disentangled stochastic attribute: position $z_{p}$ , scale $z_{s}$ , depth $z_{d}$ , transparency $z_{t}$ , and visual features $z_{f}$ . Further details are provided in Section 3 of the main text and Section A.3 of the Appendix.

Figure 9: Encoding and decoding particles in DLP. Input image is first used to generate keypoint proposals , that, jointly with input image are used to encode a set of particles by the Encoder , which is then decoded by the Decoder . Further details are provided in Section 4 of the main text and Section A.4 of the Appendix.

Figure 10: Encoding particles in DLP. The encoding process involves 4 steps: (1) keypoint proposals - a patch-based network with a spatial softmax layer generates candidate particle locations; (2) attribute encoding - offsets, scales, depths, and transparencies are inferred for each particle; (3) appearance encoding - foreground particle features, and (4) background encoding - global background features are extracted from the keypoint-masked image.

Figure 11: Decoding particles in DLP. Each particle is decoded into a localized appearance patch, positioned and scaled according to its spatial attributes, while transparency and depth resolve visibility and occlusions. The background is decoded with a standard upsampling CNN-based network. Finally, the foreground and background components are stitched using the effective alpha masks.

Figure 13: Particle-grid regime. Each particle is constrained to move only within a local region around its original patch center, and when it reaches the limits of this region, its features are transferred to nearby particles.

Figure 14: Spatio-temporal transformer block. It consists of (1) a spatial block that independently processes all $M$ particles at each timestep, with shape $[B\times T,M,D]$ , and (2) a temporal block that captures each particle’s temporal dependencies across the full horizon, with shape $[B\times M,T,D]$ . Here, $B$ is batch size, $T$ is temporal horizon, $M$ is number of particles, and $D$ is the embedding dimension.

Attribute	Distribution	Parameters ( $\text{glimpse\_ratio}=0.25$ )	Parameters ( $\text{glimpse\_ratio}=0.125$ )
Position Offset $z_{o}$	Normal, $\mathcal{N}(\mu,\sigma^{2})$	$\mu=0,\,\sigma=0.2$	$\mu=0,\,\sigma=0.1$
Scale $z_{s}$	Normal, $\mathcal{N}(\mu,\sigma^{2})$	$\mu=\text{Sigmoid}^{-1}(0.25),\,\sigma=0.3$	$\mu=\text{Sigmoid}^{-1}(0.125),\,\sigma=0.15$
Depth $z_{d}$	Normal, $\mathcal{N}(\mu,\sigma^{2})$	$\mu=0,\,\sigma=1$	$\mu=0,\,\sigma=1$
Transparency $z_{t}$	Beta, $\text{Beta}(a,b)$	$a=0.01,\,b=0.01$	$a=0.01,\,b=0.01$
Appearance Features $z_{f}$ , $z_{\text{bg}}$	Normal, $\mathcal{N}(\mu,\sigma^{2})$	$\mu=0,\,\sigma=1$	$\mu=0,\,\sigma=1$

Appendix

Table 6: Prior distribution parameters for different glimpse (patch) ratios. Glimpses are patches taken around keypoints, where

\text{glimpse\_ratio}=\frac{\text{glimpse size}}{\text{image size}}

		OBJ3D
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
DLP	39.23 $\pm$ 3.33	0.982 $\pm$ 0.009	0.085 $\pm$ 0.018
DLPv2	41.97 $\pm$ 3.74	0.985 $\pm$ 0.006	0.019 $\pm$ 0.01
DLPv3	43.87 $\pm$ 4.45	0.990 $\pm$ 0.006	0.011 $\pm$ 0.005

Appendix

Table 7: DLPv3, DLPv2, and DLP image reconstruction performance comparison in the single-image setting, evaluated on the test set.

Dataset	OBJ3D			PHYRE
Setting	$t:20,c:6,p:44$			$t:15,c:10,p:40$
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
DVAE	31.44 $\pm$ 5.69	0.923 $\pm$ 0.05	0.085 $\pm$ 0.07	26.61 $\pm$ 6.01	0.94 $\pm$ 0.04	0.047 $\pm$ 0.04
G-SWM	31.7 $\pm$ 6.2	0.924 $\pm$ 0.05	0.118 $\pm$ 0.07	24.64 $\pm$ 6.25	0.93 $\pm$ 0.05	0.078 $\pm$ 0.06
SlotFormer / OCVP	31.2 $\pm$ 4.91	0.925 $\pm$ 0.04	0.135 $\pm$ 0.05	21.26 $\pm$ 3.54	0.89 $\pm$ 0.05	0.108 $\pm$ 0.05
DDLP	31.29 $\pm$ 5.22	0.923 $\pm$ 0.04	0.088 $\pm$ 0.06	26.98 $\pm$ 5.3	0.95 $\pm$ 0.04	0.055 $\pm$ 0.04

Appendix

Table 8: Quantitative results on video prediction for datasets with deterministic dynamics.

t

is the training horizon,

c

is the conditional frames at inference and

p

is the predicted frames at inference.

BAIR-64 ( $64\times 64$ )	FVD $\downarrow$
LVT ( Rakhimov et al., 2020 )	125.8
DVD-GAN-FP ( Clark et al., 2019 )	109.8
TrIVD-GAN-FP ( Luc et al., 2020 )	103.3
VideoGPT ( Yan et al., 2021 )	103.3
CCVS ( Le Moing et al., 2021 )	99.0
FitVid ( Babaeizadeh et al., 2021 )	93.6

Appendix

Table 9: Video prediction results on BAIR-64 (

64\times 64

) conditioning on one past frame and predicting

15

frames in the future. Table adapted from Shrivastava and Shrivastava (2024) .

Figure 16: Multi-modal future sampling by LPWM. Starting from the same initial frame, LPWM produces diverse possible future trajectories, illustrated on the Mario (left) and BAIR (right) datasets.

Dataset	Bridge-L				LanguageTable-L
	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$
DVAE	19.37 $\pm$ 3.8	0.75 $\pm$ 0.09	0.177 $\pm$ 0.078	146.85	36.0 $\pm$ 4.14	0.97 $\pm$ 0.01	0.019 $\pm$ 0.01	26.78
LPWM (Ours)	26.38 $\pm$ 4.1	0.87 $\pm$ 0.08	0.077 $\pm$ 0.05	47.78	36.57 $\pm$ 3.01	0.97 $\pm$ 0.007	0.016 $\pm$ 0.006	15.96

Appendix

Table 10: Quantitative results on language-conditioned ( L ) video generation. PSNR, SSIM and LPIPS are calculated on latent-action-conditioned video prediction. FVD is reported for stochastic generation by sampling from the latent policy.

Ablation Variant	Latent	Latent Actions	Positional	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FVD $\downarrow$
	Actions $d_{\text{ctx}}$	Type	Embeddings
Original	7	Per-Particle	Learned AdaLN	28.55 $\pm$ 3.30	0.91 $\pm$ 0.05	0.072 $\pm$ 0.03	120.32
$d_{\text{ctx}}=1$	1	Per-Particle	Learned AdaLN	27.71 $\pm$ 3.50	0.89 $\pm$ 0.06	0.081 $\pm$ 0.03	177.64
$d_{\text{ctx}}=3$	3	Per-Particle	Learned AdaLN	29.08 $\pm$ 3.15	0.92 $\pm$ 0.05	0.070 $\pm$ 0.03	117.46
$d_{\text{ctx}}=10$	10	Per-Particle	Learned AdaLN	28.97 $\pm$ 3.28	0.91 $\pm$ 0.05	0.068 $\pm$ 0.03	117.54
$d_{\text{ctx}}=14$	14	Per-Particle	Learned AdaLN	28.81 $\pm$ 3.22	0.91 $\pm$ 0.05	0.069 $\pm$ 0.03	121.02

Appendix

Table 11: Ablation results: impact of latent action dimensions and type, and positional embeddings on LPWM performance. Results are reported on the Sketchy dataset after 10 epochs of training. Results do not reflect final performance.

Figure 17: LPWM generated goal-conditioned imagined trajectories (top) and actual environment executions (bottom) through a learned mapping to actions on PandaPush from two views. LPWM generates dynamics in both views simultaneously, handling occlusions by the gripper.

Task	VQ-BeT	Diffuser	EIT+BC	EC Diffusion Policy	EC Diffuser	LPWM (Ours)
1 Cube	$93\pm 3$	$36.7\pm 2.7$	$89\pm 2$	$88.7\pm 3$	94.8 $\pm$ 1.5	92.7 $\pm$ 4.5
2 Cubes	$5.2\pm 1$	$1.3\pm 1$	$14.6\pm 12.5$	$38.8\pm 10.6$	91.7 $\pm$ 3	$74\pm 4$
3 Cubes	$0.6\pm 0.1$	$0.2\pm 0.4$	$14\pm 16.4$	$66.8\pm 17$	89.4 $\pm$ 2.5	$62.1\pm 4.4$

Appendix

Table 12: Performance results on PandaPush , a physics-based tabletop benchmark in IsaacGym where a Franka arm must arrange multiple cubes to match target goal images. Reported values are success rates over 500 trajectories across 5 seeds; results within one standard deviation of the best are shown in bold.

Task	GCBC	GCIVL	GCIQL	QRL	CRL	HIQL	LPWM (Ours)
task1	$59$ $\pm 7$	$84$ $\pm 4$	$56$ $\pm 4$	$44$ $\pm 6$	$52$ $\pm 6$	$80$ $\pm 6$	$\mathbf{100}$ $\pm 0$
task2	$0$ $\pm 0$	$24$ $\pm 8$	$1$ $\pm 1$	$2$ $\pm 2$	$1$ $\pm 1$	$\mathbf{81}$ $\pm 7$	$6$ $\pm 9$
task3	$0$ $\pm 0$	$16$ $\pm 8$	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$61$ $\pm 11$	$\mathbf{89}$ $\pm 9$
task4	$2$ $\pm 1$	$0$ $\pm 0$	$3$ $\pm 4$	$2$ $\pm 1$	$1$ $\pm 1$	$\mathbf{20}$ $\pm 8$	$3$ $\pm 5$
task5	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$0$ $\pm 0$	$\mathbf{3}$ $\pm 2$	$0$ $\pm 0$
overall	$12$ $\pm 2$	$25$ $\pm 3$	$12$ $\pm 2$	$10$ $\pm 1$	$11$ $\pm 2$	$\mathbf{49}$ $\pm 4$	$40$ $\pm 1$

Appendix

Table 13: Full results on Visual Scene with the visual-scene-play-v0 dataset. Results for baselines were taken from the original OGBench benchmark ( Park et al., 2025 ) and represent success rates across 4 seeds. Results within a standard deviation are highlighted in bold.

Explore similar work

Mar 11, 2026

World2Act: Latent Action Post-Training from World Model Dynamics

World Models (WMs) offer a promising mechanism for post-training Vision-Language-Action (VLA) policies by providing dynamics priors that improve generalization under task and scene variation. However, most WM-based post-training methods rely on pixel-space supervision, making policies sensitive to visual artifacts introduced by imperfect WM rollouts. We present World2Act, a latent-space post-training framework that transfers WM dynamics to the VLA policy without pixel-space supervision. World2Act operates in two stages: 1) it induces a shared video-action latent space by contrastively aligning WM-dynamics latents with action embeddings, and 2) it post-trains the VLA by guiding policy action representations toward WM-imagined dynamics rather than decoded pixels. Built on GR00T-N1.6, World2Act delivers absolute success-rate gains of up to +2.5% on simulation benchmarks (RoboCasa, LIBERO, Bridge-SIMPLER) and +6.7% on a real robot over finetuned VLA baselines. Notably, it outperforms pixel-space WM supervision by up to +6.0%, including on LIBERO where pixel supervision degrades the baseline, suggesting that latent WM dynamics offer a more stable WM-based post-training alternative to pixel-space transfer.

An Dinh Vuong, Tuan Van Vo, Abdullah Sohail +6

MBZUAI

Vision-Language-Action Models VLMs for Robotics Latent Action Learning

Ask PDF

Aug 24, 2026

MOSH-WM: Mask-Grounded Soft-Hamiltonian Dynamics for Object-Centric World Models

Object-centric world models forecast future videos by evolving a set of entity slots, but the variables receiving dynamics supervision are often unconstrained visual features. We introduce \method{}, a mask-grounded soft-Hamiltonian world model that makes its position-like state explicitly depend on slot-owned image support. A frozen video-slot encoder produces slots and masks; spatial moments of mask-owned support form a canonical state

Q

, temporal differences form

P

, and a learned energy supplies a soft directional bias to a bounded learned increment. Decoder-relevant appearance and identity are stored separately in a causal visual context. A gated composer and bounded residual then combine this context with the propagated phase state to reconstruct decoder-compatible slots. On OBJ3D, given six observed frames and evaluated over the following 30 frames, \method{} reduces LPIPS by 25.0% and spatial MSE by 33.7% relative to the strongest object-centric baseline. On CLEVRER, given six observed frames and evaluated over the following ten frames, the corresponding reductions are 14.5% and 18.7%. Horizon-resolved visual and object-state measurements show that the complete model accumulates error more slowly throughout the 30-frame closed-loop rollout. Project page:https://github.com/moshwm-anon/-moshwm-anon.github.io.

Zhekai Wang, Haoxiang Huang, Xiang Liu +6

Beijing Institute of Technology · University of Science and Technology of China · Tsinghua University +1

Object-Centric World Models Video World Models Video Prediction Hamiltonian Dynamics

Ask PDF

Apr 13, 2026

Grounded World Model: Latent Planning with Language Goals

World models such as DINO-WM and LeWM specify the goal with an image, which is difficult to obtain in advance for novel tasks. We present the Grounded World Model (GWM), a latent world model that enables zero-shot planning in the real world from language goals alone. Given a candidate action sequence and the current observation, GWM predicts the future in the visual space of a pretrained video-language embedding model. The frozen readout of this embedding model maps this imagined future and the task description into the same embedding space, where their negative cosine similarity serves as the planning cost. Training GWM requires only offline and task-agnostic video-action pairs and no language labels. In simulated experiments on WISER, planning with GWM, which executes the candidate action of lowest cost, solves 87% of 288 tasks with unseen instructions and visual signals, while ten fine-tuned VLAs average 22%. We then scale GWM up with real robot data, and use it for zero-shot planning in realistic simulation and real scenes. In the IsaacSim evaluation, planning with GWM completes all 70 trials across 14 tasks that require reasoning over referring expressions, matching a modular planner grounded by a frontier VLM, while pi0.5 reaches 37/70. Deployed on a real Franka, the same stack completes 55/60 separately evaluated pick-and-place sub-tasks, comparable to the modular planner's 52/60, with the full system running locally on a single consumer GPU. Project website: https://quanyili.github.io/gwm-wiser/.

Quanyi Li, Lan Feng, Haonan Zhang +4

EPFL · NUS

Zero-Shot Robotic Manipulation World Models for Robotics World Model-Based Planning Action-Conditioned World Models Generalization in Robotic Manipulation Vision-Language Grounding+5

Ask PDF