World models are useful for robotic manipulation because robots can predict how actions change the states of objects before executing them. We present PointCast, a point-set world model that spans rigid, articulated, and deformable object manipulation. Its state is a set of 3D points on the object and the end-effector, mesh-free and topology-agnostic. Each point keeps its identity and is supervised on its own trajectory, which teaches the model where every point goes rather than only the shape the points form. Its backbone is a diffusion transformer that denoises a short window of future point positions, conditioned on the points' recent history and the commanded end-effector motion. The backbone's attention alternates between local and global, and cross-attention to the end-effector carries the coupling. This one architecture at 19.8M parameters and one training recipe cover four regimes, rigid objects, cloth, rope, and multi-joint cabinets, with a separate checkpoint trained for each. Trained on randomized simulation and scored against four baselines on the same metric, it is best on three of four regimes and second on rigid. Trained on a real-world robot teleoperation dataset, it has the lowest mean error in four of its six categories, is second in the other two, and improves on the dataset's own model in all six; zero-shot, its simulation checkpoints are best on two of four captures. Frozen inside sampling-based model-predictive control at one network evaluation per window, it plans four simulated tasks over 64 episodes, competitive with or outperforming every baseline on each. Project website at https://pointcast-wm.github.io.
Figures & tables
Fig. 1: PointCast frozen inside sampling-based MPC, planning from a point-set goal. One recorded episode each, at the first window the planner controls and after the last; the real panel is qualitative. Goal green, the executed end-effector path magenta, the pose goal flattened to the footprint the block must cover. Sec. IV-D plans three further tasks.
Fig. 2: Overview of PointCast . Every object point, actor point, and register enters as one token. The eight blocks alternate kNN-local and global self-attention, and each reads the actor tokens through cross-attention. At deployment, one Euler step turns a noise sample into the predicted window.
Fig. 3: The three attention patterns of the backbone. kNN-local restricts a query point (red) to its k nearest neighbors, the global panel is drawn from a register token (gold), which attends to every object token, and cross-attention reads the actor tokens (magenta).
Fig. 4: Each model rolled forward from its own predictions through one held-out episode per regime, near the end of the interaction. Ground truth blue, predictions orange, actor points magenta. The cabinet row highlights the moving part in yellow.
Fig. 5: Real-world rollouts on the PGND benchmark at step 30. GT is the photograph, and each prediction column shows the episode’s pre-interaction Gaussian splats carried by that method’s predicted point motion [ 4 ] . AdaptiGraph is shown but not scored (Sec. IV-B2 ).
Method
Rigid
Cloth
Rope
Cabinet ∗
PGND [ 4 ]
47.82
524.33
68.76
54.45
AdaptiGraph [ 3 ]
55.42
153.14
49.24
36.64
PTv3 [ 10 ]
52.06
42.82
72.30
10.34
ParticleFormer [ 8 ]
20.88
38.26
21.78
26.73
PointCast (ours)
24.40
31.90
14.67
6.84
n (interactions)
2796
1047
1119
996
TABLE I: Simulation benchmark, mean rollout error (mm)
Method
Cloth
Rope
Box
Bread
Bag
Sloth
PTv3 [ 10 ]
42.24
25.85
21.76
22.37
21.40
48.43
ParticleFormer [ 8 ]
31.65
24.10
24.54
22.61
18.62
44.32
PGND [ 4 ] (native)
44.49
39.14
27.79
20.33
16.16
42.94
PointCast (ours)
32.20
15.83
23.51
18.65
11.64
37.83
n (test episodes)
40
40
20
20
20
20
TABLE II: Real-world PGND benchmark, mean displacement error at the 3 s horizon (mm)
Fig. 6: Zero-shot rollouts, one episode per capture of Table III . GT is a point cloud on Push-T and YCB objects and the photograph on rope and cloth, and the end-effector drawn is the recorded one, not the actor points the models read.
Method
Push-T replay
Franka YCB
Rope
Cloth
PGND [ 4 ]
37.27
75.38
58.07
291.12
PTv3 [ 10 ]
24.67
60.30
83.45
54.77
ParticleFormer [ 8 ]
18.07
72.50
42.47
32.43
PointCast (ours)
15.35
52.86
49.05
42.20
n (interactions)
70
35
123
31
TABLE III: Zero-shot transfer, rollout error (mm)
Variant
Rigid
Cloth
Rope
Cabinet ∗
PointCast (Table I , flow, S=1 )
24.40
31.90
14.67
6.84
What decides articulation
ParticleFormer set-level loss
−14.5
+3.5
−2.2
+79.3
no distance prior
+1.7
−3.1
−2.1
+18.0
part-aware prior off (cabinet)
–
–
–
+15.6
Attention and tokenization
TABLE IV: Simulation ablations, percent change from the shaded row
Task
Metric
init
AdaptiGraph
PGND
PTv3
ParticleFormer
PointCast (ours)
SE(2) pose push
per-point (cm)
–
9.52 (0.19)
2.31 (5.98)
2.27 (0.25)
1.57 (0.17)
1.39 (0.10)
Sequential articulation
part err (cm)
–
7.97 / 2.67 (0.07)
2.25 / 0.38 (1.90)
0.72 / 0.97 (0.21)
0.88 / 0.41 (0.15)
0.42 / 0.38 (0.09)
Cloth drag
Chamfer (cm)
27.8
27.75 (0.27)
2.83 (2.23)
2.41 (0.03)
2.08 (0.03)
2.09 (0.04)
Rope drag
Chamfer (cm)
15.4
3.28 (0.31)
1.91 (12.08)
5.87 (0.03)
2.16 (0.03)
1.71 (0.04)
TABLE V: Planning across four tasks
Fig. 7: One recorded episode per task, the start and the final state each rollout model leaves under the same planner, cost, budget, seed, and goal. Object blue, goal green, end-effector magenta with its executed path, the pose goal flattened to the footprint the block must cover. Table V carries the statistics.
Fig. 8: Goal distance per replanning step, log axes, band a bootstrap 95% interval over the 16 episodes.
World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions. Existing approaches typically represent the world as RGB frames or latent counterparts while predicting actions as end-effector poses or joint angles, but they often struggle to capture the 3D spatial structure and contact geometry central to dexterous manipulation. We introduce Point World Action Model (PointWAM), a 3D world action model that decomposes the world into a scene (i.e., environment) and hands (i.e., actor), and jointly forecasts both as 3D point trajectories within a shared space-time coordinate frame. This explicit, disentangled representation enables effective pre-training on large-scale human demonstration videos without requiring any task-specific object or keypoint selection. Given a colored point cloud and a language instruction, PointWAM predicts how the scene and hands co-evolve in 3D space over time, then retargets the forecast hand motion to robot actions. Pre-training on human videos improves average DexJoCo success by 56.9 percentage points, and scene-trajectory supervision adds 10.9 points over forecasting the hands alone. With both, PointWAM surpasses the prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong VLAs on a real robot.
Predicting object dynamics (i.e., world modeling) is a fundamental challenge for robotic manipulation, and modeling deformable objects presents a particularly difficult case due to their high-dimensional state spaces and complex material properties. While current world models approach this through two distinct paradigms: learning the dynamics over the 2D pixel space or more explicit 3D geometric space. A systematic understanding of their relative strengths and limitations remains elusive due to the lack of diverse, large-scale real-world data. To address this, we present Deform360, a large-scale visuotactile dataset featuring 198 daily-life objects, 1,980 interaction sequences, and over 215 hours of observations from 41 surround-view cameras and bimanual tactile grippers to capture both global motion and contact-induced local deformations. Leveraging a novel markerless visuotactile 3D tracking pipeline to extract dense geometry and motion, we systematically evaluate current state-of-the-art world models, comparing 2D video models against 3D particle models. Finally, we provide a preliminary demonstration indicating the real-world applicability of our dataset by performing robot planning tasks on deformable objects. Our analysis reveals key insights into the trade-offs between structural priors and scalability, providing a solid benchmark for future research in generalizable deformable object-centric world modeling. Project website: https://deform360.lhy.xyz
Hongyu Li, Wanjia Fu, Xiaoyan Cong +11
Brown University, Providence, RI · Columbia University, New York, NY · Massachusetts Institute of Technology, Cambridge, MA
Latent action world models let agents plan new behaviors at test time by predicting how actions change the environment, and joint-embedding predictive architectures (JEPAs) do so by forecasting future latent states rather than pixels. Yet nearly all such models see the world through a camera, even though robotic manipulation is fundamentally geometric: in robotics goals for manipulation are traditionally specified by target object poses, not by images of the object once placed. We ask whether latent planning survives a shift from appearance to geometry, on the observation side as well as on the goal specifications side. To answer this, we extend the stable-worldmodel evaluation platform with simulated LiDAR-style raycast point clouds as a new sensor modality, and adapt three JEPA designs to point clouds: a frozen-encoder model built on Utonia features, a distribution-prior model based on LeWM, and an action-sensitive model based on Delta-JEPA. We further introduce a goal-encoding mechanism that constructs the goal latent from the current latent and a 3D target pose, removing the need for goal images or goal point clouds. A comparative evaluation of the different anti-collapse mechanisms shows that point-cloud world models can match their image-based counterparts, demonstrating that the modality shift from appearance to geometry is achievable. All models are released as open weights with open-source training and inference code, to make world-model planning accessible for LiDAR-driven and pose-directed robotic tasks.
Fabio F. Oberweger, Michael Schwingshackl, Markus Murschitz
AIT Austrian Institute of Technology Assistive & Autonomous Systems