World models are useful for robotic manipulation because robots can predict how actions change the states of objects before executing them. We present PointCast, a point-set world model that spans rigid, articulated, and deformable object manipulation. Its state is a set of 3D points on the object and the end-effector, mesh-free and topology-agnostic. Each point keeps its identity and is supervised on its own trajectory, which teaches the model where every point goes rather than only the shape the points form. Its backbone is a diffusion transformer that denoises a short window of future point positions, conditioned on the points' recent history and the commanded end-effector motion. The backbone's attention alternates between local and global, and cross-attention to the end-effector carries the coupling. This one architecture at 19.8M parameters and one training recipe cover four regimes, rigid objects, cloth, rope, and multi-joint cabinets, with a separate checkpoint trained for each. Trained on randomized simulation and scored against four baselines on the same metric, it is best on three of four regimes and second on rigid. Trained on a real-world robot teleoperation dataset, it has the lowest mean error in four of its six categories, is second in the other two, and improves on the dataset's own model in all six; zero-shot, its simulation checkpoints are best on two of four captures. Frozen inside sampling-based model-predictive control at one network evaluation per window, it plans four simulated tasks over 64 episodes, competitive with or outperforming every baseline on each. Project website at https://pointcast-wm.github.io.
Figures & tables
Fig. 1: PointCast frozen inside sampling-based MPC, planning from a point-set goal. One recorded episode each, at the first window the planner controls and after the last; the real panel is qualitative. Goal green, the executed end-effector path magenta, the pose goal flattened to the footprint the block must cover. Sec. IV-D plans three further tasks.
Fig. 2: Overview of PointCast . Every object point, actor point, and register enters as one token. The eight blocks alternate kNN-local and global self-attention, and each reads the actor tokens through cross-attention. At deployment, one Euler step turns a noise sample into the predicted window.
Fig. 3: The three attention patterns of the backbone. kNN-local restricts a query point (red) to its k nearest neighbors, the global panel is drawn from a register token (gold), which attends to every object token, and cross-attention reads the actor tokens (magenta).
Fig. 4: Each model rolled forward from its own predictions through one held-out episode per regime, near the end of the interaction. Ground truth blue, predictions orange, actor points magenta. The cabinet row highlights the moving part in yellow.
Fig. 5: Real-world rollouts on the PGND benchmark at step 30. GT is the photograph, and each prediction column shows the episode’s pre-interaction Gaussian splats carried by that method’s predicted point motion [ 4 ] . AdaptiGraph is shown but not scored (Sec. IV-B2 ).
Method
Rigid
Cloth
Rope
Cabinet ∗
PGND [ 4 ]
47.82
524.33
68.76
54.45
AdaptiGraph [ 3 ]
55.42
153.14
49.24
36.64
PTv3 [ 10 ]
52.06
42.82
72.30
10.34
ParticleFormer [ 8 ]
20.88
38.26
21.78
26.73
PointCast (ours)
24.40
31.90
14.67
6.84
n (interactions)
2796
1047
1119
996
TABLE I: Simulation benchmark, mean rollout error (mm)
Method
Cloth
Rope
Box
Bread
Bag
Sloth
PTv3 [ 10 ]
42.24
25.85
21.76
22.37
21.40
48.43
ParticleFormer [ 8 ]
31.65
24.10
24.54
22.61
18.62
44.32
PGND [ 4 ] (native)
44.49
39.14
27.79
20.33
16.16
42.94
PointCast (ours)
32.20
15.83
23.51
18.65
11.64
37.83
n (test episodes)
40
40
20
20
20
20
TABLE II: Real-world PGND benchmark, mean displacement error at the 3 s horizon (mm)
Fig. 6: Zero-shot rollouts, one episode per capture of Table III . GT is a point cloud on Push-T and YCB objects and the photograph on rope and cloth, and the end-effector drawn is the recorded one, not the actor points the models read.
Method
Push-T replay
Franka YCB
Rope
Cloth
PGND [ 4 ]
37.27
75.38
58.07
291.12
PTv3 [ 10 ]
24.67
60.30
83.45
54.77
ParticleFormer [ 8 ]
18.07
72.50
42.47
32.43
PointCast (ours)
15.35
52.86
49.05
42.20
n (interactions)
70
35
123
31
TABLE III: Zero-shot transfer, rollout error (mm)
Variant
Rigid
Cloth
Rope
Cabinet ∗
PointCast (Table I , flow, S=1 )
24.40
31.90
14.67
6.84
What decides articulation
ParticleFormer set-level loss
−14.5
+3.5
−2.2
+79.3
no distance prior
+1.7
−3.1
−2.1
+18.0
part-aware prior off (cabinet)
–
–
–
+15.6
Attention and tokenization
TABLE IV: Simulation ablations, percent change from the shaded row
Task
Metric
init
AdaptiGraph
PGND
PTv3
ParticleFormer
PointCast (ours)
SE(2) pose push
per-point (cm)
–
9.52 (0.19)
2.31 (5.98)
2.27 (0.25)
1.57 (0.17)
1.39 (0.10)
Sequential articulation
part err (cm)
–
7.97 / 2.67 (0.07)
2.25 / 0.38 (1.90)
0.72 / 0.97 (0.21)
0.88 / 0.41 (0.15)
0.42 / 0.38 (0.09)
Cloth drag
Chamfer (cm)
27.8
27.75 (0.27)
2.83 (2.23)
2.41 (0.03)
2.08 (0.03)
2.09 (0.04)
Rope drag
Chamfer (cm)
15.4
3.28 (0.31)
1.91 (12.08)
5.87 (0.03)
2.16 (0.03)
1.71 (0.04)
TABLE V: Planning across four tasks
Fig. 7: One recorded episode per task, the start and the final state each rollout model leaves under the same planner, cost, budget, seed, and goal. Object blue, goal green, end-effector magenta with its executed path, the pose goal flattened to the footprint the block must cover. Table V carries the statistics.
Fig. 8: Goal distance per replanning step, log axes, band a bootstrap 95% interval over the 16 episodes.