cs.RONov 5, 2026
SaveGPWM: Modeling the World as 3D Particle Motion
Organizations: Carnegie Mellon University · Toyota Research Institute
Abstract
We introduce GPWM, a Generative Particle World Model that predicts how 3D scenes evolve by generating the motion of particles and rigid bodies over time. Unlike video world models that generate scene appearance, GPWM operates directly in 3D physical space. A single representation and architecture cover rigid, deformable, cloth, and granular materials, in both passive physics and robotic interaction. GPWM is an object-aware multi-resolution transformer that combines flow matching with diffusion forcing to generate coherent physical trajectories far beyond its training horizon. We train and evaluate GPWM on ParticleWorlds, a 25.5-million-frame corpus spanning diverse geometries, materials, interactions, and robot embodiments. GPWM outperforms the best prior learned-dynamics baseline in each of six physical domains, reducing prediction error by 31% on average. Breaking from the domain-specific setting of prior methods, a single Unified GPWM models these heterogeneous regimes with one set of weights, largely outperforming per-domain baselines and even surpassing our own specialist models in two domains. GPWM further generalizes to unseen geometries, physical properties, object counts, and rollout horizons. Beyond prediction, GPWM enables contact-rich planning by learning a generative prior over joint actuator-object motion. Instead of searching over unconstrained actions, planning searches the model's latent noise for interactions that achieve a desired object-space goal. With the same world model, noise-space planning substantially outperforms action-space search on pushing and picking, where complex contact sequences are hard to discover. Executed open-loop in simulation, the resulting plans achieve goal-directed manipulation without reference action trajectories or goal-specific policies.
Figures & tables
Figure 1: GPWM learns 3D particle dynamics across six physical domains. Top: samples from the six domains of ParticleWorlds . Middle: given the observed frames and a noisy future, a U-shaped multi-resolution spatio-temporal transformer denoises the future motion of every particle. Bottom: GPWM (the better of its specialist and unified models in each domain) lowers prediction error relative to the strongest baseline in every domain, by 31% on average, rolls out 55 past its 16-frame training window, simulates 3D dynamics from images, and plans via guided diffusion, succeeding at picking where action-space search fails.
Figure 2: GPWM architecture. Noisy per-particle tokens, which embed displacement, actuator motion, geometry, physical parameters, and material, pass through a multi-resolution DiT. PointNet++ layers pool particles into per-object anchors on the way down and interpolate back on the way up, and long skips connect matching levels. Each block (right) applies temporal attention, 3D relative spatial attention, and pointwise feed-forward layers, modulated by AdaLN-Zero from the per-frame noise level , and the network outputs clean future displacements.
| Method | Free Fall | Dragging | Garment Folding † | Pushing | Pick and Place | Dexterous Hands |
| Graph/grid-based | 97.2 | 30.4 | 20.5 | 23.9 | 68.2 | 26.7 |
| PointWorld | – | – | 10.6 | 28.7 | 39.8 | 27.4 |
| PhysCtrl | 53.0 | 7.8 | 10.7 | 24.5 | 34.1 | 24.8 |
| Ours (Specialist) | 40.0 | 6.5 | 10.0 | 13.3 | 12.6 | 18.7 |
| Ours (Unified) | 41.5 | 6.1 | 10.8 | 18.0 | 11.1 | 22.6 |
Table 1: Prediction across diverse physical settings. Prediction error in millimeters ( ). Baselines and Ours (Specialist) are trained separately for each setting; Ours (Unified) uses a single model across heterogeneous settings. Graph/grid-based denotes GNS or PGND, depending on the setting. –: not applicable. † Garment Folding reports symmetric Chamfer distance averaged over 898 predicted frames on 20 test trajectories.
| 1–4 objects | 5–7 objects | Stiffness | ||||
| Method | Seen shapes | Unseen shapes | Seen shapes | Unseen shapes | Seen | Unseen |
| GNS | 96.8 | 97.2 | 100.9 | 97.6 | 116.7 | 113.2 |
| PhysCtrl | 48.5 | 53.0 | 122.7 | 120.9 | 51.2 | 51.7 |
| Ours (Specialist) | 37.5 | 40.0 | 45.1 | 45.2 | 47.3 | 48.9 |
| Ours (Unified) | 39.0 | 41.5 | 47.8 | 47.5 | 54.5 | 53.1 |
Table 2: Generalization on Free Fall. Mean particle position error in millimeters ( ). Training uses one to four objects per scene. Stiffness values are seen (10, 100, 1,000 kPa) or unseen within the training range (30, 300 kPa). Material types are weighted equally.
Figure 3: Predictions from a single image. Each row shows the input with the reconstructed objects outlined, their particles, and the predicted motion. Top, a tower of a toy car (elastic, 1 MPa), a pepper (sand, 100 kPa), a lemon (plasticine, 100 kPa), and a ball (elastic, 100 kPa). Bottom, an elastic ball (10 kPa) dropped onto a sand or a plasticine castle (100 kPa).
Figure 4: Planning performance. Task progress versus planning budget, measured after open-loop execution in simulation. Results are averaged over 50 scenarios for Pushing and Pile Manipulation and 25 problems for Picking. Reported values are interpolated at each task’s planning budget (dashed lines).
Figure 5: Planning with a generative interaction prior. Latent noise optimization guides a frozen GPWM toward an object-space goal (green ). When a wall blocks the direct path, the generated actuator trajectory lifts the object over it.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Examples from the ParticleWorlds domains. Each row shows one recorded episode at four moments, labeled with the time since its first frame; Pushing has one row for each of its two sources. From the top, four objects of three materials fall onto the table, a piggy bank is lifted by a patch on its back and released, two grippers fold a shirt, a rod pushes a block, a gripper pushes an object, a gripper picks up an object from a cluttered table and places it, and a hand turns a cube. Actuated parts are orange, and objects keep the colors of their meshes where available. Rigid bodies are drawn with extra points that follow their recorded poses, denser than the points the model receives. Rod Pushing records no time step, so its panels give frame indices.
| Domain | Materials | Actuation | Training frames |
| Free Fall | Elastic, elastoplastic, granular | — | 2,317,112 |
| Dragging | Elastic, elastoplastic | Material-point velocities | 183,949 |
| Garment Folding | Cloth | Two grippers | 4,068,020 |
| Pushing | Rigid | Rod or gripper | 5,958,397 |
| Pick and Place | Rigid | Gripper | 7,257,150 |
| Dexterous Hands | Rigid | Articulated hand | 5,679,049 |
Table 3: Overview of ParticleWorlds . Pushing combines rod and gripper recordings, and Pick and Place combines single-object and cluttered recordings. Frame counts refer to usable training episodes at each domain’s training sampling rate, including recorded settling frames.
| Specialist | Unified | |
| Parameters | 103.1 M | 502.2 M |
| Anchors per object at the second level | 256 deformable, 128 rigid body or link, 512 cloth | |
| Reduction to the third and fourth level | 4 and 2 | |
| Feature width per level | 64, 128, 256, 512 | 128, 320, 640, 1280 |
| Encoder blocks per level | 1, 2, 4, 9 | 1, 2, 4, 8 |
| Decoder blocks per level | 1, 2, 4 | 1, 2, 4 |
Table 4: Network sizes. The six specialists share one configuration. Levels run from the particles (first) to the coarsest anchors (fourth).
| Column | Clips | Held out | Predicted frames | Frame interval |
| Free Fall, 1 to 4 objects | 100 | shapes | 45 | 1/24 s |
| Free Fall, 5 to 7 objects (Table 2 ) | 100 | shapes and object count | 45 | 1/24 s |
| Dragging | 100 | shapes | 45 to 109 | 1/24 s |
| Pushing | 200 | objects | 16 to 93 | 0.05 s |
| Pick and Place | 200 | scenes | 233 to 921 | 0.05 s |
| Dexterous Hands | 50 | objects | 35 to 396 | 0.05 s |
Table 5: Test clips. The Held out column indicates how the test set differs from training. Predicted frames give the range of rollout lengths.
Figure 7: Free Fall. Two test clips with unseen shapes. Rows show the simulator’s recording and three models, and columns show the last given frame, three moments of the fall and the last predicted frame, labeled with the time since release. Ours has the lowest clip error on 172 of the 200 Free Fall test clips, 84 of 100 with one to four objects and 88 of 100 with five to seven.
Figure 8: Free Fall, continued.
Figure 9: Dragging. Two test clips with unseen shapes, laid out as in Figure 7 . The orange arrow shows the direction in which the dragged patch (orange) is pulled at that moment, from the recorded command that every model receives. Ours has the lowest clip error on 81 of the 100 Dragging test clips.
Figure 10: Dragging, continued. On this clip PhysCtrl has the lowest error.
Figure 11: Garment Folding, rendered back to the simulation mesh. A jacket and a T-shirt not seen in training, 0 to 30 s after the given history frames. Rows show the simulator and each model’s particles driving the same mesh, with the grippers (orange) at their recorded poses. The error under each panel is the mean position error of the cloth particles at that moment. All rows are smoothed for better mesh visualization, which moves vertices by 1.5 to 6 mm on average. Gaps in the PhysCtrl rows are where its cloth sinks below the floor.
Figure 12: Garment Folding, continued. Pants and a hooded jacket.
Figure 13: Pushing. Test clips with objects held out from training, laid out as in Figure 7 . The gripper (orange) is given to every model and drawn at its recorded pose. Ours has the lowest clip error on 165 of the 200 Pushing test clips.
Figure 14: Pushing, continued.
Figure 15: Pick and Place. Successful picks in held-out cluttered scenes, laid out as in Figure 13 . Ours has the lowest clip error on 185 of the 200 Pick and Place test clips.
Figure 16: Pick and Place, continued.
Figure 17: Dexterous Hands. A held-out screwdriver turned by the Sharpa hand. On this clip PhysCtrl and PointWorld have lower mean error than ours, but both move every point freely and thin the screwdriver.
Figure 18: Planning tasks. One scenario per task: the initial configuration with its target (top) and the outcome of the LNO plan executed open-loop in the simulator (bottom). The rod moves around the block before pushing it; the paddle sweeps part of the pile onto the target; the gripper grasps the jar before carrying it to the target.
Figure 19: Guidance step with streaming lookahead. For a gradient step on noisy sample , we get an estimate of the clean trajectory using streaming lookahead. Backpropagating through the lookahead allows guidance to optimize for long horizon tasks with sparse rewards only on the final frame.
| Planner | Pushing (100 s) | Pile Manipulation (200 s) | Picking (200 s) |
| LNO | pool 4–32; 1–48 | pool 4–32; 1–6 | pool 4–32; 1–12 |
| Noise CEM | 4–192 | 4–64 | 4–96 |
| Action CEM (GPWM) | 4–192 | 4–64 | 4–96 |
| Action CEM (PointWorld) | , 3–48 | , 3–36 | , 3–15 |
Table 6: Planner settings swept by each curve in Fig. 4 ; settings not listed are fixed. : CEM candidates per round; : CEM rounds; : LNO members refined with 5 gradient steps each. LNO’s smallest budgets decrease the screening pool and take no gradient steps.
| Planner | Pushing | Pile Manipulation | Picking |
| LNO | 89 57 | 27 23 | 89 56 |
| Noise CEM | 94 48 | 21 21 | 70 40 |
| Action CEM (GPWM) | 23 0 | 7 1 | 78 0 |
| Action CEM (PointWorld) | 37 1 | 29 19 | 98 0 |
Table 7: Predicted and executed task progress (%), at each task’s planning budget: progress in the world model’s own rollout of the chosen plan progress after open-loop execution in simulation.
Figure 20: Pushing. Executed plans at the 100 s budget on two scenarios. LNO in GPWM moves the rod around the block and pushes it onto the target. Action CEM, with GPWM or PointWorld, produces jittering rod motions that leave the block where it is.
Figure 21: Pile Manipulation. Each planner’s predicted outcome and the outcome of executing the same plan in Genesis, at the 200 s budget. LNO’s predicted push carries over to the simulator almost unchanged.
Figure 22: Picking. Executed plans at the 200 s budget for two objects. LNO in GPWM grasps and lifts each object to the target. Neither Action CEM plan establishes a grasp, and the objects are not lifted. The full Franka arm is rendered for visualization only; plans are executed with a floating hand.
Figure 23: PointWorld’s predictions for its Picking plans. The predicted object rises to the target while the predicted gripper stays 3–12 cm away from it; executed in simulation, the same plans leave the object on the table (Fig. 22 , right).
Figure 24: More drops from images. Each row shows the input image with the simulated objects outlined, their particles at the last input frame, and three predicted frames. Row 1, the lemon from the desk photograph falls onto the toy car as an elastic, a plasticine, and a sand object, shown 1.53 s after release. Row 2, a plasticine ball lands on the pepper, rolls off and pushes it aside. Rows 3 to 7 use generated images. Row 3, an elastic gummy bear falls onto a plasticine donut on a dorm desk, flattens and slides off. It stays flattened, a limitation of the model for soft elastic objects. Row 4, an elastic gel cube falls onto a pile of Mars regolith simulated as sand and scatters it. Row 5, an elastic jelly cube falls onto sprinkles simulated as sand, and the gummy bear beside it slumps by 1.53 s, the same limitation. Row 6, in a claymation scene, a plasticine tomato lands on an elastic one, and the two topple and come to rest apart. Row 7, in a 1970s photograph, an elastic lemon lands on an elastic jelly mold and comes to rest on top. Rows 6 and 7 are stylized images beyond the training data. Young’s moduli are 10 kPa for the gummy bears, donut, gel cube, bread roll, jelly cube, and jelly mold, 1 MPa for the toy car and the block, and 100 kPa for all other objects. The car, pepper, block, and tennis ball are elastic. Times are at each scene’s original scale.
Figure 25: Changing the observed history. Every row continues the same drop of the blue ball onto the pepper. In the top row the ball is released, lands, and rests on top at 0.35 s, when we freeze the scene. Without an edit it stays there. In each row below, only the ball’s four input frames at the freeze are rewritten so that it moves in the direction of the green arrow (the edited ball is drawn orange), and the model predicts the rest. Times are since the release at the original scale. The last column is 1.44 s after the freeze.
Figure 26: Dragging a sloth into a foam star. A sloth reconstructed from a photograph is dragged into a foam star, and every model gets the same start and drag. The arrow shows the commanded drag. Objects are drawn with dense points that follow each model’s predicted particles.
Figure 27: Folding garments from generated images. The grippers (orange) follow the motion recorded for a training garment, and their points are given to the model.
More in this topic
World Model Learning
World model learning builds predictive models of how environments change under actions to support planning, simulation, and decision-making.
See all 434 papers