cs.RONov 5, 2026

GPWM: Modeling the World as 3D Particle Motion

Authors: Robert Ren, Matthew Bronars, He Zhu, Yikai Tang, Jianing Zhang, Sergey Zakharov, Pavel Tokmakov, Katerina Fragkiadaki

Organizations: Carnegie Mellon University · Toyota Research Institute

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We introduce GPWM, a Generative Particle World Model that predicts how 3D scenes evolve by generating the motion of particles and rigid bodies over time. Unlike video world models that generate scene appearance, GPWM operates directly in 3D physical space. A single representation and architecture cover rigid, deformable, cloth, and granular materials, in both passive physics and robotic interaction. GPWM is an object-aware multi-resolution transformer that combines flow matching with diffusion forcing to generate coherent physical trajectories far beyond its training horizon. We train and evaluate GPWM on ParticleWorlds, a 25.5-million-frame corpus spanning diverse geometries, materials, interactions, and robot embodiments. GPWM outperforms the best prior learned-dynamics baseline in each of six physical domains, reducing prediction error by 31% on average. Breaking from the domain-specific setting of prior methods, a single Unified GPWM models these heterogeneous regimes with one set of weights, largely outperforming per-domain baselines and even surpassing our own specialist models in two domains. GPWM further generalizes to unseen geometries, physical properties, object counts, and rollout horizons. Beyond prediction, GPWM enables contact-rich planning by learning a generative prior over joint actuator-object motion. Instead of searching over unconstrained actions, planning searches the model's latent noise for interactions that achieve a desired object-space goal. With the same world model, noise-space planning substantially outperforms action-space search on pushing and picking, where complex contact sequences are hard to discover. Executed open-loop in simulation, the resulting plans achieve goal-directed manipulation without reference action trajectories or goal-specific policies.

Figures & tables

Figure 1: GPWM learns 3D particle dynamics across six physical domains. Top: samples from the six domains of ParticleWorlds . Middle: given the observed frames and a noisy future, a U-shaped multi-resolution spatio-temporal transformer denoises the future motion of every particle. Bottom: GPWM (the better of its specialist and unified models in each domain) lowers prediction error relative to the strongest baseline in every domain, by 31% on average, rolls out 55 ×\times past its 16-frame training window, simulates 3D dynamics from images, and plans via guided diffusion, succeeding at picking where action-space search fails.
Figure 2: GPWM architecture. Noisy per-particle tokens, which embed displacement, actuator motion, geometry, physical parameters, and material, pass through a multi-resolution DiT. PointNet++ layers pool particles into per-object anchors on the way down and interpolate back on the way up, and long skips connect matching levels. Each block (right) applies temporal attention, 3D relative spatial attention, and pointwise feed-forward layers, modulated by AdaLN-Zero from the per-frame noise level τ\tau , and the network outputs clean future displacements.
MethodFree FallDraggingGarment Folding †PushingPick and PlaceDexterous Hands
Graph/grid-based97.230.420.523.968.226.7
PointWorld––10.628.739.827.4
PhysCtrl53.07.810.724.534.124.8
Ours (Specialist)40.06.510.013.312.618.7
Ours (Unified)41.56.110.818.011.122.6
Table 1: Prediction across diverse physical settings. Prediction error in millimeters ( ↓\downarrow ). Baselines and Ours (Specialist) are trained separately for each setting; Ours (Unified) uses a single model across heterogeneous settings. Graph/grid-based denotes GNS or PGND, depending on the setting. –: not applicable. † Garment Folding reports symmetric Chamfer distance averaged over 898 predicted frames on 20 test trajectories.
 1–4 objects5–7 objectsStiffness
MethodSeen shapesUnseen shapesSeen shapesUnseen shapesSeenUnseen
GNS96.897.2100.997.6116.7113.2
PhysCtrl48.553.0122.7120.951.251.7
Ours (Specialist)37.540.045.145.247.348.9
Ours (Unified)39.041.547.847.554.553.1
Table 2: Generalization on Free Fall. Mean particle position error in millimeters ( ↓\downarrow ). Training uses one to four objects per scene. Stiffness values are seen (10, 100, 1,000 kPa) or unseen within the training range (30, 300 kPa). Material types are weighted equally.
Figure 3: Predictions from a single image. Each row shows the input with the reconstructed objects outlined, their particles, and the predicted motion. Top, a tower of a toy car (elastic, 1 MPa), a pepper (sand, 100 kPa), a lemon (plasticine, 100 kPa), and a ball (elastic, 100 kPa). Bottom, an elastic ball (10 kPa) dropped onto a sand or a plasticine castle (100 kPa).
Figure 4: Planning performance. Task progress versus planning budget, measured after open-loop execution in simulation. Results are averaged over 50 scenarios for Pushing and Pile Manipulation and 25 problems for Picking. Reported values are interpolated at each task’s planning budget (dashed lines).
Figure 5: Planning with a generative interaction prior. Latent noise optimization guides a frozen GPWM toward an object-space goal (green ×\times ). When a wall blocks the direct path, the generated actuator trajectory lifts the object over it.
Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Examples from the ParticleWorlds domains. Each row shows one recorded episode at four moments, labeled with the time since its first frame; Pushing has one row for each of its two sources. From the top, four objects of three materials fall onto the table, a piggy bank is lifted by a patch on its back and released, two grippers fold a shirt, a rod pushes a block, a gripper pushes an object, a gripper picks up an object from a cluttered table and places it, and a hand turns a cube. Actuated parts are orange, and objects keep the colors of their meshes where available. Rigid bodies are drawn with extra points that follow their recorded poses, denser than the points the model receives. Rod Pushing records no time step, so its panels give frame indices.
DomainMaterialsActuationTraining frames
Free FallElastic, elastoplastic, granular—2,317,112
DraggingElastic, elastoplasticMaterial-point velocities183,949
Garment FoldingClothTwo grippers4,068,020
PushingRigidRod or gripper5,958,397
Pick and PlaceRigidGripper7,257,150
Dexterous HandsRigidArticulated hand5,679,049
Appendix
Table 3: Overview of ParticleWorlds . Pushing combines rod and gripper recordings, and Pick and Place combines single-object and cluttered recordings. Frame counts refer to usable training episodes at each domain’s training sampling rate, including recorded settling frames.
 SpecialistUnified
Parameters103.1 M502.2 M
Anchors per object at the second level256 deformable, 128 rigid body or link, 512 cloth
Reduction to the third and fourth level4 and 2
Feature width per level64, 128, 256, 512128, 320, 640, 1280
Encoder blocks per level1, 2, 4, 91, 2, 4, 8
Decoder blocks per level1, 2, 41, 2, 4
Appendix
Table 4: Network sizes. The six specialists share one configuration. Levels run from the particles (first) to the coarsest anchors (fourth).
ColumnClipsHeld outPredicted framesFrame interval
Free Fall, 1 to 4 objects100shapes451/24 s
Free Fall, 5 to 7 objects (Table 2 )100shapes and object count451/24 s
Dragging100shapes45 to 1091/24 s
Pushing200objects16 to 930.05 s
Pick and Place200scenes233 to 9210.05 s
Dexterous Hands50objects35 to 3960.05 s
Appendix
Table 5: Test clips. The Held out column indicates how the test set differs from training. Predicted frames give the range of rollout lengths.
Figure 7: Free Fall. Two test clips with unseen shapes. Rows show the simulator’s recording and three models, and columns show the last given frame, three moments of the fall and the last predicted frame, labeled with the time since release. Ours has the lowest clip error on 172 of the 200 Free Fall test clips, 84 of 100 with one to four objects and 88 of 100 with five to seven.
Figure 8: Free Fall, continued.
Figure 9: Dragging. Two test clips with unseen shapes, laid out as in Figure 7 . The orange arrow shows the direction in which the dragged patch (orange) is pulled at that moment, from the recorded command that every model receives. Ours has the lowest clip error on 81 of the 100 Dragging test clips.
Figure 10: Dragging, continued. On this clip PhysCtrl has the lowest error.
Figure 11: Garment Folding, rendered back to the simulation mesh. A jacket and a T-shirt not seen in training, 0 to 30 s after the given history frames. Rows show the simulator and each model’s particles driving the same mesh, with the grippers (orange) at their recorded poses. The error under each panel is the mean position error of the cloth particles at that moment. All rows are smoothed for better mesh visualization, which moves vertices by 1.5 to 6 mm on average. Gaps in the PhysCtrl rows are where its cloth sinks below the floor.
Figure 12: Garment Folding, continued. Pants and a hooded jacket.
Figure 13: Pushing. Test clips with objects held out from training, laid out as in Figure 7 . The gripper (orange) is given to every model and drawn at its recorded pose. Ours has the lowest clip error on 165 of the 200 Pushing test clips.
Figure 14: Pushing, continued.
Figure 15: Pick and Place. Successful picks in held-out cluttered scenes, laid out as in Figure 13 . Ours has the lowest clip error on 185 of the 200 Pick and Place test clips.
Figure 16: Pick and Place, continued.
Figure 17: Dexterous Hands. A held-out screwdriver turned by the Sharpa hand. On this clip PhysCtrl and PointWorld have lower mean error than ours, but both move every point freely and thin the screwdriver.
Figure 18: Planning tasks. One scenario per task: the initial configuration with its target (top) and the outcome of the LNO plan executed open-loop in the simulator (bottom). The rod moves around the block before pushing it; the paddle sweeps part of the pile onto the target; the gripper grasps the jar before carrying it to the target.
Figure 19: Guidance step with streaming lookahead. For a gradient step on noisy sample xtx_{t} , we get an estimate of the clean trajectory using streaming lookahead. Backpropagating through the lookahead allows guidance to optimize for long horizon tasks with sparse rewards only on the final frame.
PlannerPushing (100 s)Pile Manipulation (200 s)Picking (200 s)
LNOpool 4–32; PP 1–48pool 4–32; PP 1–6pool 4–32; PP 1–12
Noise CEMKK 4–192KK 4–64KK 4–96
Action CEM (GPWM)KK 4–192KK 4–64KK 4–96
Action CEM (PointWorld)K=64K{=}64 , RR 3–48K=64K{=}64 , RR 3–36K=64K{=}64 , RR 3–15
Appendix
Table 6: Planner settings swept by each curve in Fig. 4 ; settings not listed are fixed. KK : CEM candidates per round; RR : CEM rounds; PP : LNO members refined with 5 gradient steps each. LNO’s smallest budgets decrease the screening pool and take no gradient steps.
PlannerPushingPile ManipulationPicking
LNO89 →\rightarrow 5727 →\rightarrow 2389 →\rightarrow 56
Noise CEM94 →\rightarrow 4821 →\rightarrow 2170 →\rightarrow 40
Action CEM (GPWM)23 →\rightarrow 07 →\rightarrow 178 →\rightarrow 0
Action CEM (PointWorld)37 →\rightarrow 129 →\rightarrow 1998 →\rightarrow 0
Appendix
Table 7: Predicted and executed task progress (%), at each task’s planning budget: progress in the world model’s own rollout of the chosen plan →\rightarrow progress after open-loop execution in simulation.
Figure 20: Pushing. Executed plans at the 100 s budget on two scenarios. LNO in GPWM moves the rod around the block and pushes it onto the target. Action CEM, with GPWM or PointWorld, produces jittering rod motions that leave the block where it is.
Figure 21: Pile Manipulation. Each planner’s predicted outcome and the outcome of executing the same plan in Genesis, at the 200 s budget. LNO’s predicted push carries over to the simulator almost unchanged.
Figure 22: Picking. Executed plans at the 200 s budget for two objects. LNO in GPWM grasps and lifts each object to the target. Neither Action CEM plan establishes a grasp, and the objects are not lifted. The full Franka arm is rendered for visualization only; plans are executed with a floating hand.
Figure 23: PointWorld’s predictions for its Picking plans. The predicted object rises to the target while the predicted gripper stays 3–12 cm away from it; executed in simulation, the same plans leave the object on the table (Fig. 22 , right).
Figure 24: More drops from images. Each row shows the input image with the simulated objects outlined, their particles at the last input frame, and three predicted frames. Row 1, the lemon from the desk photograph falls onto the toy car as an elastic, a plasticine, and a sand object, shown 1.53 s after release. Row 2, a plasticine ball lands on the pepper, rolls off and pushes it aside. Rows 3 to 7 use generated images. Row 3, an elastic gummy bear falls onto a plasticine donut on a dorm desk, flattens and slides off. It stays flattened, a limitation of the model for soft elastic objects. Row 4, an elastic gel cube falls onto a pile of Mars regolith simulated as sand and scatters it. Row 5, an elastic jelly cube falls onto sprinkles simulated as sand, and the gummy bear beside it slumps by 1.53 s, the same limitation. Row 6, in a claymation scene, a plasticine tomato lands on an elastic one, and the two topple and come to rest apart. Row 7, in a 1970s photograph, an elastic lemon lands on an elastic jelly mold and comes to rest on top. Rows 6 and 7 are stylized images beyond the training data. Young’s moduli are 10 kPa for the gummy bears, donut, gel cube, bread roll, jelly cube, and jelly mold, 1 MPa for the toy car and the block, and 100 kPa for all other objects. The car, pepper, block, and tennis ball are elastic. Times are at each scene’s original scale.
Figure 25: Changing the observed history. Every row continues the same drop of the blue ball onto the pepper. In the top row the ball is released, lands, and rests on top at 0.35 s, when we freeze the scene. Without an edit it stays there. In each row below, only the ball’s four input frames at the freeze are rewritten so that it moves in the direction of the green arrow (the edited ball is drawn orange), and the model predicts the rest. Times are since the release at the original scale. The last column is 1.44 s after the freeze.
Figure 26: Dragging a sloth into a foam star. A sloth reconstructed from a photograph is dragged into a foam star, and every model gets the same start and drag. The arrow shows the commanded drag. Objects are drawn with dense points that follow each model’s predicted particles.
Figure 27: Folding garments from generated images. The grippers (orange) follow the motion recorded for a training garment, and their points are given to the model.
More in this topic

World Model Learning

World model learning builds predictive models of how environments change under actions to support planning, simulation, and decision-making.

See all 434 papers

Explore similar work

CardsList
Jun 30, 2026

3D Point World Models: Point Completion Enables More Accurate Dynamics Learning

Learning predictive models of the world enables robotic control through planning, potentially allowing robots to improvise solutions on new tasks. However, large video-based dynamics models lack explicit 3D spatial structure and suffer from geometrically inconsistent long-term rollouts with compounding errors. Emerging 3D dynamics models based on partial point clouds improve geometric consistency but remain sensitive to occlusions and accumulated prediction drift. To address these challenges, we present 3D Point World Models (3DPWM) - a task-agnostic world model that operates entirely in 3D space by first completing partial point clouds and then learning action-conditioned dynamics in this completed 3D scene. By operating on completed geometry, 3DPWM enables reliable long-horizon rollouts and more accurate cost evaluation for model-based planning while supporting adaptation to new tasks. Experiments across different robotic embodiments and tabletop manipulation benchmarks demonstrate that 3DPWM achieves significantly more reliable long-horizon rollouts (100-300+ steps), supports both open-loop and closed-loop planning, and enables successful sim-to-real transfer.

Skand Peri, Hung Nguyen, Chanho Kim +2

Oregon State University

World Model LearningWorld Models for RoboticsWorld Model-Based PlanningRobot Dynamics LearningAction-Conditioned World ModelsRobotic Manipulation+1
AskPDF
Apr 13, 2026

Grounded World Model: Latent Planning with Language Goals

World models such as DINO-WM and LeWM specify the goal with an image, which is difficult to obtain in advance for novel tasks. We present the Grounded World Model (GWM), a latent world model that enables zero-shot planning in the real world from language goals alone. Given a candidate action sequence and the current observation, GWM predicts the future in the visual space of a pretrained video-language embedding model. The frozen readout of this embedding model maps this imagined future and the task description into the same embedding space, where their negative cosine similarity serves as the planning cost. Training GWM requires only offline and task-agnostic video-action pairs and no language labels. In simulated experiments on WISER, planning with GWM, which executes the candidate action of lowest cost, solves 87% of 288 tasks with unseen instructions and visual signals, while ten fine-tuned VLAs average 22%. We then scale GWM up with real robot data, and use it for zero-shot planning in realistic simulation and real scenes. In the IsaacSim evaluation, planning with GWM completes all 70 trials across 14 tasks that require reasoning over referring expressions, matching a modular planner grounded by a frontier VLM, while pi0.5 reaches 37/70. Deployed on a real Franka, the same stack completes 55/60 separately evaluated pick-and-place sub-tasks, comparable to the modular planner's 52/60, with the full system running locally on a single consumer GPU. Project website: https://quanyili.github.io/gwm-wiser/.

Quanyi Li, Lan Feng, Haonan Zhang +4

EPFL · NUS

Language-Conditioned Robot PlanningWorld Models for RoboticsWorld Model-Based PlanningLatent World ModelsZero-Shot Robotic ManipulationAction-Conditioned World Models+8
AskPDF
Jul 2, 2026

PhysMani: Physics-principled 3D World Model for Dynamic Object Manipulation

Manipulating fast and dynamically moving targets in unstructured 3D environments remains challenging for embodied AI. Existing visual-language-action models and world models struggle with accurate 3D geometry and physically meaningful forecasting. We propose PhysMani, a framework that couples a physics-principled 3D Gaussian world model with a future-aware action policy model. The world model learns a divergence-free Gaussian velocity field via online optimization for fast and physically grounded future dynamics prediction. The policy model integrates the predicted 3D scene future dynamics through a learnable token based cross-attention module. We introduce PhysMani-Bench, a dynamic manipulation benchmark with 16 tasks, and demonstrate a superior success rate over strong baselines in both simulation and real-world robot experiments.

Peng Yun, Shouwang Huang, Hao Li +3

vLAR Group, The Hong Kong Polytechnic University, Hong Kong SAR, China · Astribot, China

World Models for RoboticsWorld ModelsWorld Model-Based PlanningRobot Manipulation Benchmarks
AskPDF