cs.ROOct 16, 2026

ORD-WM: Learning Object-Centric Rigid Dynamics for 3D World Modeling in Robotic Manipulation

Authors: Yueyang Weng, Xiaopeng Zhang, Yuxi Ou, Yongjin Mu, Chengzhi Zhao, Yanjie Li

Organizations: Harbin Institute of Technology, Shenzhen

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

World models provide robots with a predictive mechanism for reasoning about how actions change object states in manipulation tasks. However, existing image-based and dense 3D world models do not explicitly impose object-level rigid motion, leaving rigid-object geometry preservation to be learned implicitly and making multi-step rollouts susceptible to shape drift or object-level motion inconsistency. To address this, we propose ORD-WM, a 3D world model that learns object-centric rigid dynamics for robotic manipulation. ORD-WM learns a vector-neuron variational autoencoder supervised by signed distance fields, which encodes each object point cloud as a geometry-aware latent representation composed of 3D vector features, and uses an action-conditioned dynamics model to predict object-wise SE(3) motion increments. Rather than generating unconstrained next latent states, ORD-WM analytically applies predicted object-wise rigid transformations to the current latent states, separating persistent object geometry from pose changes. Experiments show that ORD-WM enables stable, geometry-preserving multi-step prediction and supports efficient real-time replanning for closed-loop model predictive control in simulated and real-world manipulation scenarios.

Figures & tables

Figure 1: In contrast to video-based rollout and point-based scene-flow prediction, which may suffer from visual artifacts, shape drift, or object-level inconsistency, ORD-WM performs multi-step prediction by analytically applying predicted object-wise SE(3) updates to geometry-aware object latents composed of 3D vector features, preserving object geometry while updating pose. This latent-space rollout enables real-world object motion prediction and closed-loop model predictive control, where candidate actions are evaluated by their predicted effects on object states.
Figure 2: Overview of SDF-VNVAE. The VN encoder maps an object-centered point cloud to a latent representation composed of 3D vector features that rotate equivariantly with the input. The decoder predicts signed distance values from the latent sample ztiz_{t}^{i} and query points, and is invariant under their shared 3D rotation. For visual simplicity, the encoder output is drawn as ztiz_{t}^{i} ; during training, ztiz_{t}^{i} is sampled from the posterior parameterized by (μti,σti)(\mu_{t}^{i},\sigma_{t}^{i}) .
Figure 3: Overview of the dynamics model. The current scene state, robot proprioception, and action are encoded into tokens, which are processed to predict object-wise rigid motions and update the scene state.
Method1-step10-stepFull traj.Shape Cons. CD ↓\downarrow
ℓ2\ell_{2} mov.ℓ2\ell_{2} stat.ℓ2\ell_{2} mov.ℓ2\ell_{2} stat.ℓ2\ell_{2} mov.ℓ2\ell_{2} stat.
ORD-WM1.580.127.710.8637.608.500.00
PointWorld13.726.6183.188.75321.11216.1519.87
PointWorld (ft)10.041.0979.625.50225.6818.843.58
Table 1: Multi-step prediction comparison with PointWorld. Point-wise ℓ2\ell_{2} errors are reported for moving (mov.) and static (stat.) points at each horizon; Shape Consistency Chamfer Distance (CD) measures full-rollout drift from the initial object shape. All distances are in mm, and lower is better. PointWorld (ft) is fine-tuned on LIBERO. Bold marks the best result per metric.
Figure 4: Qualitative rollout predictions. Top: simulation rollouts and object position trajectories compared with ground truth. Bottom: real-world rollouts from observed scenes.
MethodSurface CD ↓\downarrowShape Cons. CD ↓\downarrowRollout FPS ↑\uparrow
1-step10-stepFull traj.B=1B=1B=8B=8
ORD-WM0.2520.6704.7290†\mathbf{0}^{\dagger}7846179
w/o SE(3) update0.6202.588131.6004.6839026443
w/o VN and SE(3) update8.5019.76352.40134.61112327439
w/o VN1.0051.5929.2180†\mathbf{0}^{\dagger}162124
Decoupled Geo.+Pose0.9941.56211.2870†\mathbf{0}^{\dagger}8916988
Table 2: Multi-step rollout prediction and efficiency. Surface CD is computed using points sampled from decoded object surfaces and evaluated at 1 step, 10 steps, and over full trajectories (avg. ∼\sim 170 steps). Shape Consistency CD measures full-rollout drift from the initial object shape. All CD values are in mm; rollout FPS uses batch sizes B=1,8B=1,8 . Bold marks the best reported result per metric.
CandidatesSuccess Rate ↑\uparrowPos. Error ↓\downarrowAvg. Steps ↓\downarrowPlanning Time ↓\downarrow
6410%3.72 cm36755.4 ms
25670%1.64 cm171115.7 ms
102490%0.972 cm85269.9 ms
Table 3: Real-world pushing MPC performance and planning efficiency. A trial is successful if the final object position error is below 2 cm. Pos. Error and Avg. Steps report the mean final position error and mean number of executed task steps, respectively. Planning Time is measured per MPC replanning step. Bold marks the best result per metric.
Figure 5: Closed-loop MPC with ORD-WM. The black-bordered panel shows the initial scene. At each control step, ORD-WM rolls out candidate trajectories in latent space (blue), selects the lowest-cost trajectory (red), and executes the first action in a receding-horizon manner; the accumulated executed trajectory is shown in black.
Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

ComponentOperationOutput shape
Inputcentered surface point cloud2048×32048\times 3
VN formatcoordinates as vector features1×3×20481\times 3\times 2048
Point embeddingVN layer with width 323232×3×204832\times 3\times 2048
VN trunkVN layers with widths [32,64,64,64][32,64,64,64]64×3×204864\times 3\times 2048
Global featurepoint-wise VN feature extraction and pooling64×364\times 3
Feature fusionconcatenate point and global features128×3×2048128\times 3\times 2048
Appendix
Table A.1: VN encoder architecture. Dimensions are reported as vector channels, and the batch dimension is omitted.
ComponentOperationOutput shape
Latent inputlatent vector set32×332\times 3
Query inputquery point as vector channel1×31\times 3
Concatenationlatent and query vectors33×333\times 3
VN trunkVN layers with widths [128,128,128][128,128,128]128×3128\times 3
VN standardizationcanonical-frame projection128×3128\times 3
Post-standardization VN layerVN layer with width 128128128×3128\times 3
Appendix
Table A.2: SDF decoder architecture. VN dimensions are reported as vector channels, while MLP dimensions are scalar channels. The batch and query dimensions are omitted.
HyperparameterValue
Surface points per object sample NpN_{p}20482048
SDF query points per object sample QQ20482048
Query sampling strategypre-generated mixed samples
Sampling rationear / volume / far / shell = 0.75/0.12/0.08/0.050.75/0.12/0.08/0.05
Near-surface noiseσ∼Uniform(0.001,0.05)\sigma\sim\mathrm{Uniform}(0.001,0.05)
SDF truncation / clampingnone; raw SDF values are used
Appendix
Table A.3: Training hyperparameters of the SDF-VNVAE.
DatasetMethodOriginal PoseRandom SO(3) PoseRotation Consistency CD
SDF L1Surface CDSDF L1Surface CD
LIBEROSDF-VAE1.2552.3349.0109.99011.112
SDF-VAE + pose aug.1.9383.4571.9133.4063.023
SDF-VNVAE0.8541.4410.8531.4630.228
OakInk2SDF-VAE3.3144.0117.3189.0999.856
SDF-VAE + pose aug.2.1762.9602.1713.6924.673
Appendix
Table A.4: SDF reconstruction accuracy on seen and unseen object sets. LIBERO is used as the seen evaluation set, and OakInk2 is used as the unseen evaluation set. All distance-based metrics are reported in millimeters. Lower is better for all metrics. Bold indicates the best result for each metric.
Figure A.1: Reconstruction visualization of SDF-VNVAE on seen and unseen objects. For each example, we show a predicted SDF slice and the reconstructed zero-level surface.
Figure A.2: Latent interpolation with SDF-VNVAE. The leftmost and rightmost shapes are reconstructed from endpoint object latents, while the shapes inside the dashed box are decoded from linearly interpolated latents. The smooth transitions indicate that the learned vector-valued latent space captures continuous object geometry.
Figure A.3: t-SNE visualization of SDF-VNVAE object latents. Points are colored by object category. The latent space reflects geometric similarity: visually similar categories may appear close to each other, typical instances form compact local groups, and geometrically unusual instances can appear as outliers.
ComponentConfiguration
BackboneTransformer encoder
Object/Gripper token dimension9999 ( 32×332\times 3 VN latent ++ 3D center)
Action token dimension99 (6D rotation delta ++ 3D translation delta)
Object/Gripper token projection99→51299\rightarrow 512
Action token projection9→5129\rightarrow 512
Transformer hidden dimension512512
Appendix
Table B.1: Architecture details of the dynamics model.
HyperparameterValue
OptimizerAdamW
Learning rate1×10−41\times 10^{-4}
Weight decay1×10−61\times 10^{-6}
Learning-rate scheduleCosine decay
Batch size256256
Epochs1000010000
Appendix
Table B.2: Training hyperparameters of the dynamics model.
Figure C.1: Additional multi-step rollout results on held-out trajectories, part 1.
Figure C.2: Additional multi-step rollout results on held-out trajectories, part 2.
Figure C.3: Additional multi-step rollout results on held-out trajectories, part 3.
Figure C.4: Additional multi-step rollout results on held-out trajectories, part 4.
Figure C.5: Counterfactual action rollouts. Starting from the branching time step indicated by the arrow, ORD-WM is rolled out with a nominal action sequence, a larger-amplitude motion sequence, and a gripper-release sequence. The resulting futures respond differently to the action interventions, showing action-dependent object motion prediction.
Figure C.6: Object generalization in counterfactual rollout. We replace the original bowl representation with unseen RoboCasa bowl instances while keeping the same action sequence. ORD-WM produces different rollouts according to the substituted object geometry.
Figure C.7: Geometry preservation during rollout. ORD-WM preserves object geometry by updating object latents through predicted SE(3) transformations, whereas the variant without the SE(3) update directly predicts future latents and accumulates shape distortion over time.
ItemValue
Optimization settings
Action spacerelative xyxy motion
Horizon HH88
Normalized action bounds[−1,1]2[-1,1]^{2}
Physical action range±5\pm 5 cm
Gripper stateclosed
Appendix
Table D.1: Pushing MPC optimization and cost settings.
More in this topic

World Model Learning

World model learning builds predictive models of how environments change under actions to support planning, simulation, and decision-making.

See all 414 papers

Explore similar work

CardsList

  1. 3D Point World Models: Point Completion Enables More Accurate Dynamics Learning

    Jun 30, 2026Skand Peri, Hung Nguyen, Chanho Kim +2World Model LearningWorld Models for Robotics

    Oregon State University

  2. PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation

    Oct 2, 2026Chunghyun Park, Beomjun Kim, Seungcheol Park +5Robotic ManipulationEmbodied World Models

    POSTECH · KAIST · RLWRLD +1

  3. World Models for Robotic Manipulation: A Survey

    May 27, 2026Fangyuan Wang, Ziyuan Wang, Guorui Pei +15World Model LearningWorld Models for Robotics

    Department of Mechanical Engineering, The Hong Kong Polytechnic University, Kowloon, Hong Kong SAR, China · Department of Mechanical Engineering and Automation, Harbin Institute of Technology, Shenzhen, China · School of Advanced Engineering, Great Bay University, Dongguan, Guangdong, China +9