Organizations: School of Computer Science and Technology, Harbin Institute of Technology at Weihai, Weihai 264209, China · Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 101408, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China
Modeling object dynamics from limited visual observations is a fundamental problem for enabling accurate motion trajectory prediction in embodied interaction scenarios. Existing dynamics modeling methods first compress reconstructed particle representations into sparse Key Points and model their evolution using locally constrained interactions, thereby discarding fine-grained local details and obscuring discriminative interaction modeling across spatial and temporal scales, leading to drifting trajectories and inaccurate appearance prediction. To tackle these issues, we propose DyG2T, a dynamics modeling framework that infers object motion trajectories by spatially completing and temporally discriminating Key Point representations and modeling multi-scale interaction over particle graphs. Spatially, DyG2T enriches each Key Point by aggregating neighboring raw particle positions to recover fine-grained local details, while explicitly encoding relative offsets among Key Points to enhance geometric structure perception. Temporally, we introduce a Temporal Disentangling Network (TDN) to identify dominant cross-frame variations in latent space and amplify inter-frame differences, yielding temporally discriminative representations that are subsequently aggregated via Temporal Attention to capture frame-wise temporal evolution cues. For comprehensive interaction modeling, a Particle Graph Transformer leverages global attention to preserve discriminative long-range dependencies among Key Points, mitigating representation homogenization induced by locality-constrained modeling and providing a robust basis for accurate trajectory prediction. Experiments on both synthetic and real-world datasets demonstrate that DyG2T achieves accurate dynamics modeling and reasoning, and exhibits strong cross-object and real-world generalization.
Figures & tables
Fig. 1: DyG 2 T vs. differentiable network-based methods for dynamics modeling of a falling Banana. Colored curves denote motion trajectories. Differentiable network-based methods rely on Key Point-level spatial attributes and propagate features node-wise, leading to appearance artifacts and trajectory drift. In contrast, DyG 2 T constructs discriminative embeddings by modeling multi-scale interaction based on completed particle features, enabling accurate trajectory and appearance decoding.
Fig. 2: Overview of the DyG 2 T. (a) DyG 2 T utilizes dynamic reconstruction to extract trackable particle representations. (b) The Spatial-Temporal Feature Completion and Aggregation performs semantic completion and temporal aggregation at both particle and object levels, which spatiotemporally enhance particle representations. (c) With the global attention of the Particle Graph Transformer, DyG 2 T enables discriminative multi-scale dependencies modeling.
CD↓
EMD↓
Objects
Spring-Gaus
DyG 2 T(Ours)
Spring-Gaus
DyG 2 T(Ours)
Torus
0.012
0.008
0.003
0.001
Cross
0.016
0.010
0.005
0.002
Cream
0.014
0.012
0.007
0.005
Apple
0.014
0.011
0.006
0.003
Paste
0.011
0.008
0.003
0.002
TABLE I: Quantitative results of dynamic reconstruction for DyG 2 T and baselines on Spring-Gaus synthetic dataset.
Metrics
Methods
Torus
Cross
Cream
Apple
Paste
Chess
Banana
Mean
CD↓
Spring-Gaus
0.033
0.046
0.032
0.047
0.068
0.053
0.184
0.066
GS-Dynamics
0.073
0.122
0.154
0.043
0.227
0.284
0.328
0.176
DyG 2 T(Ours)
0.029
0.038
0.027
0.028
0.038
0.050
0.055
0.039
EMD↓
Spring-Gaus
0.014
0.024
0.023
0.029
0.035
0.027
0.091
0.035
GS-Dynamics
0.033
0.062
0.097
0.021
0.164
0.200
0.171
0.107
DyG 2 T(Ours)
0.013
0.018
0.013
0.015
0.020
0.021
0.029
0.019
TABLE II: Quantitative results of dynamics modeling and reasoning on Spring-Gaus synthetic dataset.
Metrics
Methods
Spring-Gaus real-world
Unity3D-H
Dog
Potato
Pig
Burger
Bun
Polyhedron
PSNR↑
Spring-Gaus
21.499
20.881
21.136
21.026
20.456
31.027
GS-Dynamics
26.141
28.623
27.114
27.969
26.929
31.352
DyG 2 T(Ours)
27.676
27.933
27.750
30.645
27.197
31.768
SSIM↑
Spring-Gaus
0.987
0.985
0.986
0.985
0.984
0.985
GS-Dynamics
0.988
0.989
0.989
0.988
0.988
0.987
TABLE III: Quantitative results of dynamics modeling and reasoning on Spring-Gaus real-world dataset and our Unity3D-H dataset.
Methods
CD↓
EMD↓
PSNR↑
SSIM↑
LPIPS↓
Spring-Gaus
0.030
0.012
12.197
0.850
0.350
GS-Dynamics
0.116
0.049
13.342
0.861
0.235
PAC-NeRF
0.284
0.203
15.188
0.900
0.187
GIC
0.116
0.132
14.009
0.882
0.217
DyG 2 T noisy
0.035
0.016
13.647
0.878
0.188
DyG 2 T(Ours)
0.015
0.000
14.080
0.893
0.129
TABLE IV: Quantitative results of DyG 2 T, its variants, and baselines for heterogeneous torus dynamics modeling and reasoning on the Spring-Gaus synthetic dataset.
Fig. 3: Qualitative results of dynamics reasoning. (a), (b), (c), and (d) correspond to Banana, Toothpaste, Apple, and Cross, respectively.
Fig. 4: Qualitative results of dynamics modeling and reasoning on the Heterogeneous Torus. The spatial positions of dynamic objects are calibrated using a consistent coordinate system.
Methods
LSE↓
SPC↑
ACE↓
GS-Dynamics
0.499
-0.248
0.017
DyG 2 T(Ours)
0.225
0.089
0.016
TABLE V: Quantitative results of dynamics modeling and reasoning physical consistency evaluation on the Heterogeneous Torus.
Fig. 5: Qualitative results of module ablation study. The vertical axis shows the object motion’s temporal evolution.
Methods
CD↓
EMD↓
PSNR↑
SSIM↑
LPIPS↓
w/o spatial
0.354
0.203
16.174
0.875
0.220
w/o temporal
0.227
0.120
16.493
0.882
0.198
w/o transformer
0.197
0.110
16.264
0.875
0.201
DyG 2 T(Ours)
0.055
0.029
17.904
0.919
0.122
TABLE VI: Quantitative results of module ablation study.
Fig. 6: Qualitative results of the Particle Graph Transformer ablation study. (a) DyG 2 T and its w/o transformer variants on the dynamics modeling and reasoning task of the Heterogeneous Torus. (b) Local zoom-in views for evaluating the fine-grained local details.
Fig. 7: Visualization of the Temporal Disentangling Net (TDN) ablation. The Key Point feature embeddings before (a) and after (b) passing through TDN from frames t−2 , t−1 , and t are projected into 2D using t-SNE and plotted as scatter plots, with colors indicating frame indices. Cosine similarities among the 3 frames’ Key Point embeddings are also computed before (c) and after (d) TDN and visualized as heatmaps.
Methods
CD↓
EMD↓
PSNR↑
SSIM↑
LPIPS↓
1
DyG 2 T k=8
0.244
0.125
16.368
0.878
0.207
2
DyG 2 T k=32
0.169
0.092
16.365
0.879
0.190
3
PN Avg
0.107
0.240
16.071
0.871
0.242
4
PN Att
0.116
0.246
16.012
0.869
0.246
5
w/o PosDiff
0.372
0.178
16.742
0.886
0.195
6
w/o PosAware
0.107
0.056
17.613
0.912
0.139
TABLE VII: Ablation study on neighborhood ranges k (Row 1&2), Raw Points neighborhood aggregation strategy (Row 3&4), the Particle-level Spatial Semantic Completion (Row 5&6), and the Object-level Dynamic Temporal Aggregation (Row 7 ∼ 10).
Obs. Windows
CD↓
EMD↓
PSNR↑
SSIM↑
LPIPS↓
5
0.018
0.090
16.709
0.887
0.186
3
0.055
0.029
17.904
0.919
0.122
TABLE VIII: Quantitative results of dynamics modeling and reasoning under different observation window sizes.
Frames
PSNR↑
SSIM↑
LPIPS↓
Collision Frame
18.410
0.927
0.109
Rest Frame
17.687
0.916
0.128
TABLE IX: Quantitative results of dynamics modeling and reasoning on collision and non-collision frames for DyG 2 T.
Variants
CD↓
EMD↓
PSNR↑
SSIM↑
LPIPS↓
N∗=50
0.470
0.247
16.003
0.869
0.248
N∗=100
0.055
0.029
17.904
0.919
0.122
N∗=150
0.098
0.050
17.509
0.909
0.151
TABLE X: Ablation study of the number of Key Points N∗ in the Particle Graph.
Variants
CD↓
EMD↓
PSNR↑
SSIM↑
LPIPS↓
kG=2
0.564
0.255
18.436
0.910
0.183
kG=5
0.055
0.029
17.904
0.919
0.122
kG=10
0.186
0.107
16.398
0.881
0.193
TABLE XI: Ablation study of the presence of edges between Key Points and their top- kG nearest neighbors.
ϵ
CD↓
EMD↓
PSNR↑
SSIM↑
LPIPS↓
ϵ=2
0.106
0.056
17.047
0.897
0.161
ϵ=5
0.055
0.029
17.904
0.919
0.122
ϵ=8
4.844
2.406
19.062
0.933
0.143
TABLE XII: Ablation study of the size of the training temporal window ϵ .
Learning data-efficient object dynamics models for robotic manipulation remains challenging, especially for deformable objects. A popular approach is to model objects as sets of 3D particles and learn their motion using graph neural networks. In practice, this is not enough to maintain physical feasibility over long horizons and may require large amounts of interaction data to learn. We introduce PIEGraph, a novel approach to combining analytical physics and data-driven models to capture object dynamics for both rigid and deformable bodies using limited real-world interaction data. PIEGraph consists of two components: (1) a Physically Informed particle-based analytical model (implemented as a spring--mass system) to enforce physically feasible motion, and (2) an Equivariant Graph Neural Network with a novel action representation that exploits symmetries in particle interactions to guide the analytical model. We evaluate PIEGraph in simulation and on robot hardware for reorientation and repositioning tasks with ropes, cloth, stuffed animals and rigid objects. We show that our method enables accurate dynamics prediction and reliable downstream robotic manipulation planning, which outperforms state of the art baselines.
Sergio Orozco, Tushar Kusnur, Brandon May +2
Brown University, Providence, RI · Robotics and AI (RAI) Institute, Cambridge, MA · General Motors
Reconstructing dynamic 3D scenes from multi-view videos is a foundational task for robotics, AR/VR, and digital twins. While 3D Gaussian Splatting (3DGS) provides state-of-the art scene representation, its dynamic adaptations (4DGS) face significant challenges. Early MLP-based deformation models excel at interpolating within training frames but fail at temporal extrapolation --- important feature for forecasting in predictive systems. To address this, physically grounded approaches have been proposed to enable robust future forecasting, while, concurrently, semantic 4DGS methods have been developed for scene analysis and language-driven prompting. However, there is currently no framework that unites both physical consistency and semantic understanding. In this paper, we propose the first, to the best of our knowledge, unified 4DGS approach that is both physically grounded and semantically aware. Furthermore, to optimize computational efficiency, we introduce a novel rigid-body constraint mechanism. Instead of predicting the physical dynamics for every Gaussian particle, our model predicts the motion of a single anchor Gaussian per semantic object, updating the remaining Gaussians via relative transformations. This architectural design significantly accelerates novel view synthesis compared to an unconstrained baseline, while enabling object-accurate extrapolation and intuitive semantic interaction. Our project page: https://be2rlab.github.io/r5dgs/.
Data-driven learning approaches for physics simulation, sometimes referred to as world models, have emerged as promising alternatives to traditional physics simulators due to their differentiable nature. Prior work has demonstrated impressive results in predicting the motions of rigid and non-rigid objects in complex scenes involving multiple interacting bodies. However, these models are typically trained in simulated environments because obtaining perfect state information such as complete scene point clouds and point correspondences over time is challenging in real-world settings. This reliance on synthetic data can limit their applicability when the sim-to-real gap is large. In this work, we aim to overcome these limitations by introducing a novel framework for training neural object dynamics models directly from unlabeled real-world videos. Specifically, we propose to learn a particle-based dynamics model compatible with a Gaussian splatting framework, which operates on dense particles derived from Gaussians (i.e., particles with scales and rotations) and predicts their position and rotation changes over time. The model is trained via rendering supervision, enabling learning from real-world videos without requiring particle-level labeled states. Our model operates directly on dense Gaussians without relying on heuristic subsampling anchor points. To enable this study, we also present a real-world dataset consisting of about 500 videos capturing diverse object interactions.