DyG$^2$T: Modeling Object Dynamics with 3D Gaussian Temporal-Spatial Particle Graph Transformer

Yansong Wang; Zhaobo Qi; Xinyan Liu; Beichen Zhang; Shuhui Wang; Weigang Zhang; Qingming Huang

cs.CVAug 19, 2026

DyG $^2$ T: Modeling Object Dynamics with 3D Gaussian Temporal-Spatial Particle Graph Transformer

Authors: Yansong Wang, Zhaobo Qi, Xinyan Liu, Beichen Zhang, Shuhui Wang, Weigang Zhang, Qingming Huang

Organizations: School of Computer Science and Technology, Harbin Institute of Technology at Weihai, Weihai 264209, China · Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing 101408, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China

Abstract

Modeling object dynamics from limited visual observations is a fundamental problem for enabling accurate motion trajectory prediction in embodied interaction scenarios. Existing dynamics modeling methods first compress reconstructed particle representations into sparse Key Points and model their evolution using locally constrained interactions, thereby discarding fine-grained local details and obscuring discriminative interaction modeling across spatial and temporal scales, leading to drifting trajectories and inaccurate appearance prediction. To tackle these issues, we propose DyG $^2$ T, a dynamics modeling framework that infers object motion trajectories by spatially completing and temporally discriminating Key Point representations and modeling multi-scale interaction over particle graphs. Spatially, DyG $^2$ T enriches each Key Point by aggregating neighboring raw particle positions to recover fine-grained local details, while explicitly encoding relative offsets among Key Points to enhance geometric structure perception. Temporally, we introduce a Temporal Disentangling Network (TDN) to identify dominant cross-frame variations in latent space and amplify inter-frame differences, yielding temporally discriminative representations that are subsequently aggregated via Temporal Attention to capture frame-wise temporal evolution cues. For comprehensive interaction modeling, a Particle Graph Transformer leverages global attention to preserve discriminative long-range dependencies among Key Points, mitigating representation homogenization induced by locality-constrained modeling and providing a robust basis for accurate trajectory prediction. Experiments on both synthetic and real-world datasets demonstrate that DyG $^2$ T achieves accurate dynamics modeling and reasoning, and exhibits strong cross-object and real-world generalization.

Figures & tables

Fig. 1: DyG 2 T vs. differentiable network-based methods for dynamics modeling of a falling Banana. Colored curves denote motion trajectories. Differentiable network-based methods rely on Key Point-level spatial attributes and propagate features node-wise, leading to appearance artifacts and trajectory drift. In contrast, DyG 2 T constructs discriminative embeddings by modeling multi-scale interaction based on completed particle features, enabling accurate trajectory and appearance decoding.

Fig. 2: Overview of the DyG 2 T. (a) DyG 2 T utilizes dynamic reconstruction to extract trackable particle representations. (b) The Spatial-Temporal Feature Completion and Aggregation performs semantic completion and temporal aggregation at both particle and object levels, which spatiotemporally enhance particle representations. (c) With the global attention of the Particle Graph Transformer, DyG 2 T enables discriminative multi-scale dependencies modeling.

	CD↓		EMD↓
Objects	Spring-Gaus	DyG 2 T(Ours)	Spring-Gaus	DyG 2 T(Ours)
Torus	0.012	0.008	0.003	0.001
Cross	0.016	0.010	0.005	0.002
Cream	0.014	0.012	0.007	0.005
Apple	0.014	0.011	0.006	0.003
Paste	0.011	0.008	0.003	0.002

TABLE I: Quantitative results of dynamic reconstruction for DyG 2 T and baselines on Spring-Gaus synthetic dataset.

Metrics	Methods	Torus	Cross	Cream	Apple	Paste	Chess	Banana	Mean
CD↓	Spring-Gaus	0.033	0.046	0.032	0.047	0.068	0.053	0.184	0.066
	GS-Dynamics	0.073	0.122	0.154	0.043	0.227	0.284	0.328	0.176
	DyG 2 T(Ours)	0.029	0.038	0.027	0.028	0.038	0.050	0.055	0.039
EMD↓	Spring-Gaus	0.014	0.024	0.023	0.029	0.035	0.027	0.091	0.035
	GS-Dynamics	0.033	0.062	0.097	0.021	0.164	0.200	0.171	0.107
	DyG 2 T(Ours)	0.013	0.018	0.013	0.015	0.020	0.021	0.029	0.019

TABLE II: Quantitative results of dynamics modeling and reasoning on Spring-Gaus synthetic dataset.

Metrics	Methods	Spring-Gaus real-world					Unity3D-H
Metrics	Methods	Dog	Potato	Pig	Burger	Bun	Polyhedron
PSNR↑	Spring-Gaus	21.499	20.881	21.136	21.026	20.456	31.027
	GS-Dynamics	26.141	28.623	27.114	27.969	26.929	31.352
	DyG 2 T(Ours)	27.676	27.933	27.750	30.645	27.197	31.768
SSIM↑	Spring-Gaus	0.987	0.985	0.986	0.985	0.984	0.985
	GS-Dynamics	0.988	0.989	0.989	0.988	0.988	0.987

TABLE III: Quantitative results of dynamics modeling and reasoning on Spring-Gaus real-world dataset and our Unity3D-H dataset.

Methods	CD↓	EMD↓	PSNR↑	SSIM↑	LPIPS↓
Spring-Gaus	0.030	0.012	12.197	0.850	0.350
GS-Dynamics	0.116	0.049	13.342	0.861	0.235
PAC-NeRF	0.284	0.203	15.188	0.900	0.187
GIC	0.116	0.132	14.009	0.882	0.217
DyG 2 T ${}_{\text{noisy}}$	0.035	0.016	13.647	0.878	0.188
DyG 2 T(Ours)	0.015	0.000	14.080	0.893	0.129

TABLE IV: Quantitative results of DyG 2 T, its variants, and baselines for heterogeneous torus dynamics modeling and reasoning on the Spring-Gaus synthetic dataset.

Fig. 3: Qualitative results of dynamics reasoning. (a), (b), (c), and (d) correspond to Banana, Toothpaste, Apple, and Cross, respectively.

Fig. 4: Qualitative results of dynamics modeling and reasoning on the Heterogeneous Torus. The spatial positions of dynamic objects are calibrated using a consistent coordinate system.

Methods	LSE↓	SPC↑	ACE↓
GS-Dynamics	0.499	-0.248	0.017
DyG 2 T(Ours)	0.225	0.089	0.016

TABLE V: Quantitative results of dynamics modeling and reasoning physical consistency evaluation on the Heterogeneous Torus.

Fig. 5: Qualitative results of module ablation study. The vertical axis shows the object motion’s temporal evolution.

Methods	CD↓	EMD↓	PSNR↑	SSIM↑	LPIPS↓
w/o spatial	0.354	0.203	16.174	0.875	0.220
w/o temporal	0.227	0.120	16.493	0.882	0.198
w/o transformer	0.197	0.110	16.264	0.875	0.201
DyG 2 T(Ours)	0.055	0.029	17.904	0.919	0.122

TABLE VI: Quantitative results of module ablation study.

Fig. 6: Qualitative results of the Particle Graph Transformer ablation study. (a) DyG 2 T and its w/o transformer variants on the dynamics modeling and reasoning task of the Heterogeneous Torus. (b) Local zoom-in views for evaluating the fine-grained local details.

Fig. 7: Visualization of the Temporal Disentangling Net (TDN) ablation. The Key Point feature embeddings before (a) and after (b) passing through TDN from frames $t-2$ , $t-1$ , and $t$ are projected into 2D using t-SNE and plotted as scatter plots, with colors indicating frame indices. Cosine similarities among the 3 frames’ Key Point embeddings are also computed before (c) and after (d) TDN and visualized as heatmaps.

	Methods	CD↓	EMD↓	PSNR↑	SSIM↑	LPIPS↓
1	DyG 2 T k=8	0.244	0.125	16.368	0.878	0.207
2	DyG 2 T k=32	0.169	0.092	16.365	0.879	0.190
3	PN Avg	0.107	0.240	16.071	0.871	0.242
4	PN Att	0.116	0.246	16.012	0.869	0.246
5	w/o PosDiff	0.372	0.178	16.742	0.886	0.195
6	w/o PosAware	0.107	0.056	17.613	0.912	0.139

TABLE VII: Ablation study on neighborhood ranges

k

(Row 1&2), Raw Points neighborhood aggregation strategy (Row 3&4), the Particle-level Spatial Semantic Completion (Row 5&6), and the Object-level Dynamic Temporal Aggregation (Row 7

\sim

10).

Obs. Windows	CD↓	EMD↓	PSNR↑	SSIM↑	LPIPS↓
5	0.018	0.090	16.709	0.887	0.186
3	0.055	0.029	17.904	0.919	0.122

TABLE VIII: Quantitative results of dynamics modeling and reasoning under different observation window sizes.

Frames	PSNR↑	SSIM↑	LPIPS↓
Collision Frame	18.410	0.927	0.109
Rest Frame	17.687	0.916	0.128

TABLE IX: Quantitative results of dynamics modeling and reasoning on collision and non-collision frames for DyG 2 T.

Variants	CD↓	EMD↓	PSNR↑	SSIM↑	LPIPS↓
$N^{\ast}=50$	0.470	0.247	16.003	0.869	0.248
$N^{\ast}=100$	0.055	0.029	17.904	0.919	0.122
$N^{\ast}=150$	0.098	0.050	17.509	0.909	0.151

TABLE X: Ablation study of the number of Key Points

N^{\ast}

in the Particle Graph.

Variants	CD↓	EMD↓	PSNR↑	SSIM↑	LPIPS↓
$k_{G}=2$	0.564	0.255	18.436	0.910	0.183
$k_{G}=5$	0.055	0.029	17.904	0.919	0.122
$k_{G}=10$	0.186	0.107	16.398	0.881	0.193

TABLE XI: Ablation study of the presence of edges between Key Points and their top-

k_{G}

nearest neighbors.

$\epsilon$	CD↓	EMD↓	PSNR↑	SSIM↑	LPIPS↓
$\epsilon=2$	0.106	0.056	17.047	0.897	0.161
$\epsilon=5$	0.055	0.029	17.904	0.919	0.122
$\epsilon=8$	4.844	2.406	19.062	0.933	0.143

TABLE XII: Ablation study of the size of the training temporal window

\epsilon

Methods	Time(s)↓	FPS↑	Params(M)↓	FLOPs(M)↓
GS-Dynamics	6.852	4.378	2.900	1707.674
w/o spatial	2.823	10.627	2.902	2884.713
w/o temporal	2.887	10.391	4.878	4410.182
w/o transformer	2.862	10.482	4.647	4107.910
DyG 2 T (Ours)	3.074	9.761	4.650	4393.837

TABLE XIII: Inference-phase computational resource statistics.

Trajectory Prediction

Trajectory prediction forecasts how people, vehicles, robots, and other dynamic systems will move from observed histories and context.

See all 187 papers

Learning Equivariant Neural-Augmented Object Dynamics From Few Interactions
May 4, 2026Sergio Orozco, Tushar Kusnur, Brandon May +2Deformable Object Manipulation Robotic Manipulation
Brown University, Providence, RI · Robotics and AI (RAI) Institute, Cambridge, MA · General Motors
Accelerated Spatio-Temporal Gaussian Rendering via Kinematic and Semantic Priors
May 25, 2026Denis Gridusov, Maxim Popov, Sergey Kolyubin3D Gaussian Splatting 4D Reconstruction
ITMO University, St. Petersburg, Russia
Learning a Particle Dynamics Model with Real-world Videos
May 22, 2026Chanho Kim, Suhas V. Sumukh, Li FuxinWorld Model Learning Physics-Based Simulation
Oregon State University

May 4, 2026J/K move · Enter open · S save

Learning Equivariant Neural-Augmented Object Dynamics From Few Interactions

Sergio Orozco, Tushar Kusnur, Brandon May, George Konidaris+1

Brown University, Providence, RI · Robotics and AI (RAI) Institute, Cambridge, MA · General Motors

Deformable Object Manipulation Robotic Manipulation Equivariant GNNs Physics-Informed ML Physics-Informed GNNs Deformable Object Simulation

Open paper AskPDF arXiv

Learning data-efficient object dynamics models for robotic manipulation remains challenging, especially for deformable objects. A popular approach is to model objects as sets of 3D particles and learn their motion using graph neural networks. In practice, this is not enough to maintain physical feasibility over long horizons and may require large amounts of interaction data to learn. We introduce PIEGraph, a novel approach to combining analytical physics and data-driven models to capture object dynamics for both rigid and deformable bodies using limited real-world interaction data. PIEGraph consists of two components: (1) a Physically Informed particle-based analytical model (implemented as a spring--mass system) to enforce physically feasible motion, and (2) an Equivariant Graph Neural Network with a novel action representation that exploits symmetries in particle interactions to guide the analytical model. We evaluate PIEGraph in simulation and on robot hardware for reorientation and repositioning tasks with ropes, cloth, stuffed animals and rigid objects. We show that our method enables accurate dynamics prediction and reliable downstream robotic manipulation planning, which outperforms state of the art baselines.

Abstract

Trajectory Prediction

Explore similar work

Learning Equivariant Neural-Augmented Object Dynamics From Few Interactions

Accelerated Spatio-Temporal Gaussian Rendering via Kinematic and Semantic Priors

Learning a Particle Dynamics Model with Real-world Videos