PhysLDM: Latent Diffusion for High-Fidelity Deformable Simulation

Yu Zhang; Xudong Xu; Xingang Pan

cs.CVOct 6, 2026

PhysLDM: Latent Diffusion for High-Fidelity Deformable Simulation

Authors: Yu Zhang, Xudong Xu, Xingang Pan

Organizations: S-Lab, Nanyang Technological University, Singapore · Shanghai Artificial Intelligence Laboratory, Shanghai, China

Abstract

Neural simulation of high-fidelity deformable bodies is a foundational challenge in computer graphics and physical AI. Long-horizon prediction for high-resolution 3D volumetric meshes is hard: autoregressive methods are susceptible to error accumulation, while direct multi-frame prediction at native resolution is computationally prohibitive. This motivates a compact spatiotemporal latent representation, which is largely unexplored for mesh-based volumetric physics. Meanwhile, it remains unclear whether deterministic regression or generative diffusion is the more appropriate predictive paradigm. To address these coupled challenges, we introduce PhysLDM, a unified latent-diffusion paradigm for one-shot volumetric deformable simulation. Its core is a holistic spatiotemporal VAE that avoids the "staircase" artifacts of standard temporal compression (as in common video VAEs), achieving ~2.48 mm reconstruction precision on meter-scale scenes at up to 78x token compression. Based on this reliable latent space, we systematically compare regression and diffusion methods. Our experiments uncover a key modeling insight: complex deformable dynamics are often chaotic, and in this regime deterministic regression tends to produce non-physical averages, whereas diffusion better models their distribution. Accordingly, we employ a latent diffusion model that effectively learns from the chaotic data to generate physically plausible trajectories. Trained purely kinematically on an Objaverse-scale dataset, a single PhysLDM generalizes zero-shot to unseen OOD datasets (GSO and Toys4K). Its differentiability further enables efficient solution of inverse problems and higher-order design optimization. To our knowledge, PhysLDM is the first high-fidelity spatiotemporal autoencoder and latent-diffusion paradigm for volumetric deformable dynamics, offering a scalable and robust approach to neural simulation.

Figures & tables

Figure 1: PhysLDM overview. Top, Reconstruction: Ground truth (rose) versus ST-VAE reconstruction (blue) for an unseen high-resolution soft mug (11,621 nodes, ${>}2\times$ the training cap). The trajectory is compressed by ${\sim}182\times$ with only a 5.3 mm mean error. Far right: a cross-section of the final frame visualizing the internal tetrahedral structure. Bottom Left, Prediction: One-shot prediction of 64-frame trajectories for diverse out-of-distribution (OOD) objects (GSO/Toys4K) across varying materials and initial velocities (inset: initial state; main view: predicted settled state). Bottom Right, Differentiable Applications: A classic material estimation task: recovering Young’s modulus $E$ (blue sequence) by matching a target state (textured) via gradient descent. Furthermore, designing the throw that best reveals the material is a second-order inverse task (Sec. 4.2 ). PhysLDM easily solves it with native support for higher-order derivatives. By contrast, all other differentiable simulators we tested either fail or are computationally prohibitive on this task.

Figure 2: Overview of PhysLDM. The framework consists of three main components: (1) A Spatial VAE (top left) that encodes the per-frame canonical mesh into a compact set of resolution-independent latent tokens. (2) An ST-VAE (bottom) that compresses the temporal trajectory into a single joint latent $\mathbf{Z}$ , which serves as the training target of the LDM. (3) An LDM (top right) that predicts the entire latent trajectory from the initial state in a single diffusion pass. Dec $\ast$ denotes the frozen decoders, which decode the latent into the final full trajectory.

		$500\leq N\leq 5{,}000$						$5{,}000<N\leq 10{,}000$
		DCC $\uparrow$	Recon $\downarrow$			Rigid $\downarrow$		Recon $\downarrow$	Rigid $\downarrow$
	ST-VAE	mean	mean	med	P90	rot	trans	mean	rot	trans
Objaverse Held-out	Standard	0.665	5.32	3.47	12.01	0.270	0.57	6.14	0.185	0.50
	Direct world-coord. †	0.880	6.50	4.21	15.60	0.951	3.89	7.99	0.846	4.14
	Holistic (ours)	0.949	2.48	1.62	8.08	0.276	0.53	4.71	0.169	0.47
GSO	Standard	0.641	3.46	1.64	8.83	0.161	0.48	4.30	0.149	0.42

Table 1: ST-VAE reconstruction comparison across the three evaluation sets, on

300

scenes per set within the training range and

300

scenes per set beyond the

5{,}000

-vertex training cap. DCC: deformation-change correlation; Recon: deformation reconstruction error (mm); rot / trans: rigid rotation (deg) and translation (mm) errors; full definitions in Appendix B . best and second-best per column within each block. The full comparison with all baselines is reported in Table 7 . † No factoring of Eq. ( 1 ); errors computed after removing the rigid motion from its output (Appendix B ).

Panel of Figure 3 — Figure 3: ST-VAE reconstruction on four scenes (a held-out Objaverse phone, a GSO boot, a Toys4K chess piece, and a GSO toy), showing three frames sampled along the time axis. Rows: ground truth (rose), ours, and two baselines. The standard baseline looks reasonable in these static frames, yet it suffers from the staircase artifact, which is hard to see in snapshots but is exposed by the DCC in Table 1 . The direct world-coordinate baseline produces noisy geometric detail. Scene details are listed in Table 11 . More visualizations of ST-VAE reconstruction are shown in Figs. 9 and 10 .

Panel of Figure 4 — Figure 4: Prediction on four scenes (a GSO punch-drop toy, a held-out Objaverse cap and bowl, and Toys4K grapes), showing three frames sampled along the time axis. Rows: ground truth, Diffusion (ours) , Regression, and the MGN-style AR baseline. The regression method predicts geometry with visible artifacts and jitter. MGN-style AR drifts over time. Scene details are listed in Table 12 . More visualizations of prediction are shown in Fig. 11 .

	Objaverse Held-out				GSO				Toys4K
Prediction Model	Vol. $\downarrow$	SI $\downarrow$	KE $\rightarrow$ 1	Err $\downarrow$	Vol. $\downarrow$	SI $\downarrow$	KE $\rightarrow$ 1	Err $\downarrow$	Vol. $\downarrow$	SI $\downarrow$	KE $\rightarrow$ 1	Err $\downarrow$
Diffusion (ours)	7.9	0.24	0.80	28.2	2.0	0.00	0.86	14.0	10.3	0.32	0.80	35.6
Regression	22.7	0.79	0.72	37.5	5.1	0.01	0.72	19.9	24.3	1.07	0.69	42.6
MGN-style AR	55.1	3.53	0.77	80.6	47.1	0.68	0.76	78.4	57.5	4.73	0.77	79.1

Table 2: Prediction comparison across the three evaluation sets, on the same

300

scenes per set within the training vertex range (

500\leq N\leq 5{,}000

) as Table 1 . Vol.: tetrahedral volume violation %; SI: self-intersecting surface triangles %; KE: deformation-to-rigid kinetic-energy ratio

KE_{\mathrm{def}}/KE_{\mathrm{rig}}

relative to ground truth (closer to

1

is better); Err: median per-vertex error in world coordinates (mm); full definitions in Appendix B . The full set of physics checks is reported in Table 8 . best and second-best per column.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

	ST-VAE Small	ST-VAE Base
Query tokens $K$	512	512
Window $T$	32	64
Temporal tokens $M$	4	8
Temporal compression	$8\times$	$8\times$
Hidden width	512	512
Joint latent $M\times(K{+}1)\times D$	$4\times 513\times 48$	$8\times 513\times 48$

Appendix

Table 3: The two ST-VAE configurations used in this paper. ST-VAE Small is used for the reconstruction comparisons of Table 1 . ST-VAE Base is the ST-VAE behind PhysLDM, i.e. the LDM experiments and the applications.

Figure 6: Consecutive-frame silhouette difference on scene 4 of Fig. 5 (a cat) for five frame pairs around the impact (red: previous frame only; blue: next frame only; grey: overlap; number: XOR over union). Ground truth and the holistic ST-VAE change steadily from frame to frame. The standard baseline and its two variants change much less than the ground truth for the first three pairs and then much more in the last two: the reconstruction holds and then snaps. The adaptive-compression baseline changes too little over all five pairs and releases the motion even later. The direct world-coordinate baseline, which uses the holistic temporal compression, follows the ground truth until the last pair.

Scene	$N$	Holistic (ours)	Standard	+ vel. loss	+ spec. loss	Adapt. compr.	Direct world-coord.
1	5,181	0.972	0.728	0.850	0.781	0.418	0.972
2	5,913	0.969	0.589	0.717	0.587	0.583	0.928
3	6,005	0.986	0.478	0.668	0.479	0.488	0.943
4	2,245	0.974	0.822	0.761	0.769	0.551	0.917
5	1,526	0.970	0.640	0.616	0.684	0.684	0.824

Appendix

Table 4: DCC (deformation-change correlation with ground truth, computed as in Table 1 ) on the five GSO scenes of Fig. 5 . Scenes 1 to 5 are the five columns of Fig. 5 in the same order, and scene 4 is the one shown in Fig. 6 . Best per row in bold.

Figure 7: Chaos in a traditional GPU simulator. Five reruns (columns) of the same soft-racket scene from byte-identical inputs (same mesh, initial state, material, and solver configuration), shown at four sampled frames (rows): before the drop, just after impact, mid-bounce, and the last frame of the trajectory. The runs are indistinguishable until impact and then settle into visibly different poses. The complex dynamics are chaotic, so the post-impact outcome is not a deterministic function of the initial state.

Difference between the two runs	$>1\%$ of size	$>5\%$ of size	$>10\%$ of size
(a) mean error, whole trajectory	$35$ ( $11.7\%$ )	$3$ ( $1.0\%$ )	$2$ ( $0.7\%$ )
(b) final-frame error	$71$ ( $23.7\%$ )	$23$ ( $7.7\%$ )	$4$ ( $1.3\%$ )

Appendix

Table 5: Divergence between two runs of the GPU solver from byte-identical inputs on the

300

Objaverse held-out scenes of Table 1 . Each entry gives the number and the percentage of scenes whose difference between the two runs exceeds the given fraction of the object size, measured (a) as the mean per-vertex error over the whole trajectory and (b) as the per-vertex error at the last frame.

	Non-chaotic ( $229$ scenes)			Chaotic ( $71$ scenes)
	Vol. $\downarrow$	KE $\to$ 1	Err $\downarrow$	Vol. $\downarrow$	KE $\to$ 1	Err $\downarrow$
Regression	$12.6$	$0.72$	$25.0$	$38.4$	$0.69$	$63.2$
Diffusion, single sample	$3.8$	$0.82$	$18.0$	$25.4$	$0.75$	$54.0$
Diffusion, best of $8$	$4.4$	$0.83$	$14.6$	$22.8$	$0.79$	$44.7$

Appendix

Table 6: Regression versus diffusion on the

229

non-chaotic and the

71

chaotic held-out scenes (two solver runs ending less or more than

1\%

of the object size apart). Medians over scenes. Vol.: volume violation rate (%), KE:

KE_{\mathrm{def}}/KE_{\mathrm{rig}}

relative to the ground truth, Err: world-coordinate error (mm) (Appendix B.4 ). Best of

8

picks, per scene, the sample closest to the ground truth.

		$500\leq N\leq 5{,}000$						$5{,}000<N\leq 10{,}000$
		DCC $\uparrow$	Recon $\downarrow$			Rigid $\downarrow$		Recon $\downarrow$	Rigid $\downarrow$
	ST-VAE	mean	mean	med	P90	rot	trans	mean	rot	trans
Objaverse Held-out	Standard	0.665	5.32	3.47	12.01	0.270	0.57	6.14	0.185	0.50
	+ velocity loss	0.842	5.34	3.15	11.60	0.831	3.33	6.25	0.638	3.88
	+ spectral loss	0.697	5.17	3.42	11.40	0.290	0.73	5.96	0.210	0.66
	Adaptive compression	0.511	5.86	3.65	13.72	0.440	5.09	6.59	0.314	4.30

Appendix

Table 7: Full version of Table 1 with all six ST-VAEs, including the two variants of the standard baseline and the adaptive-compression baseline. ST-VAE reconstruction comparison across the three evaluation sets (row blocks: held-out in-distribution / GSO zero-shot / Toys4K zero-shot). Left six columns: scenes within the training vertex range (

500\leq N\leq 5{,}000

;

300

scenes per set). Right three columns: large-mesh subsets beyond the

5{,}000

-vertex training cap (

N>5{,}000

;

300

scenes per set), reported on mean deformation and rigid errors only. DCC is the deformation-change correlation with ground truth and directly reflects the staircase artifact (higher is better); Recon is the reconstruction error of the deformation field, i.e. of the geometry with the rigid motion removed (mm); rot and trans are the reconstruction errors of the rigid rotation (deg) and translation (mm). Full definitions in Appendix B . best and second-best per column within each block. † Direct world-coordinate baseline (no factoring). Its errors are computed after removing the rigid motion from its output.

		Volume	Self-int.	Vel.	Acc.	$\frac{KE_{\mathrm{def}}}{KE_{\mathrm{rig}}}$	World err
	Prediction Model	viol. % $\downarrow$	tris % $\downarrow$	viol. % $\downarrow$	viol. % $\downarrow$	$\times$ GT $\rightarrow$ 1	median (mm) $\downarrow$
Objaverse Held-out	ST-VAE reconstruction	8.5	0.12	0.8	16.9	0.91	1.97
	Diffusion (ours)	7.9	0.24	7.0	25.8	0.80	28.2
	Regression	22.7	0.79	8.6	29.2	0.72	37.5
	MGN-style AR	55.1	3.53	21.4	69.0	0.77	80.6
GSO	ST-VAE reconstruction	2.1	0.00	0.5	11.8	0.92	1.21

Appendix

Table 8: Full version of Table 2 with all six physics checks and the ST-VAE reconstruction reference row. Prediction comparison across the three evaluation sets, on the same

300

scenes per set within the training vertex range (

500\leq N\leq 5{,}000

) as Table 1 . Viol. %: percentage of vertices or tetrahedra, over all frames, whose predicted volume, speed, or acceleration deviates from the ground truth beyond a threshold; Self-int. is the fraction of surface triangles in a self-intersection;

KE_{\mathrm{def}}/KE_{\mathrm{rig}}

is the deformation-to-rigid kinetic-energy ratio relative to ground truth (closer to

1

is better); World err is the per-vertex error in world coordinates (mm); full definitions in Appendix B . ST-VAE reconstruction is the ground truth encoded and decoded by the ST-VAE behind the LDM, shown as a reference. best and second-best among the three models per column within each block. MGN-style AR is a MeshGraphNets-style (MGN) autoregressive graph network retrained on our scenes.

	no ST-VAE	with ST-VAE
$N$	LDM on raw tokens (GiB)	ST-VAE Base step (GiB)
500	28.9	33.5
1,000	55.4	37.4
2,000	108.3	45.8
2,500	134.5	50.7
2,600	OOM	50.7

Appendix

Table 9: Training memory versus vertex count

N

on a

140

GiB H200 (

T{=}64

, bf16 , batch

1

). No ST-VAE: our LDM architecture applied directly to the

T{\times}N

raw vertex tokens, peak memory (GiB) of one training step. With ST-VAE: peak memory (GiB) of one ST-VAE Base training step. The LDM stage with the ST-VAE costs about

4.4

GiB per trajectory at any

N

and is not listed. OOM: out of memory.

$N$	bf16	fp32	Regression	LDM-20	LDM-50	Regression	LDM-20	LDM-50	(GiB)
	Decode (ms)		bf16 fps			fp32 fps			Peak
980	318	288	195	82	44	192	30	13	4.8
2,493	327	425	189	79	42	136	28	13	4.8
5,007	328	597	188	81	43	100	26	12	4.8
7,534	332	778	186	81	43	78	25	12	5.2
9,851	384	960	162	76	42	64	23	12	5.4

Appendix

Table 10: Batch-1 inference of PhysLDM on one H200 for

64

-frame trajectories at different vertex counts

N

(held-out scenes; the last three rows exceed the

5{,}000

-vertex training cap). Decode: time of the ST-VAE decoder. fps

=64/(\text{condition build}+\text{latent model}+\text{decode})

. Peak: maximum GPU memory allocated over the whole pass.

Panel of Figure 9 — Figure 9: ST-VAE reconstruction on two more scenes (a Toys4K cake and a GSO shape-sorter toy) at four frames sampled along the trajectory. Rows: ground truth (rose), Holistic (ours) , the standard baseline, its two variants (velocity loss and spectral loss), the adaptive-compression baseline, and the direct world-coordinate baseline. Scene details are listed in Table 11 .

Panel of Figure 10 — Figure 10: ST-VAE reconstruction on two more scenes (a held-out Objaverse object and a GSO turboprop), with the same rows, sampled frames, and rendering as Fig. 9 . Scene details are listed in Table 11 .

Figure	Object	Dataset	$N$	$E$ (Pa)	$v_{0}$ (m/s)
Fig. 3	phone	Objaverse (held-out)	6,106	$2.76\times 10^{6}$	$(+0.59,\,+0.00,\,+0.94)$ , $\|v_{0}\|=1.11$
Fig. 3	boot	GSO	6,569	$5\times 10^{4}$	$0$ (released from rest)
Fig. 3	chess piece	Toys4K	6,322	$5\times 10^{4}$	$0$ (released from rest)
Fig. 3	toy	GSO	9,403	$5\times 10^{4}$	$0$ (released from rest)
Fig. 9	cake	Toys4K	6,640	$1.45\times 10^{6}$	$(-0.21,\,+0.00,\,+0.58)$ , $\|v_{0}\|=0.62$
Fig. 9	shape-sorter toy	GSO	6,426	$5\times 10^{4}$	$0$ (released from rest)

Appendix

Table 11: Scenes of Figs. 3 , 9 , and 10 : source dataset, tetrahedral node count

N

, Young’s modulus

E

, and initial velocity

v_{0}

(m/s,

y

up). The Young’s moduli were chosen to show both soft bodies with large deformation and stiff, nearly rigid ones, so some values do not match the real material of the object.

Panel of Figure 11 — Figure 11: Prediction on four more scenes (a held-out Objaverse bucket and sofa, a GSO flower pot, and a GSO plush dog) at four frames sampled along the trajectory. Rows: ground truth, Diffusion (ours) , Regression, and the MGN-style AR baseline. Scene details are listed in Table 12 .

Figure	Object	Dataset	$N$	$E$ (Pa)	$v_{0}$ (m/s)
Fig. 4	punch-drop toy	GSO	7,767	$2\times 10^{6}$	$(-0.48,\,+0.00,\,-1.78)$ , $\|v_{0}\|=1.85$
Fig. 4	cap	Objaverse (held-out)	2,435	$5\times 10^{4}$	$0$ (released from rest)
Fig. 4	bowl	Objaverse (held-out)	5,877	$5\times 10^{4}$	$0$ (released from rest)
Fig. 4	grapes	Toys4K	8,767	$5\times 10^{4}$	$0$ (released from rest)
Fig. 11	bucket	Objaverse (held-out)	9,853	$2.6\times 10^{6}$	$(-0.94,\,+0.00,\,+1.33)$ , $\|v_{0}\|=1.63$
Fig. 11	sofa	Objaverse (held-out)	4,228	$1.05\times 10^{6}$	$(-1.34,\,+0.00,\,-0.20)$ , $\|v_{0}\|=1.36$

Appendix

Table 12: Scenes of Figs. 4 and 11 : source dataset, tetrahedral node count

N

, Young’s modulus

E

, and initial velocity

v_{0}

(m/s,

y

up). The Young’s moduli were chosen to show both soft bodies with large deformation and stiff, nearly rigid ones, so some values do not match the real material of the object.

Panel of Figure 12 — Figure 12: ST-VAE reconstruction of the extreme case, a soft birdcage (held-out Objaverse, $N{=}9{,}021$ nodes, $1.8\times$ the $5{,}000$ -vertex training cap, $E=5\times 10^{4}$ Pa) with dozens of slender bars. Top: four frames sampled along the trajectory, ground truth (rose) and Holistic (ours) . Bottom: the zoom-in view of the last frame, ground truth (left) and reconstruction (right), with the surface wireframe shown. The overall shape of the collapsed cage is recovered, while the thin bars are reconstructed distorted and tangled. Mean reconstruction error over the trajectory: $9.9$ mm, against $5.3$ mm for the compact teaser mug of $11{,}621$ nodes.

Panel of Figure 13 — Figure 13: Scene-level ST-VAE reconstruction on held-out multi-body scenes (two two-body and two three-body scenes, each body with a random Young’s modulus and initial velocity) at four sampled frames: just before and just after the mid-air collision, the first frame after ground impact, and a settled frame. Rows: ground truth (rose) and reconstruction.

Panel of Figure 14 — Figure 14: One-shot prediction on held-out multi-body scenes (two two-body and two three-body scenes, each body with a random Young’s modulus and initial velocity), sampled frames as in Fig. 13 . Rows: ground truth and our prediction conditioned on the initial state, rendered by driving each body’s original textured asset with the predicted tetrahedral trajectory.

Simulator	Differentiation Support	1st-Order Task (Material Estimation)	2nd-Order Task (Throw Design)
Ours	Autograd (higher order)	Success (40 seconds)	Success (7 minutes) Efficient 2nd-order grads.
DiffIPC	Analytic adjoint (1st order only)	26 minutes Unstable under defaults. Needs manual tuning.	Infeasible (Multi-day) ${\sim}635\times$ slower via finite diff.
Newton/Warp	Recorded tape	Fails Gradients violently explode.	Infeasible No usable 1st-order gradients.

Appendix

Table 13: Comparison of our model with two differentiable simulators on downstream applications. Our approach supports higher-order derivatives, enabling complex optimization tasks that are either computationally infeasible or numerically unstable for existing solvers.

Figure 15: Basic material estimation. (a) Four matched time samples of the observed trajectory ( $E^{\star}=60$ kPa), the initial model prediction ( $E_{0}=250$ kPa), and the prediction after 40 Adam steps ( $\hat{E}=52.4$ kPa). All views share the same camera and scale. (b) Absolute log-stiffness error for seven ground-truth moduli, with the illustrated case highlighted in blue. All seven final estimates are within $0.06$ dex; each optimization takes approximately 40 seconds. Curves show the recorded iterates without smoothing.

Figure 16: Beyond material estimation: designing a more informative throw. (a) Ground-truth replays of the initial and optimized release velocities at $E^{\star}=60$ kPa. Orange and blue overlays represent stiffness perturbations of $-0.05$ and $+0.05$ dex. The first three columns show true motion at matched times and a shared camera scale. In the final column, rigid translation and rotation offsets from the nominal trajectory are magnified by $5\times$ , using equal magnification in both rows. (b) Cumulative rigid-pose sensitivity, normalized by the initial throw’s final score, increases by $13.4\times$ . This score is computed from the unamplified GT trajectories. (c) Material estimation from the same distant initial guess, $E_{0}=1{,}000$ kPa. After 60 estimation steps, the initial throw remains near $901$ kPa, whereas the optimized throw reaches $73.7$ kPa, much closer to the true $60$ kPa. Throw design uses 30 Adam steps and approximately 7 minutes.

$N$	tokens	Concurrent (DiT-L)	Concurrent (DiT-B)
	( $T{\times}N$ )	GiB / ms	GiB / ms
500	32k	64.4 / 669	24.0 / 212
1,000	64k	122.1 / 1317	46.0 / 412
1,200	77k	OOM	54.7 / 495
2,000	128k	OOM	89.9 / 879
2,500	160k	OOM	111.8 / 1147

Appendix

Table 14: Memory and resolution limits of the concurrent raw-coordinate model on a

140

GiB H200: peak training memory (GiB) and GPU time per training step (ms) at batch

1

, i.e., per trajectory,

T{=}64

, bf16 , flash attention, for vertex counts

N

from

500

5{,}000

. OOM: out of memory at batch

1

. For reference, our LDM trains at

{\sim}4.4

GiB (Table 9 ) and

{\sim}27

ms per trajectory at any

N

Model	Stage	Training time
Concurrent (DiT-B)	(single)	74.6 h
Ours	spatial VAE	1.4 h
	ST-VAE	40.0 h
	LDM	19.6 h
	total	61.1 h

Appendix

Table 15: Training time of the two models on the same scenes (

\leq 1{,}000

vertices) and the same GPUs. Even with three stages, our pipeline takes less time than the concurrent model.

Panel of Figure 17 — Figure 17: The two models on four held-out scenes (a block, a spoon, a screwdriver and a bottle; all $\leq 1{,}000$ vertices) at the last frame of the trajectory. For each scene the full object is followed by a zoom-in view of the boxed region. Rows: ground truth (rose), our three-stage pipeline trained on the same scenes with the same budget (Table 15 ), and the concurrent DiT-B model. Our model preserves the geometric detail of every object. The concurrent model places the objects correctly, but its geometric detail is visibly noisy: the block bends and its faces ripple, the spoon and the shaft of the screwdriver are corrugated, and the bottle is lumpy. The zoom-in views show this noise on every surface.

Data	Output	Scenes	Vol. (%) $\downarrow$	$KE_{\mathrm{def}}/KE_{\mathrm{rig}}$ $\to 1$	Err (mm) $\downarrow$
Stable Neo-Hookean	ST-VAE reconstruction	$40$	$6.9$	$0.92$	$4.5$
Stable Neo-Hookean	LDM prediction	$40$	$8.9$	$0.83$	$23.9$
ARAP	ST-VAE reconstruction	$40$	$3.1$	$0.93$	$4.6$
ARAP	LDM prediction	$40$	$4.9$	$0.79$	$37.5$
Real captured	ST-VAE reconstruction	$22$	–	–	$6.0$
Real captured	LDM prediction	$22$	–	–	$874.9$

Appendix

Table 16: The pretrained PhysLDM on three data sources without adaptation. Medians over scenes. The volume and kinetic energy metrics need tetrahedra and are not defined on the point clouds.

Panel of Figure 18 — Figure 18: PhysLDM on real captured point clouds ( Jiang et al., 2025 ) : a plush sloth and a plush dinosaur (top), two pieces of cloth (middle), and two ropes (bottom), each at four frames sampled along the trajectory and rendered as points. Rows of each block: the tracked points (rose), the ST-VAE reconstruction, the LDM prediction of the pretrained model without adaptation (zero-shot), and the LDM prediction after fine-tuning (Table 17 ). The zero-shot prediction lets the resting object fall through the table, so it is not visible in the camera.

		Vol. (%) $\downarrow$		$KE_{\mathrm{def}}/KE_{\mathrm{rig}}$ $\to 1$		Err (mm) $\downarrow$
Data	Scenes	before	after	before	after	before	after
Stable Neo-Hookean	$32$	$8.9$	$6.2$	$0.82$	$0.91$	$22.9$	$7.6$
ARAP	$32$	$4.9$	$3.1$	$0.80$	$0.92$	$34.4$	$8.8$
Real captured	$16$	–	–	–	–	$921.3$	$6.2$

Appendix

Table 17: Fine-tuning the LDM of PhysLDM on each data source with the unchanged trainer. Medians over the scenes of each fine-tuning subset, before and after fine-tuning.

Explore similar work

Jun 25, 2026cs.CV

PhysiFormer: Learning to Simulate Mechanics in World Space

We present PhysiFormer, a diffusion transformer for physically-plausible 3D object motion. Unlike video world models that operate in view-dependent pixel space, PhysiFormer represents objects as 3D meshes expressed in world coordinates. Given the initial vertex positions and velocities, as well as object material type, rigid or elastic, the model samples future vertex trajectories. While related neural physics approaches build on ad-hoc latent spaces or explicitly enforce rigidity and causality, PhysiFormer shows that excellent results can be obtained without any such inductive biases, by casting vertex trajectory prediction as a single denoising diffusion process directly in world coordinates. The probabilistic formulation captures uncertainty in the learned dynamics, enabling diverse plausible futures from initial conditions, making this framework potentially useful for applications with unobserved uncertainty. The model features attention factorised over time, space, and objects for efficiency, enabling permutation-invariant multi-object reasoning without needing explicit object encoding. Trained on over 100k simulated trajectories, PhysiFormer generates rigid and elastic mechanics, and generalises to mixed-material settings, unseen real-world geometries, and larger object counts. It substantially outperforms autoregressive baselines in trajectory accuracy, rigidity preservation, and momentum-based physical consistency. Our results position coordinate-space diffusion as a promising step toward view-invariant, geometry-aware world modelling for robotics, graphics, and physical design. Visualisations, code, and models are available at https://yimingc9.github.io/physiformer.

Yiming Chen, Yushi Lan, Andrea Vedaldi

Visual Geometry Group, University of Oxford

Physics Simulation 3D Motion Diffusion Transformers Diffusion Models

Ask PDF

Sep 7, 2026cs.RO

PhysReal: Learning Real-World Deformable Object Physics via Hybrid Constitutive Modeling

Learning physically plausible dynamics from visual observations is essential for interactive world models and embodied agents. However, modeling real-world deformable objects remains challenging because their dynamics often arise from complex, spatially heterogeneous material responses. To address this challenge, we propose PhysReal, a video-driven framework for learning and simulating the underlying physics of real deformable objects. PhysReal integrates a spatially varying hybrid expert-neural constitutive model with a differentiable MPM simulator and 3DGS renderer. Analytical expert models provide interpretable physical priors, while neural constitutive residuals capture material responses beyond predefined formulations. Spatially distributed patches parameterize the constitutive field, enabling a continuous representation of local material variations. To organize the identification of this model from sparse visual observations, we adopt a progressive curriculum that sequentially optimizes global material properties, spatially varying local parameters, and neural constitutive residuals, together with complementary motion and mask supervision. Extensive experiments on diverse deformable-object interactions demonstrate that PhysReal achieves superior performance in dynamic reconstruction and future-state prediction, while showing strong potential for downstream robotic applications.

Yinan Deng, Jianqiao Song, Yisi Zhang +3

Deformable Linear Objects Physics Simulation Sim-To-Real Reinforcement Learning Objects

Ask PDF

Jun 16, 2026cs.CV

Adaptive Volumetric Mechanical Property Fields Invariant to Resolution

Accurate mechanical properties (or materials) Young's modulus (

E

), Poisson's ratio (

ν

) and density (

ρ

) are essential for reliable physics simulation of digital worlds, but most 3D assets lack this information. We propose AdaVoMP, a method for predicting accurate dense spatially-varying (

E

ν

ρ

) for input 3D objects across representations, improving the resolution, accuracy, and memory efficiency over the state-of-the-art. The foundation of our technique is a sparse and adaptive voxel structure SAV that efficiently represents both the input 3D shape and the material field output. We replace the fixed-voxel model of the most accurate prior method, VoMP, with a novel sparse transformer encoder-decoder model that learns to generate a unique SAV autoregressively for every input shape to represent its materials, achieving a resolution

16^3\times

higher than prior art. Experiments show that AdaVoMP estimates more accurate volumetric properties, even with lesser test-time compute than all prior art. This allows us to convert high-resolution complex 3D objects into simulation-ready assets, resulting in realistic deformable simulations.

Rishit Dagli, Donglai Xiang, Vismay Modi +4

NVIDIA · University of Toronto

Volume Voxel Deformable Linear Objects

Ask PDF