PhysLDM: Latent Diffusion for High-Fidelity Deformable Simulation

Yu Zhang; Xudong Xu; Xingang Pan

	ST-VAE Small	ST-VAE Base
Query tokens $K$	512	512
Window $T$	32	64
Temporal tokens $M$	4	8
Temporal compression	$8\times$	$8\times$
Hidden width	512	512
Joint latent $M\times(K{+}1)\times D$	$4\times 513\times 48$	$8\times 513\times 48$

Appendix

Table 3: The two ST-VAE configurations used in this paper. ST-VAE Small is used for the reconstruction comparisons of Table 1 . ST-VAE Base is the ST-VAE behind PhysLDM, i.e. the LDM experiments and the applications.

Figure 5: The staircase artifact on five GSO scenes. Top: ground truth at the impact frame. Bottom: deformation-change curve (defined in Appendix B.3 ) for ground truth and the six ST-VAEs of Table 7 over the impact window. DCC in Table 1 is the correlation of this curve with the ground-truth curve. On every scene the standard baseline, its two variants, and the adaptive-compression baseline stay below ground truth for several frames and then overshoot it in one frame, while the holistic ST-VAE follows the ground-truth curve. The direct world-coordinate baseline also follows the ground-truth curve, because it uses the same holistic temporal compression.

Figure 6: Consecutive-frame silhouette difference on scene 4 of Fig. 5 (a cat) for five frame pairs around the impact (red: previous frame only; blue: next frame only; grey: overlap; number: XOR over union). Ground truth and the holistic ST-VAE change steadily from frame to frame. The standard baseline and its two variants change much less than the ground truth for the first three pairs and then much more in the last two: the reconstruction holds and then snaps. The adaptive-compression baseline changes too little over all five pairs and releases the motion even later. The direct world-coordinate baseline, which uses the holistic temporal compression, follows the ground truth until the last pair.

Scene	$N$	Holistic (ours)	Standard	+ vel. loss	+ spec. loss	Adapt. compr.	Direct world-coord.
1	5,181	0.972	0.728	0.850	0.781	0.418	0.972
2	5,913	0.969	0.589	0.717	0.587	0.583	0.928
3	6,005	0.986	0.478	0.668	0.479	0.488	0.943
4	2,245	0.974	0.822	0.761	0.769	0.551	0.917
5	1,526	0.970	0.640	0.616	0.684	0.684	0.824

Appendix

Table 4: DCC (deformation-change correlation with ground truth, computed as in Table 1 ) on the five GSO scenes of Fig. 5 . Scenes 1 to 5 are the five columns of Fig. 5 in the same order, and scene 4 is the one shown in Fig. 6 . Best per row in bold.

Figure 7: Chaos in a traditional GPU simulator. Five reruns (columns) of the same soft-racket scene from byte-identical inputs (same mesh, initial state, material, and solver configuration), shown at four sampled frames (rows): before the drop, just after impact, mid-bounce, and the last frame of the trajectory. The runs are indistinguishable until impact and then settle into visibly different poses. The complex dynamics are chaotic, so the post-impact outcome is not a deterministic function of the initial state.

Difference between the two runs	$>1\%$ of size	$>5\%$ of size	$>10\%$ of size
(a) mean error, whole trajectory	$35$ ( $11.7\%$ )	$3$ ( $1.0\%$ )	$2$ ( $0.7\%$ )
(b) final-frame error	$71$ ( $23.7\%$ )	$23$ ( $7.7\%$ )	$4$ ( $1.3\%$ )

Appendix

Table 5: Divergence between two runs of the GPU solver from byte-identical inputs on the

300

Objaverse held-out scenes of Table 1 . Each entry gives the number and the percentage of scenes whose difference between the two runs exceeds the given fraction of the object size, measured (a) as the mean per-vertex error over the whole trajectory and (b) as the per-vertex error at the last frame.

	Non-chaotic ( $229$ scenes)			Chaotic ( $71$ scenes)
	Vol. $\downarrow$	KE $\to$ 1	Err $\downarrow$	Vol. $\downarrow$	KE $\to$ 1	Err $\downarrow$
Regression	$12.6$	$0.72$	$25.0$	$38.4$	$0.69$	$63.2$
Diffusion, single sample	$3.8$	$0.82$	$18.0$	$25.4$	$0.75$	$54.0$
Diffusion, best of $8$	$4.4$	$0.83$	$14.6$	$22.8$	$0.79$	$44.7$

Appendix

Table 6: Regression versus diffusion on the

229

non-chaotic and the

71

chaotic held-out scenes (two solver runs ending less or more than

1\%

of the object size apart). Medians over scenes. Vol.: volume violation rate (%), KE:

KE_{\mathrm{def}}/KE_{\mathrm{rig}}

relative to the ground truth, Err: world-coordinate error (mm) (Appendix B.4 ). Best of

8

picks, per scene, the sample closest to the ground truth.

Figure 8: The key scene of Appendix D.6 (a held-out scene) at four sampled frames (rows; ground contact at frame

31

). Columns: the two runs of the GPU solver from byte-identical inputs, which fall over in different directions; the regression prediction, which settles in between and resembles neither run; and the two diffusion samples closest to run

1

and to run

2

among the eight samples.

		$500\leq N\leq 5{,}000$						$5{,}000<N\leq 10{,}000$
		DCC $\uparrow$	Recon $\downarrow$			Rigid $\downarrow$		Recon $\downarrow$	Rigid $\downarrow$
	ST-VAE	mean	mean	med	P90	rot	trans	mean	rot	trans
Objaverse Held-out	Standard	0.665	5.32	3.47	12.01	0.270	0.57	6.14	0.185	0.50
	+ velocity loss	0.842	5.34	3.15	11.60	0.831	3.33	6.25	0.638	3.88
	+ spectral loss	0.697	5.17	3.42	11.40	0.290	0.73	5.96	0.210	0.66
	Adaptive compression	0.511	5.86	3.65	13.72	0.440	5.09	6.59	0.314	4.30

Appendix

Table 7: Full version of Table 1 with all six ST-VAEs, including the two variants of the standard baseline and the adaptive-compression baseline. ST-VAE reconstruction comparison across the three evaluation sets (row blocks: held-out in-distribution / GSO zero-shot / Toys4K zero-shot). Left six columns: scenes within the training vertex range (

500\leq N\leq 5{,}000

;

300

scenes per set). Right three columns: large-mesh subsets beyond the

5{,}000

-vertex training cap (

N>5{,}000

;

300

scenes per set), reported on mean deformation and rigid errors only. DCC is the deformation-change correlation with ground truth and directly reflects the staircase artifact (higher is better); Recon is the reconstruction error of the deformation field, i.e. of the geometry with the rigid motion removed (mm); rot and trans are the reconstruction errors of the rigid rotation (deg) and translation (mm). Full definitions in Appendix B . best and second-best per column within each block. † Direct world-coordinate baseline (no factoring). Its errors are computed after removing the rigid motion from its output.

		Volume	Self-int.	Vel.	Acc.	$\frac{KE_{\mathrm{def}}}{KE_{\mathrm{rig}}}$	World err
	Prediction Model	viol. % $\downarrow$	tris % $\downarrow$	viol. % $\downarrow$	viol. % $\downarrow$	$\times$ GT $\rightarrow$ 1	median (mm) $\downarrow$
Objaverse Held-out	ST-VAE reconstruction	8.5	0.12	0.8	16.9	0.91	1.97
	Diffusion (ours)	7.9	0.24	7.0	25.8	0.80	28.2
	Regression	22.7	0.79	8.6	29.2	0.72	37.5
	MGN-style AR	55.1	3.53	21.4	69.0	0.77	80.6
GSO	ST-VAE reconstruction	2.1	0.00	0.5	11.8	0.92	1.21

Appendix

Table 8: Full version of Table 2 with all six physics checks and the ST-VAE reconstruction reference row. Prediction comparison across the three evaluation sets, on the same

300

scenes per set within the training vertex range (

500\leq N\leq 5{,}000

) as Table 1 . Viol. %: percentage of vertices or tetrahedra, over all frames, whose predicted volume, speed, or acceleration deviates from the ground truth beyond a threshold; Self-int. is the fraction of surface triangles in a self-intersection;

KE_{\mathrm{def}}/KE_{\mathrm{rig}}

is the deformation-to-rigid kinetic-energy ratio relative to ground truth (closer to

1

is better); World err is the per-vertex error in world coordinates (mm); full definitions in Appendix B . ST-VAE reconstruction is the ground truth encoded and decoded by the ST-VAE behind the LDM, shown as a reference. best and second-best among the three models per column within each block. MGN-style AR is a MeshGraphNets-style (MGN) autoregressive graph network retrained on our scenes.

	no ST-VAE	with ST-VAE
$N$	LDM on raw tokens (GiB)	ST-VAE Base step (GiB)
500	28.9	33.5
1,000	55.4	37.4
2,000	108.3	45.8
2,500	134.5	50.7
2,600	OOM	50.7

Appendix

Table 9: Training memory versus vertex count

N

on a

140

GiB H200 (

T{=}64

, bf16 , batch

1

). No ST-VAE: our LDM architecture applied directly to the

T{\times}N

raw vertex tokens, peak memory (GiB) of one training step. With ST-VAE: peak memory (GiB) of one ST-VAE Base training step. The LDM stage with the ST-VAE costs about

4.4

GiB per trajectory at any

N

and is not listed. OOM: out of memory.

$N$	bf16	fp32	Regression	LDM-20	LDM-50	Regression	LDM-20	LDM-50	(GiB)
	Decode (ms)		bf16 fps			fp32 fps			Peak
980	318	288	195	82	44	192	30	13	4.8
2,493	327	425	189	79	42	136	28	13	4.8
5,007	328	597	188	81	43	100	26	12	4.8
7,534	332	778	186	81	43	78	25	12	5.2
9,851	384	960	162	76	42	64	23	12	5.4

Appendix

Table 10: Batch-1 inference of PhysLDM on one H200 for

64

-frame trajectories at different vertex counts

N

(held-out scenes; the last three rows exceed the

5{,}000

-vertex training cap). Decode: time of the ST-VAE decoder. fps

=64/(\text{condition build}+\text{latent model}+\text{decode})

. Peak: maximum GPU memory allocated over the whole pass.

Figure 9: ST-VAE reconstruction on two more scenes (a Toys4K cake and a GSO shape-sorter toy) at four frames sampled along the trajectory. Rows: ground truth (rose), Holistic (ours) , the standard baseline, its two variants (velocity loss and spectral loss), the adaptive-compression baseline, and the direct world-coordinate baseline. Scene details are listed in Table 11 .

Figure 10: ST-VAE reconstruction on two more scenes (a held-out Objaverse object and a GSO turboprop), with the same rows, sampled frames, and rendering as Fig. 9 . Scene details are listed in Table 11 .

Figure	Object	Dataset	$N$	$E$ (Pa)	$v_{0}$ (m/s)
Fig. 3	phone	Objaverse (held-out)	6,106	$2.76\times 10^{6}$	$(+0.59,\,+0.00,\,+0.94)$ , $\|v_{0}\|=1.11$
Fig. 3	boot	GSO	6,569	$5\times 10^{4}$	$0$ (released from rest)
Fig. 3	chess piece	Toys4K	6,322	$5\times 10^{4}$	$0$ (released from rest)
Fig. 3	toy	GSO	9,403	$5\times 10^{4}$	$0$ (released from rest)
Fig. 9	cake	Toys4K	6,640	$1.45\times 10^{6}$	$(-0.21,\,+0.00,\,+0.58)$ , $\|v_{0}\|=0.62$
Fig. 9	shape-sorter toy	GSO	6,426	$5\times 10^{4}$	$0$ (released from rest)

Appendix

Table 11: Scenes of Figs. 3 , 9 , and 10 : source dataset, tetrahedral node count

N

, Young’s modulus

E

, and initial velocity

v_{0}

(m/s,

y

up). The Young’s moduli were chosen to show both soft bodies with large deformation and stiff, nearly rigid ones, so some values do not match the real material of the object.

Figure 11: Prediction on four more scenes (a held-out Objaverse bucket and sofa, a GSO flower pot, and a GSO plush dog) at four frames sampled along the trajectory. Rows: ground truth, Diffusion (ours) , Regression, and the MGN-style AR baseline. Scene details are listed in Table 12 .

Figure	Object	Dataset	$N$	$E$ (Pa)	$v_{0}$ (m/s)
Fig. 4	punch-drop toy	GSO	7,767	$2\times 10^{6}$	$(-0.48,\,+0.00,\,-1.78)$ , $\|v_{0}\|=1.85$
Fig. 4	cap	Objaverse (held-out)	2,435	$5\times 10^{4}$	$0$ (released from rest)
Fig. 4	bowl	Objaverse (held-out)	5,877	$5\times 10^{4}$	$0$ (released from rest)
Fig. 4	grapes	Toys4K	8,767	$5\times 10^{4}$	$0$ (released from rest)
Fig. 11	bucket	Objaverse (held-out)	9,853	$2.6\times 10^{6}$	$(-0.94,\,+0.00,\,+1.33)$ , $\|v_{0}\|=1.63$
Fig. 11	sofa	Objaverse (held-out)	4,228	$1.05\times 10^{6}$	$(-1.34,\,+0.00,\,-0.20)$ , $\|v_{0}\|=1.36$

Appendix

Table 12: Scenes of Figs. 4 and 11 : source dataset, tetrahedral node count

N

, Young’s modulus

E

, and initial velocity

v_{0}

(m/s,

y

up). The Young’s moduli were chosen to show both soft bodies with large deformation and stiff, nearly rigid ones, so some values do not match the real material of the object.

Figure 12: ST-VAE reconstruction of the extreme case, a soft birdcage (held-out Objaverse,

N{=}9{,}021

nodes,

1.8\times

the

5{,}000

-vertex training cap,

E=5\times 10^{4}

Pa) with dozens of slender bars. Top: four frames sampled along the trajectory, ground truth (rose) and Holistic (ours) . Bottom: the zoom-in view of the last frame, ground truth (left) and reconstruction (right), with the surface wireframe shown. The overall shape of the collapsed cage is recovered, while the thin bars are reconstructed distorted and tangled. Mean reconstruction error over the trajectory:

9.9

mm, against

5.3

mm for the compact teaser mug of

11{,}621

nodes.

Figure 13: Scene-level ST-VAE reconstruction on held-out multi-body scenes (two two-body and two three-body scenes, each body with a random Young’s modulus and initial velocity) at four sampled frames: just before and just after the mid-air collision, the first frame after ground impact, and a settled frame. Rows: ground truth (rose) and reconstruction.

Figure 14: One-shot prediction on held-out multi-body scenes (two two-body and two three-body scenes, each body with a random Young’s modulus and initial velocity), sampled frames as in Fig. 13 . Rows: ground truth and our prediction conditioned on the initial state, rendered by driving each body’s original textured asset with the predicted tetrahedral trajectory.

Simulator	Differentiation Support	1st-Order Task (Material Estimation)	2nd-Order Task (Throw Design)
Ours	Autograd (higher order)	Success (40 seconds)	Success (7 minutes) Efficient 2nd-order grads.
DiffIPC	Analytic adjoint (1st order only)	26 minutes Unstable under defaults. Needs manual tuning.	Infeasible (Multi-day) ${\sim}635\times$ slower via finite diff.
Newton/Warp	Recorded tape	Fails Gradients violently explode.	Infeasible No usable 1st-order gradients.

Appendix

Table 13: Comparison of our model with two differentiable simulators on downstream applications. Our approach supports higher-order derivatives, enabling complex optimization tasks that are either computationally infeasible or numerically unstable for existing solvers.

Figure 15: Basic material estimation. (a) Four matched time samples of the observed trajectory (

E^{\star}=60

kPa), the initial model prediction (

E_{0}=250

kPa), and the prediction after 40 Adam steps (

\hat{E}=52.4

kPa). All views share the same camera and scale. (b) Absolute log-stiffness error for seven ground-truth moduli, with the illustrated case highlighted in blue. All seven final estimates are within

0.06

dex; each optimization takes approximately 40 seconds. Curves show the recorded iterates without smoothing.

Figure 16: Beyond material estimation: designing a more informative throw. (a) Ground-truth replays of the initial and optimized release velocities at

E^{\star}=60

kPa. Orange and blue overlays represent stiffness perturbations of

-0.05

and

+0.05

dex. The first three columns show true motion at matched times and a shared camera scale. In the final column, rigid translation and rotation offsets from the nominal trajectory are magnified by

5\times

, using equal magnification in both rows. (b) Cumulative rigid-pose sensitivity, normalized by the initial throw’s final score, increases by

13.4\times

. This score is computed from the unamplified GT trajectories. (c) Material estimation from the same distant initial guess,

E_{0}=1{,}000

kPa. After 60 estimation steps, the initial throw remains near

901

kPa, whereas the optimized throw reaches

73.7

kPa, much closer to the true

60

kPa. Throw design uses 30 Adam steps and approximately 7 minutes.

$N$	tokens	Concurrent (DiT-L)	Concurrent (DiT-B)
	( $T{\times}N$ )	GiB / ms	GiB / ms
500	32k	64.4 / 669	24.0 / 212
1,000	64k	122.1 / 1317	46.0 / 412
1,200	77k	OOM	54.7 / 495
2,000	128k	OOM	89.9 / 879
2,500	160k	OOM	111.8 / 1147

Appendix

Table 14: Memory and resolution limits of the concurrent raw-coordinate model on a

140

GiB H200: peak training memory (GiB) and GPU time per training step (ms) at batch

1

, i.e., per trajectory,

T{=}64

, bf16 , flash attention, for vertex counts

N

from

500

to

5{,}000

. OOM: out of memory at batch

1

. For reference, our LDM trains at

{\sim}4.4

GiB (Table 9 ) and

{\sim}27

ms per trajectory at any

N

.

Model	Stage	Training time
Concurrent (DiT-B)	(single)	74.6 h
Ours	spatial VAE	1.4 h
	ST-VAE	40.0 h
	LDM	19.6 h
	total	61.1 h

Appendix

Table 15: Training time of the two models on the same scenes (

\leq 1{,}000

vertices) and the same GPUs. Even with three stages, our pipeline takes less time than the concurrent model.

Figure 17: The two models on four held-out scenes (a block, a spoon, a screwdriver and a bottle; all

\leq 1{,}000

vertices) at the last frame of the trajectory. For each scene the full object is followed by a zoom-in view of the boxed region. Rows: ground truth (rose), our three-stage pipeline trained on the same scenes with the same budget (Table 15 ), and the concurrent DiT-B model. Our model preserves the geometric detail of every object. The concurrent model places the objects correctly, but its geometric detail is visibly noisy: the block bends and its faces ripple, the spoon and the shaft of the screwdriver are corrugated, and the bottle is lumpy. The zoom-in views show this noise on every surface.

Data	Output	Scenes	Vol. (%) $\downarrow$	$KE_{\mathrm{def}}/KE_{\mathrm{rig}}$ $\to 1$	Err (mm) $\downarrow$
Stable Neo-Hookean	ST-VAE reconstruction	$40$	$6.9$	$0.92$	$4.5$
Stable Neo-Hookean	LDM prediction	$40$	$8.9$	$0.83$	$23.9$
ARAP	ST-VAE reconstruction	$40$	$3.1$	$0.93$	$4.6$
ARAP	LDM prediction	$40$	$4.9$	$0.79$	$37.5$
Real captured	ST-VAE reconstruction	$22$	–	–	$6.0$
Real captured	LDM prediction	$22$	–	–	$874.9$

Appendix

Table 16: The pretrained PhysLDM on three data sources without adaptation. Medians over scenes. The volume and kinetic energy metrics need tetrahedra and are not defined on the point clouds.

Figure 18: PhysLDM on real captured point clouds ( Jiang et al., 2025 ) : a plush sloth and a plush dinosaur (top), two pieces of cloth (middle), and two ropes (bottom), each at four frames sampled along the trajectory and rendered as points. Rows of each block: the tracked points (rose), the ST-VAE reconstruction, the LDM prediction of the pretrained model without adaptation (zero-shot), and the LDM prediction after fine-tuning (Table 17 ). The zero-shot prediction lets the resting object fall through the table, so it is not visible in the camera.

		Vol. (%) $\downarrow$		$KE_{\mathrm{def}}/KE_{\mathrm{rig}}$ $\to 1$		Err (mm) $\downarrow$
Data	Scenes	before	after	before	after	before	after
Stable Neo-Hookean	$32$	$8.9$	$6.2$	$0.82$	$0.91$	$22.9$	$7.6$
ARAP	$32$	$4.9$	$3.1$	$0.80$	$0.92$	$34.4$	$8.8$
Real captured	$16$	–	–	–	–	$921.3$	$6.2$

Appendix

Table 17: Fine-tuning the LDM of PhysLDM on each data source with the unchanged trainer. Medians over the scenes of each fine-tuning subset, before and after fine-tuning.

PhysLDM: Latent Diffusion for High-Fidelity Deformable Simulation

Abstract

Figures & tables

Appendix

Explore similar work

PhysiFormer: Learning to Simulate Mechanics in World Space

PhysReal: Learning Real-World Deformable Object Physics via Hybrid Constitutive Modeling

Adaptive Volumetric Mechanical Property Fields Invariant to Resolution

	Objaverse Held-out				GSO				Toys4K
Prediction Model	Vol. $\downarrow$	SI $\downarrow$	KE $\rightarrow$ 1	Err $\downarrow$	Vol. $\downarrow$	SI $\downarrow$	KE $\rightarrow$ 1	Err $\downarrow$	Vol. $\downarrow$	SI $\downarrow$	KE $\rightarrow$ 1	Err $\downarrow$
Diffusion (ours)	7.9	0.24	0.80	28.2	2.0	0.00	0.86	14.0	10.3	0.32	0.80	35.6
Regression	22.7	0.79	0.72	37.5	5.1	0.01	0.72	19.9	24.3	1.07	0.69	42.6
MGN-style AR	55.1	3.53	0.77	80.6	47.1	0.68	0.76	78.4	57.5	4.73	0.77	79.1