RigidBench: Evaluating Rigid-Body Physics in Video Generation Models

Swarnim Jain; Shangzhe Wu

cs.CVAug 16, 2026

RigidBench: Evaluating Rigid-Body Physics in Video Generation Models

Authors: Swarnim Jain, Shangzhe Wu

Organizations: University of Cambridge

Abstract

Video models are increasingly used to predict what happens next in a scene, yet the metrics commonly used to compare their outputs say little about whether the predicted objects move correctly. Motion, geometry, identity, background stability, and visual similarity can fail independently, but whole-frame scores often mix these errors together. We introduce RigidBench, a simulator-grounded benchmark that compares a generated continuation with a reference rollout from the same initial frame and motion description. Its five rigid-body tasks vary objects, materials, viewpoints, and indoor and outdoor scenes, with per-frame masks, depth, 6-DoF trajectories, and contacts available for scoring. We evaluate eight models on the same 100 examples with ten measurements that keep these aspects separate. The resulting rankings depend strongly on what is measured: no model leads on all ten, and across model means, higher SSIM accompanies larger 3D trajectory error (r = 0.89). RigidBench also includes 5,000 training videos with exact simulator state, which we use to fine-tune and analyze Wan 2.2 TI2V-5B. Full fine-tuning reduces 3D trajectory error by about 20% with almost no change in SSIM, while teacher-forced probes and targeted interventions show that object position is represented throughout Wan's diffusion transformer and used by its denoising computation.

Figures & tables

Panel of Figure 1 — Figure 1: The five RigidBench tasks. Each image is the first frame of an example in the fixed evaluation set. Objects, scenes, surfaces, dimensions, and materials vary between examples.

Task	Setup	Main interaction
Free fall	One object drops toward a support	Free flight
Bounce	A high-restitution object drops onto a surface	Restitution
Ramp collision	Two objects roll down opposing ramps and collide	Rolling collision
Ball chain	A rolling object strikes a row of three objects	Contact chain
Drop into cluster	One object falls onto a three-object cluster	Multi-body impact

Table 1: The five RigidBench tasks. Each is instantiated with different objects, geometry, materials, viewpoints, and scenes.

Condition	Scenes	Objects	Examples
In-distribution	training	training	25
OOD-scene	held out	training	25
OOD-object	training	held out	25
OOD-both	held out	held out	25

Table 2: Evaluation conditions. Every row contains five examples from each task.

Figure 2: Model and task results. Left: model means on SSIM and ATE-3D. The ATE-3D axis is reversed, and the correlation is computed against negative ATE-3D. Right: mean ATE-3D for each model and task, with one horizontal bar per model. Bars within each task follow the order shown in the key.

	Object motion and geometry						Visual similarity and stability
Model	IoU $\uparrow$	L2 $\downarrow$	Cham. $\downarrow$	ATE $\downarrow$	ATE-3D $\downarrow$	SI-MSE $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	IdDrift $\downarrow$	BGDrift $\downarrow$
Wan 2.2	.183	.220	.370	.200	2.64	.0288	.925	.056	.501	.739
Cosmos 2.5	.171	.300	.527	.282	2.84	.0331	.909	.095	.484	1.094
Veo 3.1	.153	.230	.380	.203	2.47	.0283	.879	.112	.516	2.368
Veo 3.1 Fast	.157	.229	.382	.204	2.63	.0352	.902	.100	.517	1.491
Kling 3.0	.169	.230	.387	.186	2.76	.0364	.912	.130	.553	1.632

Table 3: Full model comparison. Means over the 100-example evaluation set, omitting non-finite measurements. ATE denotes 2D ATE. BGDrift is multiplied by

10^{3}

. Best and second-best values are bold and underlined.

Figure 3: Fine-tuning across checkpoints. Mean paired change from base Wan with bootstrap 95% confidence intervals. Filled points mark the checkpoint with the lowest ATE-3D in each sweep on this evaluation set.

Figure 4: Probing and intervention. Left: actor-aligned hidden states are probed for simulator state. Right: a fitted subspace is removed and Wan’s flow-matching loss is measured through two readouts.

Figure 5: Physical state is accessible throughout Wan. Linear-probe scores by block, averaged over noise levels, with shading spanning their range.

Figure 6: Wan depends on position-aligned directions. Mean loss increase after subspace removal, with bootstrap 95% confidence intervals. The symmetric-log axis is linear between

-0.01

and

0.01

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Physical setup	Prompt and camera direction
Free fall	One object is placed $0.50$ – $0.70$ m above the support.	“{object} is dropping.” Any azimuth.
Bounce	One object tagged bouncy is placed $0.50$ – $0.70$ m above the support. The support restitution is $0.6$ and its friction is $0.3$ .	“{object} is dropping.” Any azimuth.
Ramp collision	Two opposing ramps each have length $0.35$ – $0.45$ m, height $0.15$ – $0.25$ m, and width $0.20$ – $0.30$ m. Their gap is $0.10$ – $0.15$ m, with one rolling object placed on each ramp.	“{object} and {object} are rolling down two ramps facing each other.” Relative azimuth $70$ – $110^{\circ}$ or $250$ – $290^{\circ}$ .
Drop into cluster	Three stackable objects form a cluster with gap $0.003$ – $0.008$ m. A fourth object is placed $0.50$ – $0.70$ m above the support.	“{object} is dropping onto a cluster of objects.” Any azimuth.
Ball chain	A ramp has length $0.50$ – $0.70$ m, height $0.30$ – $0.45$ m, and width $0.30$ – $0.40$ m. One object rolls toward a line of three objects, placed $0.08$ – $0.12$ m beyond the ramp with gaps of $0.02$ – $0.04$ m.	“{object} is rolling down the ramp toward a row of objects.” Relative azimuth $60$ – $150^{\circ}$ or $210$ – $300^{\circ}$ .

Appendix

Table 4: Task configurations. Camera directions are measured relative to the resolved task rotation. Ramp material is sampled from light wood, dark wood, polished white marble, and rough gray concrete. Prompt placeholders expand to the selected object’s name with an indefinite article.

Subset	Configuration identifiers
Training scenes	hallway_interior , modern_loft , modern_luxury_kitchen , minimal_white_kitchen , modern_japandi_dining , dining_room_set , modular_flat , coastal_living_room
Held-out scenes	beach_gazebo , butterfly_mural_room , cozy_living_room , modern_warm_kitchen
Training objects	apple , orange , lemon , kiwi , pomegranate , pear , mango , banana , watermelon , baseball , bouncy_ball , rugby_ball , red_bull_can , plastic_bottle , large_dice , food_can
Held-out objects	peach , tomato , coconut , tennis_ball , toy_duck , cricket_ball , rubiks_cube , soda_can

Appendix

Table 5: Scene and object partitions. The identifiers correspond directly to the released YAML files.

Partition	Scene set	Object set	Examples
Training	Training	Training	5,000
In-distribution	Training	Training	25
OOD scene	Held-out	Training	25
OOD object	Training	Held-out	25
OOD both	Held-out	Held-out	25

Appendix

Table 6: Data partitions. Each evaluation partition contains five examples from each task. Training examples and evaluation examples are disjoint.

Component	Setting
Video	$1280\times 704$ , 49 frames, 24 frames per second, PNG frames and H.264 video
Simulation	Blender 4.5.9, 30 Bullet substeps per frame, 20 solver iterations
Rigid bodies	Per-object mass, friction, and restitution, with convex-hull active collisions, linear damping $0.1$ , angular damping $0.5$ , and collision margin $0.001$ m
Passive geometry	Mesh collisions with margin $0.0001$ m, default friction $0.4$ , and restitution $0.3$ , unless a task overrides the support
Cycles	32 adaptive samples with threshold $0.05$ , OptiX denoising, six maximum bounces, and at most three diffuse, glossy, or transmission bounces
Camera	$35$ mm lens, $10^{\circ}$ elevation, $1.4$ m minimum distance, framing multiplier $1.4$ , bounding-box padding $0.2$ m

Appendix

Table 7: Simulation and rendering configuration. These values come from the released render configuration, container image, and Blender setup code.

Artifact	Contents
frames/ , video.mp4	The 49 rendered RGB frames and their encoded video
EXR pass streams	Per-frame 32-bit object-index and Z passes under masks/ and depth/ , written by the Cycles compositor
masks.npz	Boolean array of shape $(T,N,H,W)$ obtained by thresholding the object-index render pass, together with actor names
depth.npz	Float32 array of shape $(T,H,W)$ containing the Cycles Z pass used by the trajectory code
trajectories.npz	Per-frame world positions, Blender-order quaternions, and finite differences of position and rotation for every actor
contacts.json	Per-frame mesh-overlap events for actor pairs and passive geometry, with up to five contact points and normals per pair

Appendix

Table 8: Per-sample artifact bundle. Here

T=49

H=704

W=1280

, and

N

is the number of task actors, including passive task geometry such as ramps.

Model	Identifier	Generation and scoring configuration
Wan 2.2 TI2V-5B	Wan-AI/Wan2.2-TI2V-5B , local DiffSynth	$1280\times 704$ , 49 frames at 24 Hz, 50 steps
Cosmos Predict 2.5-2B	nvidia/Cosmos-Predict2.5-2B , revision diffusers/base/post-trained , local Diffusers	$1280\times 704$ , 33 frames at 16 Hz, 50 steps
Veo 3.1 / Fast	google/veo-3.1 , google/veo-3.1-fast	4 s, 720p, $16{:}9$ , audio disabled, scored as 24 Hz
Kling 3.0	kwaivgi/kling-v3-video	3 s, pro mode, $16{:}9$ , audio disabled, scored as 24 Hz
Seedance 2.0 / Fast	bytedance/seedance-2.0 , bytedance/seedance-2.0-fast	4 s, 720p, $16{:}9$ , audio disabled, scored as 24 Hz
Grok Imagine	xai/grok-imagine-video	4 s, 720p, automatic aspect ratio, scored as 24 Hz

Appendix

Table 9: Generation settings. Model identifiers are shown verbatim. Wan, Cosmos, Veo, and Kling receive a negative prompt. The Seedance and Grok calls do not.

Quantity	Model	Input and settings
Actor masks	SAM 2.1 Hiera Large, facebook/sam2.1-hiera-large	One frame-zero mask for each active actor. Propagated logits are thresholded at zero.
Point tracks	CoTracker3 offline	Twenty points per active actor, sampled without replacement from a three-pixel erosion of its reference mask. Generated and reference visibility must both be true.
Disparity	Video Depth Anything Large, depth-anything/Video-Depth-Anything-Large	ViT-L encoder, input size 384, target rate 24 Hz. The saved output is disparity.
Identity features	DINOv2 ViT-L/14	$64\times 64$ track-centered patches resized to $224\times 224$ , ImageNet normalization, bfloat16 inference in batches of 256, and L2-normalized features.
Full-frame features	LPIPS with AlexNet	Reference and generated RGB mapped from $[0,255]$ to $[-1,1]$ .
Background tracks	CoTracker3 offline	At most 200 Shi-Tomasi corners with quality threshold $0.01$ and minimum separation 10 pixels, sampled within the background after a 15-pixel erosion.

Appendix

Table 10: Perception and feature settings. Each generated clip is processed independently. The corresponding reference sample supplies the frame-zero inputs.

Metric	Per-sample calculation
IoU $\uparrow$	Per-actor mask intersection over union, with an empty union assigned one.
Centroid L2 $\downarrow$	Mask-centroid distance divided by image height. Empty actor-frames are omitted.
Chamfer $\downarrow$	Sum of the two mean nearest-neighbor distances between foreground-pixel sets, divided by image height. Empty actor-frames are omitted.
ATE $\downarrow$	Mean track-point distance over jointly visible point-frames, divided by image height.
ATE-3D $\downarrow$	Actor-position RMSE divided by the mean reference displacement of actors moving more than $0.01$ m.
SI-MSE $\downarrow$	Per-frame variance of the log-depth residual after one clip-level affine disparity alignment.

Appendix

Table 11: All reported measurements. Arrows show the direction of better performance. The object-level scores use active actors only.

Figure 7: Controlled metric perturbations. Left: equal 3D offsets are applied along the viewing ray and perpendicular to it. Center: coherent translation and independent corner jitter have matched raw displacement. Right: patches undergo an identity swap or a gamma change. Shading gives one standard error from the saved test output.

Figure 8: Uncertainty across all ten measurements. Points are model means and bars are bootstrap 95% confidence intervals over evaluation examples. Models follow the same fixed row order in every panel. BGDrift is multiplied by

10^{3}

Figure 9: Matched probability of improvement. Each cell is the probability that the row model is better than the column model on the same example. Rows and columns are ordered by the corresponding model mean, best first. Bold entries have a paired bootstrap 95% confidence interval that excludes $0.5$ .

Model	Ball chain	Ramp collision	Drop into cluster	Bounce	Free fall
Wan 2.2	1.02	1.39	1.78	3.70	5.30
Cosmos 2.5	1.12	1.32	2.01	4.02	5.74
Veo 3.1	0.75	1.86	1.69	3.53	4.54
Veo 3.1 Fast	0.86	1.59	1.80	3.66	5.22
Kling 3.0	1.52	1.56	1.79	3.89	5.02
Seedance 2.0	1.07	1.27	1.21	2.55	3.97

Appendix

Table 12: ATE-3D by task. Each cell is the mean over the 20 examples for that task. Lower is better; best and second-best values are bold and underlined.

Setting	LoRA	Full fine-tuning
Trainable weights	Rank $32$ , $\alpha=32$ , dropout $0$	All non-conditioning DiT weights
Updated modules	q , k , v , o , ffn.0 , ffn.2	All remaining DiT modules; text_embedding , time_embedding , and time_projection fixed
Distributed strategy	DDP	FSDP FULL_SHARD , wrapped by DiT block
Batch per GPU	2	16
Gradient accumulation	8 batches	1 batch
Effective batch	128 examples	128 examples

Appendix

Table 13: Fine-tuning configurations. Both runs use eight H100 GPUs, bf16 mixed precision, fused AdamW with weight decay

0.01

\epsilon=10^{-8}

, default

\beta

values, gradient clipping at

1.0

, and random seed 42.

		Object motion and geometry						Visual similarity and stability
Regime	Step	IoU $\uparrow$	L2 $\downarrow$	Cham. $\downarrow$	ATE $\downarrow$	ATE-3D $\downarrow$	SI-MSE $\downarrow$	SSIM $\uparrow$	LPIPS $\downarrow$	IdDrift $\downarrow$	BGDrift $\downarrow$
Base	–	.183	.220	.370	.200	2.64	.0288	.925	.056	.501	.739
LoRA	500	.185	.168	.272	.158	2.34	.0247	.929	.056	.525	.726
LoRA	1,000	.207	.164	.264	.154	2.31	.0250	.929	.056	.518	.718
LoRA	1,500	.195	.174	.282	.163	2.40	.0261	.925	.058	.519	.770
LoRA	2,000	.193	.165	.264	.159	2.44	.0260	.925	.058	.523	.736

Appendix

Table 14: Complete checkpoint sweep. Means over the 100-example evaluation set. Bold checkpoint labels mark the minimum ATE-3D within each run and are the checkpoints used in the subsequent comparisons. Within each metric, the best and second-best values are bold and underlined. BGDrift is multiplied by

10^{3}

Metric	LoRA 1k $-$ base	Full FT 2.5k $-$ base	Full FT $-$ LoRA
IoU $\uparrow$	$+.024\;[+.008,+.041]$	$+.016\;[+.001,+.030]$	$-.009\;[-.027,+.009]$
L2 $\downarrow$	$-.056\;[-.076,-.036]$	$-.055\;[-.076,-.035]$	$+.001\;[-.019,+.021]$
Cham. $\downarrow$	$-.106\;[-.146,-.068]$	$-.107\;[-.148,-.067]$	$-.000\;[-.039,+.038]$
ATE $\downarrow$	$-.045\;[-.064,-.027]$	$-.050\;[-.069,-.031]$	$-.004\;[-.024,+.015]$
ATE-3D $\downarrow$	$-.33\;[-.58,-.10]$	$-.53\;[-.86,-.25]$	$-.20\;[-.50,+.07]$
SI-MSE $\downarrow$	$-.00376\;[-.00741,-.00091]$	$-.00262\;[-.00583,-.00002]$	$+.00114\;[+.00005,+.00243]$

Appendix

Table 15: Paired changes for the selected checkpoints. Each entry is the mean paired change followed by its bootstrap 95% confidence interval. Positive is better for IoU and SSIM; negative is better for every other row. BGDrift is multiplied by

10^{3}

Model	Ball chain	Ramp collision	Drop into cluster	Bounce	Free fall
Base	1.02	1.39	1.78	3.70	5.30
LoRA 1k	0.94	1.14	1.41	3.20	4.85
Full FT 2.5k	0.78	1.12	1.51	3.21	3.94

Appendix

Table 16: Selected checkpoints by task. Mean ATE-3D over 20 examples per task. Lower is better; best and second-best values are bold and underlined.

	ATE-3D $\downarrow$				SSIM $\uparrow$
Model	ID	Scene	Object	Both	ID	Scene	Object	Both
Base	2.64	2.31	3.63	1.98	0.946	0.906	0.946	0.901
LoRA 1k	2.20	2.18	3.36	1.50	0.948	0.916	0.946	0.906
Full FT 2.5k	1.95	1.99	3.03	1.48	0.947	0.910	0.948	0.902

Appendix

Table 17: Selected checkpoints by evaluation partition. ID uses training scenes and objects; Scene, Object, and Both use the corresponding held-out sets. Each cell averages 25 examples. Best and second-best values are bold and underlined.

Component	Configuration
Backbones	Randomly initialized Wan architecture, base Wan 2.2 TI2V-5B, and the full fine-tuning checkpoint at step 2,500
Extraction sites	30 blocks at five schedule fractions, giving 150 cells per backbone
Probe rows	148,303 training, 19,696 validation, and 18,637 test actor–time rows after visibility filtering
Linear probe	One affine map from 3,072 dimensions to the target
MLP probe	3,072–64–target with a ReLU hidden layer
Optimization	Adam for 20 epochs, batch size 4,096, learning rate $10^{-3}$ , seed 0

Appendix

Table 18: Probe configuration. Every layer, schedule fraction, target, probe class, and label condition is fit separately.

	Position ( $R^{2}$ )		Contact (AUC)
Model	Linear	MLP	Linear	MLP
Random init	$0.388\;(-0.007)$	$0.769\;(-0.011)$	$0.696\;(0.489)$	$0.766\;(0.489)$
Base Wan	$\underline{0.709}\;(-0.006)$	$\underline{0.878}\;(-0.003)$	$\underline{0.799}\;(0.506)$	$\underline{0.816}\;(0.515)$
Full fine-tuning	$\mathbf{0.785}\;(-0.008)$	$\mathbf{0.924}\;(-0.006)$	$\mathbf{0.837}\;(0.505)$	$\mathbf{0.864}\;(0.509)$

Appendix

Table 19: Probe scores across layers and noise levels. Each entry is the mean test score over 150 layer–noise cells. The score from a probe trained on shuffled labels is shown in parentheses. Best and second-best trained scores are bold and underlined.

Figure 10: Linear probes at every block and schedule fraction. Each point is one of the 150 fitted cells per model and target. Faint horizontal lines mark chance. The random network approaches the trained models for position at

\tau=0.9

, while the separation is clear at the other schedule positions.

Figure 11: MLP probes at every block and schedule fraction. The nonlinear readout narrows the difference between random and trained networks, especially for position. Contact retains a clearer ordering between the three backbones.

	Linear		MLP
Simulator target	Mean	Best cell	Mean	Best cell
Position	0.709	0.772	0.879	0.926
Orientation	0.024	0.055	0.035	0.065
Linear velocity	-14.958	-1.497	-0.010	0.278
Linear acceleration	-0.001	0.353	0.149	0.493
Angular velocity	-0.095	-0.023	-0.002	0.001

Appendix

Table 20: Continuous-state probes on base Wan. Entries are test

R^{2}

, averaged over the 150 layer–noise cells or taken at the best cell. A negative value is worse than predicting the test-set mean.

Removed subspace	Full model $\Delta\mathcal{L}$	Direct head $\Delta\mathcal{L}$
Position (INLP)	$0.602\;[0.563,0.644]$	$1.229\;[1.157,1.304]$
Variance-matched PC	$0.023\;[0.020,0.026]$	$0.055\;[0.052,0.059]$
Random subspace	$0.012\;[0.012,0.012]$	$0.012\;[0.011,0.012]$

Appendix

Table 21: Aggregate intervention effects. Each entry is the mean loss increase followed by a bootstrap 95% confidence interval. We first average each of the 100 clips over 30 blocks and three noise levels, then resample clips 20,000 times.

Figure 12: Intervention effects by block and schedule fraction. Lines are means over 100 clips and shading gives bootstrap 95% confidence intervals. The symmetric-log axis is linear between

-0.01

and

0.01

, allowing the near-zero controls and the position intervention to remain visible in the same panels.

Physical Consistency in Video Generation

Physical consistency in video generation studies how to make and evaluate videos whose geometry, motion, interactions, and state changes obey real-world laws.

See all 191 papers

PVBench: Quantifying Physical Plausibility in Video Generation with Object-Level Physical Measurements
Oct 19, 2026Kaiwei Chen, Moran Li, Zhenye Gan +5Physical Consistency in Video Generation Video Generation Evaluation
APRIL Lab, Zhejiang University · Huazhong University of Science and Technology · Tencent Youtu Lab
RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation
Oct 20, 2026Siyuan Ma, Yu Qi, Cheng Ye +8Vision-Language Model-as-a-Judge Video Generation Evaluation
The Chinese University of Hong Kong · Northeastern University · University of California, San Diego +3
Quantitative Video World Model Evaluation for Geometric-Consistency
May 14, 2026Jiaxin Wu, Yihao Pi, Yinling Zhang +2Physical Consistency in Video Generation Video Generation
1Tsinghua University - IEI Lab · 2UW-Madison · 3Adobe Research

Oct 19, 2026J/K move · Enter open · S save

PVBench: Quantifying Physical Plausibility in Video Generation with Object-Level Physical Measurements

Kaiwei Chen, Moran Li, Zhenye Gan, Yabiao Wang+4

APRIL Lab, Zhejiang University · Huazhong University of Science and Technology · Tencent Youtu Lab

Physical Consistency in Video Generation Video Generation Evaluation Geometric Consistency in Video Generation Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Recent video generation models produce visually compelling outputs, yet their physical plausibility remains poorly measured. Physical plausibility asks whether generated objects obey geometry, kinematics, dynamics, and conservation laws. Existing benchmarks rely on coarse visual-language model judgments or target embodied world models rather than video generation itself. We introduce PVBench, a physical video benchmark that quantitatively measures the plausibility of generated videos across 46 dimensions organized into seven domains, spanning: i) instruction and semantic fidelity, ii) behavioral and causal consistency, iii) perceptual and temporal quality, iv) geometric and spatial consistency, v) material properties and field states, vi) kinematic and spatiotemporal persistence, vii) and mechanical plausibility with conservation laws. PVBench comprises 600 carefully crafted prompts, of which 30% involve physical embodied agents such as robot arms, humanoids, quadrupeds, and multi-agent systems. For challenging physically tractable dimensions, it pairs each with specialized vision models that derive verifiable, physically grounded measurements, such as object-level 6D pose, metric scale, surface normals, kinematics, and numerical conservation-law residuals. It thus avoids relying solely on subjective scoring, while supporting both text-to-video and image-to-video generation. Finally, we validate the agreement between automatic metrics and human judgments through human evaluation. Meanwhile, we evaluate mainstream open-source and closed-source models and present several insights. By releasing the benchmark, prompts, and calibration protocol, we aim to foster physically grounded video generation.

Abstract

Physical Consistency in Video Generation

Explore similar work

PVBench: Quantifying Physical Plausibility in Video Generation with Object-Level Physical Measurements

RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation

Quantitative Video World Model Evaluation for Geometric-Consistency