PhysInOne: Visual Physics Learning and Reasoning in One Suite

Siyuan Zhou; Hejun Wang; Hu Cheng; Jinxi Li; Dongsheng Wang; Junwei Jiang; Yixiao Jin; Jiayue Huang

cs.CVApr 10, 2026

PhysInOne: Visual Physics Learning and Reasoning in One Suite

Authors: Siyuan Zhou, Hejun Wang, Hu Cheng, Jinxi Li, Dongsheng Wang, Junwei Jiang, Yixiao Jin, Jiayue Huang,

Organizations: vLAR Group · The Hong Kong Polytechnic University · Syai Singapore · Meta

Abstract

We present PhysInOne, a large-scale synthetic dataset addressing the critical scarcity of physically-grounded training data for AI systems. Unlike existing datasets limited to merely hundreds or thousands of examples, PhysInOne provides 2 million videos across 153,810 dynamic 3D scenes, covering 71 basic physical phenomena in mechanics, optics, fluid dynamics, and magnetism. Distinct from previous works, our scenes feature multiobject interactions against complex backgrounds, with comprehensive ground-truth annotations including 3D geometry, semantics, dynamic motion, physical properties, and text descriptions. We demonstrate PhysInOne's efficacy across four emerging applications: physics-aware video generation, long-/short-term future frame prediction, physical property estimation, and motion transfer. Experiments show that fine-tuning foundation models on PhysInOne significantly enhances physical plausibility, while also exposing critical gaps in modeling complex physical dynamics and estimating intrinsic properties. As the largest dataset of its kind, orders of magnitude beyond prior works, PhysInOne establishes a new benchmark for advancing physics-grounded world models in generation, simulation, and embodied AI.

Figures & tables

	Year	# Physical Phenomena	# Physical Scenes	# Dynamic Videos	Multiobject Interactions	Multiphysics Activities	Complex Objects	Complex Backgrounds	# Fixed / Monocular Cameras	Annotations ( G eometry, S emantics, M otion, P roperties, T exts)
Physics101 [ 88 ]	BMVC’16	4	-	17,408	✗	✗	✗	✗	3, 0	P
ShapeStacks [ 36 ]	ECCV’18	1	20,000	20,000	✓	✗	✗	✗	1, 0	S, M, T
ScalarFlow [ 29 ]	TOG’19	1	100	100	✗	✗	✗	✗	- , -	M, P
CLEVRER [ 92 ]	ICLR’20	1	10,000	10,000	✓	✗	✗	✗	- , -	M, T
Physion [ 9 ]	NeurIPS’21	8	-	24,000	✓	✗	✗	✓	- , -	M, T
ComPhy [ 18 ]	ICLR’22	1	12,000	12,000	✓	✗	✗	✗	1, 0	M, T

Table 1: A comparison between PhysInOne and existing datasets that are relevant to learning physics from dynamic videos. PhysInOne provides massive 3D scenes with complex objects and backgrounds across diverse physical phenomena, surpassing all prior works.

Figure 2: Spanning 71 basic physical phenomena scaled to 3,284 multiphysics activities, PhysInOne comprises 153,810 unique scenes featuring 2,231 objects across 528 backgrounds and 623 materials, yielding 2 million videos with comprehensive annotations.

Figure 3: Qualitative examples demonstrating improved physical plausibility in videos generated after fine-tuning on PhysInOne.

Figure 4: Qualitative examples of long-term future frame prediction by current methods for trained viewpoints.

			Human	Mechanics	Magnetism	Optics	Fluid
	PMF $\uparrow$	FVD $\downarrow$	Rating $\uparrow$	PMF $\uparrow$	PMF $\uparrow$	PMF $\uparrow$	PMF $\uparrow$
SVD [ 10 ]	2.753	203	6.09	2.763	3.758	2.582	3.278
SVD lora	2.446	150	5.82	2.473	3.283	2.303	2.815
SVD sft	3.147	143	6.08	3.139	3.948	3.261	3.868
SVD flt	2.464	147	5.45	2.463	3.426	2.485	3.061
CogVideoX [ 90 ]	2.877	165	2.98	2.912	3.025	2.495	3.033

Table 2: The left part shows quantitative results of video generation models with and without fine-tuning on PhysInOne. The right part shows PMF scores across four physical domains separately.

Figure 5: Qualitative resimulation results using estimated physical properties. Both baselines fail to accurately infer properties for complex objects against intricate backgrounds, leading to physically implausible outcomes.

	PMF $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
TiNeuVox [ 31 ]	3.710 / 2.885	21.49 / 15.20	0.633 / 0.452	0.517 / 0.665
DefGS [ 91 ]	3.980 / 3.347	22.85 / 17.95	0.833 / 0.598	0.192 / 0.348
TRACE [ 53 ]	3.869 / 3.242	22.42 / 17.44	0.756 / 0.599	0.295 / 0.422
FreeGave [ 54 ]	3.897 / 3.265	22.57 / 17.75	0.818 / 0.619	0.219 / 0.355
ExtDM [ 98 ]	3.363 / -	19.55 / -	0.657 / -	0.771 / -
MAGI-1 [ 1 ]	4.086 / -	23.14 / -	0.788 / -	0.364 / -

Table 3: Quantitative results of long-term future frame prediction from seen / novel viewpoints on PhysInOne.

	PMF $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
DefGS [ 91 ]	4.536 / 3.728	26.02 / 20.92	0.861 / 0.739	0.206 / 0.322
FreeGave [ 54 ]	4.742 / 3.706	27.09 / 20.80	0.876 / 0.715	0.199 / 0.336
ExtDM [ 98 ]	3.774 / -	22.14 / -	0.717 / -	0.715 / -
MAGI-1 [ 1 ]	4.696 / -	26.75 / -	0.886 / -	0.116 / -

Table 4: Quantitative results of short-term future frame prediction from seen / novel viewpoints on PhysInOne.

Figure 6: Qualitative motion transfer results from GoWithTheFlow and MotionPro. Generated frames retain visual realism but fail to transfer complex physical motions ( e . g ., moving cars, falling ball).

	Elastic Solids
	$\log_{10}(E)$	$\nu$		$\boldsymbol{v}$
PAC-NeRF [ 55 ]	117.18 $\pm$ 68.44	14.26 $\pm$ 7.94		4.04 $\pm$ 1.11
GIC [ 14 ]	49.76 $\pm$ 8.78	16.35 $\pm$ 1.36		3.32 $\pm$ 1.34
	Plasticine
	$\log_{10}(E)$	$\nu$	$\log_{10}(\tau_{Y})$	$\boldsymbol{v}$
PAC-NeRF [ 55 ]	68.38 $\pm$ 44.14	15.79 $\pm$ 11.07	25.51 $\pm$ 18.34	3.25 $\pm$ 0.23

Table 5: Quantitative results of physical parameters estimation for five types of materials on PhysInOne.

	PMF $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
PAC-NeRF [ 55 ]	5.617	24.12	0.942	0.086
GIC [ 14 ]	5.938	26.90	0.950	0.074

Table 6: Quantitative results of resimulation based on estimated physical properties.

	PMF $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
GoWithTheFlow [ 13 ]	3.309	18.98	0.691	0.410
MotionPro [ 99 ]	3.484	20.28	0.775	0.467

Table 7: Quantitative results of transferring physical motions.

Index	Physical Phenomena	Physical Laws
1	Object collide with static, stationary objects	Laws of Momentum
2	Moving objects collide with non-static stationary objects	Laws of Momentum
3	Two moving objects collide	Laws of Momentum
4	Objects in equilibrium of wind and gravity	Equilibrium, Aerodynamics, Gravity
5	Wind applied to a stationary object	Aerodynamics
6	Wind applied to objects moving in same direction	Aerodynamics

Table 8: The list of Physical Phenomena and Related Physical Laws

Figure 7: Exemplary 3D asset under CC BY License

Figure 8: Exemplary 3D asset under UE Standard License

Figure 11: The pipeline to create 3D scenes in Unreal Engine.

Figure 12: The pipeline to create 3D scenes concerning liquid.

Figure 13: The pipeline to create 3D scenes concerning special materials.

	Friction Coefficient	Restitution	Density	SPH Viscosity	SPH Surface Tension	Fluid Viscosity	Plasticity Viscosity	Yield Stress	Friction Angle	..
min $\sim$ max	0. $\sim$ 0.9	0.1 $\sim$ 0.8	0.1 $\sim$ 21.3	0.0 $\sim$ 5.0	0.5 $\sim$ 3.0	25 $\sim$ 200	3 $\sim$ 10	0.1 $\sim$ 10	15 $\sim$ 60	..

Table 9: Ranges of Example Physical Parameters.

Figure 16: Sinusoidal Interpolation Sampling

Figure 17: Circular Loop Trajectory Sampling on Sphere

Figure 18: The distribution of video lengths.

Figure 19: Demonstration for PMF. In the top-left pair, the only variance is the initial spatial location where the object begins to fall. As expected, PMF yields a perfect score, reflecting identical motion patterns regardless of absolute positioning. Conversely, in the top-right pair, the motion is completely reversed ( e . g ., upward vs. downward fall). The metric degrades severely due to the starkly mismatched spectral energy distributions, correctly indicating poor motion similarity. In the bottom-left case, the initial position is fixed but the velocity is halved. Here, the PMF value increases substantially, effectively distinguishing the altered temporal dynamics through the energy spectrum. Finally, in the bottom-right example, where the object shape differs but the underlying motion dynamics are preserved, the PMF remains relatively high. This confirms that PMF is largely invariant to visual appearance variations and focuses primarily on the consistency of motion dynamics.

	Elastic Solids	Plasticine	Newtonian Fluids	Non-Newtonian Fluids	Granular Substances
PAC-NeRF [ 55 ]	$E=316228$ $\nu=0.25$	$E=10000$ $\nu=0.25$ $\tau_{y}=1000$	$\mu=10$ $\kappa=1000$	$\mu=100$ $\kappa=100000$ $\tau_{y}=10$ $\eta=1$	$\theta_{fric}=10$
GIC [ 14 ]	$E=316228$ $\nu=0.25$	$E=10000$ $\nu=0.25$ $\tau_{y}=1000$	$\mu=10$ $\kappa=1000$	$\mu=100$ $\kappa=100000$ $\tau_{y}=10$ $\eta=1$	$\theta_{fric}=10$

Table 10: The initial guess of physical parameters estimation for five types of materials on PhysInOne.

		PMF $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Elastic Solids	PAC-NeRF [ 55 ]	5.158	27.909	0.966	0.062
Elastic Solids	GIC [ 14 ]	5.570	30.143	0.968	0.057
Plasticine	PAC-NeRF [ 55 ]	5.071	27.429	0.967	0.051
Plasticine	GIC [ 14 ]	5.535	30.191	0.971	0.044
Newtonian Fluids	PAC-NeRF [ 55 ]	4.201	24.797	0.957	0.061
Newtonian Fluids	GIC [ 14 ]	5.047	29.381	0.967	0.050

Table 11: Additional quantitative results of novel view synthesis based on estimated physical properties.

		PMF $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Elastic Solids	PAC-NeRF [ 55 ]	4.701	25.463	0.964	0.062
Elastic Solids	GIC [ 14 ]	5.136	27.886	0.967	0.055
Plasticine	PAC-NeRF [ 55 ]	4.614	24.850	0.963	0.058
Plasticine	GIC [ 14 ]	5.206	28.332	0.969	0.047
Newtonian Fluids	PAC-NeRF [ 55 ]	4.166	24.253	0.956	0.061
Newtonian Fluids	GIC [ 14 ]	4.900	28.144	0.964	0.051

Table 12: Additional quantitative results of resimulation based on estimated physical properties.

Physical Reasoning in Video

Physical reasoning in video studies how models infer, predict, and generate motion and interactions consistent with physical laws.

See all 74 papers

PhyWorld: Physics-Faithful World Model for Video Generation
May 19, 2026Pu Zhao, Juyi Lin, Timothy Rupprecht +10Physics-Informed Generative Modeling Physical Consistency in Video Generation
Northeastern University · University of Georgia · Tulane University +1
PhysRAG: Enhancing Physics-Awareness in Video Generation via Retrieval-Augmented Generation
Jun 25, 2026Kexu Cheng, Zicheng Liu, Mingju Gao +2Multimodal RAG Physical Consistency in Video Generation
School of Computer Science, Peking University, Beijing, China · Institute of AI for Industries, Chinese Academy of Sciences, Nanjing, China
PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objects
May 20, 2026Ziang Cao, Yinghao Liu, Haitian Li +53D Asset Generation Physics-Based Simulation
S-Lab, Nanyang Technological University · ACE Robotics

May 19, 2026J/K move · Enter open · S save

PhyWorld: Physics-Faithful World Model for Video Generation

Pu Zhao, Juyi Lin, Timothy Rupprecht, Arash Akbari+9

Northeastern University · University of Georgia · Tulane University+1

Physics-Informed Generative Modeling Physical Consistency in Video Generation Video Generation Temporal Consistency in Video Generation Video World Models

Open paper AskPDF arXiv

World simulators can provide safe and scalable environments for training Physical AI systems before real-world deployment. Large video generation models are emerging as a promising basis for such simulators because they can generate diverse and realistic visual futures. However, using them as world simulators requires physically faithful video continuations, namely, generated videos that preserve the physical state implied by the conditioning input, and evolve in ways consistent with basic physical principles. We propose PhyWorld, a video generation world model designed to produce temporally coherent and physically faithful scene continuations through two-stage post-training. In the first stage, we improve video-to-video continuation with flow matching fine-tuning, encouraging stable visual attributes and coherent motion dynamics across frames. In the second stage, we align generated dynamics with physical principles using Direct Preference Optimization (DPO) over physics preference pairs, guiding the model toward outputs with higher physical plausibility. To evaluate PhyWorld, we use both standard video-quality benchmarks and a dedicated physical-faithfulness benchmark with per-law scoring. Experiments show that PhyWorld improves video consistency, achieving an average score of 0.769 on VBench compared with 0.756 or below for state-of-the-art baselines. PhyWorld also improves physical plausibility, reaching an average score of 3.09 on our physical-faithfulness benchmark compared with 2.99 for the strongest baseline. These results suggest that post-training large video generation models with continuation and physics-preference signals can make them more effective world simulators for Physical AI.

Abstract

Physical Reasoning in Video

Explore similar work

PhyWorld: Physics-Faithful World Model for Video Generation

PhysRAG: Enhancing Physics-Awareness in Video Generation via Retrieval-Augmented Generation

PhysX-Omni: Unified Simulation-Ready Physical 3D Generation for Rigid, Deformable, and Articulated Objects