We present PhysInOne, a large-scale synthetic dataset addressing the critical scarcity of physically-grounded training data for AI systems. Unlike existing datasets limited to merely hundreds or thousands of examples, PhysInOne provides 2 million videos across 153,810 dynamic 3D scenes, covering 71 basic physical phenomena in mechanics, optics, fluid dynamics, and magnetism. Distinct from previous works, our scenes feature multiobject interactions against complex backgrounds, with comprehensive ground-truth annotations including 3D geometry, semantics, dynamic motion, physical properties, and text descriptions. We demonstrate PhysInOne's efficacy across four emerging applications: physics-aware video generation, long-/short-term future frame prediction, physical property estimation, and motion transfer. Experiments show that fine-tuning foundation models on PhysInOne significantly enhances physical plausibility, while also exposing critical gaps in modeling complex physical dynamics and estimating intrinsic properties. As the largest dataset of its kind, orders of magnitude beyond prior works, PhysInOne establishes a new benchmark for advancing physics-grounded world models in generation, simulation, and embodied AI.
Figures & tables
Year
# Physical Phenomena
# Physical Scenes
# Dynamic Videos
Multiobject Interactions
Multiphysics Activities
Complex Objects
Complex Backgrounds
# Fixed / Monocular Cameras
Annotations ( G eometry, S emantics, M otion, P roperties, T exts)
Physics101 [ 88 ]
BMVC’16
4
-
17,408
✗
✗
✗
✗
3, 0
P
ShapeStacks [ 36 ]
ECCV’18
1
20,000
20,000
✓
✗
✗
✗
1, 0
S, M, T
ScalarFlow [ 29 ]
TOG’19
1
100
100
✗
✗
✗
✗
- , -
M, P
CLEVRER [ 92 ]
ICLR’20
1
10,000
10,000
✓
✗
✗
✗
- , -
M, T
Physion [ 9 ]
NeurIPS’21
8
-
24,000
✓
✗
✗
✓
- , -
M, T
ComPhy [ 18 ]
ICLR’22
1
12,000
12,000
✓
✗
✗
✗
1, 0
M, T
Table 1: A comparison between PhysInOne and existing datasets that are relevant to learning physics from dynamic videos. PhysInOne provides massive 3D scenes with complex objects and backgrounds across diverse physical phenomena, surpassing all prior works.
Figure 2: Spanning 71 basic physical phenomena scaled to 3,284 multiphysics activities, PhysInOne comprises 153,810 unique scenes featuring 2,231 objects across 528 backgrounds and 623 materials, yielding 2 million videos with comprehensive annotations.
Figure 3: Qualitative examples demonstrating improved physical plausibility in videos generated after fine-tuning on PhysInOne.
Figure 4: Qualitative examples of long-term future frame prediction by current methods for trained viewpoints.
Human
Mechanics
Magnetism
Optics
Fluid
PMF ↑
FVD ↓
Rating ↑
PMF ↑
PMF ↑
PMF ↑
PMF ↑
SVD [ 10 ]
2.753
203
6.09
2.763
3.758
2.582
3.278
SVD lora
2.446
150
5.82
2.473
3.283
2.303
2.815
SVD sft
3.147
143
6.08
3.139
3.948
3.261
3.868
SVD flt
2.464
147
5.45
2.463
3.426
2.485
3.061
CogVideoX [ 90 ]
2.877
165
2.98
2.912
3.025
2.495
3.033
Table 2: The left part shows quantitative results of video generation models with and without fine-tuning on PhysInOne. The right part shows PMF scores across four physical domains separately.
Figure 5: Qualitative resimulation results using estimated physical properties. Both baselines fail to accurately infer properties for complex objects against intricate backgrounds, leading to physically implausible outcomes.
PMF ↑
PSNR ↑
SSIM ↑
LPIPS ↓
TiNeuVox [ 31 ]
3.710 / 2.885
21.49 / 15.20
0.633 / 0.452
0.517 / 0.665
DefGS [ 91 ]
3.980 / 3.347
22.85 / 17.95
0.833 / 0.598
0.192 / 0.348
TRACE [ 53 ]
3.869 / 3.242
22.42 / 17.44
0.756 / 0.599
0.295 / 0.422
FreeGave [ 54 ]
3.897 / 3.265
22.57 / 17.75
0.818 / 0.619
0.219 / 0.355
ExtDM [ 98 ]
3.363 / -
19.55 / -
0.657 / -
0.771 / -
MAGI-1 [ 1 ]
4.086 / -
23.14 / -
0.788 / -
0.364 / -
Table 3: Quantitative results of long-term future frame prediction from seen / novel viewpoints on PhysInOne.
PMF ↑
PSNR ↑
SSIM ↑
LPIPS ↓
DefGS [ 91 ]
4.536 / 3.728
26.02 / 20.92
0.861 / 0.739
0.206 / 0.322
FreeGave [ 54 ]
4.742 / 3.706
27.09 / 20.80
0.876 / 0.715
0.199 / 0.336
ExtDM [ 98 ]
3.774 / -
22.14 / -
0.717 / -
0.715 / -
MAGI-1 [ 1 ]
4.696 / -
26.75 / -
0.886 / -
0.116 / -
Table 4: Quantitative results of short-term future frame prediction from seen / novel viewpoints on PhysInOne.
Figure 6: Qualitative motion transfer results from GoWithTheFlow and MotionPro. Generated frames retain visual realism but fail to transfer complex physical motions ( e . g ., moving cars, falling ball).
Elastic Solids
log10(E)
ν
v
PAC-NeRF [ 55 ]
117.18 ± 68.44
14.26 ± 7.94
4.04 ± 1.11
GIC [ 14 ]
49.76 ± 8.78
16.35 ± 1.36
3.32 ± 1.34
Plasticine
log10(E)
ν
log10(τY)
v
PAC-NeRF [ 55 ]
68.38 ± 44.14
15.79 ± 11.07
25.51 ± 18.34
3.25 ± 0.23
Table 5: Quantitative results of physical parameters estimation for five types of materials on PhysInOne.
PMF ↑
PSNR ↑
SSIM ↑
LPIPS ↓
PAC-NeRF [ 55 ]
5.617
24.12
0.942
0.086
GIC [ 14 ]
5.938
26.90
0.950
0.074
Table 6: Quantitative results of resimulation based on estimated physical properties.
PMF ↑
PSNR ↑
SSIM ↑
LPIPS ↓
GoWithTheFlow [ 13 ]
3.309
18.98
0.691
0.410
MotionPro [ 99 ]
3.484
20.28
0.775
0.467
Table 7: Quantitative results of transferring physical motions.
Index
Physical Phenomena
Physical Laws
1
Object collide with static, stationary objects
Laws of Momentum
2
Moving objects collide with non-static stationary objects
Laws of Momentum
3
Two moving objects collide
Laws of Momentum
4
Objects in equilibrium of wind and gravity
Equilibrium, Aerodynamics, Gravity
5
Wind applied to a stationary object
Aerodynamics
6
Wind applied to objects moving in same direction
Aerodynamics
Table 8: The list of Physical Phenomena and Related Physical Laws
Figure 7: Exemplary 3D asset under CC BY License
Figure 8: Exemplary 3D asset under UE Standard License
Figure 9: Examples of materials.
Figure 10: Examples of 3D assets.
Figure 11: The pipeline to create 3D scenes in Unreal Engine.
Figure 12: The pipeline to create 3D scenes concerning liquid.
Figure 13: The pipeline to create 3D scenes concerning special materials.
Friction Coefficient
Restitution
Density
SPH Viscosity
SPH Surface Tension
Fluid Viscosity
Plasticity Viscosity
Yield Stress
Friction Angle
..
min ∼ max
0. ∼ 0.9
0.1 ∼ 0.8
0.1 ∼ 21.3
0.0 ∼ 5.0
0.5 ∼ 3.0
25 ∼ 200
3 ∼ 10
0.1 ∼ 10
15 ∼ 60
..
Table 9: Ranges of Example Physical Parameters.
Figure 14: Static Camera Sampling
Figure 15: Linear Drift Sampling
Figure 16: Sinusoidal Interpolation Sampling
Figure 17: Circular Loop Trajectory Sampling on Sphere
Figure 18: The distribution of video lengths.
Figure 19: Demonstration for PMF. In the top-left pair, the only variance is the initial spatial location where the object begins to fall. As expected, PMF yields a perfect score, reflecting identical motion patterns regardless of absolute positioning. Conversely, in the top-right pair, the motion is completely reversed ( e . g ., upward vs. downward fall). The metric degrades severely due to the starkly mismatched spectral energy distributions, correctly indicating poor motion similarity. In the bottom-left case, the initial position is fixed but the velocity is halved. Here, the PMF value increases substantially, effectively distinguishing the altered temporal dynamics through the energy spectrum. Finally, in the bottom-right example, where the object shape differs but the underlying motion dynamics are preserved, the PMF remains relatively high. This confirms that PMF is largely invariant to visual appearance variations and focuses primarily on the consistency of motion dynamics.
Elastic Solids
Plasticine
Newtonian Fluids
Non-Newtonian Fluids
Granular Substances
PAC-NeRF [ 55 ]
E=316228ν=0.25
E=10000ν=0.25τy=1000
μ=10κ=1000
μ=100κ=100000τy=10η=1
θfric=10
GIC [ 14 ]
E=316228ν=0.25
E=10000ν=0.25τy=1000
μ=10κ=1000
μ=100κ=100000τy=10η=1
θfric=10
Table 10: The initial guess of physical parameters estimation for five types of materials on PhysInOne.
PMF ↑
PSNR ↑
SSIM ↑
LPIPS ↓
Elastic Solids
PAC-NeRF [ 55 ]
5.158
27.909
0.966
0.062
GIC [ 14 ]
5.570
30.143
0.968
0.057
Plasticine
PAC-NeRF [ 55 ]
5.071
27.429
0.967
0.051
GIC [ 14 ]
5.535
30.191
0.971
0.044
Newtonian Fluids
PAC-NeRF [ 55 ]
4.201
24.797
0.957
0.061
GIC [ 14 ]
5.047
29.381
0.967
0.050
Table 11: Additional quantitative results of novel view synthesis based on estimated physical properties.
PMF ↑
PSNR ↑
SSIM ↑
LPIPS ↓
Elastic Solids
PAC-NeRF [ 55 ]
4.701
25.463
0.964
0.062
GIC [ 14 ]
5.136
27.886
0.967
0.055
Plasticine
PAC-NeRF [ 55 ]
4.614
24.850
0.963
0.058
GIC [ 14 ]
5.206
28.332
0.969
0.047
Newtonian Fluids
PAC-NeRF [ 55 ]
4.166
24.253
0.956
0.061
GIC [ 14 ]
4.900
28.144
0.964
0.051
Table 12: Additional quantitative results of resimulation based on estimated physical properties.
World simulators can provide safe and scalable environments for training Physical AI systems before real-world deployment. Large video generation models are emerging as a promising basis for such simulators because they can generate diverse and realistic visual futures. However, using them as world simulators requires physically faithful video continuations, namely, generated videos that preserve the physical state implied by the conditioning input, and evolve in ways consistent with basic physical principles. We propose PhyWorld, a video generation world model designed to produce temporally coherent and physically faithful scene continuations through two-stage post-training. In the first stage, we improve video-to-video continuation with flow matching fine-tuning, encouraging stable visual attributes and coherent motion dynamics across frames. In the second stage, we align generated dynamics with physical principles using Direct Preference Optimization (DPO) over physics preference pairs, guiding the model toward outputs with higher physical plausibility. To evaluate PhyWorld, we use both standard video-quality benchmarks and a dedicated physical-faithfulness benchmark with per-law scoring. Experiments show that PhyWorld improves video consistency, achieving an average score of 0.769 on VBench compared with 0.756 or below for state-of-the-art baselines. PhyWorld also improves physical plausibility, reaching an average score of 3.09 on our physical-faithfulness benchmark compared with 2.99 for the strongest baseline. These results suggest that post-training large video generation models with continuation and physics-preference signals can make them more effective world simulators for Physical AI.
Pu Zhao, Juyi Lin, Timothy Rupprecht +10
Northeastern University · University of Georgia · Tulane University +1
Developing physically aware video generation models remains a significant challenge due to the difficulty in capturing diverse physical phenomena, such as thermal dynamics, mechanics, and optics. In this work, we introduce PhysRAG, a novel pipeline that enhances physical awareness in video generation through Retrieval-Augmented Generation (RAG). To address the issue of limited high-quality data, we design a two-stage data filtering pipeline based on the WISA-80K dataset, resulting in a curated set of 7K high-quality videos for training. Furthermore, we construct a physical video database and develop a mechanism to inject physical knowledge into a video diffusion model using learnable queries. Our method achieves state-of-the-art performance in both visual quality and physical rule compliance, surpassing existing models in benchmarks such as PhyGenBench and VBench. We conduct extensive ablation studies to validate the effectiveness of our key components, including the data filtering pipeline, RAG mechanism, and method for physical information extraction. To facilitate future research, our code, data, and models are prepared for release at https://github.com/sediment1024/PhysRAG.
Kexu Cheng, Zicheng Liu, Mingju Gao +2
School of Computer Science, Peking University, Beijing, China · Institute of AI for Industries, Chinese Academy of Sciences, Nanjing, China
Simulation-ready physical 3D assets have emerged as a promising direction owing to their broad applicability in downstream tasks. However, most existing 3D generation methods either neglect physical properties or are limited to a single asset category, e.g., rigid, deformable, or articulated objects. To address these limitations, we introduce PhysX-Omni, a unified framework for simulation-ready physical 3D generation across diverse asset types. Specifically, we develop a novel and efficient geometry representation tailored for Vision-Language Models, which directly encodes high-resolution 3D structures without compression, significantly improving generation performance. In addition, we construct the first general simulation-ready 3D dataset, PhysXVerse, covering diverse indoor and outdoor categories. Furthermore, to comprehensively and flexibly evaluate both generative and understanding capabilities in the wild, we propose PhysX-Bench, which encompasses six key attributes: geometry, absolute scale, material, affordance, kinematics, and function description. Extensive experiments with conventional metrics and PhysX-Bench show that PhysX-Omni performs strongly in both generation and understanding. Moreover, additional studies further validate the potential of PhysX-Omni for applications in simulation-ready scene generation and robotic policy learning. We believe PhysX-Omni can significantly advance a wide range of downstream applications, particularly in embodied AI and physics-based simulation.
Ziang Cao, Yinghao Liu, Haitian Li +5
S-Lab, Nanyang Technological University · ACE Robotics