Video generation models can produce highly realistic videos while generating physically inconsistent object motion. This limitation suggests that visual modeling alone may not provide a transferable representation of physical dynamics. We introduce MotionCore, a framework for learning physical dynamics via a learnable partial differential equation solver and focuses on rigid-body motion. We introduce a latent neural simulator learning to predict the evolution of object-level physical states from observed motion. Its predictions are then converted into spatially aligned latent controls for a controllable video diffusion model. The generator serves as a visual interface: it provides appearance and scene detail, while the learned dynamics model specifies the physical trajectory. To preserve this trajectory through the interface, MotionCore combines a diffusion-matching objective with a low-capacity decodability constraint on object identity and position. We evaluate MotionCore on public physical-consistency benchmarks and on paired data with known trajectories. Results indicate that MotionCore improves physical consistency and trajectory obedience in the evaluated rigid-body settings, while maintaining comparable visual quality. A controlled ablation protocol compares learned dynamics with an analytic prior and studies the choice of latent evolution processor.
Figures & tables
Figure 1: Overview of MotionCore. (a) Stage 1 pretrains the occupancy autoencoder and the surface operator. Stage 2 freezes them and trains the neural simulator on rollouts of K frames. At inference the simulator rolls out the scene, and the decoded occupancy is rendered into a control video for the frozen Wan2.1-VACE [ 12 ] . (b) State adapter and latent neural operator. (c) Free-motion proposal. (d) Surface operator. (e) Collision operator: learned contact probabilities and coefficients enter a closed-form impulse update. Snowflakes mark frozen modules, and dashed red boxes mark losses.
Physics-IQ-Verified subset
RigidBench [ 11 ]
Method
#Params
Score ↑
ST-IoU ↑
MSE ↓
IoU ↑
L2 ↓
Cham. ↓
ATE ↓
ATE-3D ↓
SI-MSE ↓
CogVideoX-5B-I2V
5B
30.59
.297
.0046
.146
.222
.361
.186
2.63
.0349
HunyuanVideo1.5-480P-I2V
8.3B
30.08
.151
.0057
.170
.211
.345
.181
2.19
.0372
MiniMax-H3
33B
39.36
.236
.0048
.202
.198
.328
.186
2.00
.0375
Wan2.2-TI2V
5B
24.70
.165
.0057
.183
.220
.370
.200
2.64
.0288
Cosmos-Predict-2.5 2B
2B
27.05
.190
.0060
.171
.300
.527
.282
2.84
.0331
Table 1: Physical plausibility on the subset of Physics-IQ-Verified [ 38 ] and RigidBench [ 11 ] . Physics-IQ-Verified block uses the rigid-body subset. Both are tested on four random seeds (0, 42, 1989704766, 177674710).
Figure 2: Qualitative comparison on three Physics-IQ-Verified scenes. Blocks show (a) collision, (b) falling, and (c) rolling. Each block compares the reference video, MotionCore, native VACE-1.3B, Wan2.2-TI2V-5B, and LTX-Video 0.9.5. Columns show the shared input and frames nearest the indicated times after the switch frame. “VACE-1.3B w/o simulator” denotes native VACE without our adapter or control-branch LoRA. These are selected examples rather than a controlled simulator ablation.
Dynamics
Params (M)
ADE/ Req↓
FDE/ Req↓
Full dynamics (LNO)
1.283
0.5181
0.9699
Analytic, same priors
0
0.5204
0.9823
Transolver processor
1.298
0.5189
0.9715
GNOT-CGPT processor
1.338
0.5184
0.9702
Table 2: Shared latent-slot early-optimization pilot. All learned variants use 50 updates and seed 0; rollouts span 20 steps at 60 Hz. Test errors use the same 16 scenes, normalized by equivalent radius Req (floor 1 mm). Parameters count the active processor.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: The rollout bound of Theorem A.4 , measured term by term on held-out MOVi-Spheres scenes. (a) Centre error against rollout horizon, with the bound evaluated from the measured per-step defects and the amplification Lloc . (b) The bound’s two factors across 50 epochs: the mean per-step defect ηˉ (left, log scale) rises fivefold after epoch 6 while the amplification implied by the observed accumulation, L⋆ (right), falls through one. The perturbation rate Lloc stays flat and tracks neither. (c) Substituting the encoded truth into one factor of the latent state at every step. Correcting the occupancy latent leaves the rollout unchanged, while correcting the object state collapses it to the per-step defect.
Error ↓
Error at k=20↓
Sphere IoU ↑
Occ. IoU ↑
Contact F1 ↑
Latent neural simulator
0.0384
0.0908
0.901
0.892
0.508
Analytic impulse solver
0.0654
0.1669
0.855
0.846
0.651
Ballistic (gravity, no contact)
0.2264
0.6248
0.693
0.705
0.597
Persistence (frozen state)
0.2049
0.5008
0.670
0.673
0.597
Appendix
Table 3: The simulator against explicit alternatives , on 200 held-out windows of 20 frames. Error is the centre distance in scene units, averaged over the rollout and at its end. Contact F1 scores predicted against true contact events under one geometric criterion applied to all four rows, so it reads the raw predicted geometry rather than any learned contact head.
Figure 4: Early state-error accumulation. Saved radius-normalized position errors, averaged over valid objects at each prediction time on the same 16 test scenes. The vertical axis starts at zero. This is one training seed and a 20-step ( ≃0.333 s) rollout under the shared latent-slot interface, not a convergence or long-horizon video comparison.
Figure 5: Actual loss over the 50 training updates. Each update uses its prescribed training window; all three learned arms see the same sequence. These raw state losses mix changing window difficulty with optimization, and are not fixed-validation convergence curves. The analytic arm is not trained.
Figure 6: Training the learned adapter on both corpora. The zero-initialised condition opens as training proceeds (centre), and the decodability constraint tightens (right); the flow-matching loss itself does not separate the variants, which is why obedience is judged by measurement after generation rather than by validation loss.
Physical consistency in video generation studies how to make and evaluate videos whose geometry, motion, interactions, and state changes obey real-world laws.
Modern video diffusion models excel at appearance synthesis but still struggle with physical consistency: objects drift, collisions lack realistic rebound, and material responses seldom match their underlying properties. We present PhyCo, a framework that introduces continuous, interpretable, and physically grounded control into video generation. Our approach integrates three key components: (i) a large-scale dataset of over 100K photorealistic simulation videos where friction, restitution, deformation, and force are systematically varied across diverse scenarios; (ii) physics-supervised fine-tuning of a pretrained diffusion model using a ControlNet conditioned on pixel-aligned physical property maps; and (iii) VLM-guided reward optimization, where a fine-tuned vision-language model evaluates generated videos with targeted physics queries and provides differentiable feedback. This combination enables a generative model to produce physically consistent and controllable outputs through variations in physical attributes-without any simulator or geometry reconstruction at inference. On the Physics-IQ benchmark, PhyCo significantly improves physical realism over strong baselines, and human studies confirm clearer and more faithful control over physical attributes. Our results demonstrate a scalable path toward physically consistent, controllable generative video models that generalize beyond synthetic training environments.
Recent advances in video generation have achieved impressive visual quality, yet they often fail to produce physically consistent dynamics due to the lack of explicit modeling of underlying physical processes. Existing approaches either rely on simulators with strong assumptions or use coarse semantic guidance, and further struggle to effectively inject physical priors into generation models. In this work, we propose Seeing the Unseen, a unified framework for physics-grounded video generation. Our approach decomposes the problem into two stages: learning physical dynamics and injecting them into video synthesis. First, we introduce a lightweight Particle-based Graph Dynamics Simulator (PGDS) that learns generalizable physical interactions from data and predicts plausible 3D motion trajectories. Second, we propose a Visible--Invisible Motion Field (VIMF) that captures both motion observable in the input frame and motion that emerges over time due to object dynamics. This representation enables more complete and structured motion guidance compared to conventional physical signals. By integrating these trajectories into a diffusion-based generator, our method produces videos that are both visually coherent and physically consistent. Extensive experiments demonstrate improved motion accuracy, interaction consistency, and reduced physical artifacts compared to strong baselines.
Mingyang Bao, Fangda Ye, Xihao Chen +3
National University of Singapore · Nanyang Technological University · University of Oxford
Recent advances in 3D content generation have amplified demand for dynamic models that are both visually realistic and physically consistent. However, state-of-the-art video diffusion models frequently produce implausible results such as momentum violations and object interpenetrations. Existing physics-aware approaches often rely on task-specific fine-tuning or supervised data, which limits their scalability and applicability. To address the challenge, we present PhyMAGIC, a training-free framework that generates physically consistent motion from a single image. PhyMAGIC integrates a pre-trained image-to-video diffusion model, confidence-guided reasoning via LLMs, and a differentiable physics simulator to produce 3D assets ready for downstream physical simulation without fine-tuning or manual supervision. By iteratively refining motion prompts using LLM-derived confidence scores and leveraging simulation feedback, PhyMAGIC steers generation toward physically consistent dynamics. Comprehensive experiments demonstrate that PhyMAGIC outperforms state-of-the-art video generators and physics-aware baselines, enhancing physical property inference and motion-text alignment while maintaining visual fidelity.
Siwei Meng, Yawei Luo, Ping Liu
Department of Computer Science & Engineering, University of Nevada, Reno, USA · School of Software Technology, Zhejiang University, China