MotionCore: A Latent Neural Simulator for Video Generation with Rigid-Body Dynamics

Zheng Wang; Zhengwei Gong; Yuanzhi Liang; Chi Zhang

cs.CVNov 6, 2026

MotionCore: A Latent Neural Simulator for Video Generation with Rigid-Body Dynamics

Authors: Zheng Wang, Zhengwei Gong, Yuanzhi Liang, Chi Zhang

Organizations: Institute of Artificial Intelligence, China Telecom (TeleAI) · The University of Tokyo · Shanghai Jiao Tong University

Abstract

Video generation models can produce highly realistic videos while generating physically inconsistent object motion. This limitation suggests that visual modeling alone may not provide a transferable representation of physical dynamics. We introduce MotionCore, a framework for learning physical dynamics via a learnable partial differential equation solver and focuses on rigid-body motion. We introduce a latent neural simulator learning to predict the evolution of object-level physical states from observed motion. Its predictions are then converted into spatially aligned latent controls for a controllable video diffusion model. The generator serves as a visual interface: it provides appearance and scene detail, while the learned dynamics model specifies the physical trajectory. To preserve this trajectory through the interface, MotionCore combines a diffusion-matching objective with a low-capacity decodability constraint on object identity and position. We evaluate MotionCore on public physical-consistency benchmarks and on paired data with known trajectories. Results indicate that MotionCore improves physical consistency and trajectory obedience in the evaluated rigid-body settings, while maintaining comparable visual quality. A controlled ablation protocol compares learned dynamics with an analytic prior and studies the choice of latent evolution processor.

Figures & tables

Figure 1: Overview of MotionCore. (a) Stage 1 pretrains the occupancy autoencoder and the surface operator. Stage 2 freezes them and trains the neural simulator on rollouts of $K$ frames. At inference the simulator rolls out the scene, and the decoded occupancy is rendered into a control video for the frozen Wan2.1-VACE [ 12 ] . (b) State adapter and latent neural operator. (c) Free-motion proposal. (d) Surface operator. (e) Collision operator: learned contact probabilities and coefficients enter a closed-form impulse update. Snowflakes mark frozen modules, and dashed red boxes mark losses.

		Physics-IQ-Verified subset			RigidBench [ 11 ]
Method	#Params	Score $\uparrow$	ST-IoU $\uparrow$	MSE $\downarrow$	IoU $\uparrow$	L2 $\downarrow$	Cham. $\downarrow$	ATE $\downarrow$	ATE-3D $\downarrow$	SI-MSE $\downarrow$
CogVideoX-5B-I2V	5B	30.59	.297	.0046	.146	.222	.361	.186	2.63	.0349
HunyuanVideo1.5-480P-I2V	8.3B	30.08	.151	.0057	.170	.211	.345	.181	2.19	.0372
MiniMax-H3	33B	39.36	.236	.0048	.202	.198	.328	.186	2.00	.0375
Wan2.2-TI2V	5B	24.70	.165	.0057	.183	.220	.370	.200	2.64	.0288
Cosmos-Predict-2.5 2B	2B	27.05	.190	.0060	.171	.300	.527	.282	2.84	.0331

Table 1: Physical plausibility on the subset of Physics-IQ-Verified [ 38 ] and RigidBench [ 11 ] . Physics-IQ-Verified block uses the rigid-body subset. Both are tested on four random seeds (0, 42, 1989704766, 177674710).

Figure 2: Qualitative comparison on three Physics-IQ-Verified scenes. Blocks show (a) collision, (b) falling, and (c) rolling. Each block compares the reference video, MotionCore, native VACE-1.3B, Wan2.2-TI2V-5B, and LTX-Video 0.9.5. Columns show the shared input and frames nearest the indicated times after the switch frame. “VACE-1.3B w/o simulator” denotes native VACE without our adapter or control-branch LoRA. These are selected examples rather than a controlled simulator ablation.

Dynamics	Params (M)	ADE/ $R_{\mathrm{eq}}\downarrow$	FDE/ $R_{\mathrm{eq}}\downarrow$
Full dynamics (LNO)	1.283	0.5181	0.9699
Analytic, same priors	0	0.5204	0.9823
Transolver processor	1.298	0.5189	0.9715
GNOT-CGPT processor	1.338	0.5184	0.9702

Table 2: Shared latent-slot early-optimization pilot. All learned variants use 50 updates and seed 0; rollouts span 20 steps at 60 Hz. Test errors use the same 16 scenes, normalized by equivalent radius

R_{\mathrm{eq}}

(floor 1 mm). Parameters count the active processor.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: The rollout bound of Theorem A.4 , measured term by term on held-out MOVi-Spheres scenes. (a) Centre error against rollout horizon, with the bound evaluated from the measured per-step defects and the amplification

L_{\mathrm{loc}}

. (b) The bound’s two factors across

50

epochs: the mean per-step defect

\bar{\eta}

(left, log scale) rises fivefold after epoch

6

while the amplification implied by the observed accumulation,

L^{\star}

(right), falls through one. The perturbation rate

L_{\mathrm{loc}}

stays flat and tracks neither. (c) Substituting the encoded truth into one factor of the latent state at every step. Correcting the occupancy latent leaves the rollout unchanged, while correcting the object state collapses it to the per-step defect.

	Error $\downarrow$	Error at $k{=}20$ $\downarrow$	Sphere IoU $\uparrow$	Occ. IoU $\uparrow$	Contact F1 $\uparrow$
Latent neural simulator	0.0384	0.0908	0.901	0.892	0.508
Analytic impulse solver	0.0654	0.1669	0.855	0.846	0.651
Ballistic (gravity, no contact)	0.2264	0.6248	0.693	0.705	0.597
Persistence (frozen state)	0.2049	0.5008	0.670	0.673	0.597

Appendix

Table 3: The simulator against explicit alternatives , on

200

held-out windows of

20

frames. Error is the centre distance in scene units, averaged over the rollout and at its end. Contact F1 scores predicted against true contact events under one geometric criterion applied to all four rows, so it reads the raw predicted geometry rather than any learned contact head.

Figure 4: Early state-error accumulation. Saved radius-normalized position errors, averaged over valid objects at each prediction time on the same 16 test scenes. The vertical axis starts at zero. This is one training seed and a 20-step (

\simeq 0.333

s) rollout under the shared latent-slot interface, not a convergence or long-horizon video comparison.

Figure 5: Actual loss over the 50 training updates. Each update uses its prescribed training window; all three learned arms see the same sequence. These raw state losses mix changing window difficulty with optimization, and are not fixed-validation convergence curves. The analytic arm is not trained.

Figure 6: Training the learned adapter on both corpora. The zero-initialised condition opens as training proceeds (centre), and the decodability constraint tightens (right); the flow-matching loss itself does not separate the variants, which is why obedience is judged by measurement after generation rather than by validation loss.

Physical Consistency in Video Generation

Physical consistency in video generation studies how to make and evaluate videos whose geometry, motion, interactions, and state changes obey real-world laws.

See all 196 papers

PhyCo: Learning Controllable Physical Priors for Generative Motion
Apr 30, 2026Sriram Narayanan, Ziyu Jiang, Srinivasa Narasimhan +1Physics-Informed Generative Modeling Physical Consistency in Video Generation
1Carnegie Mellon University · 2NEC Labs America · 3UC San Diego
Seeing the Unseen: Unified Visible--Invisible Motion for Physically Consistent Video Generation
Oct 30, 2026Mingyang Bao, Fangda Ye, Xihao Chen +3Physical Consistency in Video Generation Trajectory-Guided Video Generation
National University of Singapore · Nanyang Technological University · University of Oxford
PhyMAGIC: Physical Motion-Aware Generative Inference with Confidence-guided LLM
May 22, 2025Siwei Meng, Yawei Luo, Ping LiuPhysics-Informed Generative Modeling Physical Consistency in Video Generation
Department of Computer Science & Engineering, University of Nevada, Reno, USA · School of Software Technology, Zhejiang University, China

Apr 30, 2026J/K move · Enter open · S save

PhyCo: Learning Controllable Physical Priors for Generative Motion

Sriram Narayanan, Ziyu Jiang, Srinivasa Narasimhan, Manmohan Chandraker

1Carnegie Mellon University · 2NEC Labs America · 3UC San Diego

Physics-Informed Generative Modeling Physical Consistency in Video Generation Video Diffusion Models Video Generation Diffusion Models for Motion Generation Controllable Video Generation

Open paper AskPDF arXiv

Modern video diffusion models excel at appearance synthesis but still struggle with physical consistency: objects drift, collisions lack realistic rebound, and material responses seldom match their underlying properties. We present PhyCo, a framework that introduces continuous, interpretable, and physically grounded control into video generation. Our approach integrates three key components: (i) a large-scale dataset of over 100K photorealistic simulation videos where friction, restitution, deformation, and force are systematically varied across diverse scenarios; (ii) physics-supervised fine-tuning of a pretrained diffusion model using a ControlNet conditioned on pixel-aligned physical property maps; and (iii) VLM-guided reward optimization, where a fine-tuned vision-language model evaluates generated videos with targeted physics queries and provides differentiable feedback. This combination enables a generative model to produce physically consistent and controllable outputs through variations in physical attributes-without any simulator or geometry reconstruction at inference. On the Physics-IQ benchmark, PhyCo significantly improves physical realism over strong baselines, and human studies confirm clearer and more faithful control over physical attributes. Our results demonstrate a scalable path toward physically consistent, controllable generative video models that generalize beyond synthetic training environments.

Abstract

Physical Consistency in Video Generation

Explore similar work

PhyCo: Learning Controllable Physical Priors for Generative Motion

Seeing the Unseen: Unified Visible--Invisible Motion for Physically Consistent Video Generation

PhyMAGIC: Physical Motion-Aware Generative Inference with Confidence-guided LLM