Physical Simulator In-the-Loop Video Generation

Lin Geng Foo; Mark He Huang; Alexandros Lattas; Stylianos Moschoglou; Thabo Beeler; Christian Theobalt

cs.CVMar 6, 2026

Physical Simulator In-the-Loop Video Generation

Authors: Lin Geng Foo, Mark He Huang, Alexandros Lattas, Stylianos Moschoglou, Thabo Beeler, Christian Theobalt

Organizations: Max Planck Institute for Informatics, Saarland Informatics Campus · Saarbrücken Research Center for Visual Computing, Interaction and Artificial Intelligence · Singapore University of Technology and Design · A*STAR ·

Abstract

Recent advances in diffusion-based video generation have achieved remarkable visual realism but still struggle to obey basic physical laws such as gravity, inertia, and collision. Generated objects often move inconsistently across frames, exhibit implausible dynamics, or violate physical constraints, limiting the realism and reliability of AI-generated videos. We address this gap by introducing Physical Simulator In-the-loop Video Generation (PSIVG), a novel framework that integrates a physical simulator into the video diffusion process. Starting from a template video generated by a pre-trained diffusion model, PSIVG reconstructs the 4D scene and foreground object meshes, initializes them within a physical simulator, and generates physically consistent trajectories. These simulated trajectories are then used to guide the video generator toward spatio-temporally physically coherent motion. To further improve texture consistency during object movement, we propose a Test-Time Texture Consistency Optimization (TTCO) technique that adapts text and feature embeddings based on pixel correspondences from the simulator. Comprehensive experiments demonstrate that PSIVG produces videos that better adhere to real-world physics while preserving visual quality and diversity. Project Page: https://vcai.mpi-inf.mpg.de/projects/PSIVG/

Figures & tables

Figure 1: A baseline video generator (left) produces a physically implausible bowling collision with chaotic motion vectors across the video frames. Our PSIVG framework (right) integrates a physical simulator into the generation loop, guiding the video generator to produce a physically plausible and temporally coherent video.

Figure 2: Overview of our Physical Simulator In-the-loop Video Generation (PSIVG) framework. From an input prompt, a template video is first generated, and is processed by our perception pipeline (Sec. 3.2.1 ). The outputs of the perception pipeline are further processed before being passed into the physical simulator (Sec. 3.2.2 ). The rendered outputs from the simulator are then used for video generation (Sec. 3.3 ), and this video generation can be improved with TTCO (see Sec. 3.3.1 ) for better texture consistency.

Figure 3: Visualization of the sub-steps in our perception.

Figure 4: Overview of TTCO. To improve the consistencies of textures, during test time, we add learnable zero-initialized embeddings to the text prompt and features, and optimize them with the outputs from the physical simulator. This allows the generated video to adhere to the simulator trajectories and rotations better, thereby improving the texture consistency.

Type	Method	Motion Controllability		General Video Generation Quality
		SAM mIoU $\uparrow$	Corr. Pixel MSE $\downarrow$	CLIP Text $\uparrow$	CLIP Img $\uparrow$	Subject Consistency $\uparrow$	Background Consistency $\uparrow$	Motion Smoothness $\uparrow$	Temporal Flickering $\uparrow$
Text - based	[ 52 ] CogVideoX	0.47	0.032	0.34	0.99	0.93	0.95	0.98	0.97
	[ 24 ] HunyuanVideo	0.46	0.017	0.35	0.99	0.95	0.96	0.99	0.98
	[ 25 ] PISA-Base	0.50	0.012	0.35	0.99	0.95	0.96	0.99	0.99
	[ 25 ] PISA-Seg	0.50	0.012	0.35	0.99	0.95	0.96	0.99	0.99
	[ 25 ] PISA-Depth	0.51	0.017	0.35	0.99	0.85	0.92	0.98	0.98

Table 1: Quantitative comparison with existing methods for text-to-video generation.

Figure 5: Qualitative comparisons, showing a teddy bear being dropped (left) and objects colliding (right). See Supp. Mat. for more results.

Figure 6: Impact of Prompt-based Optimization Design.

Physical Consistency in Video Generation

Physical consistency in video generation studies how to make and evaluate videos whose geometry, motion, interactions, and state changes obey real-world laws.

See all 182 papers

PhyCo: Learning Controllable Physical Priors for Generative Motion
Apr 30, 2026Sriram Narayanan, Ziyu Jiang, Srinivasa Narasimhan +1Physics-Informed Generative Modeling Physical Consistency in Video Generation
1Carnegie Mellon University · 2NEC Labs America · 3UC San Diego
PhysAlign: Physics-Coherent Image-to-Video Generation through Feature and 3D Representation Alignment
Mar 14, 2026Zhexiao Xiong, Yizhi Song, Liu He +4Physical Consistency in Video Generation Image-to-Video Generation
Washington University in St. Louis · Purdue University · NVIDIA
PhysChoreo: Physics-Controllable Video Generation with Part-Aware Semantic Grounding
Nov 25, 2025Haoze Zhang, Tianyu Huang, Zichen Wan +4Physical Consistency in Video Generation Physical Reasoning
1Harbin Institute of Technology, China

Apr 30, 2026J/K move · Enter open · S save

PhyCo: Learning Controllable Physical Priors for Generative Motion

Sriram Narayanan, Ziyu Jiang, Srinivasa Narasimhan, Manmohan Chandraker

1Carnegie Mellon University · 2NEC Labs America · 3UC San Diego

Physics-Informed Generative Modeling Physical Consistency in Video Generation Video Diffusion Models Video Generation Diffusion Models for Motion Generation Controllable Video Generation

Open paper AskPDF arXiv

Modern video diffusion models excel at appearance synthesis but still struggle with physical consistency: objects drift, collisions lack realistic rebound, and material responses seldom match their underlying properties. We present PhyCo, a framework that introduces continuous, interpretable, and physically grounded control into video generation. Our approach integrates three key components: (i) a large-scale dataset of over 100K photorealistic simulation videos where friction, restitution, deformation, and force are systematically varied across diverse scenarios; (ii) physics-supervised fine-tuning of a pretrained diffusion model using a ControlNet conditioned on pixel-aligned physical property maps; and (iii) VLM-guided reward optimization, where a fine-tuned vision-language model evaluates generated videos with targeted physics queries and provides differentiable feedback. This combination enables a generative model to produce physically consistent and controllable outputs through variations in physical attributes-without any simulator or geometry reconstruction at inference. On the Physics-IQ benchmark, PhyCo significantly improves physical realism over strong baselines, and human studies confirm clearer and more faithful control over physical attributes. Our results demonstrate a scalable path toward physically consistent, controllable generative video models that generalize beyond synthetic training environments.

Abstract

Physical Consistency in Video Generation

Explore similar work

PhyCo: Learning Controllable Physical Priors for Generative Motion

PhysAlign: Physics-Coherent Image-to-Video Generation through Feature and 3D Representation Alignment

PhysChoreo: Physics-Controllable Video Generation with Part-Aware Semantic Grounding