The ultimate goal of video generation is to satisfy a fundamental trilemma: achieving high visual quality, maintaining rigorous physical consistency, and enabling precise controllability. While recent models can maintain this balance in simple, isolated scenarios, we observe that this equilibrium is fragile and often breaks down as scene complexity increases (e.g., involving collisions or dense traffic). To address this, we introduce Motion Forcing, a framework designed to stabilize this trilemma even in complex generative tasks. Our key insight is to explicitly decouple physical reasoning from visual synthesis via a hierarchical ``Point-Shape-Appearance'' paradigm. This approach decomposes generation into verifiable stages: modeling complex dynamics as sparse geometric anchors (Point), expanding them into dynamic depth maps that explicitly resolve 3D geometry (Shape), and finally rendering high-fidelity textures (Appearance). Furthermore, to foster robust physical understanding, we employ a Masked Point Recovery strategy. By randomly masking input anchors during training and enforcing the reconstruction of complete dynamic depth, the model is compelled to move beyond passive pattern matching and learn latent physical laws (e.g., inertia) to infer missing trajectories. Extensive experiments on autonomous driving benchmarks show that Motion Forcing significantly outperforms state-of-the-art baselines, maintaining trilemma stability across complex scenes. Evaluations on physics and robotics further confirm our framework's generality.
Figures & tables
Figure 1: Motion Forcing takes user input motion and the first frame to generate motion-coherent future frames. (a-d) show reactive ego-trajectories responding to dangerous scenarios initiated by other vehicles. (e) shows a physics scene where different actions lead to different collision results. (f) shows embodied AI, where directional inputs control a robotic hand to move an object in corresponding directions.
Figure 2: Overview of the Motion Forcing framework. (a) Preparation of Motion Representations: Input control signals are processed and subjected to spatial and temporal masking before being fed into the model. (b) Training: The model is trained by randomly sampling between two independent stages: "Point → Shape" (depth generation) and "Shape → Appearance" (RGB rendering). (c) Inference: The generation follows a complete two-stage hierarchical process, sequentially mapping sparse points to structural depth (Shape), and finally to the target RGB video (Appearance).
Figure 3: Comparison with state-of-the-art models in synthesizing cut-in and evasive driving scenarios. While the leading text-conditioned models (Seed Dance2.0, Wan2.6, Vista) rely on text prompts, trajectory-controllable baselines like MOFA-Video require object trajectories coupled with scene trajectories to simulate camera movement. Our method Motion Forcing , however, effectively captures these complex dynamics given only the initial ego and object motion trajectories.
Figure 4: Validation of physical capabilities. ✓ indicate physically coherent. MOFA-Video is fine-tuned on our exact dataset for fairness. As the control commands become more complex, only our method maintains stable and physically coherent generation.
Figure 5: Controllable robotic hand manipulation. By providing different motion control inputs, our model can control the movement of the robotic arm and the grasped object in specified directions.
Figure 6: Qualitative comparison of ego motion control. Compared to the baseline AdaLN approach, our Depth Warping-based method demonstrates significantly superior accuracy and flexibility in controlling both the direction and speed.
Table 7
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Generation pipeline and point control robustness — Scene 1. Top row: shared camera-warped depth condition. Rows 2–10: three point control variants (A/B/C), each showing point input, generated depth, and generated RGB (5 uniformly sampled frames per row).
Figure 8: Generation pipeline and point control robustness — Scene 2. Same layout as Figure 7 . The model faithfully propagates structurally distinct trajectories through the depth stage before RGB rendering.
Figure 9: Generation pipeline and point control robustness — Scene 3. Same layout as Figure 7 . Across all three scenes the depth-first hierarchy cleanly separates physical reasoning from visual rendering, enabling robust control with diverse sparse inputs.
Recent advances in diffusion-based video generation have significantly improved visual quality and short-term temporal coherence. However, existing methods still struggle to produce videos with physically consistent and causally plausible dynamics, especially in scenarios involving long-horizon interactions. This limitation arises from the fact that video diffusion models primarily learn physical consistency implicitly, while vision-language models can directly model physical laws. Based on this idea, in this work, we propose CausalMotion, a training-free framework that injects explicit physical reasoning into video generation through structured intermediate representations. Our key idea is to decouple reasoning from generation by leveraging a vision-language model to decompose a text prompt into a sequence of causally consistent keyframes and object-centric motion trajectories. These representations are then aligned and integrated as soft constraints to guide a pretrained video diffusion model during inference. This design enables explicit modeling of object dynamics and causal transitions without requiring additional training or supervision. Extensive experiments show that our method consistently improves physical plausibility and temporal coherence, particularly in dynamics-intensive scenarios, while maintaining high perceptual video quality.
Sihan Zhuang, Xinyuan Chen, Tianfan Xue +1
Shanghai Artificial Intelligence Laboratory · ShanghaiTech University · Work done during an internship at Shanghai AI Laboratory. +1
Video generation models achieve high visual quality but often struggle to generate physics-aware videos. Unlike rigid-body motion, which can be described by explicit trajectories or formulas, complex deformation dynamics remain challenging to synthesize. We observe that a lack of physical reasoning for localizing dynamic areas allows irrelevant regions to dilute the model's attention, leading to generation failure. In this paper, we propose DeforM, a reasoning-guided image-to-video generation framework that directs the model's focus toward physics-critical regions. To reason about and localize these critical regions, we introduce a VLM-guided physical reasoning module, DeforM-Reason, to identify target objects and generate spatial-temporal masks. For physical guidance, we develop two alternative strategies: DeforM-Free for training-free mechanism analysis and DeforM-Injection as a powerful training-based generator. Experimental results demonstrate that DeforM improves the realism of generated deformation scenarios, outperforming baseline models in both visual quality and physical consistency.
Yunyi Li, Yu Qiao, Yaohui Wang +1
Shanghai Artificial Intelligence Laboratory, Shanghai, China
Recent advances in generative video modeling, driven by large-scale datasets and powerful architectures, have yielded remarkable visual realism. However, emerging evidence suggests that simply scaling data and model size does not endow these systems with an understanding of the underlying physical laws that govern real-world dynamics. Existing approaches often fail to capture or enforce such physical consistency, resulting in unrealistic motion and dynamics. In his work, we investigate whether integrating the inference of latent physical properties directly into the video generation process can equip models with the ability to produce physically plausible videos. To this end, we propose Phantom, a Physics-Infused Video Generation model that jointly models the visual content and latent physical dynamics. Conditioned on observed video frames and inferred physical states, Phantom jointly predicts latent physical dynamics and generates future video frames. Phantom leverages a physics-aware video representation that serves as an abstract yet informaive embedding of the underlying physics, facilitating the joint prediction of physical dynamics alongside video content without requiring an explicit specification of a complex set of physical dynamics and properties. By integrating the inference of physical-aware video representation directly into the video generation process, Phantom produces video sequences that are both visually realistic and physically consistent. Quantitative and qualitative results on both standard video generation and physics-aware benchmarks demonstrate that Phantom not only outperforms existing methods in terms of adherence to physical dynamics but also delivers competitive perceptual fidelity.