cs.CVOct 30, 2026
SaveSeeing the Unseen: Unified Visible--Invisible Motion for Physically Consistent Video Generation
Organizations: National University of Singapore · Nanyang Technological University · University of Oxford
Abstract
Recent advances in video generation have achieved impressive visual quality, yet they often fail to produce physically consistent dynamics due to the lack of explicit modeling of underlying physical processes. Existing approaches either rely on simulators with strong assumptions or use coarse semantic guidance, and further struggle to effectively inject physical priors into generation models. In this work, we propose Seeing the Unseen, a unified framework for physics-grounded video generation. Our approach decomposes the problem into two stages: learning physical dynamics and injecting them into video synthesis. First, we introduce a lightweight Particle-based Graph Dynamics Simulator (PGDS) that learns generalizable physical interactions from data and predicts plausible 3D motion trajectories. Second, we propose a Visible--Invisible Motion Field (VIMF) that captures both motion observable in the input frame and motion that emerges over time due to object dynamics. This representation enables more complete and structured motion guidance compared to conventional physical signals. By integrating these trajectories into a diffusion-based generator, our method produces videos that are both visually coherent and physically consistent. Extensive experiments demonstrate improved motion accuracy, interaction consistency, and reduced physical artifacts compared to strong baselines.
Figures & tables
Figure 1: Seeing the Unseen. Our framework predicts physically consistent motion from a single image by modeling both reference-visible and future-emergent trajectories. By reasoning about occluded dynamics and structured interactions, it enables accurate simulation of complex events such as collisions and domino chains.
Figure 2: Overview of our proposed framework. Given an input image, we first lift it into a 3D scene particle representation. A particle-based graph dynamics simulator then predicts physically plausible 3D motion trajectories, which are rendered into reference-visible and future-emergent motions to guide physically consistent video generation.
Figure 3: Architecture of the physics-guided video generation model. The visible and future visible motion fields are encoded by a frozen 3D VAE and concatenated with the noisy video latent along the channel dimension. A zero-initialized injection strategy preserves the pretrained model behavior while progressively incorporating motion guidance.
| Method | Domino | Collision | Free Fall | Coll.+Dom. | F.F.+Dom. | F.F.+Coll. | Translation | |||||||
| SA | PC | SA | PC | SA | PC | SA | PC | SA | PC | SA | PC | SA | PC | |
| Open-source Video Generation Models | ||||||||||||||
| Wan2.2-I2V-14B | 2.483 | 3.017 | 2.750 | 3.083 | 2.317 | 3.033 | 2.217 | 3.050 | 2.533 | 2.950 | 2.517 | 3.000 | 3.000 | 4.000 |
| Wan2.2-TI2V-5B | 2.650 | 3.033 | 2.917 | 3.067 | 2.183 | 3.083 | 2.200 | 3.033 | 2.467 | 3.033 | 2.183 | 3.083 | 3.000 | 4.100 |
| HunyuanVideo-1.5 | 2.550 | 3.017 | 2.867 | 3.067 | 2.317 | 3.017 | 2.233 | 3.000 | 2.617 | 2.950 | 2.467 | 3.000 | 3.050 | 3.933 |
| CogVideoX1.5-5B | 2.450 | 3.200 | 2.833 | 3.017 | 2.333 | 3.000 | 2.300 | 2.983 | 2.650 | 3.000 | 2.333 | 3.000 | 3.050 | 3.933 |
Table 1: Quantitative results on seven physical video generation scenarios. Best results are bold . “–” indicates unsupported cases.
Figure 4: Qualitative comparison with existing methods. Our method produces physically plausible dynamics with accurate object interactions (e.g., correct collision responses and domino chain propagation), while baseline models frequently exhibit artifacts such as object interpenetration, implausible motion trajectories, or failure to trigger expected physical events.
Table 4: PGDS motion prediction on seven Physion++ test scenarios. Vel. RMSE is the root mean squared particle velocity error over complete rollouts. Acc-V reports the percentage of particles meeting strict ( m/s or ) or relaxed ( m/s or ) velocity-error thresholds. Kinetic-energy bias is .
| Method | Type | Manual Phys. | Precise 3D | Total Runtime | Unseen |
| Params | Mesh | (incl. manual tuning) | Shapes | ||
| PhysGen [ 31 ] | Simulator | 6–10 | Required | 10 min | Limited |
| PhysMotion [ 46 ] | Simulator | 6–8 | Required | 10 min | Limited |
| PhysGen3D [ 8 ] | Simulator | 5–7 | Required | 10 min | Limited |
| PhysCtrl [ 50 ] | Data-driven | 3–5 | Partial | 8 min | Moderate |
| NewtonGen [ 63 ] | Data-driven | 2–3 | Not required | 6 min | Limited |
Table 5: Configuration efficiency of trajectory generation methods. We compare setup cost and generalization with time and parameter estimates from public implementations.
Figure 5: Pairwise human evaluation comparing our method against baseline models. For each comparison, human annotators judge which method produces better results across three criteria: (a) instruction-following, measuring how faithfully the generated video adheres to the given text prompt; (b) video quality, assessing visual fidelity, temporal coherence, and artifact severity; and (c) overall preference. Results are reported as win/tie/lose rates (%).
| Method | Domino | Collision | Free Fall | Translation | ||||
| SA | PC | SA | PC | SA | PC | SA | PC | |
| Go with Flow [ 5 ] | 2.583 | 3.389 | 3.275 | 3.691 | 3.670 | 3.928 | 3.568 | 4.419 |
| Wan-Move [ 12 ] | 2.672 | 3.351 | 3.594 | 3.771 | 3.892 | 4.017 | 4.167 | 4.602 |
| Diffusion as Shader [ 21 ] | 2.640 | 3.295 | 3.697 | 3.829 | 3.764 | 3.923 | 3.821 | 4.493 |
| Ours (visible-only) | 2.693 | 3.348 | 3.693 | 3.717 | 3.972 | 4.004 | 3.837 | 4.521 |
| Ours (full) | 3.976 | 4.013 | 4.160 | 4.394 | 4.347 | 4.328 | 4.013 | 4.578 |
Table 6: Comparison of physical prior injection strategies and the visible-only ablation across four scenarios. Our visible-only and full variants use the same diffusion backbone; the full variant uses both reference-visible and future-emergent tracking videos. Best results are bold .
Figure 6: Comparison with different physical prior injection methods.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Architecture of the Particle-based Graph Dynamics Simulator (PGDS). The simulator follows an Encode–Process–Decode pipeline. Given particle states, a graph is first constructed based on spatial proximity. Node and edge features are encoded to capture material properties and geometric relations. A message-passing processor then iteratively propagates interaction effects across the graph. Finally, a decoder predicts particle velocities, which are used to update positions and roll out future dynamics.
Figure 8: Unified motion encoding and injection pipeline. The visible tracking video and future-visible tracking video are encoded using a frozen 3D VAE to obtain two latent sequences. These are concatenated with the noisy video latent to form a multi-channel input to the diffusion transformer. To preserve pretrained behavior, the patch embedding weights corresponding to tracking channels are zero-initialized, while the original weights are copied from the pretrained model. The first frame is injected via noise replacement to provide appearance conditioning.
| Stage | Component | 1 Object | 3 Objects | 5 Objects | Peak VRAM |
| S1 | 3D lifting (segmentation + monocular depth + single-image 3D reconstruction) | 10.1 s | 15.8 s | 24.7 s | 28.7–40.8 GiB |
| S2 | Mesh-to-particle conversion | 4.0 s | 4.9 s | 5.1 s | GiB |
| S2 | PGDS simulation | 1.3 s | 1.3 s | 1.3 s | 526 MiB |
| S3 | VIMF dual-stream tracking rendering | 15.9 s | 16.0 s | 15.9 s | GiB |
| S4 | Video diffusion generation (Ours) | 150.6 s | 151.0 s | 150.6 s | 39.67 GiB |
| End-to-end total | 181.9 s | 189.0 s | 197.6 s | 40.8 GiB | |
Table 7: End-to-end runtime and peak GPU memory usage on a single NVIDIA H200 GPU. The total covers stages S1–S4.
| Metric | Base TI2V-5B | Ours | Additional Overhead |
| Peak VRAM | 39.59 GiB | 39.67 GiB | MiB |
| 35 denoising steps | 133.54 s | 133.74 s | s |
| Dual-stream tracking encoding | – | 5.95 s | s |
| Total generation time | 144.20 s | 150.28 s | s ( ) |
Table 8: Additional cost of dual-stream conditioning in the video-generation stage on a single NVIDIA H200 GPU. The backbone and our model use 48 and 144 input latent channels, respectively.
Figure 9: Ablation on future-emergent motion signals. Top: our full model with both reference-visible and future-emergent tracking signals. Bottom: model using only reference-visible tracking.
Figure 10: Qualitative comparison of domino motion.
Figure 11: Qualitative comparison on free fall & collision motion.
Figure 12: Qualitative comparison of free-fall motion.
Figure 13: Qualitative comparison of collision motion.
Figure 14: Qualitative comparison of translation motion.
| Scenario | Ours (clean) | Ours (Tracks) | Ours (Objects) | Ours (Both) | Wan2.2 TI2V-5B |
| Domino | 3.976 / 4.013 | 3.777 / 3.852 | 3.578 / 3.732 | 3.459 / 3.612 | 2.650 / 3.033 |
| Collision | 4.160 / 4.394 | 3.952 / 4.218 | 3.744 / 4.086 | 3.619 / 3.955 | 2.917 / 3.067 |
| Free Fall | 4.347 / 4.328 | 4.130 / 4.155 | 4.086 / 4.155 | 3.912 / 4.025 | 2.183 / 3.083 |
| Coll.+Dom. | 3.854 / 4.016 | 3.661 / 3.855 | 3.469 / 3.735 | 3.353 / 3.614 | 2.200 / 3.033 |
| F.F.+Dom. | 3.985 / 4.017 | 3.786 / 3.856 | 3.587 / 3.736 | 3.467 / 3.615 | 2.467 / 3.033 |
| F.F.+Coll. | 4.327 / 4.589 | 4.111 / 4.405 | 3.894 / 4.268 | 3.764 / 4.130 | 2.183 / 3.083 |
Table 9: Robustness to incomplete motion guidance and upstream perception errors. Each entry reports SA / PC ( ). The Tracks variant drops approximately of object tracks, the Objects variant removes – segmented objects, and Both applies both perturbations. Coll., Dom., and F.F. denote Collision, Domino, and Free Fall. reports the relative change in average SA / PC (%) with respect to the clean full pipeline.
More in this topic
Physical Consistency in Video Generation
Physical consistency in video generation studies how to make and evaluate videos whose geometry, motion, interactions, and state changes obey real-world laws.
See all 196 papers