Text-to-video (T2V) diffusion models have made remarkable progress in generating high-fidelity visuals, yet their outputs still often exhibit physically implausible phenomena, including discontinuous dynamics and implausible event interactions. Existing representation alignment methods mainly transfer generic video features or short-range motion cues, treating physical plausibility as a single-level representation problem. To address this issue, we propose MiMaAlign, a hierarchical micro-macro alignment framework that transfers complementary dynamics-oriented and event-oriented knowledge into different abstraction depths of a pretrained video diffusion model. Specifically, micro-level continuous dynamics alignment imposes constraints by aligning diffusion features with a continuous-time motion representation learned by a Neural Ordinary Differential Equation (Neural ODE), while macro-level event-commonsense alignment provides representations of object interactions and event semantics through an instruction-prompted vision-language model (VLM). With training-only auxiliary objectives, we align the complementary supervision from the two teachers with shallower and deeper generator features, respectively, enabling the generator to internalize the corresponding physical priors. Empirical evaluations show that, on CogVideoX-2B, MiMaAlign achieves a relative improvement of up to 21.0% in the Physical Commonsense (PC) score on the VideoPhy benchmark over the baseline, and also delivers substantial gains on the highly challenging VideoPhy2 benchmark, while maintaining general video quality on VBench and being supported by blind user comparisons.
Figures & tables
Figure 1: A comparison of alignment frameworks. (a) VideoREPA uses a single teacher with entangled features. (b) MoAlign disentangles short-term motion for alignment. (c) Our MiMaAlign employs hierarchical micro-macro alignment to synergistically enhance physical plausibility.
Methods
link
roll
contain
dominoes
VideoMAEv2 (Optical Flow)
59.9%
51.6%
60.8%
69.2%
VideoMAEv2 (Neural ODE)
65.0%
59.2%
64.1%
71.7%
Table 1: Physics understanding evaluation on Physion ( Bear et al. 2021 ) . Our proposed Neural-ODE-based continuous dynamics modeling significantly outperforms optical-flow-based short-term motion modeling on the Object Contact Prediction (OCP) task.
Figure 2: Overview of the proposed MiMaAlign framework. Stage 1 : A Neural-ODE-based micro-dynamics teacher is trained to model continuous-time motion dynamics. Stage 2 : The complementary supervision from the micro-dynamics teacher and the VLM-based macro-event teacher is aligned with shallower and deeper layers, respectively, to improve physical plausibility.
Methods
VideoPhy
VideoPhy2
Solid–Solid
Solid–Fluid
Fluid–Fluid
Overall
SA
PC
SA
PC
SA
PC
SA
PC
SA
PC
HunyuanVideo
55.2
16.1
70.5
32.2
63.6
54.5
63.1
29.1
26.8
66.8
LongCat-Video
50.4
16.1
73.3
30.1
69.1
40.0
63.1
25.9
26.6
67.0
Cosmos-Predict2.5-2B
32.2
4.2
54.1
15.8
41.8
23.6
43.0
12.2
21.4
56.1
Wan2.1-14B
49.7
15.4
71.2
27.4
72.7
43.6
62.5
25.0
27.5
59.2
Table 2: Quantitative comparison on VideoPhy and VideoPhy2 benchmarks. SA denotes Semantic Adherence and PC denotes Physical Commonsense. Best results are in bold , and second best are underlined .
Figure 3: Qualitative comparison of CogVideoX, VideoREPA, MoAlign, and MiMaAlign (Ours).
Methods
Aesthetic
Background
Imaging
Motion
Dynamic
Subject
Temporal
Quality
Quality
Consistency
Quality
Smoothness
Degree
Consistency
Flickering
Score
CogVideoX-2B
60.7
95.0
60.5
97.9
63.9
92.0
97.6
80.6
+VideoREPA
59.7
95.4
59.3
98.5
63.9
93.0
98.2
81.1
+MoAlign
60.7
95.5
59.6
98.4
59.7
93.6
98.2
81.1
+MiMaAlign (ours)
61.0
95.6
61.1
98.3
63.9
93.6
98.0
81.6
Table 3: Results on VBench quality score. For each method, the best performance relative to its base model is highlighted in bold .
Comparison
MiMaAlign Wins
Baseline Wins
Tie
CogVideoX-2B
SA
57.1%
18.9%
24.0%
PC
66.9%
20.6%
12.6%
VideoREPA
SA
54.3%
20.6%
25.1%
PC
59.4%
26.9%
13.7%
MoAlign
SA
35.4%
22.3%
42.3%
PC
42.3%
30.3%
27.4%
Table 4: Blind user study with pairwise Good–Same–Bad (GSB) comparisons. We report the fraction of pairs where MiMaAlign is preferred (Wins), the baseline is preferred, or the two are judged similar (Tie), under Semantic Adherence (SA) and Physical Commonsense (PC) criteria.
Figure 8
Mi@
Ma@
Ma Source
VLM Embedding Layer
SA
PC
8
18
Vision Encoder
–
64.2
26.7
8
18
LM Decoder
0
63.4
27.3
8
18
LM Decoder
1
63.1
27.6
8
18
LM Decoder
15
63.4
28.2
8
18
LM Decoder
Final
64.8
31.7
Table 6: Ablation study comparing VLM visual encoders and language decoders on VideoPhy. The results demonstrate that the hidden states of the language decoder, particularly those from the final layer, provide superior guidance, significantly outperforming representations derived solely from visual features.
Figure 5: Ablation on the alignment weight λ in terms of the PC score on VideoPhy. Left: fixing λMacro=0.5 and varying λMicro ; right: fixing λMicro=0.5 and varying λMacro .
Figure 6: System prompt used to guide the macro-level physical event teacher (Qwen2.5-VL) during alignment.
Figure 7: Qualitative comparison of MiMaAlign against the CogVideoX-2B baseline, VideoREPA, and MoAlign, all built on the CogVideoX-2B backbone.
Figure 8: Qualitative comparison. The first row shows the outputs of CogVideoX-5B, and the second row presents the results of our MiMaAlign.
Figure 9: Qualitative comparison. The first row shows the outputs of Wan2.1-1.3B, and the second row presents the results of our MiMaAlign.
Figure 10: Qualitative generation results of MiMaAlign based on CogVideoX-2B.
Figure 11: Qualitative generation results of MiMaAlign based on CogVideoX-5B.
Figure 12: Qualitative generation results of MiMaAlign based on Wan2.1-1.3B.
Figure 13: Upsampled Detailed Prompts for Video Data Generation.
Figure 14: Upsampled Detailed Prompts for Video Data Generation.
Physical consistency in video generation studies how to make and evaluate videos whose geometry, motion, interactions, and state changes obey real-world laws.
Video Diffusion Models (VDMs) offer a promising approach for simulating dynamic scenes and environments, with broad applications in robotics and media generation. However, existing models often generate temporally incoherent content that violates basic physical intuition, significantly limiting their practical applicability. We propose PhysAlign, an efficient framework for physics-coherent image-to-video (I2V) generation that explicitly addresses this limitation. To overcome the critical scarcity of physics-annotated videos, we first construct a fully controllable synthetic data generation pipeline based on rigid-body simulation, yielding a highly-curated dataset with accurate, fine-grained physics and 3D annotations. Leveraging this data, PhysAlign constructs a unified physical latent space by coupling explicit 3D geometry constraints with a Gram-based spatio-temporal relational alignment that extracts kinematic priors from video foundation models. Extensive experiments demonstrate that PhysAlign significantly outperforms existing VDMs on tasks requiring complex physical reasoning and temporal stability, without compromising zero-shot visual quality. PhysAlign shows the potential to bridge the gap between raw visual synthesis and rigid-body kinematics, establishing a practical paradigm for genuinely physics-grounded video generation. The project page is available at https://physalign.github.io/PhysAlign.
Zhexiao Xiong, Yizhi Song, Liu He +4
Washington University in St. Louis · Purdue University · NVIDIA
Despite remarkable advances in video generative models, they still struggle to generate physically realistic videos, frequently exhibiting appearance drift, implausible motion, and temporal inconsistencies. In this work, we address this limitation by transferring relational knowledge encoded in spatio-temporal self-similarity (STSS) from visual foundation models into video generative models. STSS represents pairwise similarities among features across space and time, revealing the relational structure of how objects interact with other entities throughout a video, effectively capturing real-world dynamics, including object motion and semantic transformations. To transfer this relational knowledge, we propose Tempered Self-similarity Alignment (TSA) loss, which transforms STSS into probabilistic correspondence distributions and trains the video generative model to align its correspondence distributions with those of the visual foundation model on dynamically changing regions. Evaluated on VideoPhy and VideoPhy2 benchmarks, our method demonstrates substantial improvements in physical plausibility across diverse interaction scenarios, validating the effectiveness of transferring relational knowledge for physically realistic video generation.
Manjin Kim, Suha Kwak, Minsu Cho
Pohang University of Science and Technology (POSTECH)
Despite advancements in generating visually stunning content, video diffusion models (VDMs) often yield physically inconsistent results due to pixel-only reconstruction. To address this, we propose MMPhysVideo, the first study to enhance physical plausibility in video generation through joint multimodal modeling. We recast perceptual cues, specifically semantics, geometry, and spatio-temporal trajectories, into a unified pseudo-RGB format, enabling VDMs to directly capture complex physical dynamics. To mitigate cross-modal interference, we propose a Bidirectionally Controlled Teacher architecture, which utilizes parallel branches to fully decouple RGB and perception processing and adopts two zero-initialized control links to gradually establish pixel-wise consistency. For inference efficiency, the teacher's physical prior is distilled into a single-stream student model via representation alignment. Furthermore, we present MMPhysPipe, an end-to-end data curation and annotation pipeline tailored for constructing physics-rich multimodal datasets. MMPhysPipe employs a vision-language model (VLM) guided by a chain-of-visual-evidence rule to pinpoint physical subjects, enabling expert models to extract multi-granular perceptual information. Without additional inference costs, MMPhysVideo consistently improves physical plausibility of advanced models on the Videophy and PhyGenbench benchmarks and achieves superior performance among existing methods.
Shubo Lin, Xuanyang Zhang, Wei Cheng +3
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, University of Chinese Academy of Sciences · StepFun +2