Text-to-video (T2V) diffusion models have made remarkable progress in generating high-fidelity visuals, yet their outputs still often exhibit physically implausible phenomena, including discontinuous dynamics and implausible event interactions. Existing representation alignment methods mainly transfer generic video features or short-range motion cues, treating physical plausibility as a single-level representation problem. To address this issue, we propose MiMaAlign, a hierarchical micro-macro alignment framework that transfers complementary dynamics-oriented and event-oriented knowledge into different abstraction depths of a pretrained video diffusion model. Specifically, micro-level continuous dynamics alignment imposes constraints by aligning diffusion features with a continuous-time motion representation learned by a Neural Ordinary Differential Equation (Neural ODE), while macro-level event-commonsense alignment provides representations of object interactions and event semantics through an instruction-prompted vision-language model (VLM). With training-only auxiliary objectives, we align the complementary supervision from the two teachers with shallower and deeper generator features, respectively, enabling the generator to internalize the corresponding physical priors. Empirical evaluations show that, on CogVideoX-2B, MiMaAlign achieves a relative improvement of up to 21.0% in the Physical Commonsense (PC) score on the VideoPhy benchmark over the baseline, and also delivers substantial gains on the highly challenging VideoPhy2 benchmark, while maintaining general video quality on VBench and being supported by blind user comparisons.
Figures & tables
Figure 1: A comparison of alignment frameworks. (a) VideoREPA uses a single teacher with entangled features. (b) MoAlign disentangles short-term motion for alignment. (c) Our MiMaAlign employs hierarchical micro-macro alignment to synergistically enhance physical plausibility.
Methods
link
roll
contain
dominoes
VideoMAEv2 (Optical Flow)
59.9%
51.6%
60.8%
69.2%
VideoMAEv2 (Neural ODE)
65.0%
59.2%
64.1%
71.7%
Table 1: Physics understanding evaluation on Physion ( Bear et al. 2021 ) . Our proposed Neural-ODE-based continuous dynamics modeling significantly outperforms optical-flow-based short-term motion modeling on the Object Contact Prediction (OCP) task.
Figure 2: Overview of the proposed MiMaAlign framework. Stage 1 : A Neural-ODE-based micro-dynamics teacher is trained to model continuous-time motion dynamics. Stage 2 : The complementary supervision from the micro-dynamics teacher and the VLM-based macro-event teacher is aligned with shallower and deeper layers, respectively, to improve physical plausibility.
Methods
VideoPhy
VideoPhy2
Solid–Solid
Solid–Fluid
Fluid–Fluid
Overall
SA
PC
SA
PC
SA
PC
SA
PC
SA
PC
HunyuanVideo
55.2
16.1
70.5
32.2
63.6
54.5
63.1
29.1
26.8
66.8
LongCat-Video
50.4
16.1
73.3
30.1
69.1
40.0
63.1
25.9
26.6
67.0
Cosmos-Predict2.5-2B
32.2
4.2
54.1
15.8
41.8
23.6
43.0
12.2
21.4
56.1
Wan2.1-14B
49.7
15.4
71.2
27.4
72.7
43.6
62.5
25.0
27.5
59.2
Table 2: Quantitative comparison on VideoPhy and VideoPhy2 benchmarks. SA denotes Semantic Adherence and PC denotes Physical Commonsense. Best results are in bold , and second best are underlined .
Figure 3: Qualitative comparison of CogVideoX, VideoREPA, MoAlign, and MiMaAlign (Ours).
Methods
Aesthetic
Background
Imaging
Motion
Dynamic
Subject
Temporal
Quality
Quality
Consistency
Quality
Smoothness
Degree
Consistency
Flickering
Score
CogVideoX-2B
60.7
95.0
60.5
97.9
63.9
92.0
97.6
80.6
+VideoREPA
59.7
95.4
59.3
98.5
63.9
93.0
98.2
81.1
+MoAlign
60.7
95.5
59.6
98.4
59.7
93.6
98.2
81.1
+MiMaAlign (ours)
61.0
95.6
61.1
98.3
63.9
93.6
98.0
81.6
Table 3: Results on VBench quality score. For each method, the best performance relative to its base model is highlighted in bold .
Comparison
MiMaAlign Wins
Baseline Wins
Tie
CogVideoX-2B
SA
57.1%
18.9%
24.0%
PC
66.9%
20.6%
12.6%
VideoREPA
SA
54.3%
20.6%
25.1%
PC
59.4%
26.9%
13.7%
MoAlign
SA
35.4%
22.3%
42.3%
PC
42.3%
30.3%
27.4%
Table 4: Blind user study with pairwise Good–Same–Bad (GSB) comparisons. We report the fraction of pairs where MiMaAlign is preferred (Wins), the baseline is preferred, or the two are judged similar (Tie), under Semantic Adherence (SA) and Physical Commonsense (PC) criteria.
Figure 8
Mi@
Ma@
Ma Source
VLM Embedding Layer
SA
PC
8
18
Vision Encoder
–
64.2
26.7
8
18
LM Decoder
0
63.4
27.3
8
18
LM Decoder
1
63.1
27.6
8
18
LM Decoder
15
63.4
28.2
8
18
LM Decoder
Final
64.8
31.7
Table 6: Ablation study comparing VLM visual encoders and language decoders on VideoPhy. The results demonstrate that the hidden states of the language decoder, particularly those from the final layer, provide superior guidance, significantly outperforming representations derived solely from visual features.
Figure 5: Ablation on the alignment weight λ in terms of the PC score on VideoPhy. Left: fixing λMacro=0.5 and varying λMicro ; right: fixing λMicro=0.5 and varying λMacro .
Figure 6: System prompt used to guide the macro-level physical event teacher (Qwen2.5-VL) during alignment.
Figure 7: Qualitative comparison of MiMaAlign against the CogVideoX-2B baseline, VideoREPA, and MoAlign, all built on the CogVideoX-2B backbone.
Figure 8: Qualitative comparison. The first row shows the outputs of CogVideoX-5B, and the second row presents the results of our MiMaAlign.
Figure 9: Qualitative comparison. The first row shows the outputs of Wan2.1-1.3B, and the second row presents the results of our MiMaAlign.
Figure 10: Qualitative generation results of MiMaAlign based on CogVideoX-2B.
Figure 11: Qualitative generation results of MiMaAlign based on CogVideoX-5B.
Figure 12: Qualitative generation results of MiMaAlign based on Wan2.1-1.3B.
Figure 13: Upsampled Detailed Prompts for Video Data Generation.
Figure 14: Upsampled Detailed Prompts for Video Data Generation.
Physical consistency in video generation studies how to make and evaluate videos whose geometry, motion, interactions, and state changes obey real-world laws.
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA · School of Artificial Intelligence, University of Chinese Academy of Sciences · StepFun +2