Compositional video generation aims to synthesize multiple instances with diverse appearance and motion. However, current approaches mainly focus on binding semantics, neglecting to understand diverse motion categories specified in prompts. In this paper, we propose a motion factorization framework that decomposes complex motion into three primary categories: motionlessness, rigid motion, and non-rigid motion. Specifically, our framework follows a planning before generation paradigm. (1) During planning, we reason about motion laws on the motion graph to obtain frame-wise changes in the shape and position of each instance. This alleviates semantic ambiguities in the user prompt by organizing it into a structured representation of instances and their interactions. (2) During generation, we modulate the synthesis of distinct motion categories in a disentangled manner. Conditioned on the motion cues, guidance branches stabilize appearance in motionless regions, preserve rigid-body geometry, and regularize local non-rigid deformations. Crucially, our two modules are model-agnostic, which can be seamlessly incorporated into various diffusion model architectures. Extensive experiments demonstrate that our framework achieves impressive performance in motion synthesis on real-world benchmarks. Code is available at https://github.com/ZixuanWang0525/MF-CVG.
Figures & tables
Figure 2: Overview of our Structured Motion Reasoning (SMR) module ( Sec. 3.2 ). (a) Given a user prompt, we organize it into a motion graph describing instances and their interactions. (b) For each instance, conditioned on its motion category, we infer a bounding box sequence from graph-derived motion cues. All bounding box sequences are then composed into a coherent spatial-temporal layout.
Figure 3: Overview of Disentangled Motion Guidance (DMG) module ( Sec. 3.3 ). (a) For motionless instances, we enforce each frame interacts only with a designated anchor frame. (b) For rigidly moving instances, we restrict cross-frame interactions of a foreground within the shape aligned regions. (c) For instances undergoing non-rigid movements, we minimize pixel-wise discrepancies between perceptual deformations and box-induced deformations.
Models
CVGBench-m
CVGBench-p
Subject
Background
Temporal
Motion
Dynamic
Subject
Background
Temporal
Motion
Dynamic
Consistency
Consistency
Flickering
Smoothness
Degree
Consistency
Consistency
Flickering
Smoothness
Degree
LVDM [ 16 ]
88.74%
91.27%
89.37%
91.76%
84.55%
91.05%
92.60%
91.26%
93.79%
68.78%
modelScope [ 46 ]
93.17%
93.93%
94.69%
96.23%
51.82%
95.71%
95.28%
96.01%
97.27%
30.81%
ZeroScope [ 56 ]
96.41%
95.69%
97.40%
98.66%
30.77%
97.40%
96.24%
97.73%
98.84%
16.10%
LATTE [ 36 ]
90.91%
94.33%
92.79%
94.78%
77.44%
95.21%
96.11%
95.42%
97.02%
51.66%
Table 1: Performance comparison of cross-modal compositional video generation approaches on our CVGBench-m and CVGBench-p datasets. Best/2nd best scores are bolded / underlined . † indicates compositional generation models.
Figure 4: Visual cases: comparisons under diverse motion categories. In 3D Unet architecture, we compare our framework with baseline VideoCrafter-v2.0 [ 7 ] and compositional approach A&R [ 39 ] . While in DiT architecture, we compare our framework with baseline CogVideoX-2B [ 54 ] and compositional approach R&P [ 5 ] . Our framework yields improved cross-frame consistency and motion fidelity.
Figure 5: Visual cases: generation of complex motion generation.
Backbones
Subject
Background
Temporal
Motion
Dynamic
Consistency
Consistency
Flickering
Smoothness
Degree
Baseline: VideoCrafter-v2.0 [ 7 ]
LLaMA-8B [ 10 ]
97.55%
97.35%
96.55%
98.22%
75.34%
LLaMA-70B [ 33 ]
98.40%
98.11%
97.39%
98.63%
82.21%
Baseline: CogVideoX-2B [ 54 ]
LLaMA-8B [ 10 ]
96.70%
96.34%
97.32%
98.17%
94.77%
Table 2: Ablation analysis of diverse backbones for motion reasoning. Best scores are bolded .
RCG
GIG
SDG
Subject
Background
Temporal
Motion
Dynamic
Consistency
Consistency
Flickering
Smoothness
Degree
Baseline: VideoCrafter-v2.0 [ 7 ]
✗
✗
✗
97.48%
97.05%
96.43%
98.27%
38.40%
✓
✗
✗
98.11%
97.85%
97.21%
98.55%
51.60%
✗
✓
✗
98.07%
97.80%
97.15%
98.52%
53.60%
✗
✗
✓
97.71%
97.40%
96.72%
98.36%
74.85%
Table 3: Ablation analysis of diverse motion guidance components, including Reference Conditioned Guidance (RCG), Geometric Invariance Guidance (GIG), Spatial Deformation Guidance (SDG). Best scores are bolded .
Figure 6: Failure case: Overly small bounding boxes are insufficient for guiding object (motionlessness) generation.