Real-world videos naturally portray complex interactions among distinct physical objects, effectively forming dynamic compositions of visual elements. However, most current video generation models synthesize scenes holistically and therefore lack mechanisms for explicit compositional manipulation. To address this limitation, we propose HECTOR, a generative pipeline that enables fine-grained compositional control. In contrast to prior methods,HECTOR supports hybrid reference conditioning, allowing generation to be simultaneously guided by static images and/or dynamic videos. Moreover, users can explicitly specify the trajectory of each referenced element, precisely controlling its location, scale, and speed (see Figure1). This design allows the model to synthesize coherent videos that satisfy complex spatiotemporal constraints while preserving high-fidelity adherence to references. Extensive experiments demonstrate that HECTOR achieves superior visual quality, stronger reference preservation, and improved motion controllability compared with existing approaches.
Figures & tables
Figure 2: Pipeline of the Video Decompositor , which extracts video composition alongside dynamic and static references from a video. Specifically, Video SAM is first used to segment elements from the footage. Depending on the entity size, we place one or multiple anchor points on each object. A point tracking method is then used to propagate these selected anchors over time. We design a reference trajectory extraction method that converts the anchor tracks into a composition layout, capturing both the scale and translation of the entity. Finally, we crop each object from the original video using the computed spatial parameters to serve as the reference.
Figure 3: Overview of the HECTOR framework , which accepts hybrid inputs—static images and dynamic video references—alongside user-defined spatiotemporal layouts. The Spatio-Temporal Alignment Module (STAM) projects these references into the latent space using dynamic Gaussian masks to create aligned feature conditions. These conditions guide the DiT backbone to synthesize a unified video that preserves reference fidelity while strictly adhering to the specified motion trajectories.
Method
R-CLIP ↑
R-DINO ↑
CLIP-I ↑
DINO-I ↑
CLIP-T ↑
T-Cons ↑
mIoU ↑
CD ↓
Single-Object
MotionBooth
0.6277
0.2113
0.5622
0.3264
0.2855
0.9645
0.1822
0.2920
VACE (bbox)
0.6623
0.2602
0.5757
0.3357
0.3363
0.9909
0.2191
0.2740
VACE (mask)
0.6613
0.2497
0.5636
0.3048
0.3067
0.9913
0.2087
0.2821
Ours
0.6905
0.4277
0.6081
0.3735
0.3306
0.9914
0.3912
0.1130
Multi-Object
Table 1: Quantitative comparison for HECTOR vs baselines with image-based references. Best results in bold , second-best underlined .
Figure 4: Qualitative comparison against baselines. We evaluate static reference-controlled video generation, as baselines are limited to this modality. The left column displays the source reference objects; for a fair experimental setup, we apply masks to crop the objects, ensuring all approaches receive only the object appearance without background context. The right columns show the resulting generated videos, illustrating the visual quality and precise spatial alignment with the input bounding box trajectories.
Figure 5: Qualitative results for video reference. We demonstrate our framework’s versatility adopting video-based reference through three distinct applications: (a) Object Replacement, seamlessly transferring a reference object’s identity onto a moving subject, (b) Compositional Multi-Subject Generation, where distinct video references independently control separate entities, and (c) Background-Locked Motion Editing, enabling precise foreground manipulations while keeping the background region frozen.
Generative video compositing, which involves inserting external assets seamlessly into existing video sequences, is essential for content creation and visual effects. However, existing approaches suffer from a control-fidelity trade-off: they either hallucinate motion from static images, failing to preserve the dynamics of pre-animated assets, or lack fine-grained spatial control for precise asset placement along user-defined trajectories. We propose FlexComposer, a unified framework that standardizes video compositing as a trajectory-guided conditional generation task, enabling the seamless integration of both static images and dynamic footage. Our approach introduces three key designs: (1) a Unified Canonical Foreground Representation that decouples an object's intrinsic motion from its global displacement, standardizing heterogeneous inputs into a stabilized, centered latent space; (2) a Spatial-Aware Latent Injection strategy that exploits the translation equivariance of VAE latent spaces to transport canonical features onto target trajectories via a parameter-free mechanism; and (3) a Hybrid Dataset and Synthetic-to-Real Curriculum that synergizes procedural simulation, real-world cinematic footage, and generative data to implicitly learn physically plausible illumination and shadow harmonization. This unified design handles diverse inputs from product photos to dynamic subjects achieving high-fidelity motion control and environmental integration without the need for explicit 3D reconstruction or auxiliary learnable adapters. Extensive experiments demonstrate that FlexComposer outperforms state-of-the-art methods in visual quality, temporal consistency, and trajectory adherence.
Modern video generation models can synthesize visually compelling and temporally coherent clips, yet controlling their physical behavior remains difficult with standard text and image conditions. The core challenge is a conditioning bottleneck: material response, contact interaction, deformation, and motion trajectory are continuous and relational physical cues that are hard to specify exhaustively in language but can be demonstrated naturally by video. We propose VIPER, a Visual In-Context Physics Reasoning framework for reference-guided image-to-video generation. Given a target image, a brief target prompt, and a reference video, VIPER treats the reference as a dense visual demonstration of the desired physical process rather than an appearance template. It uses a Multimodal Large Language Model (MLLM) to extract reference-derived physical cues and guide a pretrained image-to-video generator through a hierarchical training strategy, enabling physical behavior transfer while preserving the visual prior of the base generator. To support this setting, we construct VIPER-19K, a curated dataset with material, trajectory, and physical-impact annotations, together with filtered reference-target pairs. Experiments on an unseen validation set show that VIPER achieves stronger reference-video physical similarity and higher human preference than representative video generation and video-as-prompt baselines, while maintaining competitive general video quality. Qualitative results further demonstrate that VIPER can transfer reference-derived physical behavior to new target scenes without requiring carefully engineered prompts.
Tianxiao Chen, Hanmo Chen, Huajin Chen +3
Zhejiang University · vivo BlueImage Lab, vivo Mobile Communication Co., Ltd.
Compositional video generation aims to synthesize multiple instances with diverse appearance and motion. However, current approaches mainly focus on binding semantics, neglecting to understand diverse motion categories specified in prompts. In this paper, we propose a motion factorization framework that decomposes complex motion into three primary categories: motionlessness, rigid motion, and non-rigid motion. Specifically, our framework follows a planning before generation paradigm. (1) During planning, we reason about motion laws on the motion graph to obtain frame-wise changes in the shape and position of each instance. This alleviates semantic ambiguities in the user prompt by organizing it into a structured representation of instances and their interactions. (2) During generation, we modulate the synthesis of distinct motion categories in a disentangled manner. Conditioned on the motion cues, guidance branches stabilize appearance in motionless regions, preserve rigid-body geometry, and regularize local non-rigid deformations. Crucially, our two modules are model-agnostic, which can be seamlessly incorporated into various diffusion model architectures. Extensive experiments demonstrate that our framework achieves impressive performance in motion synthesis on real-world benchmarks. Code is available at https://github.com/ZixuanWang0525/MF-CVG.
Zixuan Wang, Ziqin Zhou, Feng Chen +4
Sichuan University · The University of Adelaide · Nanyang Technological University +1