Recent advances in Video Foundation Models (VFMs) have revolutionized human-centric video synthesis, yet fine-grained and independent editing of subjects and scenes remains a critical challenge. Recent attempts to incorporate richer environment control through rigid 3D geometric compositions often encounter a stark trade-off between precise control and generative flexibility. Furthermore, the heavy 3D pre-processing still limits practical scalability. In this paper, we propose ONE-SHOT, a parameter-efficient framework for compositional human-environment video generation. Our key insight is to factorize the generative process into disentangled signals. Specifically, we introduce a canonical-space injection mechanism that decouples human dynamics from environmental cues via cross-attention. We also propose Dynamic-Grounded-RoPE, a novel positional embedding strategy that establishes spatial correspondences between disparate spatial domains without any heuristic 3D alignments. To support long-horizon synthesis, we introduce a Hybrid Context Integration mechanism to maintain subject and scene consistency across minute-level generations. Experiments demonstrate that our method significantly outperforms state-of-the-art methods, offering superior structural control and creative diversity for video synthesis. Our project has been available on: https://martayang.github.io/ONE-SHOT/.
Figures & tables
Figure 1: Human-Environment Synthesis by ONE-SHOT . Three kinds of conditions are decoupled and factorized within our ONE-SHOT framework to enable compositional generation. Blue ) : human dynamics. Green ) : textual prompt. Orange ) : environment space. The generated videos exhibit high consistency in human subjects and environments w.r.t. given conditions, and further accurately convey the textual interaction of “holding a longsword”.
Figure 2: Model Architecture. Our model is built upon a Pretrained VFM (gray, right) augmented with an additional Conditioning Branch (cyan, left) for concept injection. The environmental condition cenv is encoded from 2D-projected point clouds and depth maps, while identity appearance cid and context memory cmem jointly maintain visual coherence. Human dynamics cmot are disentangled from environmental inputs and injected through the proposed Decoupled Motion Cross-Attention.
Figure 3: Decoupled Motion Cross-attention. The canonical-space human pose is injected into the video through cross-attention, where the proposed Dynamic-Grounded-RoPE bridges the spatial discrepancy between the environment and human spaces.
Self-reconstruction (Traj100)
Cross-composition (Identity/Motion Swap)
Method
FID ↓
FVD ↓
MS ↑
BC ↑
SubSim ↑
FID ↓
FVD ↓
MS ↑
BC ↑
SubSim ↑
Face ↑
WanAnimate [ 5 ]
21.25
224.08
0.893
0.956
0.189
152.07
1435.3
0.892
0.950
0.368
0.196
RealisDance [ 36 ]
22.17
247.54
0.920
0.946
0.237
155.71
1416.5
0.986
0.962
0.230
0.257
Uni3C [ 3 ]
55.47
775.38
0.853
0.928
0.185
162.53
1519.5
0.924
0.950
0.321
0.203
RealisMotion [ 19 ]
18.89
193.89
0.916
0.940
0.356
151.19
1486.8
0.960
0.919
0.300
0.238
ONE-SHOT (ours)
16.88
181.17
0.940
0.959
0.317
150.98
1382.4
0.993
0.963
0.349
0.241
Table 1: Quantitative Results on Traj100 (Self-Reconstruction) and Our Constructed Testset (Cross-Composition). Lower is better for FID/FVD. Higher is better for others.
Figure 4: Self-Reconstruction and Cross-Composition Comparisons on Traj100. (a) Our method better preserves both the scene structure and human appearance, and (b) maintains stable subject placement and motion consistency when swapping identity.
Figure 5: Text-guided editing with compositional controls . Beyond composing scene, identity, and motion, our method enables instruction-based editing via text prompts, indicating strong compatibility with the pretrained VFM and minimal loss of its native text-conditioned editing ability.
Ablation
FID ↓
FVD ↓
MS ↑
ONE-SHOT (ours)
16.88
181.17
0.9404
w/o RoPE grounding
22.34
229.76
0.9268
label-only RoPE
21.36
221.66
0.9256
w/o face reference
21.37
215.43
0.9378
w/o subject-cropped reference
19.43
200.23
0.9351
w/o context memory
18.17
182.98
0.9330
Table 2: Ablations on Traj100 (Self-Reconstruction).
Figure 6: Ablations of Dynamic-Grounded-RoPE and Face Appearance Integration on Long-Horizon Generation. Removing RoPE grounding weakens motion controllability, leading to pose drift and reduced adherence to the provided motion control. Removing face reference weakens identity anchoring, causing increased identity drift over time. Our full model achieves both better motion following and identity preservation.
Figure 7: Effect of context memory on viewpoint revisit consistency . Our model maintains structural and temporal consistency even after camera loops, whereas memoryless baselines suffer from appearance drift.
Self-reconstruction (Traj100)
Cross-composition
Method
FID ↓
FVD ↓
MS ↑
BC ↑
SubSim ↑
FID ↓
FVD ↓
MS ↑
BC ↑
SubSim ↑
Face ↑
WAN-VACE (fine-tuned)
20.96
185.09
0.935
0.955
0.311
151.68
1465.3
0.912
0.947
0.182
0.099
ONE-SHOT (ours)
16.88
181.17
0.940
0.959
0.317
150.98
1382.4
0.993
0.963
0.349
0.241
Table 3: Comparison with fine-tuned WAN-VACE on self-reconstruction and cross-composition. WAN-VACE is adapted to our setting by fine-tuning on the same training data as ONE-SHOT, with SMPL-X used as compositional control input. Lower is better for FID/FVD. Higher is better for others.
Generative video compositing, which involves inserting external assets seamlessly into existing video sequences, is essential for content creation and visual effects. However, existing approaches suffer from a control-fidelity trade-off: they either hallucinate motion from static images, failing to preserve the dynamics of pre-animated assets, or lack fine-grained spatial control for precise asset placement along user-defined trajectories. We propose FlexComposer, a unified framework that standardizes video compositing as a trajectory-guided conditional generation task, enabling the seamless integration of both static images and dynamic footage. Our approach introduces three key designs: (1) a Unified Canonical Foreground Representation that decouples an object's intrinsic motion from its global displacement, standardizing heterogeneous inputs into a stabilized, centered latent space; (2) a Spatial-Aware Latent Injection strategy that exploits the translation equivariance of VAE latent spaces to transport canonical features onto target trajectories via a parameter-free mechanism; and (3) a Hybrid Dataset and Synthetic-to-Real Curriculum that synergizes procedural simulation, real-world cinematic footage, and generative data to implicitly learn physically plausible illumination and shadow harmonization. This unified design handles diverse inputs from product photos to dynamic subjects achieving high-fidelity motion control and environmental integration without the need for explicit 3D reconstruction or auxiliary learnable adapters. Extensive experiments demonstrate that FlexComposer outperforms state-of-the-art methods in visual quality, temporal consistency, and trajectory adherence.
Real-world videos naturally portray complex interactions among distinct physical objects, effectively forming dynamic compositions of visual elements. However, most current video generation models synthesize scenes holistically and therefore lack mechanisms for explicit compositional manipulation. To address this limitation, we propose HECTOR, a generative pipeline that enables fine-grained compositional control. In contrast to prior methods,HECTOR supports hybrid reference conditioning, allowing generation to be simultaneously guided by static images and/or dynamic videos. Moreover, users can explicitly specify the trajectory of each referenced element, precisely controlling its location, scale, and speed (see Figure1). This design allows the model to synthesize coherent videos that satisfy complex spatiotemporal constraints while preserving high-fidelity adherence to references. Extensive experiments demonstrate that HECTOR achieves superior visual quality, stronger reference preservation, and improved motion controllability compared with existing approaches.
Guofeng Zhang, Angtian Wang, Jacob Zhiyuan Fang +4
Johns Hopkins University · Bytedance, Intelligent Creation
Compositional video generation aims to synthesize multiple instances with diverse appearance and motion. However, current approaches mainly focus on binding semantics, neglecting to understand diverse motion categories specified in prompts. In this paper, we propose a motion factorization framework that decomposes complex motion into three primary categories: motionlessness, rigid motion, and non-rigid motion. Specifically, our framework follows a planning before generation paradigm. (1) During planning, we reason about motion laws on the motion graph to obtain frame-wise changes in the shape and position of each instance. This alleviates semantic ambiguities in the user prompt by organizing it into a structured representation of instances and their interactions. (2) During generation, we modulate the synthesis of distinct motion categories in a disentangled manner. Conditioned on the motion cues, guidance branches stabilize appearance in motionless regions, preserve rigid-body geometry, and regularize local non-rigid deformations. Crucially, our two modules are model-agnostic, which can be seamlessly incorporated into various diffusion model architectures. Extensive experiments demonstrate that our framework achieves impressive performance in motion synthesis on real-world benchmarks. Code is available at https://github.com/ZixuanWang0525/MF-CVG.
Zixuan Wang, Ziqin Zhou, Feng Chen +4
Sichuan University · The University of Adelaide · Nanyang Technological University +1