In film production, chroma keying composites an approved foreground performance into a new environment, but conventional workflows require coordinated background plates and dedicated harmonization. Recent generative methods often require pre-aligned backgrounds, focus on background-conditioned foreground relighting, or represent the environment only through an edited first frame. These restrictions limit background selection, scene composition, and camera motion while adding preparation or editing effort. We present FreeCHROMA, which adapts a pretrained video diffusion model in a parameter-efficient manner for generative video compositing from one to three unaligned background references. The model integrates the foreground into the referenced environment, preserving foreground identity, motion, and fine visual details while generating a spatiotemporally coherent background. Foreground-background harmonization occurs within the same generation process, without a separate relighting stage. Optional text instructions help disambiguate complex spatial relationships and camera behavior. To train the model, we construct pseudo-paired data from readily available real monocular videos through foreground-background separation and augmentation with existing foundation models. The original videos serve as training targets, while automated processing and quality verification enable scalable data construction. Across reference fidelity, foreground preservation, scene integration, harmonization, and overall video quality, FreeCHROMA consistently outperforms recent compositing and conditional video-generation baselines, with particularly clear gains under camera motion. FreeCHROMA relaxes the constraints of existing tools to enable end-to-end generative chroma keying from unaligned inputs, with exploratory experiments demonstrating its potential in robotics.
Figures & tables
Figure 1: FreeCHROMA and the production constraints it relaxes. Top left: Layer composition requires pre-aligned videos and primarily relights the foreground against a fixed background. First/key-frame editing adds an image-editing step and constrains the environment only in selected views. Top center and right: FreeCHROMA preserves the foreground performance while using one or more unaligned background reference images throughout generation, jointly inferring composition, interactions, camera-aware background evolution, and harmonization, with optional text guidance. Bottom: Masked inputs retain all foreground elements for environment augmentation, the robot and interacting objects for sim-to-real transfer, or only the robot body for visual world modeling.
Figure 2: Automatic pseudo-pair construction from natural videos. Clip curation: A shot detector, conventional image-quality tests, and a vision–language model (VLM) retain clips with a visible foreground and informative background. Foreground augmentation: The VLM enumerates the foreground entities; SAM 3.1 tracks their masks M ; and a temporally coherent photometric and normal-guided illumination transform produces (F,M) . Background reference construction: A motion proxy selects one to three keyframes, a frozen image editor removes all selected entities, a frozen novel-view model changes viewpoint and crop, and a VLM quality gate retains scene-consistent references R . The conditions map to the untouched source video V , yielding scalable supervision without green-screen capture, manual compositing, or 3D annotation. Tool roles and selected models are labeled in the figure; further implementation details are provided in Appendix D . Visuals are schematic.
Figure 3: Parameter-efficient FreeCHROMA architecture. During training, a frozen VAE encodes the target and foreground streams. The noisy target latent, foreground latent, and aligned mask condition are concatenated channel-wise and enter through the expanded patchify projector in eq. 5 ; background reference tokens are produced by a trainable convolution and prepended to the video sequence; optional text uses the pretrained cross-attention path. Flow-matching supervision updates the added foreground and mask input weights, the background-reference convolution, and DiT LoRA adapters. At inference, the same conditioned denoiser transforms noise into a complete video, which the frozen VAE decodes. Scene synthesis, spatiotemporal composition, and foreground harmonization occur in one denoising process, without an auxiliary relighting or frame-wise harmonization network.
Content Consistency
Temporal Quality
Visual Quality
Method
Subject Cons. ↑
Background Cons. ↑
Motion Smoothness ↑
Temporal Flickering ↑
Multi-view Cons. ↑
Aesthetic Quality ↑
Imaging Quality ↑
Lumen ( Zeng et al., 2025 ) †
94.26
93.22
98.06
96.14
19.06
53.82
71.25
VACE ( Jiang et al., 2025 )
93.94
93.28
97.73
95.98
24.56
57.33
72.42
IllumiCraft ( Lin et al., 2025 )
91.64
92.12
96.87
95.78
15.07
53.23
70.61
FlowPortal ( Gao et al., 2026b )
93.23
92.53
97.77
95.95
15.63
53.85
67.24
ID-V2V + Paste ( Xu et al., 2026 )
91.70
93.18
98.07
96.11
25.01
56.49
73.30
Table 1: Video quality on the evaluation benchmark: VBench ( Huang et al., 2024 ) , except multi-view consistency from VBench-2.0 ( Zheng et al., 2025 ) . Best results are bold; second-best are underlined. † Lumen receives a caption; other methods use visual background references.
Foreground Quality
Scene Integration
Method
FG Preservation VLM ↑
Edge Quality VLM ↑
FG–BG Fusion VLM ↑
Illum. Harmony VLM ↑
Lumen ( Zeng et al., 2025 ) †
9.778
6.722
6.470
7.040
VACE ( Jiang et al., 2025 )
9.852
7.840
7.580
7.864
IllumiCraft ( Lin et al., 2025 )
8.330
6.196
6.278
6.932
FlowPortal ( Gao et al., 2026b )
5.624
5.842
5.802
6.450
ID-V2V + Paste ( Xu et al., 2026 )
9.636
6.304
6.372
6.088
Table 2: Task-specific evaluation on the novel-background benchmark. VLM scores measure foreground preservation, boundary quality, scene integration, and illumination harmony. Scores are averaged over 1,000 generated videos (500 foreground videos, each paired with two different sets of background references). Best results are bold; second-best are underlined.
Figure 4: Qualitative comparison of generated videos. Given background reference images and a masked foreground sequence, each method generates an output video; two frames sampled at matched relative times are shown. FreeCHROMA faithfully preserves the foreground performance while producing more plausible foreground–background spatial relationships, including viewpoint consistency, relative scale, and physical support. It also generates more natural camera motion and harmonizes illumination and appearance across foreground and background. See the project page for more video examples.
Global Trajectory Error
Local Motion Error
Motion Extent
Method
ATE ↓
ATE/Path (%) ↓
RPE Translation ↓
RPE Rotation ( ∘ ) ↓
Relative Path Length →1
Lumen ( Zeng et al., 2025 ) †
0.015
9.13
0.008
0.593
0.834
VACE ( Jiang et al., 2025 )
0.006
4.50
0.005
0.176
0.936
IllumiCraft ( Lin et al., 2025 )
0.017
9.25
0.009
0.653
0.823
FlowPortal ( Gao et al., 2026b )
0.015
9.00
0.008
0.614
0.866
ID-V2V + Paste ( Xu et al., 2026 )
0.011
6.11
0.006
0.192
0.929
Table 3: Camera-trajectory consistency between generated videos and the original source videos from which the foreground inputs are extracted. Best results are bold; second-best are underlined.
VLM Evaluation
VBench Evaluation
Camera Trajectory
Method
FG–BG Fusion ↑
Illum. Harmony ↑
Subject Cons. ↑
Motion Smoothness ↑
ATE ↓
Relative Path Length →1
w/o FG illumination perturbation
6.384
6.958
94.16
98.50
0.006
0.918
w/o BG viewpoint augmentation
7.638
7.902
94.56
98.45
0.005
0.909
FreeCHROMA
7.728
7.956
94.59
98.55
0.004
0.940
Table 4: Ablation results on training augmentation components.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Qualitative ablation of the two training-time augmentations. For each sample we show the environment reference ( Target BG ), the masked foreground control ( Input FG ), and two frames of the videos generated by the two ablated models and by the full model ( Ours , orange). Without the foreground illumination perturbation, the model tends to copy the illumination of the input foreground instead of re-lighting it for the target environment. Without the background viewpoint augmentation, the model is trained only on pixel-aligned background plates; given an unaligned reference at test time, it tends to force the reference’s viewpoint into the output and composites a scene that is inconsistent with the foreground. The full model re-lights the foreground to match the target environment and uses the reference as an unaligned appearance cue, producing a consistent scene. See the project page for more video examples.
Figure 6: Sim2Real on RoboCasa ( Nasiriany et al., 2026 ) : key frames. Top: the simulated source video and the rendered robot-object interaction used as foreground control (the arm together with the manipulated objects). Below: the videos generated by FreeCHROMA from eight different environment reference images, shown at the same frames. The same simulated action is re-rendered into every environment while the arm motion and the object interaction are preserved. See the project page for more video examples.
Figure 7: Qualitative robotics applications. Rendered RoboCasa ( Nasiriany et al., 2026 ) (top) and masked DROID ( Khazatsky et al., 2024 ) (bottom) arms; green/red panels show successful/failed interactions. See the project page for more video examples.
Operator
Sampling rule
Exposure
logg∼N(0,(0.40s)2)
White balance
Channel log gains ∼N(0,(0.25sc)2) ; gains normalized by their mean
Contrast
q=1+0.40su , about linear mid-gray 0.18
Gamma
logκj∼N(0,(0.20sc)2) for each channel j
Saturation
σ=1+0.50scu , about BT.601 luma
Appendix
Table 5: Photometric augmentation, applied in the listed order. Each occurrence of u is sampled independently from U(−1,1) .
Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.
We present PAI-Studio, a new reference-conditioned video synthesis task that addresses a long-standing challenge in cinematic background replacement: generating dynamic backgrounds aligned with foreground motion while preserving foreground identity, matching reference scene appearance, and achieving globally consistent illumination with realistic foreground relighting. Existing open-source systems and commercial APIs cannot simultaneously ensure motion-consistent background generation, high-fidelity foreground relighting and foreground identity preservation, often resulting in static backgrounds, inconsistent boundaries, and noticeable compositing artifacts. To bridge this gap, we build upon a Diffusion Transformer video backbone and reformulate the problem as an in-context conditional generation task. Through bidirectional attention, our model jointly captures foreground dynamics and background reference information within a unified architecture. We further construct a 30K-scale dataset sourced from high-quality films and online videos to support this task. Extensive evaluations demonstrate that our method significantly outperforms existing open-source and commercial API solutions.
Heyuan Gao, Bangxun Tang, Yiren Song +4
Utopai Studios · Nanyang Technological University · University of California, Irvine +1
Real-world videos naturally portray complex interactions among distinct physical objects, effectively forming dynamic compositions of visual elements. However, most current video generation models synthesize scenes holistically and therefore lack mechanisms for explicit compositional manipulation. To address this limitation, we propose HECTOR, a generative pipeline that enables fine-grained compositional control. In contrast to prior methods,HECTOR supports hybrid reference conditioning, allowing generation to be simultaneously guided by static images and/or dynamic videos. Moreover, users can explicitly specify the trajectory of each referenced element, precisely controlling its location, scale, and speed (see Figure1). This design allows the model to synthesize coherent videos that satisfy complex spatiotemporal constraints while preserving high-fidelity adherence to references. Extensive experiments demonstrate that HECTOR achieves superior visual quality, stronger reference preservation, and improved motion controllability compared with existing approaches.
Guofeng Zhang, Angtian Wang, Jacob Zhiyuan Fang +4
Johns Hopkins University · Bytedance, Intelligent Creation
Generative video compositing, which involves inserting external assets seamlessly into existing video sequences, is essential for content creation and visual effects. However, existing approaches suffer from a control-fidelity trade-off: they either hallucinate motion from static images, failing to preserve the dynamics of pre-animated assets, or lack fine-grained spatial control for precise asset placement along user-defined trajectories. We propose FlexComposer, a unified framework that standardizes video compositing as a trajectory-guided conditional generation task, enabling the seamless integration of both static images and dynamic footage. Our approach introduces three key designs: (1) a Unified Canonical Foreground Representation that decouples an object's intrinsic motion from its global displacement, standardizing heterogeneous inputs into a stabilized, centered latent space; (2) a Spatial-Aware Latent Injection strategy that exploits the translation equivariance of VAE latent spaces to transport canonical features onto target trajectories via a parameter-free mechanism; and (3) a Hybrid Dataset and Synthetic-to-Real Curriculum that synergizes procedural simulation, real-world cinematic footage, and generative data to implicitly learn physically plausible illumination and shadow harmonization. This unified design handles diverse inputs from product photos to dynamic subjects achieving high-fidelity motion control and environmental integration without the need for explicit 3D reconstruction or auxiliary learnable adapters. Extensive experiments demonstrate that FlexComposer outperforms state-of-the-art methods in visual quality, temporal consistency, and trajectory adherence.