RealMaster: Lifting Rendered Scenes into Photorealistic Video

We often aim to generate images that are both photorealistic and 3D-consistent, adhering to precise geometry, material, and viewpoint controls. Typically, this is achieved by fine-tuning an image generator, pre-trained on billions of real images, using renders of synthetic 3D assets, where annotations for control signals are available. While this approach can learn the desired controls, it often compromises the realism of the images due to domain gap between photographs and renders. We observe that this issue largely arises from the model learning an unintended association between the presence of control signals and the synthetic appearance of the images. To address this, we introduce Realiz3D, a lightweight framework for training diffusion models, that decouples controls and visual domain. The key idea is to explicitly learn visual domain, real or synthetic, separately from other control signals by introducing a co-variate that, fed into small residual adapters, shifts the domain. Then, the generator can be trained to gain controllability, without fitting to specific visual domain. In this way, the model can be guided to produce realistic images even when controls are applied. We enhance control transferability to the real domain by leveraging insights about roles of different layers and denoising steps in diffusion-based generators, informing new training and inference strategies that further mitigate the gap. We demonstrate the advantages of Realiz3D in tasks as text-to-multiview generation and texturing from 3D inputs, producing outputs that are 3D-consistent and photorealistic.

Diffusion models have achieved remarkable progress in video generation, but their controllability remains a major limitation. Key scene factors such as layout, lighting, and camera trajectory are often entangled or only weakly modeled, restricting their applicability in domains like filmmaking and virtual production where explicit scene control is essential. We present LiVER, a diffusion-based framework for scene-controllable video generation. To achieve this, we introduce a novel framework that conditions video synthesis on explicit 3D scene properties, supported by a new large-scale dataset with dense annotations of object layout, lighting, and camera parameters. Our method disentangles these properties by rendering control signals from a unified 3D representation. We propose a lightweight conditioning module and a progressive training strategy to integrate these signals into a foundational video diffusion model, ensuring stable convergence and high fidelity. Our framework enables a wide range of applications, including image-to-video and video-to-video synthesis where the underlying 3D scene is fully editable. To further enhance usability, we develop a scene agent that automatically translates high-level user instructions into the required 3D control signals. Experiments show that LiVER achieves state-of-the-art photorealism and temporal consistency while enabling precise, disentangled control over scene factors, setting a new standard for controllable video generation.

Video Diffusion Models (VDMs) synthesize high-quality videos, but precise camera and motion control remains challenging when large viewpoint shifts, changing occlusions, or complex dynamics reveal unseen regions. Diffusion priors alone can yield structural inconsistencies, while warped guidance can contain incomplete geometry and misleading view-dependent appearance cues. Real2SAM2Real complements VDM priors with a coarse 3D foreground cache of topologically complete meshes for intuitive editing and animation. Generative 3D lifting estimates these object and human meshes, including unobserved surfaces, from a single reference image. Normal maps rendered under specified camera trajectories and foreground motions guide video synthesis in the VDM, while the reference image provides appearance conditioning. Soft spatial-aligned injection and minimally invasive fine-tuning incorporate this guidance while largely preserving pretrained synthesis priors. Decoupled conditioning enables independent appearance and geometry editing. We train on around 2,000 monocular clips using instance-masked normal sequences with instance-level spatial perturbations to reduce sensitivity to differences between video-estimated training normals and normals rendered from coarse 3D caches at inference. Training requires neither explicit 3D assets nor paired multi-trajectory recordings. Real2SAM2Real supports decoupled camera and multi-instance motion control, including articulated humans. Across these challenging settings, experiments show strong spatiotemporal consistency and better overall control accuracy and video fidelity than evaluated state-of-the-art methods. Preliminary demonstrations suggest potential robotics applications.