We present CamLit, the first unified video diffusion model that jointly performs novel view synthesis (NVS) and relighting from a single input image. Given one reference image, a user-defined camera trajectory, and an environment map, CamLit synthesizes a video of the scene from new viewpoints under the specified illumination. Within a single generative process, our model produces temporally coherent and spatially aligned outputs, including relit novel-view frames and corresponding albedo frames, enabling high-quality control of both camera pose and lighting. Qualitative and quantitative experiments demonstrate that CamLit achieves high-fidelity outputs on par with state-of-the-art methods in both novel view synthesis and relighting, without sacrificing visual quality in either task. We show that a single generative model can effectively integrate camera and lighting control, simplifying the video generation pipeline while maintaining competitive performance and consistent realism.
Figures & tables
Figure 2: An illustration of CamLit pipeline. At the core of our framework is a multi-modal video DiT. This model takes as input a single RGB image, a camera trajectory, and an environment map. From these inputs, the model simultaneously generates a spatially and temporally aligned triplet of videos: (i) an RGB novel-view sequence under the same illumination as the input image, (ii) the corresponding relit sequence (with full shading from the environment map), and (iii) an albedo sequence capturing the scene’s intrinsics without shading.
Figure 3: Video generation examples of CamLit. For each example, we visualize two camera trajectories, moving backward and turning right as indicated by the arrows, to reveal generated content in unseen regions. From left to right, we show the input image, a novel view frame under the original lighting, the corresponding albedo, and three relit novel view frames. The environment maps used for relighting are shown in the insets.
NVS Methods
FID ↓
SEVA [ 56 ]
5.615
GEN3C [ 35 ]
5.144
Our nvs-only
6.724
Ours
7.213
Relighting Methods
FID ↓
DiffusionRenderer [ 22 ]
14.088
Table 1: FID scores computed on RealEstate10K [ 57 ] test split. Our unified architecture performs NVS and relighting jointly, yet achieves fidelity on par with state-of-the-art approaches dedicated to each task.
NVS Methods
FVD ↓
SEVA [ 56 ]
94.95
GEN3C [ 35 ]
91.69
Our nvs-only
78.03
Ours
99.17
Relighting Methods
FVD ↓
DiffusionRenderer [ 22 ]
103.98
Table 2: FVD scores computed on RealEstate10K [ 57 ] test split. Our unified system achieves video quality comparable to state-of-the-art approaches dedicated to each task.
Figure 4: Qualitative comparison of novel view synthesis methods. For each input image, we apply two camera trajectories, moving backward ( 1 st row) and turning right ( 2 nd row), as indicated by the arrows. Our model, which performs NVS and relighting jointly, achieves NVS quality on par with state-of-the-art methods specifically dedicated to NVS.
Figure 5: Qualitative comparison of relighting methods. Our approach produces albedo and relit videos with quality comparable to DiffusionRenderer [ 22 ] , which is our theoretical performance upper bound. The environment maps used for relighting are shown in the middle insets.
Figure 6: Qualitative comparison of our NVS‑only and full models. Two camera trajectories, moving backward and turning left as indicated by arrow directions. Both models produce high‑quality, geometrically consistent novel views, demonstrating that the unified formulation preserves NVS performance while integrating relighting.
Recent diffusion models have achieved remarkable success in image relighting, and this success has quickly been extended to video relighting. However, existing methods offer limited explicit control over illumination in the relighted output. We present LightCtrl, the first controllable video relighting method that enables explicit control of video illumination through a user-supplied light trajectory in a training-free manner. Our approach combines pre-trained diffusion models: an image relighting model processes each frame individually, followed by a video diffusion prior to enhance temporal consistency. To achieve explicit control over dynamically varying lighting, we introduce two key components. First, a Light Map Injection module samples light trajectory-specific noise and injects it into the latent representation of the source video, improving illumination coherence with the conditional light trajectory. Second, a Geometry-Aware Relighting module dynamically combines RGB and normal map latents in the frequency domain to suppress the influence of the original lighting, further enhancing adherence to the input light trajectory. Experiments show that LightCtrl produces high-quality videos with diverse illumination changes that closely follow the specified light trajectory, demonstrating improved controllability over baseline methods. Code is available at: https://github.com/GVCLab/LightCtrl.
Yizuo Peng, Xuelin Chen, Kai Zhang +1
Tsinghua University · GVC Lab, Great Bay University · Adobe Research
Video relighting requires balancing long-form temporal consistency with a physically grounded understanding of light transport, which depends on accurate estimation of intrinsic scene properties such as materials, geometry, and illumination. Existing methods follow two paradigms: (1) reconstruct a video's photometric properties via inverse rendering and relight them to a target illumination via forward rendering, using physically-based rendering (PBR) or a neural renderer; these suffer from noisy reconstructions and struggle with hard-to-model effects such as global illumination. (2) Frame the task as generative video-to-video translation conditioned on relighting targets (a target environment map or text); this limits relighting control and temporal stability, since diffusion models struggle to translate long-form videos, and is constrained by the availability of input/relit training pairs. We propose LightCrafter, a hybrid pipeline that reformulates video relighting as video translation of a proxy video: rather than translating the input video directly to the target, we translate a PBR rendering of the input under the target illumination to the final target. This bakes illumination targets into the PBR proxy, removing the need to teach the diffusion model illumination concepts like environment maps, and enables more intricate lighting control while naturally providing long-form temporal consistency. We show PBR renders alone already outperform some prior art but struggle with effects like global illumination; to capture these, we leverage photometric priors in video generation models by post-training CogVideoX on synthetic video pairs and real-world unpaired videos. We outperform prior state-of-the-art on existing real-world relighting benchmarks and contribute a synthetic benchmark for further analysis. We will release our dataset, benchmark, metrics, and code.
Zixin Guo, Yehonathan Litman, Yifeng He +3
Carnegie Mellon University · University of Toronto · Bosch Research
Recent advances have shown that large-scale video diffusion models can be repurposed as neural renderers by first decomposing videos into intrinsic scene representations and then performing forward rendering under novel illumination. While promising, this paradigm fundamentally relies on accurate intrinsic decomposition, which remains highly unreliable for real-world videos and often leads to distorted appearances, broken materials, and accumulated temporal artifacts during relighting. In this work, we present Relit-LiVE, a novel video relighting framework that produces physically consistent, temporally stable results without requiring prior knowledge of camera pose. Our key insight is to explicitly introduce raw reference images into the rendering process, enabling the model to recover critical scene cues that are inevitably lost or corrupted in intrinsic representations. Furthermore, we propose a novel environment video prediction formulation that simultaneously generates relit videos and per-frame environment maps aligned with each camera viewpoint in a single diffusion process. This joint prediction enforces strong geometric-illumination alignment and naturally supports dynamic lighting and camera motion, significantly improving physical consistency in video relighting while easing the requirement of known per-frame camera pose. Extensive experiments demonstrate that Relit-LiVE consistently outperforms state-of-the-art video relighting and neural rendering methods across synthetic and real-world benchmarks. Beyond relighting, our framework naturally supports a wide range of downstream applications, including scene-level rendering, material editing, object insertion, and streaming video relighting. The Project is available at https://github.com/zhuxing0/Relit-LiVE.
Weiqing Xiao, Hong Li, Xiuyu Yang +7
Nanjing University, China · BAAI, China · BUAA, China +5