LightCtrl: Training-free Controllable Video Relighting

Yizuo Peng; Xuelin Chen; Kai Zhang; Xiaodong Cun

cs.CVMar 28, 2026

LightCtrl: Training-free Controllable Video Relighting

Authors: Yizuo Peng, Xuelin Chen, Kai Zhang, Xiaodong Cun

Organizations: Tsinghua University · GVC Lab, Great Bay University · Adobe Research

Abstract

Recent diffusion models have achieved remarkable success in image relighting, and this success has quickly been extended to video relighting. However, existing methods offer limited explicit control over illumination in the relighted output. We present LightCtrl, the first controllable video relighting method that enables explicit control of video illumination through a user-supplied light trajectory in a training-free manner. Our approach combines pre-trained diffusion models: an image relighting model processes each frame individually, followed by a video diffusion prior to enhance temporal consistency. To achieve explicit control over dynamically varying lighting, we introduce two key components. First, a Light Map Injection module samples light trajectory-specific noise and injects it into the latent representation of the source video, improving illumination coherence with the conditional light trajectory. Second, a Geometry-Aware Relighting module dynamically combines RGB and normal map latents in the frequency domain to suppress the influence of the original lighting, further enhancing adherence to the input light trajectory. Experiments show that LightCtrl produces high-quality videos with diverse illumination changes that closely follow the specified light trajectory, demonstrating improved controllability over baseline methods. Code is available at: https://github.com/GVCLab/LightCtrl.

Figures & tables

Figure 1: LightCtrl can relight an input video to produce high-quality results with strong temporal consistency and, particularly, illumination that closely follows the user-specified light trajectory.

Figure 2: Overall Pipeline. We perform controllable video relighting with a user-provided light trajectory. Where we inject the light map on the noisy latent of VDM using the light map injection module. Then, in each denosing step, we design a geometry-aware relighting module to produce the relighted results frame-wise. Thus, the VDM can help to generate consistent video results with controllable lighting.

Figure 3: The framework of Geometry-Aware Relighting Module . This module is designed for frame-wise relighting. Firstly, the normal sequence of the source video and each consistent target derived from the denoising steps are encoded into the latent space and further process them in the frequency domain. Then, their spatio-temporal low-frequency components are dynamically fused with the high-frequency components of the consistent latents, with the cut-off frequency varying at each denoising step to add details. Finally, these lantents are transformed back to the video space and produce the relighted frames using frame-wise IC-Light Zhang et al. (2025) .

Figure 4: Qualitative comparison of light trajectory control. We compare our LightCtrl with IC-Light, IC-Light + SDEdit-0.2, IC-Light + SDEdit-0.6, and LAV-Traj. Compared to baselines, LightCtrl produces high-quality, temporally coherent, and controllable relighting results, closely following the input light trajectories.

Methods	Video Quality		Control Ability		User Study
Methods	AQ $\uparrow$	FVD $\downarrow$	PSNR y $\uparrow$	PSNR light $\uparrow$	VS $\uparrow$	LC $\uparrow$	LQ $\uparrow$	ALT $\uparrow$
IC-Light Zhang et al. (2025)	0.5937	1018.5	11.059	15.850	1.00%	15.00%	3.00%	9.73%
IC-Light + SDEdit-0.2 Meng et al. (2021)	0.5907	1134.8	11.009	16.249	2.50%	2.00%	0.50%	3.00%
IC-Light + SDEdit-0.6 Meng et al. (2021)	0.5681	1630.9	10.980	16.385	4.75%	1.09%	0.50%	3.27%
LAV Zhou et al. (2025) -Traj	0.6157	1077.4	11.043	17.755	23.50%	4.00%	20.00%	10.27%
LightCtrl (Ours)	0.6114	993.1	11.768	18.532	68.25%	77.91%	74.86%	73.73%

Table 1: Quantitative Comparisons with Other Methods.

Figure 5: Ablation Study. Results of controllable video relighting with the Geometry-Aware Relighting (GAR) module or the Light Map Injection (LMI) module removed. The red box indicates the light distribution brought by the input video.

Figure 6: Failure Case. We show an example result where the light map is blended into the original input video. Our method struggles to perform effectively in scenarios involving strong shadows, such as the ”cat” case with the relight prompt ”a cat, warm sunshine.”

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Visualization of the questionnaire for user study. Users are asked to compare the all baselines at each aspect and select their preferred one.

Figure 8: Ablation Study on Feature Selection of GAR module.

Figure 9: Ablation Study on Injection Method of GAR(Left) and LMI module(Right).

Method	AQ $\uparrow$	FVD $\downarrow$
0.5 + 0.5	0.5892	1010.9
Cut-off-0.25	0.6097	998.7
Cut-off-lbd	0.6114	993.1

Appendix

Table 2: Comparison of Normal Injection Strategies on Video Quality

Methods	AQ $\uparrow$	FVD $\downarrow$
Raw Frames	0.5946	1037.4
With Residual	0.6114	993.1

Appendix

Table 3: Quantitative Results between Raw Frames and Decoded Frames + Residual.

Methods	CLIP-Score $\uparrow$
w/o Progressive Fusion	0.9200
With Progressive Fusion	0.9634

Appendix

Table 4: Effect of Progressive Fusion on Temporal CLIP-Score

Figure 10: Ablation Study on Raw Frames vs Residual.

Figure 11: Ablation Study on Progressive Fusion and Injection method of Normal.

Methods	IC-Light	SDEdit-0.2	SDEdit-0.6	LAV-Traj	Ours
$PSNR_{\text{light}}$	12.112	13.349	13.686	16.630	17.494

Appendix

Table 5: Comparison of

PSNR_{\text{light}}

(dB) under static light conditions

Panel of Figure 12 — Figure 12: More results based on CogVideoX.

Figure 13: More Results with Diverse Lighting Trajectories. The first and third columns show the results under random polyline illumination trajectories. The second column presents the results of two light sources moving independently, one from left to right and the other from top to bottom. The fourth column shows the results of two light sources moving in opposite directions along circular paths.

Panel of Figure 14 — Figure 14: More results of LightCtrl in light trajectory-conditioned video relighting .

Video Relighting

See all 13 papers

LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting
Jul 9, 2026Zixin Guo, Yehonathan Litman, Yifeng He +3Physical Consistency in Video Generation Video Diffusion Models
Carnegie Mellon University · University of Toronto · Bosch Research
Relit-LiVE: Relight Video by Jointly Learning Environment Video
May 7, 2026Weiqing Xiao, Hong Li, Xiuyu Yang +7Physical Consistency in Video Generation Video Diffusion Models
Nanjing University, China · BAAI, China · BUAA, China +5
CamLit: Unified Video Diffusion with Explicit Camera and Lighting Control
Mar 15, 2026Zhiyi Kuang, Chengan He, Egor Zakharov +6Video Relighting Camera-Controlled Video Generation
Codec Avatars Lab, Reality Labs, Meta · State Key Lab of CAD&CG, Zhejiang University · University of Tübingen

Jul 9, 2026J/K move · Enter open · S save

LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting

Zixin Guo, Yehonathan Litman, Yifeng He, John Miller+2

Carnegie Mellon University · University of Toronto · Bosch Research

Physical Consistency in Video Generation Video Diffusion Models Video Editing Controllable Video Generation Video Relighting Physically Based Rendering+1

Open paper AskPDF arXiv

Video relighting requires balancing long-form temporal consistency with a physically grounded understanding of light transport, which depends on accurate estimation of intrinsic scene properties such as materials, geometry, and illumination. Existing methods follow two paradigms: (1) reconstruct a video's photometric properties via inverse rendering and relight them to a target illumination via forward rendering, using physically-based rendering (PBR) or a neural renderer; these suffer from noisy reconstructions and struggle with hard-to-model effects such as global illumination. (2) Frame the task as generative video-to-video translation conditioned on relighting targets (a target environment map or text); this limits relighting control and temporal stability, since diffusion models struggle to translate long-form videos, and is constrained by the availability of input/relit training pairs. We propose LightCrafter, a hybrid pipeline that reformulates video relighting as video translation of a proxy video: rather than translating the input video directly to the target, we translate a PBR rendering of the input under the target illumination to the final target. This bakes illumination targets into the PBR proxy, removing the need to teach the diffusion model illumination concepts like environment maps, and enables more intricate lighting control while naturally providing long-form temporal consistency. We show PBR renders alone already outperform some prior art but struggle with effects like global illumination; to capture these, we leverage photometric priors in video generation models by post-training CogVideoX on synthetic video pairs and real-world unpaired videos. We outperform prior state-of-the-art on existing real-world relighting benchmarks and contribute a synthetic benchmark for further analysis. We will release our dataset, benchmark, metrics, and code.

Abstract

Video Relighting

Explore similar work

LightCrafter: PBR-Conditioned Video Diffusion Refinement for Controllable and Consistent Relighting

Relit-LiVE: Relight Video by Jointly Learning Environment Video

CamLit: Unified Video Diffusion with Explicit Camera and Lighting Control