Organizations: Institute of Computing Technology, Chinese Academy of Sciences, China and University of Chinese Academy of Sciences, China, China · Cardiff University, United Kingdom · Kuaishou Technology, China
Understanding and modeling light transport in human-centric scenes is a longstanding challenge. While recent progress in diffusion models has boosted research in relighting, precise illumination control still lags, primarily due to the scarcity of diverse, high-quality, precisely labeled video relighting datasets. Existing end-to-end relighting approaches model light transport implicitly through neural networks, which substantially complicates controllability. In contrast, some works resort to path-traced irradiance cues as an explicit blueprint for light transport. However, such constructed irradiance cues are prone to artifacts and hinder generative quality when directly used as control signals. With the key observation that noise corruption removes high-frequency artifacts and preserves low-frequency light transport, we introduce an asynchronous joint-modeling paradigm that facilitates fine-grained illumination control. During two-stage training, we force the model to jointly denoise RGB and ground-truth irradiance frames with separate timesteps scheduling. A domain matching strategy is applied to the second stage to bridge the domain gap between irradiance cues and ground-truth irradiance maps. At inference time, we introduce an asynchronous hijacked generation pipeline that facilitates flexible illumination control. For training data, we develop a multi-illumination high-quality synthetic video dataset with paired irradiance maps using a powerful game engine. Trained exclusively on this dataset, our model exhibits robust generalization to real- world scenarios and outperforms prior approaches in both lighting fidelity and generative quality. Project page: http://www.geometrylearning.com/JointLight/
Figures & tables
Figure 1. Our method jointly models RGB and irradiance map, enabling high-quality generation with physically accurate illumination control. This is the teaser figure for the article.
Figure 2. Overall training pipeline. Noisy RGB video, noisy irradiance video and input video are encoded and concatenated frame-wise before fed into the fully tuned Diffusion Transformer. Noisy RGB and irradiance tokens are assigned with separate noise timesteps to enable asynchronous inference (Section 5.3 ).
Figure 3. Inference pipeline. From the input video and target environment map, we render a per-frame irradiance cue using monocular point-cloud estimation and Mitsuba 3. The cue is noised to th , kept fixed until th , then jointly denoised with RGB, preserving low-frequency transport while suppressing artifacts.
Figure 4. Illustration of our introduced multi-illumination synthetic dataset.
Figure 5. Qualitative comparison on vertical SynthStatic and HumanVid videos. The last row shows synthetic ground truth and real-world irradiance cues.
Figure 6. Qualitative comparison on horizontal images.
SynthStatic
SynthDynamic
User Study
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
VQ ↑
LF ↑
IC-Light
12.79
0.508
0.329
12.76
0.512
0.317
3.28
2.10
Light-A-Video
12.18
0.390
0.401
11.82
0.375
0.440
3.41
3.01
TC-Light
10.74
0.433
0.472
9.87
0.366
0.539
2.73
2.45
DiLightNet
11.76
0.507
0.336
11.21
0.493
0.371
1.19
1.31
Neural Gaffer
13.72
0.629
0.269
12.25
0.607
0.327
2.40
2.57
Table 1. Quantitative comparison on SynthStatic and SynthDynamic , and user-study scores. VQ and LF denote visual quality and lighting fidelity. Best and second-best results are bold and underlined, respectively.
Figure 7. Results under dynamic lighting. Images are sampled every 10 frames while the environment map rotates through 360∘ .
Figure 8. More results on vertical images from indoor and outdoor scenes. In the second example, the light source appears in the upper-right region of the chrome ball. We show a black background for clarity.
Figure 9. More results and failure cases on horizontal images from indoor and outdoor scenes. The bottom-left example shows the limitation on highly reflective surfaces; the bottom-right example shows light leakage around the hand caused by inaccurate depth near a discontinuity.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Neural Gaffer
17.10
0.881
0.151
DiLightNet
15.68
0.847
0.142
Light-X
18.01
0.862
0.161
DiffusionRenderer
22.32
0.912
0.114
UniRelight
19.54
0.871
0.136
Ours
23.89
0.928
0.078
Table 2. Cross-dataset evaluation on HumanOLAT.
Figure 10. Qualitative ablation. From left: input, direct irradiance conditioning without joint modeling, without domain matching, and full model.
Figure 11. Qualitative comparison on HumanOLAT dataset.
Variant
PSNR ↑
SSIM ↑
LPIPS ↓
HDR map conditioning
13.79
0.585
0.331
irradiance conditioning
14.10
0.578
0.327
+ Joint modeling
14.06
0.572
0.321
+ Hijacked conditional generation
14.69
0.582
0.306
+ Asynchronous timestep scheduling
15.20
0.611
0.298
+ Domain matching strategy (Ours)
15.69
0.621
0.282
Table 3. Quantitative ablation on major components.
Recent diffusion models have achieved remarkable success in image relighting, and this success has quickly been extended to video relighting. However, existing methods offer limited explicit control over illumination in the relighted output. We present LightCtrl, the first controllable video relighting method that enables explicit control of video illumination through a user-supplied light trajectory in a training-free manner. Our approach combines pre-trained diffusion models: an image relighting model processes each frame individually, followed by a video diffusion prior to enhance temporal consistency. To achieve explicit control over dynamically varying lighting, we introduce two key components. First, a Light Map Injection module samples light trajectory-specific noise and injects it into the latent representation of the source video, improving illumination coherence with the conditional light trajectory. Second, a Geometry-Aware Relighting module dynamically combines RGB and normal map latents in the frequency domain to suppress the influence of the original lighting, further enhancing adherence to the input light trajectory. Experiments show that LightCtrl produces high-quality videos with diverse illumination changes that closely follow the specified light trajectory, demonstrating improved controllability over baseline methods. Code is available at: https://github.com/GVCLab/LightCtrl.
Yizuo Peng, Xuelin Chen, Kai Zhang +1
Tsinghua University · GVC Lab, Great Bay University · Adobe Research
Video relighting requires balancing long-form temporal consistency with a physically grounded understanding of light transport, which depends on accurate estimation of intrinsic scene properties such as materials, geometry, and illumination. Existing methods follow two paradigms: (1) reconstruct a video's photometric properties via inverse rendering and relight them to a target illumination via forward rendering, using physically-based rendering (PBR) or a neural renderer; these suffer from noisy reconstructions and struggle with hard-to-model effects such as global illumination. (2) Frame the task as generative video-to-video translation conditioned on relighting targets (a target environment map or text); this limits relighting control and temporal stability, since diffusion models struggle to translate long-form videos, and is constrained by the availability of input/relit training pairs. We propose LightCrafter, a hybrid pipeline that reformulates video relighting as video translation of a proxy video: rather than translating the input video directly to the target, we translate a PBR rendering of the input under the target illumination to the final target. This bakes illumination targets into the PBR proxy, removing the need to teach the diffusion model illumination concepts like environment maps, and enables more intricate lighting control while naturally providing long-form temporal consistency. We show PBR renders alone already outperform some prior art but struggle with effects like global illumination; to capture these, we leverage photometric priors in video generation models by post-training CogVideoX on synthetic video pairs and real-world unpaired videos. We outperform prior state-of-the-art on existing real-world relighting benchmarks and contribute a synthetic benchmark for further analysis. We will release our dataset, benchmark, metrics, and code.
Zixin Guo, Yehonathan Litman, Yifeng He +3
Carnegie Mellon University · University of Toronto · Bosch Research
This paper presents a method for image relighting that enables precise and continuous control over multiple illumination attributes in a photograph. We formulate relighting as a conditional image generation task and introduce attribute tokens to encode distinct lighting factors such as intensity, color, ambient illumination, diffuse level, and 3D light positions. The model is trained on a large-scale synthetic dataset with ground-truth lighting annotations, supplemented by a small set of real captures to enhance realism and generalization. We validate our approach across a variety of relighting tasks, including controlling in-scene lighting fixtures and editing environment illumination using virtual light sources, on synthetic and real images. Our method achieves state-of-the-art quantitative and qualitative performance compared to prior work. Remarkably, without explicit inverse rendering supervision, the model exhibits an inherent understanding of how light interacts with scene geometry, occlusion, and materials, yielding convincing lighting effects even in traditionally challenging scenarios such as placing lights within objects or relighting transparent materials plausibly. Project page: vrroom.github.io/tokenlight/