Organizations: Institute of Computing Technology, Chinese Academy of Sciences, China and University of Chinese Academy of Sciences, China, China · Cardiff University, United Kingdom · Kuaishou Technology, China
Understanding and modeling light transport in human-centric scenes is a longstanding challenge. While recent progress in diffusion models has boosted research in relighting, precise illumination control still lags, primarily due to the scarcity of diverse, high-quality, precisely labeled video relighting datasets. Existing end-to-end relighting approaches model light transport implicitly through neural networks, which substantially complicates controllability. In contrast, some works resort to path-traced irradiance cues as an explicit blueprint for light transport. However, such constructed irradiance cues are prone to artifacts and hinder generative quality when directly used as control signals. With the key observation that noise corruption removes high-frequency artifacts and preserves low-frequency light transport, we introduce an asynchronous joint-modeling paradigm that facilitates fine-grained illumination control. During two-stage training, we force the model to jointly denoise RGB and ground-truth irradiance frames with separate timesteps scheduling. A domain matching strategy is applied to the second stage to bridge the domain gap between irradiance cues and ground-truth irradiance maps. At inference time, we introduce an asynchronous hijacked generation pipeline that facilitates flexible illumination control. For training data, we develop a multi-illumination high-quality synthetic video dataset with paired irradiance maps using a powerful game engine. Trained exclusively on this dataset, our model exhibits robust generalization to real- world scenarios and outperforms prior approaches in both lighting fidelity and generative quality. Project page: http://www.geometrylearning.com/JointLight/
Figures & tables
Figure 1. Our method jointly models RGB and irradiance map, enabling high-quality generation with physically accurate illumination control. This is the teaser figure for the article.
Figure 2. Overall training pipeline. Noisy RGB video, noisy irradiance video and input video are encoded and concatenated frame-wise before fed into the fully tuned Diffusion Transformer. Noisy RGB and irradiance tokens are assigned with separate noise timesteps to enable asynchronous inference (Section 5.3 ).
Figure 3. Inference pipeline. From the input video and target environment map, we render a per-frame irradiance cue using monocular point-cloud estimation and Mitsuba 3. The cue is noised to th , kept fixed until th , then jointly denoised with RGB, preserving low-frequency transport while suppressing artifacts.
Figure 4. Illustration of our introduced multi-illumination synthetic dataset.
Figure 5. Qualitative comparison on vertical SynthStatic and HumanVid videos. The last row shows synthetic ground truth and real-world irradiance cues.
Figure 6. Qualitative comparison on horizontal images.
SynthStatic
SynthDynamic
User Study
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
VQ ↑
LF ↑
IC-Light
12.79
0.508
0.329
12.76
0.512
0.317
3.28
2.10
Light-A-Video
12.18
0.390
0.401
11.82
0.375
0.440
3.41
3.01
TC-Light
10.74
0.433
0.472
9.87
0.366
0.539
2.73
2.45
DiLightNet
11.76
0.507
0.336
11.21
0.493
0.371
1.19
1.31
Neural Gaffer
13.72
0.629
0.269
12.25
0.607
0.327
2.40
2.57
Table 1. Quantitative comparison on SynthStatic and SynthDynamic , and user-study scores. VQ and LF denote visual quality and lighting fidelity. Best and second-best results are bold and underlined, respectively.
Figure 7. Results under dynamic lighting. Images are sampled every 10 frames while the environment map rotates through 360∘ .
Figure 8. More results on vertical images from indoor and outdoor scenes. In the second example, the light source appears in the upper-right region of the chrome ball. We show a black background for clarity.
Figure 9. More results and failure cases on horizontal images from indoor and outdoor scenes. The bottom-left example shows the limitation on highly reflective surfaces; the bottom-right example shows light leakage around the hand caused by inaccurate depth near a discontinuity.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Neural Gaffer
17.10
0.881
0.151
DiLightNet
15.68
0.847
0.142
Light-X
18.01
0.862
0.161
DiffusionRenderer
22.32
0.912
0.114
UniRelight
19.54
0.871
0.136
Ours
23.89
0.928
0.078
Table 2. Cross-dataset evaluation on HumanOLAT.
Figure 10. Qualitative ablation. From left: input, direct irradiance conditioning without joint modeling, without domain matching, and full model.
Figure 11. Qualitative comparison on HumanOLAT dataset.
Variant
PSNR ↑
SSIM ↑
LPIPS ↓
HDR map conditioning
13.79
0.585
0.331
irradiance conditioning
14.10
0.578
0.327
+ Joint modeling
14.06
0.572
0.321
+ Hijacked conditional generation
14.69
0.582
0.306
+ Asynchronous timestep scheduling
15.20
0.611
0.298
+ Domain matching strategy (Ours)
15.69
0.621
0.282
Table 3. Quantitative ablation on major components.