Estimating scene illumination from images is challenging as lighting is inherently spatially varying, depending on geometry, occlusion, and light sources. Existing methods incorporate geometric cues by deriving local environment maps or probe-based representations, but largely rely on regression or parametric models, resulting in over-smoothed lighting and limited realism. We revisit spatially-varying illumination estimation from a generative perspective and introduce LUMEN, a geometry-conditioned diffusion framework for synthesizing high-dynamic-range (HDR) environment maps at arbitrary spatial locations from single or multi-view images. Specifically, we reconstruct a coarse 3D scene from input images and render partial environment maps at arbitrary locations to encode visibility and spatial structure. These signals are fused with multi-view image features and fed into a diffusion transformer to generate full HDR environment maps. To train the model, we build a synthetic dataset of 700 indoor scenes with dense spatial supervision, including multi-view RGB images, depth, and HDR illumination at multiple sampled locations. This dataset enables the model to learn spatially consistent lighting distributions across diverse scenes. Extensive experiments demonstrate that LUMEN significantly improves visual fidelity and spatial coherence compared to prior approaches.
Figures & tables
Figure 1. Given a set of multi-view images, LUMEN estimates spatially-varying illumination at any location with consistency and achieves seamless virtual object insertion results (the balls and the bunny are rendered using predicted HDRI maps at their positions). teaser
Figure 2. Overview of LUMEN. Given a single-view image or multi-view inputs, we first estimate the camera parameters and reconstruct a coarse point cloud using an off-the-shelf feed-forward 3D reconstruction model. We then render partial environment maps at arbitrary locations in the scene based on the reconstructed geometry, and condition a 2D diffusion model on the input views with these partial environment maps to generate a complete HDRI map.
Dataset
Method
Scale-invariant RMSE ↓
Angular Error ↓
Normalized RMSE ↓
Diffuse
Matte
Mirror
Diffuse
Matte
Mirror
Diffuse
Matte
Mirror
Laval Indoor
StyleLight
0.064
0.152
0.265
3.874
4.606
10.974
0.232
0.411
0.520
DiffusionLight
0.075
0.163
0.282
3.415
4.157
6.483
0.246
0.391
0.457
LuxDiT
0.056
0.140
0.274
2.759
3.875
6.099
0.207
0.388
0.546
Ours
0.046
0.120
0.252
2.700
3.773
6.257
0.180
0.319
0.411
Laval Outdoor
StyleLight
0.131
0.286
0.456
3.524
4.864
7.609
0.293
0.377
0.475
Table 1. Quantitative results on the Laval Indoor and Laval Outdoor datasets, underlined entries indicate the second-best results.
Figure 3. Qualitative comparison on the Laval Indoor and Laval Outdoor datasets. (Please zoom in to see details)
Scene
Method
RMSE ↓
si-RMSE ↓
Angular Error ↓
SSIM ↑
Diffuse
Matte
Mirror
Diffuse
Matte
Mirror
Diffuse
Matte
Mirror
Diffuse
Matte
Mirror
Infinigen (single)
DiffusionLight (CVPR’24)
0.157
0.228
0.460
0.066
0.180
0.428
2.935
4.053
5.394
0.947
0.679
0.471
4DLighting (SIGGRAPH’25)
0.164
0.207
0.430
0.043
0.134
0.381
1.457
2.418
3.595
0.972
0.754
0.533
Ours
0.110
0.149
0.369
0.042
0.108
0.329
2.762
3.382
4.594
0.973
0.832
0.679
In-the-wild (single)
DiffusionLight (CVPR’24)
0.172
0.236
0.351
0.081
0.193
0.310
1.592
2.681
3.856
0.923
0.654
0.447
4DLighting (SIGGRAPH’25)
0.141
0.190
0.289
0.048
0.132
0.238
1.186
1.904
2.865
0.969
0.755
0.519
Table 2. Quantitative comparison for spatially-varying lighting estimation. The results are reported in terms of RMSE, si-RMSE, angular error, and SSIM.
Figure 4. Qualitative comparison of spatially-varying lighting estimation from a single-view image.
Scene
Method
RMSE ↓
si-RMSE ↓
Angular Error ↓
N-RMSE ↓
Diffuse
Matte
Mirror
Diffuse
Matte
Mirror
Diffuse
Matte
Mirror
Diffuse
Matte
Mirror
Laval Indoor
Single-only
0.268
0.287
0.409
0.053
0.130
0.259
3.654
4.359
6.436
0.231
0.396
0.508
Infinigen-only
0.304
0.318
0.458
0.052
0.128
0.264
3.662
4.545
6.826
0.197
0.401
0.557
w/o SigLIP
0.202
0.242
0.362
0.058
0.129
0.249
4.776
5.790
8.743
0.195
0.347
0.450
Ours
0.184
0.218
0.351
0.046
0.120
0.252
2.700
3.773
6.257
0.180
0.319
0.411
In-the-wild
Single-only
0.502
0.457
0.503
0.056
0.123
0.200
2.525
2.975
3.619
0.228
0.543
0.647
Table 3. Ablation results on Laval indoor and in-the-wild scenes. Results are reported in terms of RMSE, si-RMSE, angular error, and normalized RMSE.
Figure 5. Qualitative comparison of ablations.
Figure 6. Qualitative comparison of spatially-varying lighting estimation from multi-view images.
Figure 7. Qualitative comparison of object insertion on the Laval Indoor dataset (top) and the Laval Indoor Spatially Varying HDR dataset (below). (Please zoom in to see details)
Figure 8. Visualizations of object insertion and the estimated HDRI maps from multi-view input images.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Scene
Method
DR-Error ↓
JS ↓
EMD ↓
Laval Indoor
StyleLight
5.541
0.097
0.718
DiffusionLight
3.538
0.094
0.477
LuxDiT
2.960
0.082
0.471
Ours
2.626
0.059
0.379
Laval Outdoor
StyleLight
9.996
0.106
0.663
DiffusionLight
1.688
0.102
0.460
Appendix
Table 4. Quantitative comparison for dynamic range on the Laval Indoor and Laval Outdoor datasets. Lower is better for all metrics.
Dataset
Method
Scale-invariant RMSE ↓
Angular Error ↓
Normalized RMSE ↓
Diffuse
Matte
Mirror
Diffuse
Matte
Mirror
Diffuse
Matte
Mirror
Laval Indoor
StyleLight
0.031
0.049
0.084
3.861
3.908
3.997
0.166
0.259
0.328
DiffusionLight
0.030
0.050
0.094
3.193
3.236
3.318
0.160
0.211
0.233
LuxDiT
0.026
0.043
0.086
3.014
3.066
3.137
0.140
0.227
0.303
Ours
0.025
0.041
0.083
2.957
3.011
3.101
0.126
0.212
0.257
Laval Outdoor
StyleLight
0.099
0.122
0.171
5.922
5.983
6.140
0.269
0.253
0.239
Appendix
Table 5. Quantitative results on the Laval Indoor and Laval Outdoor datasets for the 3-Sphere-Shadow setting with shadow. Underlined entries indicate the second-best results.
Method
si-RMSE ↓
Angular-Error ↓
EverLight
0.091
6.360
StyleLight
0.123
7.090
DiffusionLight
0.090
5.250
LuxDiT
0.089
4.900
Ours
0.087
4.474
Appendix
Table 6. Quantitative results under the array of spheres protocol on the Laval Indoor dataset.
Method
Chess room
Small class
PU21-PSNR ↓
PU21-PSNR ↓
PanoHDR-NeRF
27.61
28.57
GasLight
30.29
30.77
Ours
31.28
34.70
Appendix
Table 7. Quantitative comparison for spatially-varying lighting estimation on PanoHDR-NeRF Dataset.
Figure 9. Qualitative results under the array of spheres protocol on the Laval Indoor dataset.
Figure 10. Visualization of the estimated spatially-varying illumination on in-the-wild scenes. The inserted spheres on the left correspond one-to-one to the rendered results on the right: the spheres are ordered from left to right, matching the results arranged from top to bottom.
Figure 11. Visualization of the object insertion results on the Laval Indoor Spatially Varying dataset.
Figure 12. Image conditioned panorama generation results on real-world captures.
Recovering dynamic environment maps from a single in-the-wild video is crucial for photorealistic rendering, yet remains a challenge. Recent video generation models can produce photorealistic scenes with complex lighting, possessing an inherent understanding of lighting. In this paper, we introduce V-LITE (Video generation models are inherent lighting estimators), a framework that unlocks this internal knowledge by reframing lighting estimation as a guided video inpainting task. Inspired by VFX industry practices, we insert a synthetic chrome ball into the scene to compel the model to generate physically plausible reflections from the surrounding spatio-temporal context. To bridge the gap from LDR-native models to the HDR domain, we design an HDR-aware VAE and employ an efficient LoRA-based fine-tuning strategy. We then construct a mixed dataset comprising high-fidelity HDR images to provide realistic HDR priors, and in-the-wild HDR videos to provide dynamic spatio-temporal context. Extensive experiments demonstrate that V-LITE produces temporally coherent HDR environment maps, revealing that modern video diffusion models are not merely synthesizers but also powerful, inherently capable estimators of physical scene lighting.
Ziqi Cai, Shuchen Weng, Kaiqi Liu +5
State Key Laboratory of Multimedia Information Processing and National Engineering Research Center of Visual Technology, School of Computer Science, Peking University · Beijing Academy of Artificial Intelligence · School of Software and Microelectronics, Peking University +1
Ill-posed inverse problems require priors to constrain the solution space toward plausible outcomes. In inverse rendering, learned priors modeling the distribution of natural illumination improve the recovery of scene properties. However, existing models rely on the distant-illumination assumption, representing lighting as a far-field environment map. This limits their applicability to indoor scenes, where illumination is highly spatially varying due to finite-distance emitters, visibility changes, and parallax, all of which are poorly approximated by a single environment map. To address this, we introduce a spatially-aware illumination prior trained on real-world indoor panoramas and their estimated depth. Our variational autoencoder model learns a compact, optimizable latent space that decodes into HDR radiance and depth, parameterizing an area light emitter for direct integration into standard differentiable rendering pipelines. This design bridges the plausibility guarantees of a learned prior with the gradient flow required for downstream optimization. Crucially, by jointly modeling radiance and depth, our prior captures the spatial structure of indoor illumination, instead of treating the light sources as infinitely distant. We demonstrate that this formulation enables spatially-varying illumination modeling and achieves higher-fidelity recovery of indoor lighting compared to existing approaches. Project page: https://andreead-a.github.io/RoomLight
Indoor scene relighting demands photorealism, precise spatial control, and strict multi-view consistency. While diffusion-based image editing models enable semantic lighting manipulation via text prompts, enforcing exact 3D light placement often disrupts their generative priors. We propose Lume-Palette, a progressive framework that leverages semantic lighting priors for spatially controllable multi-view indoor relighting. The approach decouples relighting into two stages: (1) illumination distillation, which extracts canonical illumination palettes from a pretrained diffusion model to preserve realistic material-light interactions, and (2) illumination casting, which explicitly maps target spatial lighting conditions defined from coarse 3D geometry. To efficiently handle dense multi-view and multi-modal inputs, we introduce an asymmetric multi-view conditioning strategy that selectively injects essential spatial context. Experiments on diverse synthetic scenes and real-world scenes demonstrate that Lume-Palette produces photorealistic, spatially controllable, and multi-view consistent relighting results. Project Page: https://cjeen.github.io/lumepalette
Chenjian Gao, Linning Xu, Tianfan Xue
Multimedia Laboratory, The Chinese University of Hong Kong · Shanghai AI Laboratory · CPII under InnoHK