Directionally Factorized Illumination with Spherical Harmonic Rotation
Authors: Yimeng Dou, Adam M. Saunders, Andre T. S. Hucke, Michael E. Kim, Adam W. Anderson, Eric R. Tkaczyk, Bennett A. Landman, Daniel Moyer
Organizations: Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, USA · Department of Computer Science, Vanderbilt University, Nashville, TN, USA · Department of Biomedical Engineering, Vanderbilt University, Nashville, TN, USA · Department of Radiology and Radiological Sciences, Vanderbilt University Medical Center, Nashville, TN, USA · Vanderbilt University Institute of Imaging Science, Vanderbilt University Medical Center, Nashville, TN, USA · Department of Dermatology, Vanderbilt University Medical Center · Veterans Affairs Tennessee Valley Healthcare System
Many real-world capture scenarios involve an active light source rigidly attached to the camera: camera flashes for dark-environment imaging, underwater vehicle lights, and endoscopic/laparoscopic medical procedures. As the camera moves, the co-mounted light moves with it, causing the illumination direction at each surface point to change across views. Neural rendering methods (NeRF, Gaussian Splatting) encode this pose-dependent illumination as surface coloration, producing appearance inconsistencies and making it harder to recover the underlying scene. We present DaFISH, which can be plugged into any novel-view synthesis (NVS) pipeline such as NeRF. It factorizes per-point emitted color into surface albedo and a spherical harmonic (SH)-represented illumination factor. During training, the SH illumination is evaluated directly, enabling stable joint optimization of albedo, lighting, and geometry. DaFISH provides analytical generalization of illumination to arbitrary camera poses without retraining. Building on the closed-form SH-to-irradiance conversion, the training pipeline is fully differentiable from SH evaluation through volume rendering. Averaged over the held-out frames of the five scenes, DaFISH attains the best mean PSNR (38.91,dB), SSIM (0.9613), and LPIPS (0.0582) among all compared methods. It improves over the strongest splatting baseline as well as the closest NeRF-based baseline, while recovering the scene consistently across views compared to prior methods.
Figures & tables
Fig. 1: (a) Robots that operate in dark environments, such as subterranean exploration (SubT challenge) [ 1 ] , deep-water surveys by the remotely operated vehicle Hercules [ 2 ] , and nighttime driving [ 3 ] rely on onboard lights that move with the platform. (b) In standard novel view synthesis (NVS, left), hundreds of images are captured while the camera moves around the scene under fixed illumination field L , and a radiance field is reconstructed with methods such as NeRF, 3DGS, or their derivatives. However, in co-located capture (right), the camera and a light source are rigidly attached, so the surface illumination changes with every pose, as the light source is moving as well. We model this moving illumination through the irradiance E(n) , so the irradiance for a given normal is computed by analytically rotating the coefficients with a Wigner-D matrix of the known camera motion, where the residual spatial variation from the near-field source is absorbed by a learned per-sample MLP (see III for more details). (c) DaFISH can reconstruct the scenes while attributing frame-varying brightness to the illumination branch. The brightness of ground truth and rendered images is increased by 40% for visualization.
Fig. 2: Overview of the proposed method. Given a co-located capture that produces N images I and the extrinsic camera parameter matrix c2w (I. Capture), the method factors appearance into albedo and illumination before volumetric rendering. The albedo branch (IIa) samples a global coordinate position x through a scene MLP to predict per-sample albedo ρ(x) , density σ(x) , and a global normal n from the density gradient. The illumination branch (IIb) starts from a single trainable l=4 SH code Esh∈R25×3 shared across all frames, and rotates it analytically into each frame via the Wigner-D matrix D(l) derived from the known rotation Rt∈SO(3) , so that the rotated coefficients become Et . Albedo ρ(x) and illumination factor α(xc,nc) are multiplied into the factorized radiance ρ′=α⊙ρ and composited through volumetric rendering, producing a full render C(r) , a clean albedo render ∑iwiρ (i.e., a render without illumination effects), and an illumination map α .
Lab1
Lab2
Outdoor1
Outdoor2
FourLights
Method
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
2DGS [ 7 ]
28.52
0.9363
0.1629
29.70
0.9212
0.1844
26.23
0.9268
0.2330
36.97
0.9352
0.1717
27.28
0.7777
0.1644
3DGS [ 6 ]
30.08
0.9463
0.1017
29.82
0.8626
0.1697
25.17
0.8723
0.2117
37.86
0.9505
0.0920
27.30
0.7637
0.1620
GS-W [ 24 ]
37.03
0.9582
0.0851
29.14
0.8444
0.2393
25.69
0.8685
0.2238
33.47
0.8251
0.1508
27.84
0.8778
0.2043
DarkGS [ 8 ]
38.15
0.9508
0.1249
39.03
0.9587
0.1385
35.25
0.9546
0.1909
41.22
0.9506
0.1708
n/a
SpotlessGS [ 21 ]
35.01
0.9573
0.1107
36.80
0.9642
0.1236
30.44
0.9584
0.1683
40.93
0.9595
0.1578
25.51
0.8182
0.1720
TABLE I: NVS results on five co-located captures of the DarkRobotic dataset [ 8 ] and FourLights [ 10 ] . The relative-illumination baseline ( [ 10 ] ) failed the Outdoor2 reconstruction before reaching the maximum iteration and is reported with n/a there. Best per scene in bold , second-best in italics .
Fig. 3: Qualitative comparison across three co-located captures (rows: Lab1, Outdoor1, and FourLights) and all methods plus ground truth (GT) are split over two stacked panels of four columns (top 4: 2DGS [ 7 ] , 3DGS [ 6 ] , GS-W [ 24 ] , DarkGS [ 8 ] ; bottom 4: SpotlessGS [ 21 ] , relative-illumination [ 10 ] , DaFISH, and GT). The red bounding box highlights the reconstructed detail. Our factorization keeps albedo stable while the directional SH irradiance travels with the camera.
Fig. 4: Clean scene recovery on held-out frames across three DarkRobotic scenes (columns). For each scene, a held-out GT frame (top) is shown next to the clean albedo (no illumination effect render) of the baselines (SpotlessGS [ 21 ] , relative-illumination [ 10 ] ) and DaFISH.
Fig. 5: The row shows the original capture (left) next to the learned illumination field (right) at the same camera orientation.
Variation
PSNR ↑
SSIM ↑
LPIPS ↓
Full model (Outdoor1)
38.71
0.9687
0.0667
relative-illumination (Outdoor1)
33.67
0.9321
0.1563
SH order l=1 (FourLights)
25.83
0.9148
0.0954
SH order l=2 (FourLights)
27.30
0.9197
0.0924
SH order l=4 (FourLights)
28.46
0.9276
0.0880
TABLE II: Ablation of the DaFISH components on Outdoor1 and FourLights.
Recent advances in neural scene representations enable photorealistic novel-view synthesis, yet most methods remain tightly coupled to a single rendering paradigm, limiting their versatility and integration with conventional graphics workflows. We introduce Floating Radiance Networks (FlaRe), a neural scene representation combining explicit ray-traceable geometry with continuous neural radiance functions. A scene is represented by floating planar generalized Gaussian primitives, each carrying a compact latent descriptor of a local radiance field. A lightweight decoder shared across the scene maps this descriptor, local surface coordinates, and viewing direction to color and opacity. This formulation preserves the expressiveness of neural fields while providing an explicitly addressable structure that can be efficiently queried and manipulated. Hardware-accelerated primitive intersections enable interactive rendering and recursive ray-tracing, including reflections, refractions, transparency, and shadows. The same representation further supports primitive-level deformation, mesh extraction, and appearance stylization directly in its learned descriptor space. Experiments across standard reconstruction benchmarks demonstrate competitive rendering quality while using a compact set of primitives. Together, these results establish FlaRe as a versatile representation that brings high-fidelity neural rendering, ray-tracing, geometric manipulation, and appearance editing into a unified scene model. Source code is available online. Source code can be found at: https://github.com/KByrski/FlaRe
Krzysztof Byrski, Rafał Tobiasz, Grzegorz Wilczyński +5
Jagiellonian University, Faculty of Mathematics and Computer Science · IDEAS Research Institute · Poznań University of Technology, Institute of Robotics and Machine Intelligence
Although there has been significant progress in neural radiance fields, an issue on dynamic illumination changes still remains unsolved. Different from relevant works that parameterize time-variant/-invariant components in scenes, subjects' radiance is highly entangled with their own emitted radiance and lighting colors in spatio-temporal domain. In this paper, we present a new effective method to learn disentangled neural fields under the severe illumination changes, named RehearsalNeRF. Our key idea is to leverage scenes captured under stable lighting like rehearsal stages, easily taken before dynamic illumination occurs, to enforce geometric consistency between the different lighting conditions. In particular, RehearsalNeRF employs a learnable vector for lighting effects which represents illumination colors in a temporal dimension and is used to disentangle projected light colors from scene radiance. Furthermore, our RehearsalNeRF is also able to reconstruct the neural fields of dynamic objects by simply adopting off-the-shelf interactive masks. To decouple the dynamic objects, we propose a new regularization leveraging optical flow, which provides coarse supervision for the color disentanglement. We demonstrate the effectiveness of RehearsalNeRF by showing robust performances on novel view synthesis and scene editing under dynamic illumination conditions. Our source code and video datasets will be publicly available.
Changyeon Won, Hyunjun Jung, Jungu Cho +3
Department of AI Convergence, GIST, Gwangju, Korea. · AI R&D Division, CJ Corporation, Seoul, Korea. · Department of Artificial Intelligence, Yonsei University, Seoul, Korea.
Ill-posed inverse problems require priors to constrain the solution space toward plausible outcomes. In inverse rendering, learned priors modeling the distribution of natural illumination improve the recovery of scene properties. However, existing models rely on the distant-illumination assumption, representing lighting as a far-field environment map. This limits their applicability to indoor scenes, where illumination is highly spatially varying due to finite-distance emitters, visibility changes, and parallax, all of which are poorly approximated by a single environment map. To address this, we introduce a spatially-aware illumination prior trained on real-world indoor panoramas and their estimated depth. Our variational autoencoder model learns a compact, optimizable latent space that decodes into HDR radiance and depth, parameterizing an area light emitter for direct integration into standard differentiable rendering pipelines. This design bridges the plausibility guarantees of a learned prior with the gradient flow required for downstream optimization. Crucially, by jointly modeling radiance and depth, our prior captures the spatial structure of indoor illumination, instead of treating the light sources as infinitely distant. We demonstrate that this formulation enables spatially-varying illumination modeling and achieves higher-fidelity recovery of indoor lighting compared to existing approaches. Project page: https://andreead-a.github.io/RoomLight