LumiMotion: Improving Gaussian Relighting with Scene Dynamics
Authors: Joanna Kaleta, Piotr Wójcik, Kacper Marzol, Tomasz Trzciński, Kacper Kania, Marek Kowalski
Organizations: Warsaw University of Technology · Sano Centre for Computational Medicine · Inst. for Biomedical Informatics, Univ. Hospital of Cologne · University of Cologne · CMMC Cologne · Jagiellonian University · IDEAS Research Institute · Microsoft
In 3D reconstruction, the problem of inverse rendering, namely recovering the illumination of the scene and the material properties, is fundamental. Existing Gaussian Splatting-based methods primarily target static scenes and often assume simplified or moderate lighting to avoid entangling shadows with surface appearance. This limits their ability to accurately separate lighting effects from material properties, particularly in real-world conditions. We address this limitation by leveraging dynamic elements - regions of the scene that undergo motion - as a supervisory signal for inverse rendering. Motion reveals the same surfaces under varying lighting conditions, providing stronger cues for disentangling material and illumination. This thesis is supported by our experimental results which show we improve LPIPS by 23% for albedo estimation and by 15% for scene relighting relative to next-best baseline. To this end, we introduce LumiMotion, the first Gaussian-based approach that leverages dynamics for inverse rendering and operates in arbitrary dynamic scenes. Our method learns a dynamic 2D Gaussian Splatting representation that employs a set of novel constraints which encourage the dynamic regions of the scene to deform, while keeping static regions stable. As we demonstrate, this separation is crucial for correct optimization of the albedo. Finally, we release a new synthetic benchmark comprising five scenes under four lighting conditions, each in both static and dynamic variants, for the first time enabling systematic evaluation of inverse rendering methods in dynamic environments and challenging lighting. Link to project page: https://joaxkal.github.io/LumiMotion/
Figures & tables
Figure 1: Qualitative results for LumiMotion and IRGS [ 13 ] on a real-world dataset. Note that thanks to modeling the scene dynamics LumiMotion can successfully remove shadows from the albedo, leading to improved relighting. IRGS struggles to correctly remove shadows from the albedo, and the specular component is not properly optimized. To allow for comparison of LumiMotion and IRGS in absence of the latter’s specularity estimation errors we show some of IRGS results with diffuse component only.
Relightable methods
Method Name
Supports Dynamics
Object-Agnostic
Unknown Train Light
Natural Train Light
GS 3 [ 2 ]
✗
✓
✗
✗
R-3DGS [ 11 ] , GI-GS [ 7 ] , IR-GS [ 13 ]
✗
✓
✓
✓
TensorIR [ 20 ] , NeRFactor [ 52 ]
✗
✓
✓
✓
Relightable Neural Actor [ 30 ]
✓
✗
✗
✓
Gaussian Codec Avatars [ 36 ] & [ 39 ]
✓
✗
✗
✗
Table 1: Overview of the representative relightable methods and datasets. Existing methods often target static scenes or focus on human avatars, assume known lights or require unnatural lighting setup. In contrast, LumiMotion uniquely performs inverse rendering of generic dynamic scenes under unknown natural lighting. We also show that none of the available datasets with known GT lighting meet the requirements for the tackled setup, which we address with our newly released dataset.
Figure 2: Method overview. In Stage 1, a dynamic representation of 2D Gaussians, tailored for the relighting task, is trained. The canonical color from Stage 1 serves as a starting point for albedo and is further optimized in Stage 2, together with roughness and the unknown environment map. In Stage 2, we rasterize the albedo, roughness and normal maps, then perform stratified sampling from the environment map Lenv and apply ray tracing to compute per-pixel visibility V and indirect light Lind . L1 Loss is computed between the rendered pixels and the ground truth pixels.
Figure 3: Qualitative comparison of novel view synthesis, relighting, components and estimated lighting. All renders are shown from a novel viewpoint. Our method not only achieves stronger relighting capabilities and effectively removes shadows from albedo, but also estimates lighting more accurately. The presented environment map (top shows the map truncated to [0,1] while bottom shows it scaled so that the maximum value is 1 ) captures the main light direction more accurately and relights new item more realistically than environment maps estimated from static baselines. Please zoom in for details.
Figure 4: Qualitative evaluation of components for a dynamic scene for different timesteps. LumiMotion maintains consistent normals across timesteps, accurately separates static elements, and produces temporally consistent relighting.
Env Lights
Method
Albedo
Relight
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Dam Wall
R-3DGS
20.744 ± 0.661
0.900 ± 0.013
0.128 ± 0.031
21.220 ± 1.843
0.915 ± 0.016
0.112 ± 0.028
↓
GI-GS
20.943 ± 1.747
0.906 ± 0.014
0.105 ± 0.023
18.434 ± 1.681
0.868 ± 0.023
0.139 ± 0.032
Harbour Sunset
IR-GS
22.888 ± 1.559
0.936 ± 0.013
0.076 ± 0.023
26.177 ± 1.606
0.953 ± 0.011
0.064 ± 0.018
LumiMotion
27.268 ± 1.568
0.952 ± 0.007
0.069 ± 0.024
26.037 ± 0.579
0.928 ± 0.007
0.060 ± 0.012
Chapel Day
R-3DGS
22.463 ± 2.001
0.927 ± 0.017
0.096 ± 0.038
22.282 ± 2.806
0.943 ± 0.012
0.081 ± 0.030
Table 2: Quantitative results for albedo estimation and relighting. Static methods use only one timestep with multiple views of the scene. LumiMotion uses a dynamic scene for training while testing is done on the same views and the same single timestep as in the static setting. Results are grouped by train-test light setting. Although training on dynamic scenes is inherently more challenging , LumiMotion significantly outperforms albedo estimation tasks on all metrics across all baselines. Notably, for both tasks, we achieve a substantial improvement in LPIPS which is a metric that corresponds well to perceptual quality.
Figure 5: LumiMotion on real-world ENeRF scene. The sharp shadow cast by the dynamic actor is largely removed from albedo. Under novel lighting, the actor appears realistic, casting shadows in appropriate locations.
Figure 6: Influence of separation loss parameters. The dynamic objects in the scene remain correctly classified as dynamic across tested separation weights and initialization times. However, shadows on static parts of the scene (e.g. ground) can be misclassified as dynamic Gaussians (here mostly visible for start 10,000 iter). Increasing the separation weight or starting the separation earlier helps to reduce such misclassifications. Zoom in for details.
Figure 7: Ablation study of static-dynamic separation. The demonstrated scene is characterized with strong directional light and strong shadows. Without the static-dynamic separation (top), the Gaussians reconstruct a strong shadow that follows the object motion, which degrades albedo optimization in the second stage . When separation is enabled, the shadows are modeled by Δc instead of Δμ which enables correct albedo optimization. Please zoom in for details.
Method
Albedo
Relight
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
w/o Stage 2
23.935
0.943
0.072
23.154
0.926
0.060
w/o Δc
23.983
0.937
0.083
23.142
0.926
0.061
w/o P
26.768
0.954
0.064
25.079
0.933
0.052
full model
27.929
0.959
0.058
25.405
0.936
0.048
Table 3: Ablation study on Golden Bay → Dam Wall configuration, characterized by strong directional train light. The additional components improve albedo and relight reconstruction quality.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Roughness estimation. Baseline methods fail to recover realistic roughness maps. For IRGS, the standard loss set yields roughness values with little to no surface detail (see the ‘full model‘ column). Removing smoothness constraints causes different issues, for example: in the top row, the plate is estimated as specular and the items as rough, whereas the ground truth shows the opposite. Other baselines also fail to disentangle material properties and estimate roughness reliably. In contrast, LumiMotion produces smooth and largely accurate roughness maps.
Method
Novel View Synthesis
Albedo
Roughness
Relight
Δ PSNR NVS → Relight
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
MSE ↓
PSNR ↑
SSIM ↑
LPIPS ↓
Dam Wall → Harbour Sunset
R-3DGS
35.031
0.987
0.035
20.744
0.900
0.128
0.066±0.005
21.220
0.915
0.112
39.5%
GI-GS
26.749
0.956
0.066
20.943
0.906
0.105
0.036±0.001
18.431
0.868
0.139
31.1%
IR-GS
32.207
0.983
0.021
22.888
0.936
0.076
0.136±0.040
26.177
0.953
0.064
18.7%
IR-GS †
32.639
0.985
0.019
23.512
0.935
0.080
0.024±0.010
27.156
0.954
0.067
16.8%
Appendix
Table 4: LumiMotion for novel view synthesis, material estimation, and relighting. Static methods use only one timestep, while LumiMotion operates on a dynamic scene. Testing is performed on the same views and timestep as in the static setting. Results are grouped by train-test lighting conditions. For IRGS, we denote training scheme without smooth constraint on roughness by † . NVS : Note that the dynamic setting we use is significantly more challenging than static setup for baselines . Despite this, LumiMotion achieves strong results for albedo and relight, demonstrating the effectiveness of our approach. Please note that the high NVS scores of static baselines are also caused by overfitting to the training lighting conditions. When evaluated under novel illumination, their performance drops significantly. This observation is consistent with our qualitative results. We show PSNR drop in the last column. This highlights the effectiveness of our separation strategy and the consistent behavior of our method across both train and test lighting. Material : We achieve significantly better material estimation than the closest baseline, IRGS, regardless of its training setup. Notably, LumiMotion consistently produces higher-quality albedo and achieves at least a 2× lower roughness MSE compared to IRGS.
Figure 9: DNA - Actor going to sit on a stool. Please see attached video for this scene.
Figure 10: DNA – Actor cleaning a specular table. Although the actor’s arm casts a strong shadow on the tabletop, the Gaussians on the table are correctly classified as static. Please see attached video for this scene.
Figure 11: DNA - Actor with a hair dryer. Actor’s feet are static and only his upper body parts are moving. Please see attached video for this scene.
Figure 12: Scene 1. Our method produces much clearer albedo than the baselines. Although we train on dynamic scenes—more challenging than the static ones used by baselines—our results still show smooth and accurate geometry and normals. The presented environment map (top shows the map truncated to [0,1] while bottom shows it scaled so that the maximum value is 1 ) captures the main light direction more accurately than environment maps estimated from static baselines. We mark in color boxes fine details like artifacts including baked-in shadows, which clearly show superiority of our method. We highlight fine details—such as artifacts and baked-in shadows—in color boxes, clearly demonstrating the superiority of our method.
Figure 13: Scene 2. Our method produces much clearer albedo than the baselines. Although we train on dynamic scenes—more challenging than the static ones used by baselines—our results still show smooth and accurate geometry and normals. The presented environment map (top shows the map truncated to [0,1] , while bottom shows it scaled so that the maximum value is 1 ) captures the main light direction more accurately than environment maps estimated from static baselines. We highlight fine details—such as artifacts and baked-in shadows—in color boxes, clearly demonstrating the superiority of our method.
Figure 14: Scene 3. This scene was captured under mild training illumination, with no strong shadows present. As a result, the static baselines exhibit relatively well-separated albedo in the training image (no baked in shadows). We demonstrate that in this case our method achieves relighting and geometry quality competitive with the latest state-of-the-art IRGS, despite IRGS being a static baseline. The presented environment map (top shows the map truncated to [0,1] , while bottom shows is scaled so that the maximum value is 1 ) captures the main light direction more accurately than environment maps estimated from static baselines.
Figure 15: Separation - ablation study. Please zoom in for details.
Figure 16: Separation – impact of hyperparameters. Top: Renderings using downsized Gaussians illustrate uneven Gaussian placement on the plate when the separation loss is introduced too late, in 10,000th iteration. In thia case several Gaussians on the plate are incorrectly classified as dynamic and move with the shadows. Bottom: Renderings with full-sized Gaussians highlight the extent of this misclassification. Please zoom in for details.
Figure 17: Example visualization of the ‘jumpingjacks‘ scene at time steps 6, 46, 53, and 86. Note that each time step corresponds to a different camera pose, so the renderings are shown from varying viewpoints.
Figure 18: Environment maps used to light our scenes.
Figure 19: Example ground truth renders of our scenes.
Figure 20: Limitations of simple separation strategy. Some Gaussians between the plate surface and the shoes are neither part of the static plate nor clearly part of the dynamic shoe.
Decomposing outgoing surface radiance into material and illumination during inverse rendering is essential for applications such as relighting and augmented reality, yet it is severely ill-posed since multiple combinations can result in the same observed colour. Capturing an object under multiple lighting conditions usually helps resolve this ambiguity as it constrains the optimization towards correct solutions. In this work, we explore the potential of reconstructing rigidly moving objects -- which provides observations of diverse light-surface interactions -- to resolve the material-lighting ambiguity in inverse rendering. For this purpose, we introduce a relightable approach that marries object tracking and reconstruction with inverse rendering for general rigidly moving objects. Our experimental analysis on synthetic data demonstrates that motion can be an advantage for disentangling material and lighting: the reconstructed material is significantly more accurate when the object is observed under rigid motion than when it is static. Moreover, results on RGB videos of real hand-held objects show that our pipeline preserves this advantage even under noisy real-world conditions.
Raza Yunus, Benjamin Ummenhofer, Jan Eric Lenssen +1
University of Technology Nuremberg · Max Planck Institute for Informatics, Saarland Informatics Campus · Intel +1
We address inverse rendering with 3D Gaussian Splatting (3DGS) by introducing explicit visibility modeling for relighting as well as material and light decomposition. Our approach augments the standard rasterized 3DGS pipeline with a sparse visibility oracle that estimates transmittance-weighted occlusion for Gaussian primitives from sampled lighting directions. This oracle is implemented using a point-based ray tracer accelerated by a bounding volume hierarchy (BVH) and queried sparsely within the inverse-rendering objectives, while the original 3DGS loop is retained for training efficiency, interactive visualization, and tooling compatibility. Shading follows a physically based rendering (PBR) formulation with a principled microfacet bidirectional reflectance distribution function (BRDF) and low-resolution high-dynamic-range (HDR) environment maps. We evaluate our method on the TensoIR synthetic dataset against baselines based on neural radiance fields (NeRFs), tensor representations, and 3DGS. The results show competitive performance and improvements over several earlier baselines for novel-view synthesis (NVS), albedo reconstruction, and relighting. Ablations indicate that explicit visibility modeling is an important contributor to relighting and material-illumination disentanglement, while pure PBR supervision can favor appearance fitting in NVS. Overall, the findings suggest that sparse, point-based visibility queries effectively complement rasterized splatting for inverse rendering while preserving the practical advantages of the 3DGS ecosystem.
Sven Kluge, Oliver Staadt
University of Rostock, Albert-Einstein-Straße 22, 18059 Rostock, Germany
Robots operating in dark or poorly lit environments rely on onboard lights, which often produce uneven illumination that degrades downstream perception tasks. Prior approaches based on 2D image enhancement lack reliable supervision and fail to preserve multi-view geometric consistency. To address these limitations, we extend Dark Gaussian Splatting (DarkGS) toward a more accurate and flexible relightable 3D reconstruction framework. First, we eliminate the need for explicit light parameter calibration by jointly optimizing lighting parameters within the Gaussian Splatting framework. Second, we introduce a low-frequency illumination model based on spherical harmonics (SH) to capture spatially varying residual and ambient lighting effects. Third, we incorporate an MLP-based Bidirectional Reflectance Distribution Function (BRDF) to model non-Lambertian reflectance. Experiments on synthetic and real-world datasets demonstrate that our method effectively mitigates illumination artifacts while improving rendering quality and quantitative performance over prior approaches. We further validate its benefits for robotic perception through a downstream task.
Liang Hong, Jiaxin Wei, Simon Schaefer +2
Mobile Robotics Lab, School of Computation, Information and Technology, Technical University of Munich. · Mobile Robotics Lab, Department of Mechanical and Process Engineering, ETH Zurich.