cs.CVApr 13, 2026
SaveLumiMotion: Improving Gaussian Relighting with Scene Dynamics
Organizations: Warsaw University of Technology · Sano Centre for Computational Medicine · Inst. for Biomedical Informatics, Univ. Hospital of Cologne · University of Cologne ·
Abstract
In 3D reconstruction, the problem of inverse rendering, namely recovering the illumination of the scene and the material properties, is fundamental. Existing Gaussian Splatting-based methods primarily target static scenes and often assume simplified or moderate lighting to avoid entangling shadows with surface appearance. This limits their ability to accurately separate lighting effects from material properties, particularly in real-world conditions. We address this limitation by leveraging dynamic elements - regions of the scene that undergo motion - as a supervisory signal for inverse rendering. Motion reveals the same surfaces under varying lighting conditions, providing stronger cues for disentangling material and illumination. This thesis is supported by our experimental results which show we improve LPIPS by 23% for albedo estimation and by 15% for scene relighting relative to next-best baseline. To this end, we introduce LumiMotion, the first Gaussian-based approach that leverages dynamics for inverse rendering and operates in arbitrary dynamic scenes. Our method learns a dynamic 2D Gaussian Splatting representation that employs a set of novel constraints which encourage the dynamic regions of the scene to deform, while keeping static regions stable. As we demonstrate, this separation is crucial for correct optimization of the albedo. Finally, we release a new synthetic benchmark comprising five scenes under four lighting conditions, each in both static and dynamic variants, for the first time enabling systematic evaluation of inverse rendering methods in dynamic environments and challenging lighting. Link to project page: https://joaxkal.github.io/LumiMotion/
Figures & tables
Figure 1: Qualitative results for LumiMotion and IRGS [ 13 ] on a real-world dataset. Note that thanks to modeling the scene dynamics LumiMotion can successfully remove shadows from the albedo, leading to improved relighting. IRGS struggles to correctly remove shadows from the albedo, and the specular component is not properly optimized. To allow for comparison of LumiMotion and IRGS in absence of the latter’s specularity estimation errors we show some of IRGS results with diffuse component only.
| Relightable methods | ||||
|---|---|---|---|---|
| Method Name | Supports Dynamics | Object-Agnostic | Unknown Train Light | Natural Train Light |
| GS 3 [ 2 ] | ✗ | ✓ | ✗ | ✗ |
| R-3DGS [ 11 ] , GI-GS [ 7 ] , IR-GS [ 13 ] | ✗ | ✓ | ✓ | ✓ |
| TensorIR [ 20 ] , NeRFactor [ 52 ] | ✗ | ✓ | ✓ | ✓ |
| Relightable Neural Actor [ 30 ] | ✓ | ✗ | ✗ | ✓ |
| Gaussian Codec Avatars [ 36 ] & [ 39 ] | ✓ | ✗ | ✗ | ✗ |
Table 1: Overview of the representative relightable methods and datasets. Existing methods often target static scenes or focus on human avatars, assume known lights or require unnatural lighting setup. In contrast, LumiMotion uniquely performs inverse rendering of generic dynamic scenes under unknown natural lighting. We also show that none of the available datasets with known GT lighting meet the requirements for the tackled setup, which we address with our newly released dataset.
Figure 2: Method overview. In Stage 1, a dynamic representation of 2D Gaussians, tailored for the relighting task, is trained. The canonical color from Stage 1 serves as a starting point for albedo and is further optimized in Stage 2, together with roughness and the unknown environment map. In Stage 2, we rasterize the albedo, roughness and normal maps, then perform stratified sampling from the environment map and apply ray tracing to compute per-pixel visibility and indirect light . L1 Loss is computed between the rendered pixels and the ground truth pixels.
Figure 3: Qualitative comparison of novel view synthesis, relighting, components and estimated lighting. All renders are shown from a novel viewpoint. Our method not only achieves stronger relighting capabilities and effectively removes shadows from albedo, but also estimates lighting more accurately. The presented environment map (top shows the map truncated to while bottom shows it scaled so that the maximum value is ) captures the main light direction more accurately and relights new item more realistically than environment maps estimated from static baselines. Please zoom in for details.
Figure 4: Qualitative evaluation of components for a dynamic scene for different timesteps. LumiMotion maintains consistent normals across timesteps, accurately separates static elements, and produces temporally consistent relighting.
| Env Lights | Method | Albedo | Relight | ||||
|---|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | ||
| Dam Wall | R-3DGS | 20.744 0.661 | 0.900 0.013 | 0.128 0.031 | 21.220 1.843 | 0.915 0.016 | 0.112 0.028 |
| GI-GS | 20.943 1.747 | 0.906 0.014 | 0.105 0.023 | 18.434 1.681 | 0.868 0.023 | 0.139 0.032 | |
| Harbour Sunset | IR-GS | 22.888 1.559 | 0.936 0.013 | 0.076 0.023 | 26.177 1.606 | 0.953 0.011 | 0.064 0.018 |
| LumiMotion | 27.268 1.568 | 0.952 0.007 | 0.069 0.024 | 26.037 0.579 | 0.928 0.007 | 0.060 0.012 | |
| Chapel Day | R-3DGS | 22.463 2.001 | 0.927 0.017 | 0.096 0.038 | 22.282 2.806 | 0.943 0.012 | 0.081 0.030 |
Table 2: Quantitative results for albedo estimation and relighting. Static methods use only one timestep with multiple views of the scene. LumiMotion uses a dynamic scene for training while testing is done on the same views and the same single timestep as in the static setting. Results are grouped by train-test light setting. Although training on dynamic scenes is inherently more challenging , LumiMotion significantly outperforms albedo estimation tasks on all metrics across all baselines. Notably, for both tasks, we achieve a substantial improvement in LPIPS which is a metric that corresponds well to perceptual quality.
Figure 5: LumiMotion on real-world ENeRF scene. The sharp shadow cast by the dynamic actor is largely removed from albedo. Under novel lighting, the actor appears realistic, casting shadows in appropriate locations.
Figure 6: Influence of separation loss parameters. The dynamic objects in the scene remain correctly classified as dynamic across tested separation weights and initialization times. However, shadows on static parts of the scene (e.g. ground) can be misclassified as dynamic Gaussians (here mostly visible for start 10,000 iter). Increasing the separation weight or starting the separation earlier helps to reduce such misclassifications. Zoom in for details.
Figure 7: Ablation study of static-dynamic separation. The demonstrated scene is characterized with strong directional light and strong shadows. Without the static-dynamic separation (top), the Gaussians reconstruct a strong shadow that follows the object motion, which degrades albedo optimization in the second stage . When separation is enabled, the shadows are modeled by instead of which enables correct albedo optimization. Please zoom in for details.
| Method | Albedo | Relight | ||||
|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | |
| w/o Stage 2 | 23.935 | 0.943 | 0.072 | 23.154 | 0.926 | 0.060 |
| w/o | 23.983 | 0.937 | 0.083 | 23.142 | 0.926 | 0.061 |
| w/o | 26.768 | 0.954 | 0.064 | 25.079 | 0.933 | 0.052 |
| full model | 27.929 | 0.959 | 0.058 | 25.405 | 0.936 | 0.048 |
Table 3: Ablation study on Golden Bay → Dam Wall configuration, characterized by strong directional train light. The additional components improve albedo and relight reconstruction quality.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Roughness estimation. Baseline methods fail to recover realistic roughness maps. For IRGS, the standard loss set yields roughness values with little to no surface detail (see the ‘full model‘ column). Removing smoothness constraints causes different issues, for example: in the top row, the plate is estimated as specular and the items as rough, whereas the ground truth shows the opposite. Other baselines also fail to disentangle material properties and estimate roughness reliably. In contrast, LumiMotion produces smooth and largely accurate roughness maps.
| Method | Novel View Synthesis | Albedo | Roughness | Relight | PSNR NVS Relight | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| PSNR | SSIM | LPIPS | PSNR | SSIM | LPIPS | MSE | PSNR | SSIM | LPIPS | ||
| Dam Wall Harbour Sunset | |||||||||||
| R-3DGS | |||||||||||
| GI-GS | |||||||||||
| IR-GS | |||||||||||
| IR-GS | |||||||||||
Table 4: LumiMotion for novel view synthesis, material estimation, and relighting. Static methods use only one timestep, while LumiMotion operates on a dynamic scene. Testing is performed on the same views and timestep as in the static setting. Results are grouped by train-test lighting conditions. For IRGS, we denote training scheme without smooth constraint on roughness by . NVS : Note that the dynamic setting we use is significantly more challenging than static setup for baselines . Despite this, LumiMotion achieves strong results for albedo and relight, demonstrating the effectiveness of our approach. Please note that the high NVS scores of static baselines are also caused by overfitting to the training lighting conditions. When evaluated under novel illumination, their performance drops significantly. This observation is consistent with our qualitative results. We show PSNR drop in the last column. This highlights the effectiveness of our separation strategy and the consistent behavior of our method across both train and test lighting. Material : We achieve significantly better material estimation than the closest baseline, IRGS, regardless of its training setup. Notably, LumiMotion consistently produces higher-quality albedo and achieves at least a 2× lower roughness MSE compared to IRGS.
Figure 9: DNA - Actor going to sit on a stool. Please see attached video for this scene.
Figure 10: DNA – Actor cleaning a specular table. Although the actor’s arm casts a strong shadow on the tabletop, the Gaussians on the table are correctly classified as static. Please see attached video for this scene.
Figure 11: DNA - Actor with a hair dryer. Actor’s feet are static and only his upper body parts are moving. Please see attached video for this scene.
Figure 12: Scene 1. Our method produces much clearer albedo than the baselines. Although we train on dynamic scenes—more challenging than the static ones used by baselines—our results still show smooth and accurate geometry and normals. The presented environment map (top shows the map truncated to while bottom shows it scaled so that the maximum value is ) captures the main light direction more accurately than environment maps estimated from static baselines. We mark in color boxes fine details like artifacts including baked-in shadows, which clearly show superiority of our method. We highlight fine details—such as artifacts and baked-in shadows—in color boxes, clearly demonstrating the superiority of our method.
Figure 13: Scene 2. Our method produces much clearer albedo than the baselines. Although we train on dynamic scenes—more challenging than the static ones used by baselines—our results still show smooth and accurate geometry and normals. The presented environment map (top shows the map truncated to , while bottom shows it scaled so that the maximum value is ) captures the main light direction more accurately than environment maps estimated from static baselines. We highlight fine details—such as artifacts and baked-in shadows—in color boxes, clearly demonstrating the superiority of our method.
Figure 14: Scene 3. This scene was captured under mild training illumination, with no strong shadows present. As a result, the static baselines exhibit relatively well-separated albedo in the training image (no baked in shadows). We demonstrate that in this case our method achieves relighting and geometry quality competitive with the latest state-of-the-art IRGS, despite IRGS being a static baseline. The presented environment map (top shows the map truncated to , while bottom shows is scaled so that the maximum value is ) captures the main light direction more accurately than environment maps estimated from static baselines.
Figure 15: Separation - ablation study. Please zoom in for details.
Figure 16: Separation – impact of hyperparameters. Top: Renderings using downsized Gaussians illustrate uneven Gaussian placement on the plate when the separation loss is introduced too late, in 10,000th iteration. In thia case several Gaussians on the plate are incorrectly classified as dynamic and move with the shadows. Bottom: Renderings with full-sized Gaussians highlight the extent of this misclassification. Please zoom in for details.
Figure 17: Example visualization of the ‘jumpingjacks‘ scene at time steps 6, 46, 53, and 86. Note that each time step corresponds to a different camera pose, so the renderings are shown from varying viewpoints.
Figure 18: Environment maps used to light our scenes.
Figure 19: Example ground truth renders of our scenes.
Figure 20: Limitations of simple separation strategy. Some Gaussians between the plate surface and the shoes are neither part of the static plate nor clearly part of the dynamic shoe.
More in this topic
Inverse Rendering
Inverse rendering estimates scene geometry, materials, lighting, and other causes of images from observed visual or sensor data.
See all 68 papers