R3GW: Relightable 3D Gaussians for Outdoor Scenes in the Wild

Margherita Lea Corona; Wieland Morgenstern; Peter Eisert; Anna Hilsmann

cs.CVMar 3, 2026

R3GW: Relightable 3D Gaussians for Outdoor Scenes in the Wild

Authors: Margherita Lea Corona, Wieland Morgenstern, Peter Eisert, Anna Hilsmann

Organizations: Fraunhofer Heinrich Hertz Institute, Berlin, Germany · Humboldt Universität zu Berlin, Berlin, Germany

Abstract

3D Gaussian Splatting (3DGS) has established itself as a leading technique for 3D reconstruction and novel view synthesis of static scenes, achieving outstanding rendering quality and fast training. However, the method does not explicitly model the scene illumination, making it unsuitable for relighting tasks. Furthermore, 3DGS struggles to reconstruct scenes captured in the wild by unconstrained photo collections featuring changing lighting conditions. In this paper, we present R3GW, a novel method that learns a relightable 3DGS representation of an outdoor scene captured in the wild. Our approach separates the scene into a relightable foreground and a non-reflective background (the sky), using two distinct sets of Gaussians. R3GW models view-dependent lighting effects in the foreground reflections by combining Physically Based Rendering with the 3DGS scene representation in a varying illumination setting. We evaluate our method quantitatively and qualitatively on the NeRF-OSR dataset, offering state-of-the-art performance and enhanced support for physically-based relighting of unconstrained scenes. Our method synthesizes photorealistic novel views under arbitrary illumination conditions. Additionally, our representation of the sky mitigates depth reconstruction artifacts, improving rendering quality at the sky-foreground boundary

Figures & tables

Figure 1: R3GW reconstructs a relightable 3D Gaussian Splatting representation of an outdoor scene from a collection of unconstrained multiview images. Our method factorizes the scene’s foreground into geometry, material, and illumination, with the lighting represented as an environment map. This enables illumination editing along with novel view synthesis. The sky, as a non-reflective surface, is modeled with a dedicated set of Gaussians that remain independent of scene lighting and material.

Figure 2: Training pipeline of R3GW. R3GW learns a relightable 3DGS representation of an outdoor scene captured in the wild. The PBR color of the foreground Gaussians, depending on the surface normals and material properties at their positions, as well as on the environment light, enables relighting of the scene’s foreground. In contrast, the color of the sky Gaussians is independent of the scene illumination. The rendered image is formed by rasterizing the sky and the foreground Gaussians in a single pass. The Gaussians are regularized so that the sky Gaussians are responsible for the rendering of the sky pixels, while the foreground Gaussians contribute only to the foreground region. After training, the foreground Gaussians can be rendered under novel lighting conditions by supplying the corresponding environment map as input.

Panel of Figure 3 — Figure 3: Qualitative comparison of the depth maps predicted by our method and LumiGauss [ 16 ] for the NeRF-OSR scenes. For each novel view, the ground truth is visualized in Fig. . Compared to LumiGauss, R3GW produces more geometrically consistent depth maps with significantly reduced artifacts in the sky and in the distant regions of the foreground.

Panel of Figure 4 — Figure 4: Qualitative comparison of rendering quality at the sky-foreground boundary between our method and LumiGauss [ 16 ] for the NeRF-OSR scenes. For a clear visualization, we render the novel views for both R3GW and LumiGauss using trained illuminations. Our sky Gaussians are rendered using the corresponding trained sky color SH. R3GW yields sharper building outlines with fewer halos as a result of an improved depth reconstruction (see Fig. 3 ).

NeRF-based
Method	PSNR $\uparrow$				SSIM $\uparrow$				MSE $\downarrow$				MAE $\downarrow$
Method	lk2	lwp	st	avg	lk2	lwp	st	avg	lk2	lwp	st	avg	lk2	lwp	st	avg
$\text{NeRF-OSR}^{*}$ [ 25 ]	18.72	16.65	15.43	16.93	0.468	0.501	0.517	0.495	0.014	0.024	0.029	0.022	0.090	0.114	0.113	0.106
NeRF-OSR [ 25 ]	18.80	15.30	15.59	16.56	0.501	0.514	0.511	0.509	0.011	0.034	0.027	0.024	0.090	0.132	0.111	0.111
SR-TensoRF [ 2 ]	17.30	16.74	15.63	16.56	0.542	0.653	0.632	0.609	0.021	0.024	0.030	0.025	0.096	0.093	0.111	0.100
FEGR [ 28 ]	21.53	17.57	17.00	18.70	-	-	-	-	0.007	0.018	0.023	0.016	-	-	-	-

Table 1: Quantitative comparison of our method with the baselines on the NeRF-OSR test scenes. For LumiGauss [ 16 ] and NeRF-OSR [ 25 ] the

*

denotes the results presented in the papers by the authors. We group methods into NeRF-based and Gaussian Splatting (GS)-based approaches, as NeRF models typically achieve higher visual fidelity but are more computationally expensive to train and render. The last section shows the evaluation results of the ablation studies described in Sec. 5.3 .

Panel of Figure 5 — Figure 5: Qualitative comparison of the albedo map and surface normals predicted by our method, NeRF-OSR [ 25 ] , and LumiGauss [ 16 ] for the NeRF-OSR scene st . The ground truth novel view is visualized in Fig. . R3GW synthesizes sharper surface normal maps than LumiGauss and NeRF-OSR. Furthermore, our sky Gaussians are not attributed with spurious albedo and surface normals.

Panel of Figure 6 — Figure 6: Novel views of the NeRF-OSR scenes relighted under a novel HDR environment map. The first and second row show renderings with two different sun orientations, produced by projecting onto fourth-order SH the same environment map and rotating it around the y-axis (top-left corner of each row).

Panel of Figure 7 — Figure 7: Novel views of the NeRF-OSR scenes relighted with trained environment maps (top-left corner of the second column).

Figure 8: Initial positions of the foreground and sky Gaussians. The foreground SfM point cloud is augmented with the sky Gaussians initial positions sampled uniformly on the background of the upper hemisphere surrounding the scene.

Panel of Figure 9 — Figure 9: Rendered novel views and illuminations of the NeRF-OSR test scenes (from top to bottom: lk2 , lwp , st ) synthesized by our method, LumiGauss [ 16 ] (shadowed version), and NeRF-OSR [ 25 ] . We show novel views employed for the evaluation, rendered using ground truth environment maps. The sky is masked out in all the renderings.

Panel of Figure 10 — Figure 10: Additional qualitative comparison of the albedo map and surface normals predicted by our method, NeRF-OSR [ 25 ] , and LumiGauss [ 16 ] for the NeRF-OSR scene lk2 . The ground truth novel view is visualized in Fig. 9 .

Panel of Figure 11 — Figure 11: Comparison of two novel views of the NeRF-OSR scene lwp rendered under the same trained lighting and sky color. Insets highlight that the appearance of reflective surfaces, such as windows, varies with viewing direction.

3D Scene Relighting

See all 21 papers

F-RNG: Feed-Forward Relightable Neural Gaussians
May 25, 2026Guangming Fu, Jiahui Fan, Jian Yang +23D Gaussian Splatting 3D Reconstruction
Nankai University, China · Nanjing University, China · NVIDIA, USA
Stochastic Ray Tracing for the Reconstruction of 3D Gaussian Splatting
Mar 24, 2026Peiyu Xu, Xin Sun, Krishna Mullia +33D Gaussian Splatting Differentiable Ray Tracing
University of Illinois Urbana-Champaign · Adobe Research · Canva Research
WildSplatter: Feed-forward 3D Gaussian Splatting with Appearance Control from Unconstrained Images
Apr 23, 2026Yuki Fujimura, Takahiro Kushida, Kazuya Kitano +23D Gaussian Splatting 3D Reconstruction
NAIST · Ritsumeikan University · Kyoto University

May 25, 2026J/K move · Enter open · S save

F-RNG: Feed-Forward Relightable Neural Gaussians

Guangming Fu, Jiahui Fan, Jian Yang, Miloš Hašan+1

Nankai University, China · Nanjing University, China · NVIDIA, USA

3D Gaussian Splatting 3D Reconstruction Neural Rendering Sparse-View 3D Reconstruction Image Relighting Feed-Forward 3D Reconstruction+1

Open paper AskPDF arXiv

Capturing relightable 3D assets from real-world objects is a widely researched problem. Several per-scene optimization-based methods, based on 3D Gaussian splatting (3DGS), support relighting; however, they usually require dense input views, and their overfitting nature makes it difficult to generalize across scenes. Unlike per-scene optimization methods, generalized feed-forward models can directly reconstruct Gaussians from sparse input views. However, the resulting assets have baked-in illumination and cannot be easily used for relighting. In this paper, we present F-RNG, a feed-forward framework that directly generates relightable 3DGS assets from sparse-view inputs. Training such a model from scratch can require massive data and computing resources, and it is especially challenging to generate relightable assets in a feed-forward manner with acceptable cost. We develop F-RNG upon an existing large reconstruction model (LRM) to extract relightable representations, while also utilizing priors from an intrinsic decomposition model (IDM). Specifically, we first introduce a latent-interpolated fine-grained geometry synthesis to enhance the LRM's geometry representation. Second, we propose a prior-guided relightable appearance distillation to extract relightable neural representations by incorporating IDM priors. Finally, a universal neural renderer enables flexible and high-fidelity relighting. F-RNG requires neither re-training nor fine-tuning of the underlying LRMs, thus can automatically benefit from better LRMs and IDMs in the future. With only small networks that can be trained with affordable data and computational resources, F-RNG avoids the repetitive inference of large models under different light conditions. By comparison to the state-of-the-art LRM-based relighting method, F-RNG achieves ~~25x faster relighting, as well as superior quality (~~+2.0 dB).

Abstract

3D Scene Relighting

Explore similar work

F-RNG: Feed-Forward Relightable Neural Gaussians

Stochastic Ray Tracing for the Reconstruction of 3D Gaussian Splatting

WildSplatter: Feed-forward 3D Gaussian Splatting with Appearance Control from Unconstrained Images