LightFuse: Relightable Interactive Gaussian Scene Reconstruction via Multi-Scan Fusion and 2D Gaussian Ray Tracing

Haonan Zhou; Gaoxiang Linghu; Youlin Jia; Hongyu Cui; Kewei Wei; Kaiyue Zhou; Bruce X. B. Yu; Gaoang Wang

cs.CVAug 29, 2026

LightFuse: Relightable Interactive Gaussian Scene Reconstruction via Multi-Scan Fusion and 2D Gaussian Ray Tracing

Authors: Haonan Zhou, Gaoxiang Linghu, Youlin Jia, Hongyu Cui, Kewei Wei, Kaiyue Zhou, Bruce X. B. Yu, Gaoang Wang

Organizations: ZJU-UIUC Institute, Zhejiang University · College of Mathematics, Sichuan University · College of Computer Science and Technology, Zhejiang University · Chengdu Minto Tech

Abstract

Relightable interactive scene reconstruction aims to build an editable 3D model from scans of different object arrangements and render new layouts under novel illumination. Existing methods either bake lighting into appearance or recover material and illumination only for fixed scenes, leaving edited layouts with inconsistent shadows and indirect lighting. We present LightFuse, a 2D Gaussian framework that extends interactive scene reconstruction with explicit material-illumination decomposition and physically based relighting. LightFuse first fuses observations across states to reconstruct a shared background and movable objects. It then conducts ray-tracing-oriented geometry refinement to produce more complete and consistent surfaces. On the refined geometry, staged training with differentiable one-bounce ray tracing separates shared metallic--roughness material from state-specific environment lighting. The resulting scene supports object rearrangement, material editing, and relighting, while ray tracing recomputes appearance after each interaction. Experiments across synthetic scenes demonstrate state-of-the-art relighting quality, outperforming the strongest baseline by +9.74,dB PSNR and +0.121 SSIM on average. Project page: https://zhn202.github.io/LightFuse/

Explore similar work

F-RNG: Feed-Forward Relightable Neural Gaussians
May 25, 2026Guangming Fu, Jiahui Fan, Jian Yang +23D Gaussian Splatting 3D Reconstruction
Nankai University, China · Nanjing University, China · NVIDIA, USA
Casual Flash Lighting for Gaussian Splat Inverse Rendering
Oct 5, 2026Jiamin Xu, Dongheng Wei, Jiarong Zhao +4Inverse Rendering 3D Gaussian Splatting
Hangzhou Dianzi University, China · North China Electric Power University, China · Brown University, USA +1
φ-RIE: From Photorealistic Reconstruction to Interactive Environments
Sep 22, 2026Runyi Yang, Deheng Zhang, Xiaoye Wang +63D Gaussian Splatting 3D Scene Editing
INSAIT, Sofia University “St. Kliment Ohridski”. · vivo Mobile Communication Co., Ltd. · Karlsruhe Institute of Technology

May 25, 2026, cs.GRJ/K move · Enter open · S save

F-RNG: Feed-Forward Relightable Neural Gaussians

Guangming Fu, Jiahui Fan, Jian Yang, Miloš Hašan+1

Nankai University, China · Nanjing University, China · NVIDIA, USA

3D Gaussian Splatting 3D Reconstruction Neural Rendering Sparse-View 3D Reconstruction Image Relighting Feed-Forward 3D Reconstruction

Open paper AskPDF arXiv

Capturing relightable 3D assets from real-world objects is a widely researched problem. Several per-scene optimization-based methods, based on 3D Gaussian splatting (3DGS), support relighting; however, they usually require dense input views, and their overfitting nature makes it difficult to generalize across scenes. Unlike per-scene optimization methods, generalized feed-forward models can directly reconstruct Gaussians from sparse input views. However, the resulting assets have baked-in illumination and cannot be easily used for relighting. In this paper, we present F-RNG, a feed-forward framework that directly generates relightable 3DGS assets from sparse-view inputs. Training such a model from scratch can require massive data and computing resources, and it is especially challenging to generate relightable assets in a feed-forward manner with acceptable cost. We develop F-RNG upon an existing large reconstruction model (LRM) to extract relightable representations, while also utilizing priors from an intrinsic decomposition model (IDM). Specifically, we first introduce a latent-interpolated fine-grained geometry synthesis to enhance the LRM's geometry representation. Second, we propose a prior-guided relightable appearance distillation to extract relightable neural representations by incorporating IDM priors. Finally, a universal neural renderer enables flexible and high-fidelity relighting. F-RNG requires neither re-training nor fine-tuning of the underlying LRMs, thus can automatically benefit from better LRMs and IDMs in the future. With only small networks that can be trained with affordable data and computational resources, F-RNG avoids the repetitive inference of large models under different light conditions. By comparison to the state-of-the-art LRM-based relighting method, F-RNG achieves ~~25x faster relighting, as well as superior quality (~~+2.0 dB).

Abstract

Explore similar work

F-RNG: Feed-Forward Relightable Neural Gaussians

Casual Flash Lighting for Gaussian Splat Inverse Rendering

φ-RIE: From Photorealistic Reconstruction to Interactive Environments