GraftRay: Physically Based Mesh Insertion into Gaussian Splatting Scenes via Path Tracing

Jinxin Ai; Matthias Niessner

cs.CVOct 27, 2026

GraftRay: Physically Based Mesh Insertion into Gaussian Splatting Scenes via Path Tracing

Authors: Jinxin Ai, Matthias Niessner

Organizations: Technical University of Munich

Abstract

Inserting virtual objects into scenes reconstructed via 2D or 3D Gaussian Splatting (2DGS/3DGS) is a key capability for content creation, AR, and synthetic data generation. However, realistic insertion requires the object to be seamlessly lit by the scene and, conversely, to cast physically plausible shadows onto it. Existing approaches either harmonize appearance in image space, sacrificing multi-view consistency, or approximate scene illumination with an environment map at the insertion point, which cannot represent near-field area lights such as windows and lamps that dominate indoor lighting. To address this, we present GraftRay, a unified method for physically consistent insertion of mesh objects into indoor scenes represented by either 2D or 3D Gaussians. We recover scene materials and surface normals with diffusion-based priors, and explicitly model spatially-extended emitters, solving their radiance in closed form. A hybrid renderer then jointly path-traces the scene Gaussians and inserted meshes, producing view-consistent shading, interreflections, and soft shadows. On FIPT, GraftRay sets a new state of the art, outperforming recent baselines such as ComGS and EAG-PT in rendering fidelity and shadow accuracy, while maintaining highly competitive rendering efficiency.

Figures & tables

Figure 2: Overview of GraftRay. From LDR photographs we reconstruct a 3DGS scene (tiles: Gaussians as feature-coloured ellipsoids $|$ render) and lift materials and normals onto it ( Sec. 3.1 ). SAM-marked lights become rectangles ( Sec. 3.2 ); a Gaussian path tracer ( Sec. 3.3 ) renders one basis image per emitter, whose radiances are solved by censored least squares ( Sec. 3.5 ). Objects are inserted by differential compositing ( Sec. 3.6 ).

	Synthetic (path-traced ground truth)														Real		Generated
	FIPT							NFINITE							FIPT-Real	ScanNet++ ∗	SpAItial ∗
Method	$\Delta$ PSNR $\uparrow$	Mask $\uparrow$	IoU $\uparrow$	RMSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$\Delta$ PSNR $\uparrow$	Mask $\uparrow$	IoU $\uparrow$	RMSE $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	VQA ratio $\uparrow$	VQA ratio $\uparrow$	VQA ratio $\uparrow$
DOT3D [ 42 ]	19.33	12.27	0.18	0.39	17.40	0.785	0.409	20.75	11.31	0.16	0.33	20.45	0.861	0.217	0.401	0.443	0.464
MV-CoLight [ 36 ]	25.58	20.27	0.46	0.27	22.90	0.824	0.451	26.53	18.16	0.38	0.19	25.27	0.829	0.332	0.392	0.440	0.509
GS-IR [ 28 ]	26.71	21.18	0.47	0.18	23.81	0.853	0.387	29.10	21.49	0.45	0.17	27.14	0.893	0.210	0.385	0.412	0.478
GI-GS [ 3 ]	26.62	21.48	0.50	0.17	24.16	0.854	0.387	28.19	21.43	0.38	0.20	26.85	0.892	0.210	0.371	0.416	0.512

Table 1: Object insertion. All methods use the same objects, lights and three views per scene. Synthetic scenes are scored against path-traced ground truth (

\Delta

PSNR, Mask, IoU and RMSE measure the insertion effect); real and generated scenes, which have none, by the no-reference VQAScore ratio. GS-IR, GI-GS, IRGS: their recovered light in our renderer. –: not run. ∗ Three of the twelve real and generated scenes (ScanNet++ 8b5caf3398, a24858e51e; Plant-Wall) use preliminary object placements; GO-Renderer on ScanNet++ is evaluated on a subset of the views.

Figure 3: Qualitative comparison on synthetic scenes , cropped around the inserted objects. First column: input; last: ground truth (denoised for display). Ours reproduces object brightness and contact shadows; the baselines miss shadows or misjudge brightness.

Figure 4: Qualitative comparison on real and generated scenes (columns as in Fig. 3 without ground truth and EAG-PT, which needs linear HDR inputs).

Figure 5: Recovered materials on the FIPT bedroom (ours: RGB $\leftrightarrow$ X prior) and scores averaged over the four FIPT rooms (albedo scale-aligned). Grey tiles, –: not produced. GS-IR’s all-zero metallic scores well as only $1$ – $6\%$ of the pixels are metal.

Variant	$\Delta$ PSNR $\uparrow$	Mask $\uparrow$	IoU $\uparrow$	RMSE $\downarrow$
Point lights	26.5	20.5	0.51	0.12
Env. map (PT/3DGS)	27.6/27.4	22.0/20.7	0.56/0.55	0.11/0.12
Bounces 1/2	25.7/30.8	14.5/24.9	0.25/0.63	0.33/0.12
Bounces 3/4	32.9/34.0	28.4/30.2	0.68/0.71	0.10/0.09
GT window geometry	35.4	31.7	0.80	0.08
Size $\times$ 0.8/1.25/1.5	34.2/35.4/35.4	31.1/31.4/31.3	0.80	0.08

Table 2: Ablations on the bedroom. Shifts are in the window plane unless marked.

Panel of Figure 6 — Figure 6: Ablations, qualitative (bedroom), one row per ablation; composites unless marked PT. Point lights over-brighten, an environment map darkens; direct light is far too dark; without the object-to-room bounce the wall misses reflected light; a gray albedo distorts the room (PT), compositing hides it. Last rows: ours and GT, two views.

3D Scene Editing

3D scene editing studies how to make controlled changes to objects, appearance, layout, and motion while keeping a scene coherent across views.

See all 90 papers

LightHarmony3D: Harmonizing Illumination and Shadows for Object Insertion in 3D Gaussian Splatting
Mar 31, 2026Tianyu Huang, Zhenyang Ren, Zhenchen Wan +53D Scene Editing 3D Gaussian Splatting
The University of Sydney, Sydney, Australia · The University of Melbourne, Melbourne, Australia
VisibilitySplat: Ray-Traced Occlusion for Relighting with 3D Gaussian Splatting
Oct 14, 2026Sven Kluge, Oliver StaadtInverse Rendering 3D Gaussian Splatting
University of Rostock, Albert-Einstein-Straße 22, 18059 Rostock, Germany
Stochastic Ray Tracing for the Reconstruction of 3D Gaussian Splatting
Mar 24, 2026Peiyu Xu, Xin Sun, Krishna Mullia +33D Gaussian Splatting Differentiable Ray Tracing
University of Illinois Urbana-Champaign · Adobe Research · Canva Research

Mar 31, 2026J/K move · Enter open · S save

LightHarmony3D: Harmonizing Illumination and Shadows for Object Insertion in 3D Gaussian Splatting

Tianyu Huang, Zhenyang Ren, Zhenchen Wan, Jiyang Zheng+4

The University of Sydney, Sydney, Australia · The University of Melbourne, Melbourne, Australia

3D Scene Editing 3D Gaussian Splatting

Open paper AskPDF arXiv

3D Gaussian Splatting (3DGS) enables high-fidelity reconstruction of scene geometry and appearance. Building on this capability, inserting external mesh objects into reconstructed 3DGS scenes enables interactive editing and content augmentation for immersive applications such as AR/VR, virtual staging, and digital content creation. However, achieving physically consistent lighting and shadows for mesh insertion remains challenging, as it requires accurate scene illumination estimation and multi-view consistent rendering. To address this challenge, we present LightHarmony3D, a novel framework for illumination-consistent mesh insertion in 3DGS scenes. Central to our approach is our proposed generative module that predicts a full 360° HDR environment map at the insertion location via a single forward pass. By leveraging generative priors instead of iterative optimization, our method efficiently captures dominant scene illumination and enables physically grounded shading and shadows for inserted meshes while maintaining multi-view coherence. Furthermore, we introduce the first dedicated benchmark for mesh insertion in 3DGS, providing a standardized evaluation framework for assessing lighting consistency and photorealism. Extensive experiments across multiple real-world reconstruction datasets demonstrate that LightHarmony3D achieves state-of-the-art realism and multi-view consistency.

Abstract

3D Scene Editing

Explore similar work

LightHarmony3D: Harmonizing Illumination and Shadows for Object Insertion in 3D Gaussian Splatting

VisibilitySplat: Ray-Traced Occlusion for Relighting with 3D Gaussian Splatting

Stochastic Ray Tracing for the Reconstruction of 3D Gaussian Splatting