Gaussian Splatting is a powerful tool for reconstructing diffuse scenes, but it struggles to simultaneously model specular reflections and the appearance of objects behind semi-transparent surfaces. These specular reflections and transmittance are essential for realistic novel view synthesis, and existing methods do not properly incorporate the underlying physical processes to simulate them. To address this issue, we propose RT-GS, a unified framework that integrates a microfacet material model and ray tracing to jointly model specular reflection and transmittance in Gaussian Splatting. We accomplish this by using separate Gaussian primitives for reflections and transmittance, which allow modeling distant reflections and reconstructing objects behind transparent surfaces concurrently. We utilize a differentiable ray tracing framework to obtain the specular reflection and transmittance appearance. Our experiments demonstrate that our method successfully produces reflections and recovers objects behind transparent surfaces in complex environments, achieving significant qualitative improvements over prior methods where these specular light interactions are prominent.
Figures & tables
Figure 1: We present a method that optimizes the quality of specular reflection and transmittance concurrently in GS-based method. For the rasterization-based gaussian splatting method [ 8 ] in the left image, the specular reflection details are blurred on the surface of objects. For the reflection rays integrated method [ 23 ] in the middle image, the object behind the semi-transparent surface is blurred. We solve both the complex reflection and transmittance ray effects at the same time by integrating reflection and transmittance rays into gaussian splatting. Then, we can see the transmittance appearance and specular reflection details, as shown in the right image.
Figure 2: Overview of our method. In the first stage, we start with the rasterization process using diffuse Gaussians to obtain per-pixel material maps and diffuse color. In the second stage, we generate specular reflection rays from the surface of diffuse Gaussians and utilize ray tracing on reflection Gaussians. In the third stage, we trace first-bounce transmittance rays from the surface into the transparent objects on transmittance Gaussians to obtain the inside color and second-bounce transmittance rays from the back surface of the mesh on diffuse Gaussian again to obtain the outside color. Finally, we combine the diffuse, reflection and transmittance colors for the final image. We jointly optimize these three types of Gaussians using ground truth images for supervision.
Figure 3: Mesh-guided ray tracing for transparent objects. The surface mesh is extracted from the diffuse Gaussian. The first bounces of transmittance rays are traced from the depth positions of the diffuse Gaussian on the transmittance Gaussian. The second bounces of transmittance rays are traced from the back surface positions of the extracted mesh on the diffuse Gaussian. To obtain the back surface positions, we trace rays on the mesh to get the second nearest depth.
Figure 4: Illustration of the regularization to the depth. When reconstructing the internal object, the depth of first-bounce transmittance rays of the part of the object that exists is reconstructed well, as demonstrated by the solid red curve. But in the parts where no object is existing, the depth is trained to the outside the transparent object, as shown by the red dashes curve, because it tries to reconstruct the outside color, which should be rendered by second-bounce transmittance rays. Therefore, we add regularization to the depth to push it back into the transparent object.
Figure 5: Qualitative comparison on Ref-Real [ 21 ] Scenes.
Figure 6: Qualitative comparison on NU-NeRF [ 19 ] Scenes.
Figure 7: Ablation study on the BallStatue scene.
Figure 8: Ablation study on the Sedan scene. (a) Approximating the BSDF with separate components [ 24 , 13 ] leads to substantial errors in the reconstructions of reflections. (b) Our solution of using the full microfacet model produces better reflections.
PSNR ↑
SSIM ↑
LPIPS ↓
w/o transmittance Gaussians
24.844
0.866
0.196
w/o specular loss
25.673
0.882
0.174
w/o depth loss
25.941
0.881
0.172
w/o mesh guided ray tracing
25.889
0.880
0.174
Full Model
26.053
0.885
0.174
Table 3: Ablation study on the BallStatue scene.
Figure 9: Qualitative comparison on Ref-Real [ 21 ] Scenes.
Figure 10: Qualitative comparison on NU-NeRF [ 19 ] Scenes.
3D Gaussian Splatting (3DGS) enables real-time novel view synthesis with high visual quality. However, existing methods struggle with semi-transparent specular surfaces that exhibit both complex reflections and clear transmission, often producing blurry reflections or overly occluded transmission. To address this, we present RT-Splatting, a framework that disentangles each Gaussian's geometric occupancy from its optical opacity. This factorization yields a unified surface-volume scene representation with a single set of Gaussian primitives. Our hybrid renderer interprets this representation both as a surface to capture high-frequency reflections and as a volume to preserve clear transmission. To mitigate the ambiguity in jointly optimizing reflection and transmission, we introduce Specular-Aware Gradient Gating, which suppresses misleading gradients from highly specular regions into the transmission branch, effectively reducing distracting floaters. Experiments on challenging semi-transparent scenes show that RT-Splatting achieves state-of-the-art performance, delivering high-fidelity reflections and clear transmission with real-time rendering. Moreover, our factorization naturally enables flexible scene editing. The project page is available at https://sjj118.github.io/RT-Splatting.
Ji Shi, Xianghua Ying, Bowei Xing +2
State Key Laboratory of General Artificial Intelligence School of Intelligence Science and Technology Peking University
Transmissive scenes are ubiquitous in daily life, yet reconstructing and rendering them remains highly challenging due to the inherent entanglement between near-field reflections from the surrounding environment on the transmissive surface, and the transmitted content of the scene behind it. This coupling gives rise to dual surface geometries and dual radiance components within each observation, posing ambiguities for standard methods. We present TransmissiveGS, a novel framework for disentangled reconstruction and rendering of transmissive scenes. Specifically, we model the scene with a dual-Gaussian representation and introduce a deferred shading function to jointly render the two Gaussian components. To separate reflection and transmission, we exploit the inherent multi-view inconsistency of reflections and leverage the residuals from reconstructing multi-view consistent content as cues for disentangled geometry and appearance modeling. We further propose a reflection light field that enables high-fidelity estimation of near-field reflections. During training, we introduce a high-frequency regularization to preserve fine details. We also contribute a new synthetic dataset for evaluating transmissive surface reconstruction. Experiments on both synthetic and real-world scenes demonstrate that TransmissiveGS consistently outperforms prior Gaussian Splatting-based methods in both reconstruction and rendering quality for transmissive scenes.
The reflective appearance, especially strong and typically near-field specular reflections, poses a fundamental challenge for accurate surface reconstruction and novel view synthesis. Existing Gaussian splatting methods either fail to model near-field specular reflections or rely on explicit ray tracing at substantial computational cost. We present Ref-DGS, a reflective dual Gaussian splatting framework that addresses this trade-off by decoupling surface reconstruction from specular reflection within an efficient rasterization-based pipeline. Ref-DGS introduces a dual Gaussian scene representation consisting of geometry Gaussians and complementary local reflection Gaussians that capture near-field specular interactions without explicit ray tracing, along with a global environment reflection field for modeling far-field specular reflections. To predict specular radiance, we further propose a lightweight, physically-aware specular adaptive mixing shader that fuses global and local specular features. Experiments demonstrate that Ref-DGS achieves state-of-the-art performance on reflective scenes while training substantially faster than ray-based Gaussian methods.
Ningjing Fan, Yiqun Wang, Dong-Ming Yan +1
Chongqing University, China · MAIS, Institute of Automation, Chinese Academy of Sciences and UCAS, China · King Abdullah University of Science and Technology (KAUST), Saudi Arabia