Organizations: Shandong University, China · Peking University, Beijing, China · Sun Yat-sen University, China · Shenzhen Loop Area Institute, China · Beijing Academy of Artificial Intelligence, Beijing, China
Novel view synthesis (NVS) through non-planar refractive surfaces presents fundamental challenges due to severe, spatially varying optical distortions. While recent representations like NeRF and 3D Gaussian Splatting (3DGS) excel at NVS, their assumption of straight-line ray propagation fails under these conditions, leading to significant artifacts. To overcome this limitation, we introduce RefracGS, a framework that jointly reconstructs the refractive water surface and the scene beneath the interface. Our key insight is to explicitly decouple the refractive boundary from the target objects: the refractive surface is modeled via a neural height field, capturing wave geometry, while the underlying scene is represented as a 3D Gaussian field. We formulate a refraction-aware Gaussian ray tracing approach that accurately computes non-linear ray trajectories using Snell's law and efficiently renders the underlying Gaussian field while backpropagating the loss gradients to the parameterized refractive surface. Through end-to-end joint optimization of both representations, our method ensures high-fidelity NVS and view-consistent surface recovery. Experiments on both synthetic and real-world scenes with complex waves demonstrate that RefracGS outperforms prior refractive methods in visual quality, while achieving 15x faster training and real-time rendering at 200 FPS. The project page for RefracGS is available at https://yimgshao.github.io/refracgs/.
Figures & tables
Figure 2: Overview of the RefracGS pipeline . A proxy refractive - surface mesh is first extracted from the neural height field, after which camera rays intersect the mesh using a hierarchical global - to - local strategy to obtain intersection points and normals. The refracted directions are then computed with Snell’s law, and the resulting rays traverse the underlying scene before being rendered with Gaussian ray tracing.
Figure 2
Dataset
Method
PSNR ↑
SSIM ↑
LPIPS ↓
Train ↓
FPS ↑
RMSE(cm) ↓
NeRFrac Real
Mip-NeRF
10.385
0.221
0.916
∼ 15 h
< 1
-
TensoRF
22.973
0.809
0.139
11 min
< 1
-
Plenoxels
14.867
0.389
0.631
9 min
61
-
3DGS
27.711
0.864
0.130
3 min
857
-
3DGRT
27.732
0.842
0.164
16 min
157
-
WSGS †
22.449
0.679
0.279
106 min
129
-
Table 1: Quantitative experimental results. Our method achieves state-of-the-art novel view synthesis quality and water surface reconstruction accuracy, while maintaining fast training speed and real-time rendering frame rates. ( † indicates methods specifically designed for refractive scenes.)
Figure 5: Qualitative comparison with existing methods. On the NeRFrac dataset, our method preserves fine textures and high-frequency details better than NeRFrac. On the more challenging RefracGS dataset—featuring wider viewpoint coverage and stronger refractive effects—our method consistently produces high-fidelity renderings, whereas existing methods fail to learn accurate scene representations under such conditions, leading to noticeable artifacts and incorrect novel-view synthesis.
Method|Metric
PSNR ↑
SSIM ↑
LPIPS ↓
Train ↓
FPS ↑
(a) w/o Water Height Map
27.782
0.869
0.168
22 min
57
(b) w/o Normal Smoothing
32.643
0.928
0.133
11 min
232
(c) w/o Recursive Intersection
32.966
0.929
0.132
20 min
218
(d) w/o Opacity Loss
31.927
0.920
0.125
12 min
224
Full Model
32.876
0.929
0.132
11 min
223
Table 2: Ablation study results. The ablations show that the components of our method collectively provide a favorable balance between reconstruction quality and computational efficiency.
Method|Metric
PSNR ↑
SSIM ↑
LPIPS ↓
NeRFrac
14.950
0.380
0.563
Ours
23.151
0.869
0.110
Table 3: Metrics of water removal quality. Our method yields more reliable results compared with NeRFrac.
Figure 6: Water removal and surface editting Benefiting from our physically-based modeling approach, our method naturally decouples the water surface geometry from underwater objects. This capability enables the removal of surface refraction, as well as water surface editing using arbitrary meshes. Crucially, our precise refraction modeling allows these refraction-removed views to serve directly as inputs for methods like PGSR to achieve complete geometric reconstruction of the underlying scene.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8
Figure A3 : Boundary cases of single-refraction. (a) When cameras are fully above the water, the rays are downward or slightly oblique, single refraction holds. (b) When cameras are near water, extreme grazing angles cause multiple refractions. (c) Submerged cameras are more complicated and are beyond models of our method.
Scene
Method
PSNR↑
SSIM↑
LPIPS↓
Train↓
FPS↑
Plant
Mip-NeRF
10.350
0.322
0.945
∼ 15 h
< 1
TensoRF
21.857
0.798
0.138
∼ 11 min
< 1
Plenoxels
14.491
0.378
0.655
∼ 9 min
∼ 60
3DGS
28.520
0.801
0.198
3 min
1019
3DGRT
27.579
0.845
0.155
13 min
143
3DGS*
27.007
0.870
0.136
2 min
932
Appendix
Table A1 : Quantitative results on the Nerfrac real dataset. (* indicates models trained for 15,000 iterations; ∼ indicates that the runtime was not precisely recorded for each scene, but rather estimated. † indicates methods specifically designed for refractive scenes.)
Scene
Method
PSNR↑
SSIM↑
LPIPS↓
Train↓
FPS↑
Primary Sine
Mip-NeRF
12.482
0.402
0.852
∼ 15 h
< 1
TensoRF
16.301
0.527
0.614
∼ 11 min
< 1
Plenoxels
13.390
0.403
0.810
∼ 12 min
∼ 70
3DGS
21.279
0.705
0.311
3 min
693
3DGRT
21.135
0.663
0.390
29 min
80
3DGS*
21.459
0.726
0.346
2 min
686
Appendix
Table A2 : Quantitative results on the Nerfrac synthetic dataset. (* indicates models trained for 15,000 iterations; ∼ indicates that the runtime was not precisely recorded for each scene, but rather estimated. † indicates methods specifically designed for refractive scenes.)
Scene
Method
PSNR↑
SSIM↑
LPIPS↓
Train↓
FPS↑
RMSE(cm)↓
Toys
Mip-NeRF
12.122
0.363
0.855
∼ 15 h
< 1
-
TensoRF
12.353
0.255
0.621
∼ 13 min
< 1
-
Plenoxels
12.616
0.286
0.717
∼ 12 min
∼ 20
-
3DGS
15.230
0.297
0.525
5 min
526
-
3DGRT
16.027
0.304
0.566
16 min
171
-
3DGS*
15.085
0.303
0.528
2 min
425
-
Appendix
Table A3 : Quantitative results on the RefracGS dataset. (* indicates models trained for 15,000 iterations; ∼ indicates that the runtime was not precisely recorded for each scene, but rather estimated. † indicates methods specifically designed for refractive scenes.)
Scene
Method
PSNR↑
SSIM↑
LPIPS↓
Train↓
FPS↑
RMSE(cm)↓
Toys
Mip-NeRF
12.637
0.290
0.833
∼ 15 h
< 1
-
TensoRF
13.657
0.287
0.519
∼ 13 min
< 1
-
Plenoxels
12.261
0.271
0.701
∼ 12 min
∼ 20
-
3DGS
18.840
0.536
0.352
3 min
625
-
3DGRT
19.646
0.562
0.394
23 min
114
-
3DGS*
19.398
0.559
0.333
2 min
621
-
Appendix
Table A4 : Quantitative results of different methods on RefracGS dataset scenes under the additional nine-view setting as discussed in Sec. 0.C.3 . (* indicates models trained for 15,000 iterations; ∼ indicates that the runtime was not precisely recorded for each scene, but rather estimated. † indicates methods specifically designed for refractive scenes.)
Figure A4 : Qualitative results on the Nerfrac real dataset.
Figure A5 : Qualitative results on the Nerfrac synthetic dataset.
Figure A6 : Qualitative results on the RefracGS dataset.
Figure A7 : Qualitative results of different methods on RefracGS dataset scenes under the additional nine-view setting as discussed in Sec. 0.C.3 .
While novel view synthesis methods like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have shown impressive performance on common scenes, they often struggle with capturing view-dependent specular effects, particularly near-field reflections. Existing methods utilize environment maps, which only represent lights from infinity, or environment Gaussians, which rely solely on indirect supervision from reflections, are insufficient for accurately modeling such effects. Here, we propose a framework based on 2D Gaussian Splatting (2DGS) that leverages directly observable environment information in input images. Our approach also integrates a hybrid reflection appearance representation and view-dependent specular modeling. Experiments on public benchmarks and a newly curated near-field reflection dataset demonstrate that our method achieves superior rendering quality and geometric accuracy.
Yiming Liang, Tianyu Xiao, Hiroshi Ishikawa
Department of Computer Science and Communications Engineering, Waseda University, Japan
The reflective appearance, especially strong and typically near-field specular reflections, poses a fundamental challenge for accurate surface reconstruction and novel view synthesis. Existing Gaussian splatting methods either fail to model near-field specular reflections or rely on explicit ray tracing at substantial computational cost. We present Ref-DGS, a reflective dual Gaussian splatting framework that addresses this trade-off by decoupling surface reconstruction from specular reflection within an efficient rasterization-based pipeline. Ref-DGS introduces a dual Gaussian scene representation consisting of geometry Gaussians and complementary local reflection Gaussians that capture near-field specular interactions without explicit ray tracing, along with a global environment reflection field for modeling far-field specular reflections. To predict specular radiance, we further propose a lightweight, physically-aware specular adaptive mixing shader that fuses global and local specular features. Experiments demonstrate that Ref-DGS achieves state-of-the-art performance on reflective scenes while training substantially faster than ray-based Gaussian methods.
Ningjing Fan, Yiqun Wang, Dong-Ming Yan +1
Chongqing University, China · MAIS, Institute of Automation, Chinese Academy of Sciences and UCAS, China · King Abdullah University of Science and Technology (KAUST), Saudi Arabia
Gaussian Splatting has significantly improved the quality of novel view synthesis with explicit Gaussian representation. However, we observed that existing 3D Gaussian Splatting methods (3DGS) often suffer from surface collapse issues on reflective regions, and thus produce inferior geometry and low-quality specular. In this work, we propose a physically-based deferred rendering framework, named Reflection-aware Gaussian Splatting (RGS), that can accurately model specular regions and improve novel view synthesis performance. Specifically, we found that a powerful 3D foundation model can provide a strong 3D geometric prior to foster correct geometric modeling. Based on this, we propose a cross-view shape consistency regularization to regularize the geometry surface with the large model prior and cross-view constraints. In this manner, our RGS can produce smoother geometric surfaces on reflective regions while reducing geometric hollows. To further improve rendering results on reflective regions, we present a reflection-aware densification strategy that is designed to capture specular variations across various views. With this strategy, our RGS is able to render novel views of objects in higher quality. Extensive experiments demonstrate our method consistently renders high-quality reflective objects, achieving state-of-the-art performance.
Xiaobiao Du, Yida Wang, Cheng Bi +2
University of Technology Sydney · Li Auto Inc. · Adelaide University