Reconstructing transparent objects from a set of multi-view images is a challenging task due to the complicated nature and indeterminate behavior of light propagation. Typical methods are primarily tailored to specific scenarios, such as objects following a uniform topology, exhibiting ideal transparency and surface specular reflections, or with only surface materials, which substantially constrains their practical applicability in real-world settings. In this work, we propose a differentiable rendering framework for transparent objects, dubbed DiffTrans, which allows for efficient decomposition and reconstruction of the geometry and materials of transparent objects, thereby reconstructing transparent objects accurately in intricate scenes with diverse topology and complex texture. Specifically, we first utilize FlexiCubes with dilation and smoothness regularization as the iso-surface representation to reconstruct an initial geometry efficiently from the multi-view object silhouette. Meanwhile, we employ the environment light radiance field to recover the environment of the scene. Then we devise a recursive differentiable ray tracer to further optimize the geometry, index of refraction and absorption rate simultaneously in a unified and end-to-end manner, leading to high-quality reconstruction of transparent objects in intricate scenes. A prominent advantage of the designed ray tracer is that it can be implemented in CUDA, enabling a significantly reduced computational cost. Extensive experiments on multiple benchmarks demonstrate the superior reconstruction performance of our DiffTrans compared with other methods, especially in intricate scenes involving transparent objects with diverse topology and complex texture. The code is available at https://github.com/lcp29/DiffTrans.
Figures & tables
Figure 1: Overview of the proposed DiffTrans . Our approach reconstructs the geometry and materials of the transparent objects with diverse topology and complex texture from a set of multi-view images using differentiable mesh ray tracer, enabling scene editing capabilities such as relighting.
Figure 2: The framework of our DiffTrans . We commence by reconstructing the initial geometry from multi-view mask images, and recovering the environment light radiance field by employing pixels out of the mask regions ( Section 3.2 ). In the refine phase, we first define the light decay within absorptive medium, and the Fresnel term used for blending in-object radiance and out-of-object radiance ( Section 3.3 ). Then we optimize the geometry, IoR and absorption rate of the transparent objects simultaneously via our differentiable recursive mesh ray tracer, with geometry and material regularization ( Section 3.4 ).
Figure 3: The recursive rendering process demonstrated with a light path. LiR′ and LiR are the radiance of the reflected ray, which is abbreviated as a single variable for simplicity. Ti , Ri , Ti′ and Ri′ are calculated with Equation 7 and Equation 8 .
Method
NeRO
NU-NeRF
NeRRF
Ours(S1)
Ours
CD( ×10−4 ) ↓
36.022
7.891
13.341
4.666
3.264
F1 ( ×10−1 ) ↑
5.691
8.026
6.916
8.088
8.386
Table 1: Average quantitative comparison of geometry reconstruction across all synthetic datasets. Ours(S1) refers to the coarse geometry after the initialization.
Figure 4: Qualitative comparisons of the reconstructed geometry on bunny , horse an monkey scenes.
Figure 5: Qualitative comparison of relighting results on the horse , monkey , and flower scenes.
Dataset
horse
monkey
flower
hand
mouse
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
w/o Ltone
27.00
0.9208
0.0514
24.15
0.8942
0.0663
24.06
0.7883
0.2731
22.37
0.8719
0.1481
24.15
0.8980
0.1225
Full
27.03
0.9278
0.0457
24.62
0.8935
0.0631
24.14
0.7895
0.2717
23.39
0.8736
0.1420
24.16
0.8986
0.1219
Table 3: Quantitative results of novel view synthesis with/without the tone regularization.
Method
PSNR ↑
SSIM ↑
LPIPS ↓
NeRRF
19.25
0.8118
0.0812
NeRO
19.64
0.8500
0.0856
Ours
23.17
0.8380
0.0678
Table 4: Quantitative results of relighting averaged across all the synthetic scenes.
Figure 6: Reconstruction results with different IoRs on the monkey scene.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
horse
monkey
flowers
bunny
cow
c1
7
5
3
5
5
c2
0.5
0.1
0.3
0.5
0.1
Appendix
Table 5: Hyperparameters for geometry initialization across datasets.
Dataset
horse
monkey
flowers
γ3
0.0001
0.01
0.01
k
300
500
1000
Appendix
Table 6: Hyperparameters for freeze-geometry stage.
Dataset
Images
Material
Real-World
Object
Env.
horse
100
100
absorption
N
monkey
100
50
absorption
N
flower
104
56
absorption
Y
flower(2)
70
12
absorption
Y
flower(3)
154
69
absorption
Y
Appendix
Table 7: Overview of the datasets used in our experiments.
Figure 7: Parts of the images from the dataset used in our experiments.
Figure 8: Visualization of an extra real world test scene including the input images, recovered normal maps and relighting results.
Figure 9: Novel view synthesis results on the horse , monkey and flower scenes.
Figure 10: Reconstructed geometries on the horse , monkey and bunny scenes.
Figure 11: Per-vertex heatmap illustrating the ground truth difference in Hausdorff Distance before and after optimization.
Figure 12: Per-Vertex Chamfer Distances betweeen the meshes before and after optimization.
Figure 13: Visualization of the decomposed scene representations. Top: Recovered 2D environment maps (equirectangular projections) representing the far-field background. Bottom: 3D reconstructions of the scenes. The grey regions visible in the top-right environment map correspond to areas (zenith and nadir) that were not observed in the input footage due to the limited vertical field-of-view of the camera trajectory.
Figure 14: Rendering of the same object under different refractive indices (IOR).
Figure 15: Normal maps comparison. The Ours(S1) column shows the normal maps rendered from the first stage of our method.
Figure 16: A comparison of relighting results using the coarse mesh from stage 1 and the refined mesh from stage 2.
Method
NeRO
NU-NeRF
NeRRF
Ours(S1)
Ours(S1) ∗
Ours(S1) ∗∗
Ours
Ours ∗
Ours ∗∗
CD( ×10−4 ) ↓
36.022
7.891
13.341
4.666
4.329
4.468
3.264
3.203
3.341
F1 ( ×10−1 ) ↑
5.691
8.026
6.916
8.088
7.832
7.646
8.386
8.623
8.401
Appendix
Table 8: The extended result of Table 2 in the paper. ∗ refers to the result obtained using noised masks and ∗∗ refers to the result obtained by cropping patches off the mask.
Object
η=1.3
η=1.4
η=1.5
monkey
1.288
1.381
1.473
horse
1.305
1.421
1.512
Appendix
Table 9: Ablation study of IOR prediction under different ground-truths.
Dataset
CD( ×10−3 ) ↓
NC( ×10−1 )
Method
horse
monkey
frame
block
horse
monkey
frame
block
only Lgeo-init
3.233
4.915
1.652
6.969
2.403
2.498
2.431
2.794
+ Ldilation
0.481
1.039
14.683
31.602
1.298
1.713
0.747
0.933
+ Lsmooth
0.594
1.139
14.847
21.456
0.412
0.776
0.747
0.210
full
0.484
0.459
1.072
4.457
0.245
0.166
0.381
0.101
after processing
0.456
0.458
1.084
1.853
0.052
0.053
0.110
0.028
Appendix
Table 10: Ablation study of initial geometry reconstruction. The second best results are underlined.
Figure 17: Ablation studies for different regularization.
Figure 18: Reconstruction with/without the tone regularization.
Transparent objects pose a fundamental challenge for depth estimation and 3D reconstruction due to their violation of Lambertian assumptions, leading to severe geometry degradation in downstream tasks. To address this, we propose a novel geometry-guided preprocessing framework GHOST that leverages visual foundation models to transform transparent regions into opaque, structurally consistent representations without requiring downstream model retraining. Specifically, our pipeline utilizes (1) TransDINO and (2) TransDecomp to disentangle masks and transparency physical properties, while (3) DAF-Net recovers surface normal priors to encode geometric curvature. Subsequently, (4) GeoSemTransNet integrates these multi-modal cues to synthesize a texture-rich opaque RGB image that preserves the transparent object's 3D structure. Extensive experiments demonstrate that our method significantly enhances the accuracy of state-of-the-art depth estimation and reconstruction models on transparent objects by restoring essential photometric cues.
Langxu Zhao, Zuan Gu, Tianhan Gao
School of Software, Northeastern University, Shenyang, China.
Reconstructing translucent objects from multi-view images is a difficult problem. Previously, researchers have used differentiable path tracing and the neural implicit field, which require relatively large computational costs. Recently, many works have achieved good reconstruction results for opaque objects based on a 3DGS pipeline with much higher efficiency. However, such methods have difficulty dealing with translucent objects, because they do not consider the optical properties of translucent objects. In this paper, we propose a novel 3DGS-based pipeline (GTSR) to reconstruct the surface geometry of translucent objects. GTSR combines two sets of Gaussians, surface and interior Gaussians, which are used to model the surface and scattering color when lights pass translucent objects. To render the appearance of translucent objects, we introduce a method that uses the Fresnel term to blend two sets of Gaussians. Furthermore, to improve the reconstructed details of non-contour areas, we introduce the Disney BSDF model with deferred rendering to enhance constraints of the normal and depth. Experimental results demonstrate that our method outperforms baseline reconstruction methods on the NeuralTO Syn dataset while showing great real-time rendering performance. We also extend the dataset with new translucent objects of varying material properties and demonstrate our method can adapt to different translucent materials.
While 3D Gaussian splatting has emerged as a powerful paradigm, it fundamentally fails to model transparency such as glass panels. The core challenge lies in decoupling the intertwined radiance contributions from transparent interfaces and the transmitted geometry observed through the glass. We present GLINT, a framework that models scene-scale transparency through explicit decomposed Gaussian representation. GLINT reconstructs the primary interface and models reflected and transmitted radiance separately, enabling consistent radiance transport. During optimization, GLINT bootstraps transparency localization from geometry-separation cues induced by the decomposition, together with geometry and material priors from a pre-trained video relighting model. Extensive experiments demonstrate consistent improvements over prior methods for reconstructing complex transparent scenes.