Glass Surface Detection Grounded in 3D Visual Geometry

Yiwei Lu; Ke Xu; Tao Yan; Xiaojun Chang; Radu Timofte; Rynson W. H. Lau

cs.CVAug 27, 2026

Glass Surface Detection Grounded in 3D Visual Geometry

Authors: Yiwei Lu, Ke Xu, Tao Yan, Xiaojun Chang, Radu Timofte, Rynson W. H. Lau

Organizations: City University of Hong Kong (Dongguan) Dongguan, China

Abstract

Glass surface detection (GSD) is critical for scene understanding and reconstruction, and yet remains challenging due to the transparency and reflectivity of glass surfaces. Existing GSD methods typically rely on 2D appearance cues, which may fail in geometrically ambiguous scenes. In this paper, we propose a paradigm shift: grounding GSD in 3D visual geometry to explicitly model the physical existence of glass surfaces. Our method first distills rich 3D priors from the visual geometry grounded transformer (VGGT) and generates glass-aware 3D representations. It then exploits multi-tasking learning with a novel glass detection head, consisting of two core modules: a Frequency Self-Attention Module (FSAM) that identifies glass-specific spectral features for glass surface localization, and a Geometry Grounding Block (GeGB) that selectively grounds 2D features in 3D geometry for glass surface segmentation. Extensive experiments demonstrate that our method achieves state-of-the-art performance across seven standard GSD benchmarks, generalizes well to video/multi-modal data, and substantially improves reconstruction in glass scenes. Code is available in https://github.com/YT3DVision/VGGT_GLASS.

Figures & tables

Panel of Figure 1 — Figure 1. SOTA GSD methods typically rely on 2D appearance cues ( e.g. , by explicitly modeling the ghosting effects ( Yan et al., 2025 ) ) or implicitly learning glass-specific embeddings from other modalities ( e.g. , text) to query large-scale pre-trained image diffusion priors ( Li et al., 2025 ) . However, glass surfaces do not always exhibit strong ghosting effects ( Yan et al., 2025 ) . While the text-based image priors ( Li et al., 2025 ) show promising glass surface localization performances, they are not sufficiently discriminative for pixel-wise segmentation. In this work, we propose to shift the paradigm by grounding GSD in large-scale pre-trained 3D visual geometry. We explicitly model the physical existence of glass surfaces, resulting in accurately glass segmentation.

Figure 2. Method Overview. We present a novel two-step approach that grounds glass surface detection in 3D visual geometry. In the first step, we generate rectified 3D pseudo-ground truth of point clouds and depths by leveraging VGGT and ground truth glass surface masks. In the second step, we formulate a multi-task learning objective with a novel glass head for geometry-aware GSD. Our method achieves state-of-the-art GSD performance, generalizes well to video/multi-modal data, and substantially improves reconstruction in glass scenes.

Panel of Figure 3 — Figure 3. Depth map correction for glass surfaces improves their physical plausibility and geometric coherence.

Figure 4. Our Glass Head for precise glass surface segmentation and glass boundary delineation is built upon (1) the Frequency Self-Attention Module (FSAM) for capturing discriminative spectral representation of glass, and (2) the Geometry Grounding Block (GeGB) that reduces visual ambiguities via geometry-aware feature alignment and selective fusion.

Figure 5. Our Frequency Self-Attention Module (FSAM) operates in the frequency domain to capture prominent spectral features of glass surfaces for their preliminary localization.

Panel of Figure 6 — Figure 6. (a) Our Geometry Grounding Block (GeGB) incorporates depths and points as complementary geometric context for reasoning glass regions, and (b) our Decoder Block for segmenting glass regions and delineating their boundaries.

Methods	Venue	GDD ( Mei et al., 2020 )					GSD ( Lin et al., 2021 )					HSO ( Yu et al., 2022 )
Methods	Venue	IoU $\uparrow$	$F_{\beta}$ $\uparrow$	MAE $\downarrow$	BER $\downarrow$	ACC $\uparrow$	IoU $\uparrow$	$F_{\beta}$ $\uparrow$	MAE $\downarrow$	BER $\downarrow$	ACC $\uparrow$	IoU $\uparrow$	$F_{\beta}$ $\uparrow$	MAE $\downarrow$	BER $\downarrow$	ACC $\uparrow$
SAM2	-	82.70	0.891	0.090	8.44	0.920	73.00	0.797	0.096	9.24	0.917	71.18	0.801	0.134	12.04	0.898
VGGT	CVPR’25	85.26	0.914	0.072	6.68	0.948	78.63	0.868	0.065	7.02	0.930	77.62	0.854	0.094	8.35	0.944
SAM3	ICLR’26	92.88	0.966	0.034	3.04	0.974	89.64	0.943	0.033	3.02	0.988	87.24	0.931	0.054	4.66	0.965
GDNet	CVPR’20	88.71	0.934	0.059	5.49	0.961	83.00	0.893	0.055	5.68	0.952	79.25	0.871	0.093	8.65	0.931
GSDNet	CVPR’21	88.52	0.935	0.057	5.29	0.951	82.96	0.895	0.052	6.00	0.938	80.17	0.880	0.085	8.07	0.928

Table 1. Quantitative comparison between our method and existing GSD methods on standard single-image datasets. Foundation-model baselines are also included for reference. Best results are marked in bold .

Methods	Venue	Trans10k-stuff ( Xie et al., 2020 )
Methods	Venue	IoU $\uparrow$	$F_{\beta}$ $\uparrow$	MAE $\downarrow$	BER $\downarrow$	ACC $\uparrow$
SAM2	-	81.16	0.884	0.075	6.59	0.956
VGGT	CVPR’25	84.86	0.912	0.060	5.39	0.963
SAM3	ICLR’26	91.07	0.956	0.033	2.82	0.984
GDNet	CVPR’20	88.30	0.937	0.047	4.08	0.976
GSDNet	CVPR’21	88.62	0.940	0.043	3.87	0.974

Table 2. Quantitative comparison on the Trans10k-stuff. Best results are marked in bold .

Methods	Venue	IoU $\uparrow$	F β $\uparrow$	MAE $\downarrow$	BER $\downarrow$	ACC $\uparrow$
RGB-Thermal	TIP’23	93.80	0.965	0.027	4.08	-
Ours	-	95.71	0.976	0.023	4.19	0.970
RGB-Depth	AAAI’25	74.20	0.853	0.043	9.30	-
Ours	-	81.23	0.873	0.026	7.05	0.880

Table 3. Comparisons to multi-modal GSD methods, i.e. , RGB-Thermal ( Huo et al., 2023 ) (upper) and RGB-Depth ( Lin et al., 2025 ) (lower) on their proposed datasets. Best results are in bold .

Methods	Venue	IoU $\uparrow$	F β $\uparrow$	MAE $\downarrow$	BER $\downarrow$	ACC $\uparrow$
VGSDNet	AAAI’24	80.72	0.885	0.099	9.60	0.898
GhostingNet	TPAMI’25	80.40	0.888	0.100	9.30	0.905
GlassWizard	ICCV’25	94.20	-	0.031	2.92	-
MVGDNet	AAAI’26	86.54	0.925	0.064	6.10	0.935
Ours	-	97.36	0.982	0.016	1.97	0.986

Table 4. Comparisons on the video GSD dataset VGSD-D ( Liu et al., 2024a ) . Best results are in bold .

Panel of Figure 7 — Figure 7. Visual comparison between our method and single-image GSD methods, which highlights that 3D scene geometry information is vital for robust glass surface detection.

Panel of Figure 8 — Figure 8. Visual comparison between VGGT ( Wang et al., 2025a ) and our method in monocular depth estimation and reconstruction of 3D scenes with glass surfaces.

Methods	IoU $\uparrow$	F β $\uparrow$	MAE $\downarrow$	BER $\downarrow$	ACC $\uparrow$
w/o LoRA	88.27	0.931	0.046	4.05	0.981
w/o FSAM	92.78	0.964	0.027	2.29	0.988
w/o GeGB	92.67	0.963	0.027	2.31	0.988
w/o FFT	92.82	0.967	0.027	2.37	0.986
$F_{in}\rightarrow F_{fre}$	93.06	0.967	0.026	2.12	0.991
w/o Depth	92.86	0.967	0.026	2.28	0.988

Table 5. Ablation results of our method.

Panel of Figure 9 — Figure 9. The visual comparison of different ablated models.

Methods	Param(M)	FLOPS(G)	Time(ms)
GhostingNet	271.53	321.70	32.94
GlassWizard	1189.73	890.78	177.00
SAM3-based	471.18	2539.06	134.90
VGGT	989.17	1657.69	159.73
Ours	684.37	1260.67	117.31

Table 6. Runtime efficiency comparison on a single NVIDIA RTX 4090 GPU.

Panel of Figure 10 — Figure 10. A failure case. Our method may fail when the glass surface ( e.g. , the windshield) is too close to the camera, providing insufficient surrounding geometric contexts.

Image Segmentation

Image segmentation assigns pixels or voxels to objects and regions, supporting measurement, scene understanding, and decisions across image domains.

See all 401 papers

GlassFormer: Learning Real-time Glass Segmentation using Radar-Depth Fusion
Sep 29, 2026Suhani Grover, Astik Srivastava, Viswas Dinesh +2Robotic Perception Semantic Segmentation
Robotics Research Center, IIIT Hyderabad, India
SILICA: Repurposing Diffusion Priors for Joint Glass Segmentation and Depth Estimation
Jul 27, 2026Tarun R, Anuj Verma, Laksh Nanwani +2Image Segmentation Depth Estimation
Robotics Research Center (RRC), IIIT Hyderabad
GHOST: Geometry-Guided Hallucination of Opaque Surface Textures
Jul 13, 2026Langxu Zhao, Zuan Gu, Tianhan GaoImage-to-Image Translation 3D Reconstruction
School of Software, Northeastern University, Shenyang, China.

Sep 29, 2026J/K move · Enter open · S save

GlassFormer: Learning Real-time Glass Segmentation using Radar-Depth Fusion

Suhani Grover, Astik Srivastava, Viswas Dinesh, Avinash Sharma+1

Robotics Research Center, IIIT Hyderabad, India

Robotic Perception Semantic Segmentation Multimodal Sensor Fusion

Open paper AskPDF arXiv

Transparent surfaces are ubiquitous in built environments, yet they remain a persistent failure case for robotic perception. RGB cameras perceive the background behind glass rather than the surface itself, while depth sensors such as LiDAR, time-of-flight, and RGB-D often return invalid or background measurements in transparent regions. As a result, systems that rely solely on optical sensing may misinterpret glass walls, doors, or mirrors as free space, compromising safe and reliable navigation. Existing glass segmentation approaches address this by learning visual cues such as reflections, boundaries, and semantic context from RGB images. While effective under favourable lighting and viewing conditions, these cues degrade in low-light environments, under glare, or when glass surfaces are featureless or partially occluded. In this work, we propose a multimodal framework that fuses millimetre-wave radar with RGB-D sensing for real-time transparent surface segmentation. Radar reflects strongly off glass surfaces, providing a geometric cue that remains reliable precisely where vision and depth fail. We exploit this cross-modal inconsistency to generate a radar-guided spatial prior, which is integrated into a lightweight transformer-based segmentation network, GlassFormer, via cross-modal attention. We report results on a mixed-condition test split covering all scene types and a dedicated low-light split designed to stress vision-only methods. GlassFormer achieves 0.88 mIoU on the mixed split, and 0.59 mIoU on the low light split, demonstrating substantial robustness gains over vision-only baselines while maintaining real-time performance on resource-constrained platforms.

Abstract

Image Segmentation

Explore similar work

GlassFormer: Learning Real-time Glass Segmentation using Radar-Depth Fusion

SILICA: Repurposing Diffusion Priors for Joint Glass Segmentation and Depth Estimation

GHOST: Geometry-Guided Hallucination of Opaque Surface Textures