Organizations: School of Advanced Manufacturing, Sun Yat-sen University, Shenzhen 518107, China. · School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 518107, China. · School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006, China.
The contour estimation of transparent fragments is very important for autonomous reassembly, especially in the fields of precision optical instrument repair, cultural relic restoration, and identification of other precious device broken accidents. Different from general intact transparent objects, the contour estimation of transparent fragments face greater challenges due to strict optical properties, irregular shapes and edges. To address this issue, a general transparent fragments contour estimation framework based on visual-tactile fusion is proposed in this paper. First, we construct the transparent fragment dataset named TransFrag27K, which includes a multiscene synthetic data of broken fragments from multiple types of transparent objects, and a scalable synthetic data generation pipeline. Secondly, we propose a visual grasping position detection network named TransFragNet to identify, locate and segment the sampling grasping position. And, we use a two-finger gripper with Gelsight Mini sensors to obtain reconstructed tactile information of the lateral edge of the fragments. By fusing this tactile information with visual cues, a visual-tactile fusion material classifier is proposed. Inspired by the way humans estimate a fragment's contour combining vision and touch, we introduce a general transparent fragment contour estimation framework based on visual-tactile fusion, demonstrates strong performance in real-world validation. Finally, a multi-dimensional similarity metrics based contour matching and reassembly algorithm is proposed, providing a reproducible benchmark for evaluating visual-tactile contour estimation and fragment reassembly. The experimental results demonstrate the validity of the proposed framework. The dataset and codes are available at https://github.com/Keithllin/Transparent-Fragments-Contour-Estimation.
Figures & tables
Fig. 1: Visual-tactile fusion contour estimation inspired by the human strategy of combining vision with touch.
Fig. 2: (a) Illustrates the synthetic data generation pipeline, showcasing scene construction with elements like scene light, extra light, camera, and the projection of transparent fragments onto a surface. (b) Displays a subset of the generated dataset, featuring paired RGB images and their corresponding masks for transparent fragments.
Fig. 3: Examples of intact transparent objects used to generate synthetic fracture scenarios in the proposed TransFrag27K dataset. The collection includes a diverse range of materials, shapes, and surface textures, such as glassware, lab beakers, cups, bottles, and decorative items. The serial number corresponds to the state of the corresponding object after adding the fracture effect.
Fig. 4: This figure details the steps involved in generating the synthetic data for TransFrag27K. For various transparent objects (e.g., glass, plate, bottle), the second column shows the simulation of breakage, followed by the resulting RGB image and its corresponding mask, highlighting the visual shape of individual transparent fragments.
Fig. 5: Visual-tactile fusion based contour estimation framework for transparent fragment reassembly. (a) TransFragNet based visual grasping position detection and segmentation (b) Fragment reassembly using visual-tactile fusion based fragments contour estimation framework in mixed fragment scenes (c) Tactile information reconstruction and matching.
Fig. 6: The architecture of the proposed TransFragNet. The model employs a Swin Transformer backbone for robust feature extraction, followed by a dual-path decoder that integrates deformable transposed convolution and anti-aliased upsampling. The final segmentation head generates pixel-wise probability masks, which are further refined through morphological post-processing.
Fig. 7: Segmentation results for transparent fragments, including both single and multiple fragment scenarios. The top row shows the original input images, while the middle two rows display the annoted mask and predicted segmentation masks generated by our model. The bottom row illustrates the interaction corner points by applying morphological post-processing.
Fig. 8: Tactile reconstruction pipeline: recovering contact masks, local gradient fields, and detailed height maps from GelSight Mini data. These outputs provide fine-scale edge and curvature details critical for contour estimation and fragment matching.
Fig. 9: Architecture of the U-Net convolutional network used for segmenting tactile contact regions. This architecture is specifically designed to segment the contact regions in GelSight Mini tactile images, achieving high accuracy and sharp boundary delineation for subsequent tactile reconstruction tasks.
Fig. 10: Dual-branch late fusion network architecture for visual-tactile material classification. The visual branch employs ResNet-18 for robust feature extraction from RGB images, while the tactile branch uses a lightweight CNN to process GelSight Mini tactile data. Features are concatenated at the late fusion stage and processed through a multi-layer classifier.
Fig. 11: Visual–tactile contour estimation results for transparent fragments. Each panel shows an individual fragment with its extracted global mask (center top) and reconstructed tactile profiles: RGB tactile image, edge mask, gradient map, and height map. The color-coded label indicates the final material classification result obtained through visual–tactile fusion.
Fig. 12: Framework overview of the proposed visual-tactile fusion based contour estimation for transparent fragment reassembly.
Fig. 13: The contour matching results and scores of fragments of some different materials: including edge geometry alignment, local area gradient matching, and height extreme value complementation, are combined into a unified score for robust transparent fragment reassembly.
Fig. 14: Hardware setup: UR5 robotic arm with DH-95 two-finger gripper and integrated GelSight Mini tactile sensors.
Model
Synthetic Data (IoU)
Varying Backgrounds (IoU)
Varying Illumination (IoU)
EfficientNet
0.7875
0.7536
0.6430
MobileNet
0.7349
0.6366
0.3205
TGCNN
0.4576
0.2646
0.5968
Ours
0.8856
0.9382
0.8451
TABLE I: Comparison of mask prediction results on the synthetic dataset and real dataset. From top to bottom: Original Image, TGCNN, EfficientNet, MobileNet, and Our Model.
Fig. 15: Transparent fragment detection results and postprocessing. Including synthetic fragments detected and real fragments detected under different backgrounds.
Fig. 16: Transparent fragment detection in varying illumination, from left to right starting from 400 to 6000 Lux.
Metric
IoU
Chamfer Distance
Pair 1
0.82
4.3 px
Pair 2
0.79
5.1 px
Pair 3
0.85
3.8 px
Pair 4
0.81
2.4 px
Pair 5
0.74
4.5 px
Pair 6
0.82
5.2 px
TABLE II: Results for Partial Fracture Reassembly
Fig. 17: Visual shape-based partial fracture reassembly results: five representative cases with different notched transparent objects, showing remaining body, detached fragment, centroid alignment, and final angle-aligned overlay.
Fig. 18: Workflow of the proposed visual–tactile transparent fragment reassembly framework. The framework begins with visual detection using an RGB-D camera to segment transparent fragments and estimate grasping positions using TransFragNet. Fragments’ extracted visual features are then matched in the database. If a match is not found in the existing database, new tactile features sampling process including edge maps, gradient fields, and height profiles are extracted and stored. Matched fragments are classified by material and assigned IDs based on multi-dimensional feature similarity, then passed to the reassembly stage.
Material
Edge Score
Gradient Score
Height Score
Total Fusion
Glass (Pair 1)
0.857
0.814
0.913
0.859
Glass (Pair 2)
0.887
0.908
0.891
0.892
Acrylic (Pair 1)
0.844
0.880
0.917
0.865
Colored Crystal (Pair 1)
0.836
0.932
0.818
0.852
Plastic (Pair 1)
0.846
0.870
0.936
0.869
Colorless Crystal (Pair 1)
0.887
0.884
0.906
0.890
TABLE III: Visual-Tactile Fusion Reassembly Scores for Different Materials
Fig. 20: Experimental pipeline for complete reassembly with visual-tactile fusion, demonstrated with a colorless crystal dish: (a) full fragment set; (b) edge sampling and reconstruction; (c) random placement scene; (d) match visual ID with Database and excute reassembly; (e-f) final reassembled result with different perspectives.
Fig. 21: Generalization test for complete fracture reassembly with unseen or similar transparent fragments. (a) Experimental scene with mixed-material fragments including glass, acrylic, and colored/colorless crystal pieces. (b) Top view with highlighted examples: CF3 (colored crystal) and CL3 (colorless crystal) as shape-similar pairs; P3 as an unknown plastic fragment not stored in the database. (c) Edge sampling procedure using the GelSight Mini sensor for fallback classification. (d) Final identification map showing each fragment’s assigned material class and unique ID, providing full guidance for robust reassembly.
Fig. 22: Reassembly results of mixed fragments from completely fractured transparent objects made from different transparent materials: (a) acrylic dish; (b) plastic dish; (c) glass dish; (d) colorless crystal dish; (e) colored crystal dish. (f-j) the same as (a-e) but with different perspectives.
Scene Description
Top-1 Acc.
Top-3 Acc.
Mean IoU
Fallback Needed
Similar-Look Case (CL3 vs CF3)
50%
100%
0.79
Yes
Unknown Fragment (P3)
0%
—
—
Yes
Full Mixed Material Scene
85%
100%
0.81
Resolved
TABLE IV: Performance in Mixed Scene with Similar and Unseen Fragments
Beijing University of Posts and Telecommunications, Beijing, China · SunHDex Intelligent Technology (Beijing) Co. Ltd., Beijing, China · University of Hamburg, Hamburg, Germany