Organizations: School of Advanced Manufacturing, Sun Yat-sen University, Shenzhen 518107, China. · School of Electronics and Communication Engineering, Sun Yat-sen University, Shenzhen 518107, China. · School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou 510006, China.
The contour estimation of transparent fragments is very important for autonomous reassembly, especially in the fields of precision optical instrument repair, cultural relic restoration, and identification of other precious device broken accidents. Different from general intact transparent objects, the contour estimation of transparent fragments face greater challenges due to strict optical properties, irregular shapes and edges. To address this issue, a general transparent fragments contour estimation framework based on visual-tactile fusion is proposed in this paper. First, we construct the transparent fragment dataset named TransFrag27K, which includes a multiscene synthetic data of broken fragments from multiple types of transparent objects, and a scalable synthetic data generation pipeline. Secondly, we propose a visual grasping position detection network named TransFragNet to identify, locate and segment the sampling grasping position. And, we use a two-finger gripper with Gelsight Mini sensors to obtain reconstructed tactile information of the lateral edge of the fragments. By fusing this tactile information with visual cues, a visual-tactile fusion material classifier is proposed. Inspired by the way humans estimate a fragment's contour combining vision and touch, we introduce a general transparent fragment contour estimation framework based on visual-tactile fusion, demonstrates strong performance in real-world validation. Finally, a multi-dimensional similarity metrics based contour matching and reassembly algorithm is proposed, providing a reproducible benchmark for evaluating visual-tactile contour estimation and fragment reassembly. The experimental results demonstrate the validity of the proposed framework. The dataset and codes are available at https://github.com/Keithllin/Transparent-Fragments-Contour-Estimation.
Figures & tables
Fig. 1: Visual-tactile fusion contour estimation inspired by the human strategy of combining vision with touch.
Fig. 2: (a) Illustrates the synthetic data generation pipeline, showcasing scene construction with elements like scene light, extra light, camera, and the projection of transparent fragments onto a surface. (b) Displays a subset of the generated dataset, featuring paired RGB images and their corresponding masks for transparent fragments.
Fig. 3: Examples of intact transparent objects used to generate synthetic fracture scenarios in the proposed TransFrag27K dataset. The collection includes a diverse range of materials, shapes, and surface textures, such as glassware, lab beakers, cups, bottles, and decorative items. The serial number corresponds to the state of the corresponding object after adding the fracture effect.
Fig. 4: This figure details the steps involved in generating the synthetic data for TransFrag27K. For various transparent objects (e.g., glass, plate, bottle), the second column shows the simulation of breakage, followed by the resulting RGB image and its corresponding mask, highlighting the visual shape of individual transparent fragments.
Fig. 5: Visual-tactile fusion based contour estimation framework for transparent fragment reassembly. (a) TransFragNet based visual grasping position detection and segmentation (b) Fragment reassembly using visual-tactile fusion based fragments contour estimation framework in mixed fragment scenes (c) Tactile information reconstruction and matching.
Fig. 6: The architecture of the proposed TransFragNet. The model employs a Swin Transformer backbone for robust feature extraction, followed by a dual-path decoder that integrates deformable transposed convolution and anti-aliased upsampling. The final segmentation head generates pixel-wise probability masks, which are further refined through morphological post-processing.
Fig. 7: Segmentation results for transparent fragments, including both single and multiple fragment scenarios. The top row shows the original input images, while the middle two rows display the annoted mask and predicted segmentation masks generated by our model. The bottom row illustrates the interaction corner points by applying morphological post-processing.
Fig. 8: Tactile reconstruction pipeline: recovering contact masks, local gradient fields, and detailed height maps from GelSight Mini data. These outputs provide fine-scale edge and curvature details critical for contour estimation and fragment matching.
Fig. 9: Architecture of the U-Net convolutional network used for segmenting tactile contact regions. This architecture is specifically designed to segment the contact regions in GelSight Mini tactile images, achieving high accuracy and sharp boundary delineation for subsequent tactile reconstruction tasks.
Fig. 10: Dual-branch late fusion network architecture for visual-tactile material classification. The visual branch employs ResNet-18 for robust feature extraction from RGB images, while the tactile branch uses a lightweight CNN to process GelSight Mini tactile data. Features are concatenated at the late fusion stage and processed through a multi-layer classifier.
Fig. 11: Visual–tactile contour estimation results for transparent fragments. Each panel shows an individual fragment with its extracted global mask (center top) and reconstructed tactile profiles: RGB tactile image, edge mask, gradient map, and height map. The color-coded label indicates the final material classification result obtained through visual–tactile fusion.
Fig. 12: Framework overview of the proposed visual-tactile fusion based contour estimation for transparent fragment reassembly.
Fig. 13: The contour matching results and scores of fragments of some different materials: including edge geometry alignment, local area gradient matching, and height extreme value complementation, are combined into a unified score for robust transparent fragment reassembly.
Fig. 14: Hardware setup: UR5 robotic arm with DH-95 two-finger gripper and integrated GelSight Mini tactile sensors.
Model
Synthetic Data (IoU)
Varying Backgrounds (IoU)
Varying Illumination (IoU)
EfficientNet
0.7875
0.7536
0.6430
MobileNet
0.7349
0.6366
0.3205
TGCNN
0.4576
0.2646
0.5968
Ours
0.8856
0.9382
0.8451
TABLE I: Comparison of mask prediction results on the synthetic dataset and real dataset. From top to bottom: Original Image, TGCNN, EfficientNet, MobileNet, and Our Model.
Fig. 15: Transparent fragment detection results and postprocessing. Including synthetic fragments detected and real fragments detected under different backgrounds.
Fig. 16: Transparent fragment detection in varying illumination, from left to right starting from 400 to 6000 Lux.
Metric
IoU
Chamfer Distance
Pair 1
0.82
4.3 px
Pair 2
0.79
5.1 px
Pair 3
0.85
3.8 px
Pair 4
0.81
2.4 px
Pair 5
0.74
4.5 px
Pair 6
0.82
5.2 px
TABLE II: Results for Partial Fracture Reassembly
Fig. 17: Visual shape-based partial fracture reassembly results: five representative cases with different notched transparent objects, showing remaining body, detached fragment, centroid alignment, and final angle-aligned overlay.
Fig. 18: Workflow of the proposed visual–tactile transparent fragment reassembly framework. The framework begins with visual detection using an RGB-D camera to segment transparent fragments and estimate grasping positions using TransFragNet. Fragments’ extracted visual features are then matched in the database. If a match is not found in the existing database, new tactile features sampling process including edge maps, gradient fields, and height profiles are extracted and stored. Matched fragments are classified by material and assigned IDs based on multi-dimensional feature similarity, then passed to the reassembly stage.
Material
Edge Score
Gradient Score
Height Score
Total Fusion
Glass (Pair 1)
0.857
0.814
0.913
0.859
Glass (Pair 2)
0.887
0.908
0.891
0.892
Acrylic (Pair 1)
0.844
0.880
0.917
0.865
Colored Crystal (Pair 1)
0.836
0.932
0.818
0.852
Plastic (Pair 1)
0.846
0.870
0.936
0.869
Colorless Crystal (Pair 1)
0.887
0.884
0.906
0.890
TABLE III: Visual-Tactile Fusion Reassembly Scores for Different Materials
Fig. 20: Experimental pipeline for complete reassembly with visual-tactile fusion, demonstrated with a colorless crystal dish: (a) full fragment set; (b) edge sampling and reconstruction; (c) random placement scene; (d) match visual ID with Database and excute reassembly; (e-f) final reassembled result with different perspectives.
Fig. 21: Generalization test for complete fracture reassembly with unseen or similar transparent fragments. (a) Experimental scene with mixed-material fragments including glass, acrylic, and colored/colorless crystal pieces. (b) Top view with highlighted examples: CF3 (colored crystal) and CL3 (colorless crystal) as shape-similar pairs; P3 as an unknown plastic fragment not stored in the database. (c) Edge sampling procedure using the GelSight Mini sensor for fallback classification. (d) Final identification map showing each fragment’s assigned material class and unique ID, providing full guidance for robust reassembly.
Fig. 22: Reassembly results of mixed fragments from completely fractured transparent objects made from different transparent materials: (a) acrylic dish; (b) plastic dish; (c) glass dish; (d) colorless crystal dish; (e) colored crystal dish. (f-j) the same as (a-e) but with different perspectives.
Scene Description
Top-1 Acc.
Top-3 Acc.
Mean IoU
Fallback Needed
Similar-Look Case (CL3 vs CF3)
50%
100%
0.79
Yes
Unknown Fragment (P3)
0%
—
—
Yes
Full Mixed Material Scene
85%
100%
0.81
Resolved
TABLE IV: Performance in Mixed Scene with Similar and Unseen Fragments
Precision assembly requires sub-millimeter corrections in contact-rich "last-millimeter" regions where visual feedback fails due to occlusion from the end-effector and workpiece. We present ReTac-ACT (Reconstruction-enhanced Tactile ACT), a vision-tactile imitation learning policy that addresses this challenge through three synergistic mechanisms: (i) bidirectional cross-attention enabling reciprocal visuo-tactile feature enhancement before fusion, (ii) a proprioception-conditioned gating network that dynamically elevates tactile reliance when visual occlusion occurs, and (iii) a tactile reconstruction objective enforcing learning of manipulation-relevant contact information rather than generic visual textures. Evaluated on the standardized NIST Assembly Task Board M1 benchmark, ReTac-ACT achieves 90% peg-in-hole success, substantially outperforming vision-only and generalist baseline methods, and maintains 80% success at industrial-grade 0.1mm clearance. Ablation studies validate that each architectural component is indispensable. The ReTac-ACT codebase and a vision-tactile demonstration dataset covering various clearance levels with both visual and tactile features will be released to support reproducible research.
Minchi Ruan, LiangQing Zhou, Hongtong Li +4
Beijing University of Posts and Telecommunications, Beijing, China · SunHDex Intelligent Technology (Beijing) Co. Ltd., Beijing, China · University of Hamburg, Hamburg, Germany
Accurate 6-DoF object pose estimation is fundamental to robotic manipulation, yet vision-based methods often fail under occlusion, poor lighting, and reflective or transparent surfaces. We present YOTO, a tactile-only pose estimation system that recovers the full 6-DoF object pose from a single pair of simultaneous contacts, without requiring contact history. YOTO represents each tactile contact as a local 3D point cloud and localizes it on the object surface through a coarse-to-fine network. The two localized contacts, together with the calibrated sensor poses, are then fed to a closed-form normal-aware SVD solver that recovers the full 6-DoF object pose in one step. To reduce real-data requirements, the localization network is pretrained on virtual tactile patches sampled from the object model and fine-tuned with a small number of real contacts. We further show that YOTO can operate on object models reconstructed from consumer-grade mobile scans, and quantify the gap relative to CAD-based models. Experiments on four geometrically diverse objects demonstrate accurate tactile contact localization and pose estimation, outperforming vision-based and geometric baselines, especially when visual perception is unreliable. Code, trained models, and the real GelSight dataset will be released upon publication.
Pengfei Ye, Yuxiang Ma, Haonan Chen +5
MIT CSAIL · Harvard University · University of Cambridge
Pose estimation is essential for robotic manipulation, particularly when visual perception is occluded during gripper-object interactions. Existing tactile-based methods generally rely on tactile simulation or pre-trained models, which limits their generalizability and efficiency. In this study, we propose TacLoc, a novel tactile localization framework that formulates the problem as a one-shot point cloud registration task. TacLoc introduces a graph-theoretic partial-to-full registration method, leveraging dense point clouds and surface normals from tactile sensing for efficient and accurate pose estimation. Without requiring rendered data or pre-trained models, TacLoc achieves improved performance through normal-guided graph pruning and a hypothesis-and-verification pipeline. TacLoc is evaluated extensively on the YCB dataset. We further demonstrate TacLoc on real-world objects across two different visual-tactile sensors.