Pose estimation is essential for robotic manipulation, particularly when visual perception is occluded during gripper-object interactions. Existing tactile-based methods generally rely on tactile simulation or pre-trained models, which limits their generalizability and efficiency. In this study, we propose TacLoc, a novel tactile localization framework that formulates the problem as a one-shot point cloud registration task. TacLoc introduces a graph-theoretic partial-to-full registration method, leveraging dense point clouds and surface normals from tactile sensing for efficient and accurate pose estimation. Without requiring rendered data or pre-trained models, TacLoc achieves improved performance through normal-guided graph pruning and a hypothesis-and-verification pipeline. TacLoc is evaluated extensively on the YCB dataset. We further demonstrate TacLoc on real-world objects across two different visual-tactile sensors.
Figures & tables
Fig. 1: Real-world demonstration using TacLoc. A GelSight Mini sensor touches a textureless 3D-printed object (left), producing a tactile image that is converted into a dense point cloud and registered to a prior CAD model (right). The zoomed view shows the tactile point cloud (blue) aligned on the CAD model (red).
Fig. 2: Overview of the TacLoc pipeline for one-shot global tactile localization. The process starts with feature extraction to establish initial correspondences. A compatibility graph is constructed based on distance and normal consistency, and maximal cliques are identified for pose hypothesis generation. Finally, a hypothesis-and-verification approach is applied to refine the pose estimation, achieving partial-to-full tactile localization. It is worth noting that a CAD model is indispensable as a prior map and end-effector poses are required to build submaps during sliding touch.
Fig. 3: For each frame captured by the tactile sensor, we first convert it into a point cloud with normals. Then we construct a submap representing either a sliding touch or multi-time retouch by integrating end-effector poses. This submap is then processed using our front-end preprocessing pipeline. Best viewed zoomed in and in color.
Fig. 4: Visualization of candidate extraction pipeline with a minimal example (best viewed in color). (a) Three points in the source point cloud (blue) and three points in the target point cloud (orange), connected by six initial correspondences (gray lines). (b) The correspondences form an undirected graph G , with nodes V={ξ1,1,ξ1,2,ξ2,1,ξ2,2,ξ2,3,ξ3,3} and edges E={{ξ1,1,ξ2,2},{ξ1,1,ξ3,3},{ξ1,2,ξ2,1},{ξ2,2,ξ3,3} , constructed via pairwise consistency checks. (c) The first maximal clique C1={ξ1,1,ξ2,2,ξ3,3} . (d) The second maximal clique C2={ξ1,2,ξ2,1} .
Module
Parameters
Value
Feature Extraction
Voxel size
1mm
Radii
5mm
Corr. Pruning
Distance threshold/bound δd
6mm
Angular threshold/bound δα
30∘
Pose Estimation
Number of pose candidates K
300
Weight balance α
1
TABLE I: Key Parameters of TacLoc for YCB-Reg
Fig. 5: Selected samples from the YCB-Reg benchmark. From left to right are ten objects from the YCB dataset [ 32 ] . Gray semi-transparent objects represent the target models, while green point clouds correspond to tactile-based point clouds using the pre-processing approach in Section IV-A . Zoomed views of these sliding samples are also presented at the bottom.
Front-end
Back-end
RE ( \SIUnitSymbolDegree )
TE ( mm )
Time ( sec )
FPFH
RANSAC
128.62
99.94
1.66
FPFH
TEASER++
19.89
8.46
13.04
FPFH
3DMAC
19.07
9.54
2.06
FPFH
TacLoc
0.94
0.69
1.40
SpinNet
RANSAC
118.63
55.89
0.05⋆
SpinNet
TacLoc
130.37
28.50
0.92⋆
TABLE II: Performance Comparison on Tactile Registration
Fig. 6: Recall rate evaluated on the YCB-Reg using the FPFH descriptor for different back-end approaches under varying criterion of successful registration. The results demonstrate that the proposed TacLoc consistently achieves better performance across all tested conditions.
Fig. 7: Failure case studies on YCB-Reg benchmark. In each subfigure, the left shows the ground truth while the right displays the misalignment. TacLoc fails in (a) due to the extremely low inlier ratio; the failure in (b) is caused by repetitive patterns in symmetric objects.
Fig. 8: Parameter analysis on two YCB-Reg objects. A longer sliding length enhances the performance of tactile localization, aligning with human intuition. Additionally, the accuracy of the end-effector pose is critical: a more precise end-effector pose results in a more consistent submap, thereby improving the performance of TacLoc.
Fig. 9: Experimental results demonstrate the efficacy of normal-guided pruning. (a) Graph density, number of cliques, and computation time decrease as δα varies; (b) The number of edges and computation time exhibit an approximately polynomial relationship. When δα is 30∘ or 180∘ , the average number of edges and computation times are 7,473.82 edges with 0.20s and 15,476.75 edges with 2.87s , respectively.
Fig. 10: Rotation error and time cost with varying thresholds δd and δα .
Fig. 11: Per-stage computational profile. FPFH-based initial data association and transformation verification are the most time-intensive stages.
Fig. 12: (a) TacLoc is deployed on two real-world sensors beyond the DIGIT used in YCB simulation, despite these sensors having different characteristics. The right side shows the visual-tactile images obtained from the sensors. (b) Evaluation on five real-world objects using the GelSight Mini. We perform sliding touches on the objects 50 times, and the proposed TacLoc achieves a success rate of 33/50. Note that the red object is used for the single-touch demonstration shown in Figure 1 . The right side displays the successful cases, where the yellow points represent the submaps.
Accurate 6-DoF object pose estimation is fundamental to robotic manipulation, yet vision-based methods often fail under occlusion, poor lighting, and reflective or transparent surfaces. We present YOTO, a tactile-only pose estimation system that recovers the full 6-DoF object pose from a single pair of simultaneous contacts, without requiring contact history. YOTO represents each tactile contact as a local 3D point cloud and localizes it on the object surface through a coarse-to-fine network. The two localized contacts, together with the calibrated sensor poses, are then fed to a closed-form normal-aware SVD solver that recovers the full 6-DoF object pose in one step. To reduce real-data requirements, the localization network is pretrained on virtual tactile patches sampled from the object model and fine-tuned with a small number of real contacts. We further show that YOTO can operate on object models reconstructed from consumer-grade mobile scans, and quantify the gap relative to CAD-based models. Experiments on four geometrically diverse objects demonstrate accurate tactile contact localization and pose estimation, outperforming vision-based and geometric baselines, especially when visual perception is unreliable. Code, trained models, and the real GelSight dataset will be released upon publication.
Pengfei Ye, Yuxiang Ma, Haonan Chen +5
MIT CSAIL · Harvard University · University of Cambridge
Vision and touch are complementary modalities essential for robotic perception and manipulation. While vision provides global object context, touch offers precise local information at contact points. Integrating these modalities for contact localization, i.e., predicting the location of touch on an object's surface, poses significant challenges due to the need for accurate spatial alignment between tactile data and visual geometry. To address this challenge, we propose VTLoc, a novel visual-tactile framework that localizes contact points from tactile readings using a 3D point cloud as visual input. VTLoc introduces two key components: a geometric multi-modal alignment module, which reconstructs a pseudo-point cloud from fused visual-tactile features and aligns it with the visual point cloud to enforce spatial consistencies across modalities; and an iterative localizing updater, which iteratively refines the predicted contact location using fused visual-tactile features. Evaluated on a new benchmark of 100 real-world objects, VTLoc improves single-touch contact localization by reducing local-to-global correspondence ambiguity.
Zhiyuan Wu, Zhuo Chen, Shan Luo
Department of Engineering, King’s College London, Strand, London, WC2R 2LS, United Kingdom
Recent advances in imitation learning and vision-language models highlight the need for high-fidelity tactile perception, with 6-DoF tactile object pose estimation providing a crucial foundation for precise robotic manipulation. We introduce InvariantCloud, a 6-DoF pose estimation framework that leverages the global invariance of surface marker constellations on vision-based tactile sensors. In contrast to recent approaches, our one-shot globally invariant point cloud registration suppresses cumulative drift and overcomes long-standing limitations in accurately estimating yaw (Z-axis) rotation. Experimental verifications show that InvariantCloud achieves superior yaw tracking accuracy and re-localization repeatability compared to existing benchmarks, demonstrating its precision and robustness in long-sequence manipulation tasks.
Pengfei Ye, Yuxiang Ma, Yi Zhou +3
Department of Mechanical and Aerospace Engineering, The Hong Kong University of Science and Technology · Department of Mechanical Engineering, Massachusetts Institute of Technology · Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong