Accurate shape estimation of Deformable Linear Objects (DLOs) is a fundamental challenge in robotic manipulation, often hindered by visual occlusions and the high-dimensional configuration space of flexible materials. While Force/Torque (F/T) and vision-centric methods have been proposed, they frequently struggle with grasp-site ambiguities and environmental interference. In this letter, we investigate the hypothesis that global shape estimation can be achieved through tactile sensing and proprioception, without any visual perception. We contend that dense, high-resolution tactile imprints capture critical boundary conditions and contact gradients that are sufficient to resolve the high-dimensional configuration of the DLO. To prove this, we introduce a Physics-Informed Vision Transformer (ViT) framework that maps tactile images to a 3D vertex chain. By leveraging self-attention mechanisms and a physics-weighted loss function, our model enforces structural priors to ensure that the predicted geometry adheres to the object's physical manifold. Experimental results demonstrate that our tactile-based approach achieves superior reconstruction accuracy across Euclidean and Chamfer metrics. Furthermore, we demonstrate the efficiency of this modality, achieving model transfer to novel DLO materials and geometries. Our findings suggest that tactile feedback provides a robust and computationally efficient alternative for DLO state estimation in visually degraded or occluded environments.
Figures & tables
Fig. 1 : Overview of the tactile-based DLO shape estimation system. Dual robotic grippers ( G1,G2 ) manipulate a DLO while integrated tactile sensors provide contact images ( Iij,t ).
Fig. 2 : Architecture of the proposed ViT-Sparsh (ViT-S) -based shape estimation model. The system processes synchronous inputs from two grippers, each providing tactile images and gripper pose data. A ViT-based encoder extracts high-dimensional tactile features which are concatenated with proprioceptive embeddings to form input tokens. These tokens are processed by a cross-attention decoder to capture long-range spatial correlations between the two contact sites. Finally, a differentiable B-spline layer enforces geometric smoothness. The output is a spatial vertex chain representing the DLO configuration.
Model
Num. trainable
Num. total
RMSE
Chamfer
Inference
Freq.
param. ( ×106 )
param. ( ×106 )
(mm)
dis. (mm)
time (ms)
(Hz)
Pose-Only (MLP)
3.04
3.04
55.00±22.00
44.00±19.00
0.9
1.11×103
Pose + Direction
2.70
2.70
50.63±24.25
40.70±17.06
-
-
ResNet-18 + Cross-Attn
3.49
3.49
36.00±20.00
30.00±16.00
17.6
56.82
Propose model w/o B-spline layer
5.33
23.74
35.01±16.62
30.06±13.64
61.5
16.26
Proposed model w/o Lweighted
6.38
24.80
37.00±20.90
30.82±16.42
61.5
16.26
TABLE I : Comparison of shape estimation performance
Fig. 3 : Visual examples of ground truth and predicted shapes for the three DLOs. (a) 1.5 mm diameter steel cable with estimation error of 34.1mm , (b) 4.5 mm diameter USB cable with estimation error of 17.7mm , and (c) 2.0 mm diameter steel cable with estimation error of 41.5mm . Each row displays the experimental configuration, the corresponding four-channel raw tactile input, and the resulting 3D shape estimation (cyan: ground truth, magenta: prediction).
Fig. 4 : Shape estimation accuracy with respect to the number of samples in the training data.
Fig. 5 : Transfer estimation errors for (a) a USB cable of 4.5mm diameter and length 550mm and (b) a steel cable of 2mm diameter and 750mm length, with regard to the number of new training samples used to fine-tune the model. Dotted lines show training of the models from scratch with the new data.
Fig. 6 : Chamfer distance and RMSE for shape estimation with induced noise to the tactile images.
Num. of Sensors
RMSE
Chamfer
removed
(mm)
dis. (mm)
New trained model
1
34.2±18.0
30.1±14.2
2
37.8±21.2
34.0±16.9
3
46.3±23.3
39.6±17.6
Same model w/ fault sensors
1
45.3±23.3
36.4±16.7
2
70.9±30.8
53.4±20.2
TABLE II : Ablation study on the number of removed sensors
Pose estimation is essential for robotic manipulation, particularly when visual perception is occluded during gripper-object interactions. Existing tactile-based methods generally rely on tactile simulation or pre-trained models, which limits their generalizability and efficiency. In this study, we propose TacLoc, a novel tactile localization framework that formulates the problem as a one-shot point cloud registration task. TacLoc introduces a graph-theoretic partial-to-full registration method, leveraging dense point clouds and surface normals from tactile sensing for efficient and accurate pose estimation. Without requiring rendered data or pre-trained models, TacLoc achieves improved performance through normal-guided graph pruning and a hypothesis-and-verification pipeline. TacLoc is evaluated extensively on the YCB dataset. We further demonstrate TacLoc on real-world objects across two different visual-tactile sensors.
Accurate 6-DoF object pose estimation is fundamental to robotic manipulation, yet vision-based methods often fail under occlusion, poor lighting, and reflective or transparent surfaces. We present YOTO, a tactile-only pose estimation system that recovers the full 6-DoF object pose from a single pair of simultaneous contacts, without requiring contact history. YOTO represents each tactile contact as a local 3D point cloud and localizes it on the object surface through a coarse-to-fine network. The two localized contacts, together with the calibrated sensor poses, are then fed to a closed-form normal-aware SVD solver that recovers the full 6-DoF object pose in one step. To reduce real-data requirements, the localization network is pretrained on virtual tactile patches sampled from the object model and fine-tuned with a small number of real contacts. We further show that YOTO can operate on object models reconstructed from consumer-grade mobile scans, and quantify the gap relative to CAD-based models. Experiments on four geometrically diverse objects demonstrate accurate tactile contact localization and pose estimation, outperforming vision-based and geometric baselines, especially when visual perception is unreliable. Code, trained models, and the real GelSight dataset will be released upon publication.
Pengfei Ye, Yuxiang Ma, Haonan Chen +5
MIT CSAIL · Harvard University · University of Cambridge
Accurate object geometry estimation is essential for many downstream tasks, including robotic manipulation and physical interaction. Although vision is the dominant modality for shape perception, it becomes unreliable under occlusions or challenging lighting conditions. In such scenarios, tactile sensing provides direct geometric information through physical contact. However, reconstructing global 3D geometry from sparse local touches alone is fundamentally underconstrained. We present TouchAnything, a framework that leverages a pretrained large-scale 2D vision diffusion model as a semantic and geometric prior for 3D reconstruction from sparse tactile measurements. Unlike prior work that trains category-specific reconstruction networks or learns diffusion models directly from tactile data, we transfer the geometric knowledge encoded in pretrained visual diffusion models to the tactile domain. Given sparse contact constraints and a coarse class-level description of the object, we formulate reconstruction as an optimization problem that enforces tactile consistency while guiding solutions toward shapes consistent with the diffusion prior. Our method reconstructs accurate geometries from only a few touches, outperforms existing baselines, and enables open-world 3D reconstruction of previously unseen object instances. Our project page is https://grange007.github.io/touchanything .
Langzhe Gu, Hung-Jui Huang, Mohamad Qadri +2
Tsinghua University · University of Illinois Urbana-Champaign · Carnegie Mellon University