We present an active tactile exploration framework for joint object recognition and 6D pose estimation. The proposed method integrates wrist force/torque sensing, GelSight tactile sensing, and free-space constraints within a Bayesian inference framework that maintains a belief over object class and pose during active tactile exploration. By combining contact and non-contact evidence, the framework reduces ambiguity and improves robustness in the joint class-pose estimation problem. To enable efficient inference in the large hypothesis space, we employ a customized particle filter that progressively samples particles based on new observations. The inferred belief is further used to guide active exploration by selecting informative next touches under reachability constraints. For effective data collection, a motion planning and control framework is developed to plan and execute feasible paths for tactile exploration, handle unexpected contacts and GelSight-surface alignment with tactile servoing. We evaluate the framework in simulation and on a Franka Panda robot using 11 YCB objects. Results show that incorporating tactile and free-space information substantially improves recognition and pose estimation accuracy and stability, while reducing the number of action cycles compared with force/torque-only baselines. Code, dataset, and supplementary material will be made available online.
Figures & tables
Fig. 2 : Overview of the proposed system. The system processes measurements from the GelSight sensor, the force/torque sensor, and robot proprioception to extract contact points and contact-type labels, and to build a free-space map. These observations are then fed to a customized particle filter, which performs Bayesian inference over object class and 6D pose. The particle filter proposes new particles conditioned on the incoming observations, approximates their weights, and mixes them with the existing particle set for subsequent belief updates, yielding both the MAP estimate and the posterior distribution. The MAP estimate is used to select the next target point and approach direction for data collection. Given the target point and approach direction, our planning and control stack generates and executes a trajectory with compliance, and aligns the GelSight sensor with the contact surface once contact is detected. Each module shown in the figure is described in detail in the Section III .
Fig. 3 : Example of force/torque-based contact localization. The cross indicates the outward direction of the measured torque T . From the measured wrench (F,T) , two candidate contact points x1 and x2 are computed. The estimate x∗ is chosen as x1 since ∠(n1,F)>135∘ while ∠(n2,F)<135∘ .
Fig. 4 : Example of the three particle sampling schemes. Point-pair-based sampling: for each observed contact point pair (xi,ni) and (xj,nj) , we compute point-pair features ∥xi,j∥2 , θi,j , θi,d , θj,d , and labels li,lj , where xi,j=xj−xi . Here θi,j is the angle between ni and nj , while θi,d and θj,d are the angles between ni and xi,j , and between nj and xi,j , respectively. Model point pairs with similar features are retrieved from the database; aligning a model pair to the observed pair yields candidate class/pose hypotheses. Local random variations: additional particles are generated by applying small perturbations to the poses of the top-weighted particles (e.g., ps→ps′,ps′′ ). Iterative closest point(ICP) sampling: ICP is applied to refine the poses of top particles using the accumulated contact points, producing new hypotheses that better fit the observations (e.g., ps→ps∗ ). In the illustration, cs,ce,cr denote three object classes (square, ellipse, and rectangle), with canonical poses pso,peo,pro and aligned poses ps,pe,pr .
Fig. 5 : An overview of one action cycle in the planning and control stack. The target selection scheme proposes target points with normal vectors as inputs to the Bi-RRT* motion planner until a feasible path to the target is found. The path is executed with joint impedance control. If a contact is detected by the F/T sensor during execution, it triggers tactile servoing to align the GelSight sensor with the contact surface.
Fig. 6 : An overview of the trajectory planning process given a target point with a normal direction. The trajectory consists of three segments: from the current pose to a pre-touch pose where the sensor aligns with the normal vector and faces directly toward the target point; from the pre-touch pose moving long the opposite normal direction to a touch pose to establish contact with the estimated surface; from the touch pose to the fallback pose located at the centroid of the existing contact points.
Fig. 7 : Control scheme for tactile servoing: at each step of the control loop (60Hz), we compute a target gel-center position xd and a rotation (θ,r) to align the gel normal nG (blue) with the opposite external force direction −fE (red) while maintaining contact. The commands are derived from the estimated contact location c , current gel center xG (green), nG , and fE . A Cartesian impedance controller outputs joint torques τ to drive xG→xd and nG→−,fE , ideally converging to a force equilibrium (superscript ∗ ).
#Actions
Simulation
Real world
FT
FT+GS
FT+FS
FT+GS+FS
FT
FT+GS
FT+FS
FT+GS+FS
Classification
4.3±3.2
4.1±2.7
3.6±2.0
3.4±1.9
6.6±4.7
6.7±4.6
5.0±3.1
4.2±2.2
Pose
6.6±3.7
6.1±3.5
5.6±2.4
4.7±2.1
8.6±4.5
7.9±4.4
6.3±2.8
5.3±1.7
TABLE I: Mean and standard deviation of action cycles needed to obtain the correct class and the desired pose error on average.
Fig. 8 : The 3D-printed YCB objects used in the experiments.
Fig. 9 : Class-wise number of actions required to find the correct class and reach the desired pose estimation error in simulation and real-world experiments.
Fig. 10 : An example of how the estimated class and pose evolves as more data are collected, taken from one of the real-world experiments on the mustard bottle. The top row shows the ground truth object class and pose in blue and the MAP estimation in red, with the FT contact points (green) and GelSight contact points (black + red). The bottle row shows the belief distribution on the object class. From left to right, as more data are collected, the framework gradually became more confident about the object class and refined the object pose.
Distance (mm)
Simulation
Real world
mean
stdv
max
mean
stdv
max
GelSight
1.2
0.8
6.5
3.3
2.0
17.7
Force torque
0.7
0.8
6.4
4.5
2.4
36.1
TABLE II: Mean, standard deviation and maximum of the distance from the recorded contact points to the ground truth surface in simulation and in real-world.
Fig. 11 : Classification success rate and pose estimation success rate over time step in the simulated and real-world experiments.
ADD-S (mm)
Simulation
Real world
FT
FT+GS
FT+FS
FT+GS+FS
FT
FT+GS
FT+FS
FT+GS+FS
mean ± stdv
1.8±1.3
1.8±0.9
1.6±0.6
1.6±0.7
6.1±8.6
7.8±11.8
3.3±1.1
2.9±0.8
max
9.3
4.5
2.9
3.4
48.8
49.8
9.1
4.9
TABLE III: Mean and standard deviation of the final pose estimation error (after 15 action cycles) measured by ADD for simulation and real-world experiments average over all object classes.
Contact-rich manipulation requires pose estimates that are often more accurate than what depth-only sensing provides. Existing methods, relying on vision and contact, employ costly offline training procedures that need to be retrained for new environments and geometries. We propose BayesContact, a Simulation-Based Inference framework for visuo-tactile pose estimation in peg-in-hole insertion. BayesContact maintains a particle belief over object pose and fuses depth observations with force/torque-derived contact evidence. We employ simulation based forward models to approximate these observation likelihoods. For each pose hypothesis, a renderer predicts depth measurements and a physics simulator predicts contact outcomes under guarded probing actions; both are scored against real observations to update the belief. The resulting multimodal belief also enables information-gain-based probing for active disambiguation. Across simulated geometries and real-robot experiments, BayesContact improves pose observability and insertion success over vision-only inference by 30%
Accurate 6-DoF object pose estimation is fundamental to robotic manipulation, yet vision-based methods often fail under occlusion, poor lighting, and reflective or transparent surfaces. We present YOTO, a tactile-only pose estimation system that recovers the full 6-DoF object pose from a single pair of simultaneous contacts, without requiring contact history. YOTO represents each tactile contact as a local 3D point cloud and localizes it on the object surface through a coarse-to-fine network. The two localized contacts, together with the calibrated sensor poses, are then fed to a closed-form normal-aware SVD solver that recovers the full 6-DoF object pose in one step. To reduce real-data requirements, the localization network is pretrained on virtual tactile patches sampled from the object model and fine-tuned with a small number of real contacts. We further show that YOTO can operate on object models reconstructed from consumer-grade mobile scans, and quantify the gap relative to CAD-based models. Experiments on four geometrically diverse objects demonstrate accurate tactile contact localization and pose estimation, outperforming vision-based and geometric baselines, especially when visual perception is unreliable. Code, trained models, and the real GelSight dataset will be released upon publication.
Pengfei Ye, Yuxiang Ma, Haonan Chen +5
MIT CSAIL · Harvard University · University of Cambridge
Pose estimation is essential for robotic manipulation, particularly when visual perception is occluded during gripper-object interactions. Existing tactile-based methods generally rely on tactile simulation or pre-trained models, which limits their generalizability and efficiency. In this study, we propose TacLoc, a novel tactile localization framework that formulates the problem as a one-shot point cloud registration task. TacLoc introduces a graph-theoretic partial-to-full registration method, leveraging dense point clouds and surface normals from tactile sensing for efficient and accurate pose estimation. Without requiring rendered data or pre-trained models, TacLoc achieves improved performance through normal-guided graph pruning and a hypothesis-and-verification pipeline. TacLoc is evaluated extensively on the YCB dataset. We further demonstrate TacLoc on real-world objects across two different visual-tactile sensors.