Bayesian Active Object Recognition and 6D Pose Estimation from Multimodal Contact Sensing

Haodong Zheng; Gabriele M. Caddeo; Andrei C. Jalba; Wijnand A. IJsselsteijn; Lorenzo Natale; Raymond H. Cuijpers

cs.ROMar 22, 2026

Bayesian Active Object Recognition and 6D Pose Estimation from Multimodal Contact Sensing

Authors: Haodong Zheng, Gabriele M. Caddeo, Andrei C. Jalba, Wijnand A. IJsselsteijn, Lorenzo Natale, Raymond H. Cuijpers

Abstract

We present an active tactile exploration framework for joint object recognition and 6D pose estimation. The proposed method integrates wrist force/torque sensing, GelSight tactile sensing, and free-space constraints within a Bayesian inference framework that maintains a belief over object class and pose during active tactile exploration. By combining contact and non-contact evidence, the framework reduces ambiguity and improves robustness in the joint class-pose estimation problem. To enable efficient inference in the large hypothesis space, we employ a customized particle filter that progressively samples particles based on new observations. The inferred belief is further used to guide active exploration by selecting informative next touches under reachability constraints. For effective data collection, a motion planning and control framework is developed to plan and execute feasible paths for tactile exploration, handle unexpected contacts and GelSight-surface alignment with tactile servoing. We evaluate the framework in simulation and on a Franka Panda robot using 11 YCB objects. Results show that incorporating tactile and free-space information substantially improves recognition and pose estimation accuracy and stability, while reducing the number of action cycles compared with force/torque-only baselines. Code, dataset, and supplementary material will be made available online.

Figures & tables

Fig. 2 : Overview of the proposed system. The system processes measurements from the GelSight sensor, the force/torque sensor, and robot proprioception to extract contact points and contact-type labels, and to build a free-space map. These observations are then fed to a customized particle filter, which performs Bayesian inference over object class and 6D pose. The particle filter proposes new particles conditioned on the incoming observations, approximates their weights, and mixes them with the existing particle set for subsequent belief updates, yielding both the MAP estimate and the posterior distribution. The MAP estimate is used to select the next target point and approach direction for data collection. Given the target point and approach direction, our planning and control stack generates and executes a trajectory with compliance, and aligns the GelSight sensor with the contact surface once contact is detected. Each module shown in the figure is described in detail in the Section III .

Fig. 3 : Example of force/torque-based contact localization. The cross indicates the outward direction of the measured torque

\mathbf{T}

. From the measured wrench

(\mathbf{F},\mathbf{T})

, two candidate contact points

\mathbf{x}_{1}

and

\mathbf{x}_{2}

are computed. The estimate

\mathbf{x}^{*}

is chosen as

\mathbf{x}_{1}

since

\angle(\mathbf{n}_{1},\mathbf{F})>135^{\circ}

while

\angle(\mathbf{n}_{2},\mathbf{F})<135^{\circ}

Fig. 4 : Example of the three particle sampling schemes. Point-pair-based sampling: for each observed contact point pair $(\mathbf{x}_{i},\mathbf{n}_{i})$ and $(\mathbf{x}_{j},\mathbf{n}_{j})$ , we compute point-pair features $\|\mathbf{x}_{i,j}\|_{2}$ , $\theta_{i,j}$ , $\theta_{i,d}$ , $\theta_{j,d}$ , and labels $l_{i},l_{j}$ , where $\mathbf{x}_{i,j}=\mathbf{x}_{j}-\mathbf{x}_{i}$ . Here $\theta_{i,j}$ is the angle between $\mathbf{n}_{i}$ and $\mathbf{n}_{j}$ , while $\theta_{i,d}$ and $\theta_{j,d}$ are the angles between $\mathbf{n}_{i}$ and $\mathbf{x}_{i,j}$ , and between $\mathbf{n}_{j}$ and $\mathbf{x}_{i,j}$ , respectively. Model point pairs with similar features are retrieved from the database; aligning a model pair to the observed pair yields candidate class/pose hypotheses. Local random variations: additional particles are generated by applying small perturbations to the poses of the top-weighted particles (e.g., $\mathbf{p}_{s}\rightarrow\mathbf{p}^{\prime}_{s},\mathbf{p}^{\prime\prime}_{s}$ ). Iterative closest point(ICP) sampling: ICP is applied to refine the poses of top particles using the accumulated contact points, producing new hypotheses that better fit the observations (e.g., $\mathbf{p}_{s}\rightarrow\mathbf{p}^{*}_{s}$ ). In the illustration, $c_{s},c_{e},c_{r}$ denote three object classes (square, ellipse, and rectangle), with canonical poses $\mathbf{p}^{o}_{s},\mathbf{p}^{o}_{e},\mathbf{p}^{o}_{r}$ and aligned poses $\mathbf{p}_{s},\mathbf{p}_{e},\mathbf{p}_{r}$ .

Fig. 5 : An overview of one action cycle in the planning and control stack. The target selection scheme proposes target points with normal vectors as inputs to the Bi-RRT* motion planner until a feasible path to the target is found. The path is executed with joint impedance control. If a contact is detected by the F/T sensor during execution, it triggers tactile servoing to align the GelSight sensor with the contact surface.

Fig. 6 : An overview of the trajectory planning process given a target point with a normal direction. The trajectory consists of three segments: from the current pose to a pre-touch pose where the sensor aligns with the normal vector and faces directly toward the target point; from the pre-touch pose moving long the opposite normal direction to a touch pose to establish contact with the estimated surface; from the touch pose to the fallback pose located at the centroid of the existing contact points.

Fig. 7 : Control scheme for tactile servoing: at each step of the control loop (60Hz), we compute a target gel-center position

\mathbf{x}_{d}

and a rotation

(\theta,\mathbf{r})

to align the gel normal

\mathbf{n}_{G}

(blue) with the opposite external force direction

-\mathbf{f}_{E}

(red) while maintaining contact. The commands are derived from the estimated contact location

\mathbf{c}

, current gel center

\mathbf{x}_{G}

(green),

\mathbf{n}_{G}

, and

\mathbf{f}_{E}

. A Cartesian impedance controller outputs joint torques

\tau

to drive

\mathbf{x}_{G}\rightarrow\mathbf{x}_{d}

and

\mathbf{n}_{G}\rightarrow-,\mathbf{f}_{E}

, ideally converging to a force equilibrium (superscript ∗ ).

#Actions	Simulation				Real world
	FT	FT+GS	FT+FS	FT+GS+FS	FT	FT+GS	FT+FS	FT+GS+FS
Classification	$4.3\pm 3.2$	$4.1\pm 2.7$	$3.6\pm 2.0$	$\bm{3.4\pm 1.9}$	$6.6\pm 4.7$	$6.7\pm 4.6$	$5.0\pm 3.1$	$\bm{4.2\pm 2.2}$
Pose	$6.6\pm 3.7$	$6.1\pm 3.5$	$5.6\pm 2.4$	$\bm{4.7\pm 2.1}$	$8.6\pm 4.5$	$7.9\pm 4.4$	$6.3\pm 2.8$	$\bm{5.3\pm 1.7}$

TABLE I: Mean and standard deviation of action cycles needed to obtain the correct class and the desired pose error on average.

Fig. 8 : The 3D-printed YCB objects used in the experiments.

Fig. 9 : Class-wise number of actions required to find the correct class and reach the desired pose estimation error in simulation and real-world experiments.

Fig. 10 : An example of how the estimated class and pose evolves as more data are collected, taken from one of the real-world experiments on the mustard bottle. The top row shows the ground truth object class and pose in blue and the MAP estimation in red, with the FT contact points (green) and GelSight contact points (black + red). The bottle row shows the belief distribution on the object class. From left to right, as more data are collected, the framework gradually became more confident about the object class and refined the object pose.

Distance (mm)	Simulation			Real world
	mean	stdv	max	mean	stdv	max
GelSight	1.2	0.8	6.5	3.3	2.0	17.7
Force torque	0.7	0.8	6.4	4.5	2.4	36.1

TABLE II: Mean, standard deviation and maximum of the distance from the recorded contact points to the ground truth surface in simulation and in real-world.

Fig. 11 : Classification success rate and pose estimation success rate over time step in the simulated and real-world experiments.

ADD-S (mm)	Simulation				Real world
	FT	FT+GS	FT+FS	FT+GS+FS	FT	FT+GS	FT+FS	FT+GS+FS
mean $\pm$ stdv	$1.8\pm 1.3$	$1.8\pm 0.9$	$\bm{1.6\pm 0.6}$	$1.6\pm 0.7$	$6.1\pm 8.6$	$7.8\pm 11.8$	$3.3\pm 1.1$	$\bm{2.9\pm 0.8}$
max	$9.3$	$4.5$	$\bm{2.9}$	$3.4$	$48.8$	$49.8$	$9.1$	$\bm{4.9}$

TABLE III: Mean and standard deviation of the final pose estimation error (after 15 action cycles) measured by ADD for simulation and real-world experiments average over all object classes.

Object Pose Estimation

Object pose estimation recovers an object's 3D position and orientation from images, depth, point clouds, or contact sensing.

See all 187 papers

BayesContact: Uncertain Pose Estimation via Visuo-Tactile Proposals and Simulation-based Inference
Jul 17, 2026Aditya Kamireddypalli, Matias Mattamala, Joao Moura +3Simulation-Based Inference Contact-Rich Robotic Manipulation
School of Informatics, University of Edinburgh, Scotland. · Department of Mechanical and Mechatronics Engineering, University of Waterloo, Canada.
You Only Touch Once: 6-DoF Object Pose Estimation from Single Tactile Contact
Jun 27, 2026Pengfei Ye, Yuxiang Ma, Haonan Chen +5Object Pose Estimation Robotic Manipulation
MIT CSAIL · Harvard University · University of Cambridge
TacLoc: Global Tactile Localization on Objects from a Registration Perspective
Mar 11, 2026Zirui Zhang, Boyang Zhang, Fumin Zhang +1Object Pose Estimation Point Cloud Registration

Jul 17, 2026J/K move · Enter open · S save

BayesContact: Uncertain Pose Estimation via Visuo-Tactile Proposals and Simulation-based Inference

Aditya Kamireddypalli, Matias Mattamala, Joao Moura, Russell Buchanan+2

School of Informatics, University of Edinburgh, Scotland. · Department of Mechanical and Mechatronics Engineering, University of Waterloo, Canada.

Simulation-Based Inference Contact-Rich Robotic Manipulation Object Pose Estimation Robotic Insertion

Open paper AskPDF arXiv

Contact-rich manipulation requires pose estimates that are often more accurate than what depth-only sensing provides. Existing methods, relying on vision and contact, employ costly offline training procedures that need to be retrained for new environments and geometries. We propose BayesContact, a Simulation-Based Inference framework for visuo-tactile pose estimation in peg-in-hole insertion. BayesContact maintains a particle belief over object pose and fuses depth observations with force/torque-derived contact evidence. We employ simulation based forward models to approximate these observation likelihoods. For each pose hypothesis, a renderer predicts depth measurements and a physics simulator predicts contact outcomes under guarded probing actions; both are scored against real observations to update the belief. The resulting multimodal belief also enables information-gain-based probing for active disambiguation. Across simulated geometries and real-robot experiments, BayesContact improves pose observability and insertion success over vision-only inference by 30%

Abstract

Object Pose Estimation

Explore similar work

BayesContact: Uncertain Pose Estimation via Visuo-Tactile Proposals and Simulation-based Inference

You Only Touch Once: 6-DoF Object Pose Estimation from Single Tactile Contact

TacLoc: Global Tactile Localization on Objects from a Registration Perspective