Adaptive Manipulation Potential and Haptic Estimation for Tool-Mediated Interaction

Lin Yang; Anirvan Dutta; Yuan Ji; Yanxin Zhou; Shilin Shan; Lv Chen; Etienne Burdet; Domenico Campolo

Fig. 1: Tool-mediated haptic manipulation under visual occlusion. (A) Humans can perform skillful haptic manipulation with tools despite unreliable visual information, although sensing is limited to net forces and torques at the human–tool interface. (B) Multiple contact points may exist at the tool tip while only a single net wrench is observed, leading to contact ambiguity. (C) Humans rely on cognitive reasoning and internal models of tool use to infer task-relevant object properties, such as pose and geometry, through haptic interaction.

Fig. 2: Overall workflow of tool-mediated manipulation framework. Left: the robot uses a tool to interact with the object and senses the resulting net wrench

\overline{\bm{\mathcal{F}}}

. Middle:

\overline{\bm{\mathcal{F}}}

is fed into a haptic state estimator that incorporates a manifold-based internal physical model to predict the expected wrench

\bm{\mathcal{F}}

. The discrepancy between the observed and predicted wrenches—termed haptic mismatch —is used to update the estimated object pose

{\bm{{\theta}}}

and type

s

. Right: Based on the optimal estimates

(s^{*},{\bm{{\theta}}}^{*})

and associated uncertainty

{\bm{{\Sigma}}}_{{\bm{{\theta}}}}

, a planner and controller generate the control

{\bm{{u}}}(t)

and stiffness

{\bm{{K}}}_{c}

for robot.

A. Internal Model & Equilibrium Manifold (Sec. IV )
Symbol	Description
${\bm{{z}}}\in\mathcal{Z}$	Internal system state (e.g., spanner pose), evolving on the manifold
${\bm{{u}}}\in\mathcal{U}$	Robot control input (e.g., impedance equilibrium pose)
${\bm{{\theta}}}\in\Theta$	Continuous environment parameter (e.g., screw pose) to be estimated
$s\in\mathcal{S}$	Discrete environment parameter (e.g., spanner and screw shape/type)
$W^{(s)}({\bm{{z}}},{\bm{{u}}};{\bm{{\theta}}})$	Quasi-static manipulation potential function for shape hypothesis $s$

TABLE I: Nomenclature and Key Symbols

Fig. 3: Parameterized equilibrium manifolds and induced motion. For fixed environment parameters

(s,{\bm{{\theta}}})

, the quasi-static interaction between the robot and the environment defines an equilibrium manifold

\mathcal{M}_{eq}^{(s,{\bm{{\theta}}})}

embedded in the state-control space. Different values of the discrete parameter

s

select different manifolds, while the continuous parameter

{\bm{{\theta}}}

continuously deforms the manifold geometry within a given structural model. For fixed

(s,{\bm{{\theta}}})

, control inputs

{\bm{{u}}}

induce equilibrium configurations

{\bm{{z}}}^{*}({\bm{{u}}};{\bm{{\theta}}},s)

along the manifold. The resulting motion on

\mathcal{M}_{eq}^{(s,{\bm{{\theta}}})}

is governed by an adaptive ODE.

Fig. 4: Instantiation of the manipulation potential

W^{(s)}({\bm{{z}}},{\bm{{u}}};{\bm{{\theta}}})

through contact geometry and impedance control. Left: Schematic representation of the general case. Right: Illustrative example of a spanner–screw operation.

Fig. 5: Workflow of the hybrid inference in Haptic SLAM. For each discrete shape hypothesis

s^{(i)}

, multiple pose particles

{\bm{{\theta}}}^{(j)}[s^{(i)}]

are initialized to represent candidate configurations. These particles process sensed haptic observations

\overline{\mathcal{F}}

in parallel, independently updating their pose parameters

{\bm{{\theta}}}^{(j)}[s^{(i)}]

via a local optimizer driven by the internal physical model. A representative pose

{\bm{{\theta}}}^{*}[s^{(i)}]

is then identified for each shape, followed by a MAP selection and clustering process to determine the estimated object shape

s^{*}

and pose

{\bm{{\theta}}}^{*}

for the current batch. This inference cycle is re-executed in each subsequent batch as new haptic observations become available.

Fig. 6: Pipeline for pose and particle initialization from single-point contact.

Fig. 7: Physical and geometric interpretation of haptic SLAM: (a) Pose estimation (continuous update): For a fixed shape

s^{(2)}

, haptic mismatch

{\bm{{\epsilon}}}=\overline{\bm{\mathcal{F}}}-\bm{\mathcal{F}}

drives the deformation

\Delta{\bm{{\theta}}}

to align the expected manifold (light blue) with reality (dark blue). (b) Shape estimation (discrete selection): Parallel manifolds for distinct shapes, such as a green rectangle (

s^{(1)}

) and a blue hexagon (

s^{(2)}

), are compared to select the candidate based on

{\bm{{\epsilon}}}

.

Fig. 8: Schematic illustration of MPPI-based planning. Rollouts are evaluated using a haptic cost and an elite set is used to update the policy in a receding-horizon manner. An experimental replotted execution of this process is shown in Fig. 13 .

Fig. 9: Particle evolution in simulation during the tool insertion stage for three screw configurations. From left to right: particle initialization, intermediate evolution under tool-screw interaction, and final converged belief. The ground-truth screw geometry is shown in magenta.

Fig. 10: Position error evolution of the estimated screw pose during the insertion stage in simulation.

Fig. 11: Experimental setup for the screw-loosening task.

Fig. 12: Visualization of haptic mismatch for discrete hypotheses

s^{(1)}

-

s^{(3)}

. Red and blue point clouds represent the predicted equilibrium

{\bm{{z}}}^{*}

and observed pose

\overline{{\bm{{z}}}}

, respectively, which corresponds to haptic mismatch due to impedance control.

Fig. 13: Visualization of the online planning process under model mismatch. Gray curves denote sampled trajectories and the red curve indicates the selected minimum cost trajectory. The black contour shows current estimated screw geometry, while magenta contour represents ground truth. The discrepancy between them drives exploratory interaction for state disambiguation.

Fig. 14: Particle evolution of haptic SLAM during interaction. Each row shows the transition from the initial contact state (left) to the final converged belief (right). The system identifies the screw shape and tracks its pose during both insertion and rotation.

Exp. & Spanner	Screw Type	Condition	Success Rate		Static Error		Dynamic Error		Interaction Force (N)		Task Time (s)
Exp. & Spanner	Screw Type	Condition	$S_{id}$	$S_{mn}$	Pos. (mm)	Rot. (deg)	Pos. (mm)	Rot. (deg)	Peak	Avg.	Task Time (s)
Exp 1 (34 mm)	Hex-36	Oversized	10/10	-	$2.65\pm 1.23$	$4.09\pm 3.17$	-	-	24.17	5.53	$12.93\pm 0.28$
	Hex-34	Matched	10/10	10/10	$3.33\pm 1.51$	$2.60\pm 1.57$	$3.95\pm 1.54$	$1.48\pm 0.32$	14.97	2.35	$10.96\pm 3.36$
	Squ-19	Undersized	10/10	-	$3.02\pm 4.89$	$4.21\pm 1.85$	-	-	4.21	0.36	$13.03\pm 0.22$
Exp 2 (34 mm)	Hex-36	Oversized	10/10	-	$2.07\pm 0.45$	$5.57\pm 1.86$	-	-	24.20	7.24	$17.83\pm 0.26$
	Rec-20	Undersized	10/10	-	$4.90\pm 2.50$	$3.29\pm 2.70$	-	-	8.08	1.42	$16.89\pm 2.93$

TABLE II: Quantitative Performance of Haptic SLAM across 3-Hypothesis Experiments

Fig. 15: Probability weight evolution across four three hypothesis experiments. Solid lines and shaded regions represent the mean and variance of the screw type weight.

Fig. 16: Evolution of position error and pose uncertainty across four three-hypothesis experiments. Shaded areas represent the covariance of the particles exceeding the weight threshold. Despite variations in screw geometry, the position error converges.

Exp. & Spanner	Screw Type	Condition	Success Rate		Static Error		Dynamic Error		Interaction Force (N)		Task Time (s)
Exp. & Spanner	Screw Type	Condition	$S_{id}$	$S_{mn}$	Pos. (mm)	Rot. (deg)	Pos. (mm)	Rot. (deg)	Peak	Avg.	Task Time (s)
6-Class Batch (34 mm)	Hex-36	Oversized	10/10	–	$2.31\pm 0.93$	$3.36\pm 1.66$	–	–	24.42	6.86	$24.09\pm 0.85$
	Hex-34	Matched	8/10	10/10	$2.71\pm 2.23$	$1.94\pm 1.05$	$3.24\pm 2.64$	$4.11\pm 1.65$	10.75	2.08	$14.39\pm 5.31$
	Hex-30	Undersized	6/10	–	$1.98\pm 0.81$	$2.51\pm 2.07$	–	–	5.94	0.94	$23.17\pm 6.38$
	Squ-19	Undersized	7/10	–	$2.83\pm 1.86$	$2.51\pm 1.15$	–	–	4.45	0.73	$24.88\pm 0.68$
	Rec-20	Undersized	6/10	–	$3.57\pm 2.39$	$3.08\pm 1.65$	–	–	7.29	1.36	$28.08\pm 5.55$

TABLE III: Quantitative Results of the 6-Hypothesis Challenge (34 mm Spanner)

Fig. 17: Analysis of anisotropic pose uncertainty. Blue hexagons denote the particle distribution, while the red curve indicates the associated error ellipse. The anisotropy reflects the contrast between stiff lateral constraints and the geometric clearance along the tool opening.

Fig. 18: Typical particle evolution for the 6-class challenge. The system must simultaneously reason over six discrete shape hypotheses and their associated continuous poses. Unlike the rapid convergence in simpler tasks, the system here may compete weight of shape.

Fig. 19: Confusion matrix for 6-hypothesis stress test. The system exhibits high accuracy for oversized screw (Hex-36) but faces ambiguity where shapes are geometrically similar (Hex-34 vs. Flw-33) or interactions are under-constrained (undersized screws).

Fig. 20: Evolution of probability weights across different screw types in 6-hypothesis stress test.

Fig. 21: Quantitative results of the ablation study across different screw geometries. (a) Manipulation success rates. The proposed framework (H-SLAM+AS) consistently achieves the highest success rates compared to fixed control stiffness ablation (H-SLAM+FS) and the baseline pure impedance control (Pure IC). (b) Distributions of average interaction forces and torques. The box plots demonstrate that the full H-SLAM+AS framework maintains safe and compliant contact, generally yielding lower and more stable interaction wrenches during the tasks, whereas baselines exhibit higher variance and larger contact forces.

Fig. 22: Effect of friction on direct insertion under horizontal misalignment. Low friction enables the tool to slide into the screw head, whereas high friction causes jamming at the contact boundary.

Adaptive Manipulation Potential and Haptic Estimation for Tool-Mediated Interaction

Abstract

Figures & tables

Contact-Rich Robotic Manipulation

Haptic Sorter: A Unified Planning Framework for Online Shape Estimation and Real-Time Pose Inference

Force/Torque-Based Kinematic Adaptation for Robotic Manipulation Tasks

Relative Contact Velocity-Controlled Hand-Object Mechanism for Dexterous Tool Manipulation

Abstract

Contact-Rich Robotic Manipulation

Explore similar work

Haptic Sorter: A Unified Planning Framework for Online Shape Estimation and Real-Time Pose Inference

Force/Torque-Based Kinematic Adaptation for Robotic Manipulation Tasks

Relative Contact Velocity-Controlled Hand-Object Mechanism for Dexterous Tool Manipulation