Adaptive Manipulation Potential and Haptic Estimation for Tool-Mediated Interaction
Authors: Lin Yang, Anirvan Dutta, Yuan Ji, Yanxin Zhou, Shilin Shan, Lv Chen, Etienne Burdet, Domenico Campolo
Organizations: authors are with the School of Mechanical and Aerospace Engineering, Nanyang Technological University (NTU), Singapore. · authors are with the Imperial College of Science, Technology and Medicine, SW7 2AZ London, U.K
Achieving human-level dexterity in contact-rich, tool-mediated manipulation remains a significant challenge due to visual occlusion and the underdetermined nature of haptic sensing. This paper introduces a parameterized Equilibrium Manifold (EM) as a unified representation for tool-mediated interaction, and develops a closed-loop framework that integrates haptic estimation, online planning, and adaptive stiffness control. We establish a physical-geometric duality using an adaptive manipulation potential incorporating a differentiable contact model, which induces the manifold's geometric structure and ensures that complex physical interactions are encapsulated as continuous operations on the EM. Within this framework, we reformulate haptic estimation as a manifold parameter estimation problem. Specifically, a hybrid inference strategy (haptic SLAM) is employed in which discrete object shapes are classified via particle filtering, while the continuous object pose is estimated using analytical gradients for efficient optimization. By continuously updating the parameters of the manipulation potential, the framework dynamically reshapes the induced EM to guide online trajectory replanning and implement uncertainty-aware impedance control, thereby closing the perception-action loop. The system is validated through simulation and over 260 real-world screw-loosening trials. Experimental results demonstrate robust identification and manipulation success in standard scenarios while maintaining accurate tracking. Furthermore, ablation studies confirm that haptic SLAM and uncertainty-aware stiffness modulation outperform fixed impedance baselines, effectively preventing jamming during tight tolerance interactions.
Figures & tables
Fig. 1: Tool-mediated haptic manipulation under visual occlusion. (A) Humans can perform skillful haptic manipulation with tools despite unreliable visual information, although sensing is limited to net forces and torques at the human–tool interface. (B) Multiple contact points may exist at the tool tip while only a single net wrench is observed, leading to contact ambiguity. (C) Humans rely on cognitive reasoning and internal models of tool use to infer task-relevant object properties, such as pose and geometry, through haptic interaction.
Fig. 2: Overall workflow of tool-mediated manipulation framework. Left: the robot uses a tool to interact with the object and senses the resulting net wrench F . Middle: F is fed into a haptic state estimator that incorporates a manifold-based internal physical model to predict the expected wrench F . The discrepancy between the observed and predicted wrenches—termed haptic mismatch —is used to update the estimated object pose θ and type s . Right: Based on the optimal estimates (s∗,θ∗) and associated uncertainty Σθ , a planner and controller generate the control u(t) and stiffness Kc for robot.
Symbol
Description
A. Internal Model & Equilibrium Manifold (Sec. IV )
z∈Z
Internal system state (e.g., spanner pose), evolving on the manifold
u∈U
Robot control input (e.g., impedance equilibrium pose)
θ∈Θ
Continuous environment parameter (e.g., screw pose) to be estimated
s∈S
Discrete environment parameter (e.g., spanner and screw shape/type)
W(s)(z,u;θ)
Quasi-static manipulation potential function for shape hypothesis s
TABLE I: Nomenclature and Key Symbols
Fig. 3: Parameterized equilibrium manifolds and induced motion. For fixed environment parameters (s,θ) , the quasi-static interaction between the robot and the environment defines an equilibrium manifold Meq(s,θ) embedded in the state-control space. Different values of the discrete parameter s select different manifolds, while the continuous parameter θ continuously deforms the manifold geometry within a given structural model. For fixed (s,θ) , control inputs u induce equilibrium configurations z∗(u;θ,s) along the manifold. The resulting motion on Meq(s,θ) is governed by an adaptive ODE.
Fig. 4: Instantiation of the manipulation potential W(s)(z,u;θ) through contact geometry and impedance control. Left: Schematic representation of the general case. Right: Illustrative example of a spanner–screw operation.
Fig. 5: Workflow of the hybrid inference in Haptic SLAM. For each discrete shape hypothesis s(i) , multiple pose particles θ(j)[s(i)] are initialized to represent candidate configurations. These particles process sensed haptic observations F in parallel, independently updating their pose parameters θ(j)[s(i)] via a local optimizer driven by the internal physical model. A representative pose θ∗[s(i)] is then identified for each shape, followed by a MAP selection and clustering process to determine the estimated object shape s∗ and pose θ∗ for the current batch. This inference cycle is re-executed in each subsequent batch as new haptic observations become available.
Fig. 6: Pipeline for pose and particle initialization from single-point contact.
Fig. 7: Physical and geometric interpretation of haptic SLAM: (a) Pose estimation (continuous update): For a fixed shape s(2) , haptic mismatch ϵ=F−F drives the deformation Δθ to align the expected manifold (light blue) with reality (dark blue). (b) Shape estimation (discrete selection): Parallel manifolds for distinct shapes, such as a green rectangle ( s(1) ) and a blue hexagon ( s(2) ), are compared to select the candidate based on ϵ .
Fig. 8: Schematic illustration of MPPI-based planning. Rollouts are evaluated using a haptic cost and an elite set is used to update the policy in a receding-horizon manner. An experimental replotted execution of this process is shown in Fig. 13 .
Fig. 9: Particle evolution in simulation during the tool insertion stage for three screw configurations. From left to right: particle initialization, intermediate evolution under tool-screw interaction, and final converged belief. The ground-truth screw geometry is shown in magenta.
Fig. 10: Position error evolution of the estimated screw pose during the insertion stage in simulation.
Fig. 11: Experimental setup for the screw-loosening task.
Fig. 12: Visualization of haptic mismatch for discrete hypotheses s(1) - s(3) . Red and blue point clouds represent the predicted equilibrium z∗ and observed pose z , respectively, which corresponds to haptic mismatch due to impedance control.
Fig. 13: Visualization of the online planning process under model mismatch. Gray curves denote sampled trajectories and the red curve indicates the selected minimum cost trajectory. The black contour shows current estimated screw geometry, while magenta contour represents ground truth. The discrepancy between them drives exploratory interaction for state disambiguation.
Fig. 14: Particle evolution of haptic SLAM during interaction. Each row shows the transition from the initial contact state (left) to the final converged belief (right). The system identifies the screw shape and tracks its pose during both insertion and rotation.
Exp. & Spanner
Screw Type
Condition
Success Rate
Static Error
Dynamic Error
Interaction Force (N)
Task Time (s)
Sid
Smn
Pos. (mm)
Rot. (deg)
Pos. (mm)
Rot. (deg)
Peak
Avg.
Exp 1 (34 mm)
Hex-36
Oversized
10/10
-
2.65±1.23
4.09±3.17
-
-
24.17
5.53
12.93±0.28
Hex-34
Matched
10/10
10/10
3.33±1.51
2.60±1.57
3.95±1.54
1.48±0.32
14.97
2.35
10.96±3.36
Squ-19
Undersized
10/10
-
3.02±4.89
4.21±1.85
-
-
4.21
0.36
13.03±0.22
Exp 2 (34 mm)
Hex-36
Oversized
10/10
-
2.07±0.45
5.57±1.86
-
-
24.20
7.24
17.83±0.26
Rec-20
Undersized
10/10
-
4.90±2.50
3.29±2.70
-
-
8.08
1.42
16.89±2.93
TABLE II: Quantitative Performance of Haptic SLAM across 3-Hypothesis Experiments
Fig. 15: Probability weight evolution across four three hypothesis experiments. Solid lines and shaded regions represent the mean and variance of the screw type weight.
Fig. 16: Evolution of position error and pose uncertainty across four three-hypothesis experiments. Shaded areas represent the covariance of the particles exceeding the weight threshold. Despite variations in screw geometry, the position error converges.
Exp. & Spanner
Screw Type
Condition
Success Rate
Static Error
Dynamic Error
Interaction Force (N)
Task Time (s)
Sid
Smn
Pos. (mm)
Rot. (deg)
Pos. (mm)
Rot. (deg)
Peak
Avg.
6-Class Batch (34 mm)
Hex-36
Oversized
10/10
–
2.31±0.93
3.36±1.66
–
–
24.42
6.86
24.09±0.85
Hex-34
Matched
8/10
10/10
2.71±2.23
1.94±1.05
3.24±2.64
4.11±1.65
10.75
2.08
14.39±5.31
Hex-30
Undersized
6/10
–
1.98±0.81
2.51±2.07
–
–
5.94
0.94
23.17±6.38
Squ-19
Undersized
7/10
–
2.83±1.86
2.51±1.15
–
–
4.45
0.73
24.88±0.68
Rec-20
Undersized
6/10
–
3.57±2.39
3.08±1.65
–
–
7.29
1.36
28.08±5.55
TABLE III: Quantitative Results of the 6-Hypothesis Challenge (34 mm Spanner)
Fig. 17: Analysis of anisotropic pose uncertainty. Blue hexagons denote the particle distribution, while the red curve indicates the associated error ellipse. The anisotropy reflects the contrast between stiff lateral constraints and the geometric clearance along the tool opening.
Fig. 18: Typical particle evolution for the 6-class challenge. The system must simultaneously reason over six discrete shape hypotheses and their associated continuous poses. Unlike the rapid convergence in simpler tasks, the system here may compete weight of shape.
Fig. 19: Confusion matrix for 6-hypothesis stress test. The system exhibits high accuracy for oversized screw (Hex-36) but faces ambiguity where shapes are geometrically similar (Hex-34 vs. Flw-33) or interactions are under-constrained (undersized screws).
Fig. 20: Evolution of probability weights across different screw types in 6-hypothesis stress test.
Fig. 21: Quantitative results of the ablation study across different screw geometries. (a) Manipulation success rates. The proposed framework (H-SLAM+AS) consistently achieves the highest success rates compared to fixed control stiffness ablation (H-SLAM+FS) and the baseline pure impedance control (Pure IC). (b) Distributions of average interaction forces and torques. The box plots demonstrate that the full H-SLAM+AS framework maintains safe and compliant contact, generally yielding lower and more stable interaction wrenches during the tasks, whereas baselines exhibit higher variance and larger contact forces.
Fig. 22: Effect of friction on direct insertion under horizontal misalignment. Low friction enables the tool to slide into the screw head, whereas high friction causes jamming at the contact boundary.
Contact-rich robotic manipulation studies how robots plan and control physical interactions such as grasping, insertion, pushing, and dexterous object handling.
Robotics manipulation usually assumes that the shape and pose of the object are known to the robot prior to motion planning. However, precise geometric information is not always available in practice, and pose inference suffers from sensor uncertainties and view occlusion. In this work, we propose a unified model-based geometric framework integrating robotic haptic perception, modeling, and manipulation planning. Our novelties involve: i) Introducing Bayesian Optimization (BO) to guide the haptic exploration for object shape inference, where superellipses are used to approximate geometric boundary; ii) Adaptive formulation of manipulation potential encoding object geometry for quasi-static robot-object interaction; iii) Proposing an online Ordinary Differential Equation (ODE) for real-time pose inference based on model prediction and tactile feedback. We deploy our system on a 2D robotic sorting task, and vary object geometries to validate the robustness and generalizability of our framework in both simulation and a real-world multi-arm setup.
Zhuoyi Lu, Lin Yang, Sri Harsha Turlapati +1
School of Mechanical and Aerospace Engineering, Nanyang Technological University (NTU), Singapore
Contact-rich robotic manipulation requires an accurate model of the kinematic relationship between a robot's joints and the task features it senses. This relationship is rarely known exactly: it changes with each tool the robot picks up and shifts, sometimes almost instantaneously, as contact modes change --- especially for multi-fingered hands that make and break contact at points that are not exactly prescribed, as in full-hand grasping. This paper develops an adaptive scheme that estimates that relationship online, using only joint-angle sensing and a wrist-mounted force/torque sensor, with no exteroceptive measurement of the tool tip. We derive a provably stable kinematic update law that identifies the kinematics of an unknown tool from force/torque feedback alone, and prove stability of both the rigid case and the case with a compliance controller as an inner loop. We show that identification is confined to the directions the motion excites --- so that, for example, a tool's length is unobservable under a rigid insertion push, while a compliant loop's passive yielding partially excites it; and that with a second-order admittance the compliant certificate holds unconditionally in continuous time. We also pose the combined control and estimation problem as a Quadratic Program (QP): the formulation yields the prediction term of the update law exactly but, instructively, cannot reproduce the tracking adaptation term. We validate the scheme in simulation on a peg-in-hole insertion. This work is the first step in a research program aimed at factoring manipulation learning into a task policy which can be learned in isolation of the robot, for instance by reinforcement learning, and an adaptive kinematic component that adapts online to the particular robot, hand, or tool in use.
Carl Glen Henshaw
US Naval Research Laboratory · U.S. Naval Research Laboratory
This work investigates how to enable general multi-finger robotic hands to perform the complete tool manipulation process, which entails picking up a tool, loading it into a suitable pose, and then wielding it. Inspired by human tool manipulation and mechanical design principles, we model the hand and the tool as a unified hand-object mechanism (HOM) composed of sub-assemblies. Specifically, we define a HOM as consisting of the hand, the object, and the generalized contact frames, allowing the HOM's motions to be expressed with the same set of Cartesian-space relative contact velocities, irrespective of the hand's kinematics and geometry. Then, we define a HOM's sub-assemblies as relative contact velocity and contact force constraints between fingers. Building on these definitions, we developed a lightweight and physically interpretable motion planning and contact estimation framework using least squares and a complementary filter. We evaluated our framework in simulation by teleoperating five different robotic hands. The results show that our framework enabled all five hands to execute the complete tool manipulation process, achieving dexterous behaviors even from identical, simple reference trajectories. Furthermore, the results showcase our framework's adaptability to different hands, tools, and tasks, enabled by its kinematic and geometric foundation. Please visit website for videos: https://sunyuw.github.io/hom-sim/
Sunyu Wang, Jean Oh, Nancy S. Pollard
The authors are with the Robotics Institute at Carnegie Mellon University, Pittsburgh, USA.