How Firm Should a Grasp Be?

Matthew Beveridge; Shree K. Nayar

Figure 1: Phases of a Grasp. We consider a grasp to comprise four stages: observe, approach, grip, and lift. Our work exploits the short-lived (fraction of a second) haptic signals during the grip and lift phases to refine class-level material properties (from observation) to obtain instance-level estimates. During this refinement, we continuously adjust the applied force to ensure it is just above the minimum force needed to handle the object.

Figure 2: Soft Objects Require Less Grasp Force. For a soft object, the contact area

A

grows more rapidly with applied force

F

than it does for a hard object. As

A

grows, an additional load,

\tau A

, where

\tau

is the object’s material shear strength, supplements

F

to resist the effect of gravity ( Eq. 1 ). The theoretical reduction in minimum-required grasp force due to this additional load is shown here as a function of the elastic modulus. The object and its shear strength

\tau

are kept constant in the plot, but the ratio of mass

m

to friction

\mu

is varied. The greatest benefit ( i.e . , the possible reduction in grasp force) is for light, pliable objects.

Figure 3: The Squash Dataset. A dataset of real-world 3D object meshes and material property measurements. (a) Squash is composed of 270 instances across 52 classes of produce (fruits and vegetables). (b) We scanned each object using structured light to measure its geometry and texture at high resolution. (c) We additionally measured the material properties of each object (elastic modulus, mass, volume, and surface friction), where each ellipsoid represents the distribution of the properties within a class. Squash is a first-of-its-kind dataset that captures the natural variability of object properties, which can be used for physically-accurate simulation.

Figure 4: Framework for Gentle Grasping. From observation , we identify the object’s class to get a class-conditioned material prior and an initial minimum-required grasp force estimate. Upon first contact, our visuotactile likelihood model

\psi

(detailed in Fig. 5 ) begins to refine these priors into instance-level posteriors from temporal tactile feedback. The minimum-required grasp force is continuously reestimated in the process, and applied in real time. The result is a gentle grasp that adapts to the object instance, in situ .

Figure 5: Visuotactile Model

\bm{\psi}

(shaded green) for Material Estimation and Force Control. Tactile images

\smash{I^{(t)}}

are first encoded and passed with the applied load

\smash{F_{\text{load}}^{(t)}}

to the continuous-time sequence model (CfC), yielding the hidden state

\smash{\bm{h}^{(t)}}

. The likelihood head uses this representation to score candidate material samples,

\smash{S_{j}^{(t)}=S_{\psi}(\bm{h}^{(t)},\theta_{j})}

, where samples

\theta_{j}

are drawn from the class prior

p_{c}(\theta)

for an object’s material. Normalizing these scores yields the weighted belief

\{\smash{\theta_{j}},\smash{\alpha_{j}^{(t)}}\}_{j=1}^{N}

. The weighted samples map to force hypotheses whose posterior expected minimum force,

\smash{F_{\min}^{(t)}}

, sets the target force for the gripper controller in real time.

Open-Loop Baseline Grasping Approaches (Non-Adaptive)
	Force error (per jaw)			Control outcome
Grasp Force Controller	Bias [N]	MAE [N] $\downarrow$	WAPE $\downarrow$	Excess [N] $\downarrow$	Deficit [N] $\downarrow$	Success $\uparrow$
Naïve constant force ( $3$ N)	1.97 [1.67, 2.22]	2.18 [2.02, 2.33]	2.11 [1.56, 2.95]	2.07 [1.87, 2.26]	0.10 [0.01, 0.23]	1.00 [0.99, 1.00]
Class prior	-0.26 [-0.57, 0.10]	0.70 [0.50, 0.95]	0.68 [0.59, 0.77]	0.22 [0.14, 0.31]	0.48 [0.27, 0.74]	0.89 [0.77, 1.00]
Closed-Loop Baseline Grasping Approaches (Adaptive) a
Direct force predictor (learned)	0.29 [-0.05, 0.60]	0.96 [0.75, 1.19]	0.93 [0.74, 1.21]	0.62 [0.46, 0.80]	0.33 [0.14, 0.59]	0.96 [0.90, 1.00]

Table 1: Efficacy of Material-Aware Robot Grasping. We compare our material-aware approach for force control with open-loop and closed-loop baselines using 5k simulated objects with high material variability. “Naïve” uses a force of 3

\mathrm{N}

in each gripper jaw, sufficient to lift the largest object in Squash . “Class Prior” uses the class’s average material properties to determine the grasp force using Eq. 1 . Direct force prediction estimates the grasp force without intermediate material estimation. Slip-reactive policies modulate grasp force to hover around the boundary where incipient slip is observed. In contrast, our approach refines instance-level material properties for fine-grained force control. The best results are in bold, and second best are underlined.

(e) Average Per-Jaw Force Estimation Error

Figure 7: Internal Apple Damage by Grasping Approach. All images captured using fast separation [ 87 ] immediately after force application, revealing subsurface anomalies invisible to the human eye. High variation in subsurface scattering indicates irregular water content under the apple’s skin, which is correlated with bruising. The (a) naïve and (b) class prior approaches both leave signs of damage, while (c) our approach does not.

Figure 8: Surface Damage After 1 Day. Contrast-enhanced images 1 day after grasp force application. The force predicted by our approach leaves no visible damage. The class prior approach overexerts, leaving a bruise.

Figure S1: Extracting Material Properties from Probing Tests. The above plots show the probing results for a single instance of a peach. (a) A compression test is performed to measure the elastic modulus of the object, which is reported as a function of the Poisson ratio. (b) A tribological test is performed to measure the friction coefficient and shear strength of the object, where each point corresponds to a different trial at a different location on the object’s surface. The fitted line is used to decouple the two properties during estimation.

Figure S2: Simulating Deformable Object Grasps. We use UIPC, a GPU-accelerated incremental potential contact solver, to simulate the dynamics of a parallel-jaw gripper with deformable elastomeric gel pads grasping a deformable object. Deformable objects are taken from the Squash dataset, and their material properties and dimensions are augmented to vary the contact geometry. The simulator extracts rich visuotactile observations at each time step, such as the pressure map

\smash{P^{(t)}}

observed in the local frame of the jaw pads, which are used to train our learned model for material estimation and force control.

Figure S3: Force Estimation Error on Simulated Object Grasps. The bar plots show the (top) signed error/bias, (middle) mean absolute error, and (bottom) weighted absolute percentage error of the estimated minimum required grip force

F_{\min}

compared to the ground truth, across all simulated grasping sequences. Error bars show the confidence interval. The error is computed once the object has been lifted by 2

\mathrm{mm}

, which is 0.2

\mathrm{s}

after lifting begins. The best approach is indicated by a

\star

. The results demonstrate that our visuotactile model is able to accurately estimate the required grip force for a wide range of deformable objects with varying material properties.

Open-Loop Baseline Grasping Approaches (Non-Adaptive)
	Material estimation MAPE $\downarrow$				Force error $\downarrow$
Grasp Force Controller	$E$	$m$	$\mu$	$\tau$	Overload [ $\mathrm{N}$ ]	Work [ $\mathrm{mJ}$ ]	Impulse [ $\mathrm{N}\text{\,}\mathrm{s}$ ]
Naïve constant force ( $3$ N)	—	—	—	—	2.01 $\pm$ 0.41	3.32 $\pm$ 2.04	2.15 $\pm$ 1.28
Class prior	1.07 $\pm$ 0.25	1.50 $\pm$ 2.13	1.21 $\pm$ 0.41	0.77 $\pm$ 0.32	1.63 $\pm$ 1.42	3.63 $\pm$ 5.76	1.95 $\pm$ 2.58
Material-Aware Grasping Approaches (Adaptive), with Ablations
Ours	0.56 $\pm$ 0.23	0.49 $\pm$ 0.24	0.56 $\pm$ 0.25	0.44 $\pm$ 0.17	0.39 $\pm$ 0.26	2.25 $\pm$ 5.37	0.66 $\pm$ 0.90

Table S1: Efficacy of Material-Aware Robot Grasping in Simulation (Continuation of Tab. 1 ). We compare our material-aware approach for gripper force control with two standard approaches using 5k simulated produce items with high variability. “Naïve” uses a force of 3

\mathrm{N}

in each gripper jaw, which just exceeds the threshold to successfully lift the largest object in Squash . “Class Prior” uses the class’s average material properties to determine the grasp force using Eq. 1 . In contrast, our approach refines instance-level material properties to yield improved fine-grained force control. We additionally report the performance of our approach when removing the class prior (from observation ), and when the class prior is incorrectly identified. We see that even without an accurate estimate of the object’s class-level material prior, our approach achieves gentle grasps relative to the non-adaptive baselines. The best results are in bold, and second best are underlined.

Figure S4: Heatmap of Force and Material Estimation Errors for Real Object Grasps. The estimated force and material properties are compared against the ground truth measurements for a set of real-world grasps of fruits and vegetables. The heatmap visualizes the distribution of estimation errors of our approach. Note that the heatmap color is indicative of the range of material properties observed in the test set, and the extent of the boundary represents the range of estimation errors using a kernel density estimate. In general, we see our approach achieves low estimation errors but may struggle at the extremes of the material property range, far from the average of the class prior.

How Firm Should a Grasp Be?

Abstract

Figures & tables

Explore similar work

Control Architecture for Safe Grasping of Fragile Objects Using a Coarse Position-Controlled Gripper

A Model-based Visual Contact Localization and Force Sensing System for Compliant Robotic Grippers

Monocular Vision Based Control Framework for Grasping