Stiffness Copilot: An Impedance Policy for Contact-Rich Teleoperation
Authors: Yeping Wang, Zhengtong Xu, Pornthep Preechayasomboon, Ben Abbatematteo, Amirhossein H. Memar, Nick Colonnese, Sonny Chan
Organizations: All authors are with Meta Reality Labs Research · Yeping Wang is with University of Wisconsin-Madison, Madison, WI, USA · Zhengtong Xu is with Purdue University, West Lafayette, IN, USA
In teleoperation of contact-rich manipulation tasks, selecting robot impedance is critical but difficult. The robot must be compliant to avoid damaging the environment, but stiff to remain responsive and to apply force when needed. In this paper, we present Stiffness Copilot, a vision-based policy for shared-control teleoperation in which the operator commands robot pose and the policy adjusts robot impedance online. To train Stiffness Copilot, we first infer direction-dependent stiffness matrices in simulation using privileged contact information. We then use these matrices to supervise a lightweight vision policy that predicts robot stiffness from wrist-camera images and transfers zero-shot to real images at runtime. In a human-subject study, Stiffness Copilot achieved safety comparable to using a constant low stiffness while matching the efficiency of using a constant high stiffness.
Figures & tables
Fig. 1: Overview of Stiffness Copilot execution in the human-subject experiment. Each row shows a representative situation. (A) We collected contact forces in simulation to infer robot stiffness labels. (B) Stiffness Copilot was trained on simulated wrist-camera images (upper) and deployed on real wrist-camera images (lower), illustrating the sim-to-real gap. (C) Third-person views from our user study, where a participant teleoperated the robot. (D) Stiffness produced by Stiffness Copilot is visualized as a green ellipsoid. A longer ellipsoid axis indicates larger translation stiffness in that direction. We also annotate the stiffness magnitude along each axis. In Vase Wiping (top two rows), despite different grasp poses, the robot remained compliant perpendicular to the contact surface and stiff in other directions. In Peg-in-Hole (middle two rows), the robot was stiff while picking up the block and compliant during peg insertion. In Door Opening (bottom two rows), despite different grasp poses, the robot was stiff along the opening direction and compliant in other directions.
Fig. 2: Visualization of our study results. For box-and-whisker plots, each box spans the first to third quartile, the line marks the median, and whiskers extend to at most 1.5 times the interquartile range (IQR). ∗/∗∗/∗∗∗ indicate significance levels p<.05/.01/.001 . d is Cohen’s d for paired t -tests and r is the Wilcoxon effect size.
Metric
Low Stiffness
Stiffness Copilot
High Stiffness
Statistical Test
Mean (SD)
Mean (SD)
Mean (SD)
Objective Measures
Vase Wiping
Max Force (N)
60.58 (29.23)
59.28 (27.04)
117.46 (42.87)
F(2,34)=17.15 , p<.001 ∗∗∗ , ηp2=0.502
Success Rate
0.76 (0.36)
0.87 (0.17)
0.24 (0.28)
χ2=20.13 , p<.001 ∗∗∗ , W=0.559
Time (s)
30.96 (8.12)
25.31 (7.54)
33.62 (11.57)
F(2,16)=4.29 , p<.05 ∗ , ηp2=0.349
Peg-in-Hole
Max Force (N)
49.00 (34.25)
33.41 (11.67)
80.60 (54.44)
F(2,34)=6.99 , p<.01 ∗∗ , ηp2=0.291
Success Rate
0.67 (0.40)
0.81 (0.31)
0.70 (0.30)
χ2=3.21 , p=0.201 , W=0.089
TABLE I: Summary of user study results across stiffness conditions. Best values are shown in bold .
PRISMA Lab, Department of Electrical Engineering and Information Technology, University of Naples Federico II, Naples, Italy. · The University of Edinburgh, Edinburgh, UK.
The authors are with the Faculty of Mechanical Engineering, Sungkyunkwan University, Suwon 2066, South Korea · The authors are with the Faculty of Intelligent Robotics, Sungkyunkwan University, Suwon 2066, South Korea