Organizations: Human-Robot Interfaces and Interaction Laboratory, Istituto Italiano di Tecnologia, Genoa, Italy · Safe AI Lab, Carnegie Mellon University
Contact-rich manipulation requires robots to balance accurate motion tracking with compliant interaction, yet most visual-action policies leave compliance fixed at the controller level. We present Imp-ACT, a methodologically grounded and practical approach to incorporating direction-dependent Cartesian stiffness modulation directly into demonstration collection, without manual stiffness selection or offline target reconstruction. During teleoperation, a self-tuning impedance controller adapts stiffness along the instantaneous direction of motion while maintaining compliance in orthogonal directions. The adapted stiffness is applied and recorded alongside visual observations and motion commands, capturing motion and compliance under the same dynamics. We implement this pipeline using Action Chunking with Transformer (ACT) to predict end-effector pose, gripper action, and motion-direction stiffness from visual, proprioceptive, and wrench observations. The performance of Imp-ACT is evaluated on wiping and plug insertion using both success rate and quantitative measures of contact behavior. Compared with fixed low- and high-stiffness baselines, Imp-ACT achieves comparable or higher success while maintaining low interaction forces. In wiping, it reduces contact-force vibration by approximately 29× relative to the compliant baseline and 180× relative to the stiff baseline. In plug insertion, it reduces forces orthogonal to the insertion direction by 43% relative to the better fixed-stiffness baseline. These results highlight the benefit of maintaining sufficient stiffness along the direction needed for task execution while preserving compliance in other directions to limit contact forces and accommodate environmental constraints.
Figures & tables
Fig. 1 : Imp-ACT turns intuitive human demonstrations into compliant, adaptive robot skills. By learning from human motion while an online Cartesian impedance controller generates direction-dependent stiffness, it unites natural kinematic teaching with interaction-aware control.
Fig. 2 : Overview of (a) Demonstrations and (b) Deployment phases, where ΨB denotes the base frame. During demonstration collection, images from the cameras, measured end-effector pose Xa , external interaction wrench Fext , and the gripper state Gs are recorded as observations. The corresponding action targets comprise the desired end-effector pose Xd , the gripper command Gc=open,close , and the applied stiffness coefficient kst . During deployment, Imp-ACT predicts these action targets, including the stiffness coefficient k~st .
Fig. 3 : Cartesian stiffness ellipsoids during wiping (top) and plug insertion (bottom). The proposed approach increases the predicted stiffness k~st along the instantaneous motion direction (magenta arrow), while maintaining the two orthogonal directions at 200 N/m. The principal stiffness direction follows the task motion, lying approximately along the wiping direction or insertion axis. Each panel reports the time (upper left) and k~st (lower right). Across the illustrated episodes, k~st ranges from 200 – 642 N/m for wiping and from 200 – 329 N/m for insertion.
Fig. 4 : Predicted Cartesian stiffness during one wiping episode. Kx , Ky and Kz are the stiffness commanded w.r.t ΨB , and k~st (dashed) the inferred stiffness along the direction of motion. The pink region marks the gripper closing on the wiper while the yellow one marks contact with the board.
Fig. 5 : Task outcome and contact metrics for wiping (top) and plug insertion (bottom). Grey-background metrics are better when higher; white-background metrics are better when lower and are computed over successful episodes only. Bold text indicates the best result, including ties.
All authors are with Meta Reality Labs Research · Yeping Wang is with University of Wisconsin-Madison, Madison, WI, USA · Zhengtong Xu is with Purdue University, West Lafayette, IN, USA