Organizations: Robot Perception Laboratory, Department of Engineering, King’s College London, Strand, London, WC2R 2LS, United Kingdom · Personal Robotics Laboratory, Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, United Kingdom
Deformable objects often appear in unstructured configurations. Tracing deformable objects helps bringing them into extended states and facilitating the downstream manipulation tasks. Due to the requirements for object-specific modeling or sim-to-real transfer, existing tracing methods either lack generalizability across different categories of deformable objects or struggle to complete tasks reliably in the real world. To address this, we propose a novel visual-tactile imitation learning method to achieve one-dimensional (1D) and two-dimensional (2D) deformable object tracing with a unified model. Our method is designed from both local and global perspectives based on visual and tactile sensing. Locally, we introduce a weighted loss that emphasizes actions maintaining contact near the center of the tactile image, improving fine-grained adjustment. Globally, we propose a tracing task loss that helps the policy to regulate task progression. On the hardware side, to compensate for the limited features extracted from visual information, we integrate tactile sensing into a low-cost teleoperation system considering both the teleoperator and the robot. Extensive ablation and comparative experiments on diverse 1D and 2D deformable objects demonstrate the effectiveness of our approach, achieving an average success rate of 80% on seen objects and 65% on unseen objects.
Figures & tables
Fig. 1: (a) 1d deformable object tracing. (b) 2d deformable object tracing. Using our proposed method, an ABB YuMi traces the objects by sliding a gripper along the 1d deformable object or an edge of the 2d deformable object and transforming them from the unstructured configurations on the left to the extended states on the right.
Fig. 2: Visual-tactile teleoperation system for collecting demonstrations. On the robot side, a top-view camera is installed, and a tactile sensor is mounted on the follower robot’s gripper. On the teleoperator side, visual and tactile images are monitored in real time, and vibration motors are mounted on the leader robot’s gripper.
Fig. 3: Overview of the proposed tracing policy learning framework. The inputs include robot kinematics otK , visual image otV , and tactile image otT collected from the follower robot, while the ground truth consists of action sequence at:t+k recorded from the leader robot. Input features are first extracted using a mlp ( mlp ) and cnn ( cnn ). These features are then concatenated and fed into a Transformer-based policy network, which is trained using a combination of three loss functions: local center loss, global task loss, and regularization loss.
Fig. 4: Tactile images under different contact locations between the object and gripper. (a) Object grasped near the center of the tactile sensing region; (b) Object grasped near the front edge of the tactile sensing region; (c) Object grasped near the rear edge of the tactile sensing region. When the object is grasped near the edges of the sensing region, it is more likely to slip into an imperceptible area.
Fig. 5: 1d and 2d deformable objects used in the experiments. (a) Seen objects; (b) Unseen objects. For each seen object, 25 demonstrations were collected. The trained models were tested on both seen and unseen objects.
Fig. 6: Tactile images providing high-resolution texture information of the tested 1d deformable objects and the hemmed edges of 2d deformable objects. The folded hems of the 2d objects show textures similar to the 1d objects. The tactile textures of the unseen objects also exhibit resemblance to those of the seen objects.
Fig. 7: Success time and completion ratio for the component ablation study. Overall, the tested models showed similar success time. It is observable that all variants demonstrated lower completion ratios, indicating degraded task performance.
Methods
Success rate (Wilson 95% CI)
Robot collision
Early stopping
Over- tracing
Object dropping
Joint Space
70.0% [54.6, 81.9]
1/40
4/40
2/40
5/40
w/o Vision
65.0% [49.5, 77.9]
4/40
2/40
8/40
0/40
w/o Tactile
60.0% [44.6, 73.7]
2/40
5/40
1/40
8/40
w/o Center Loss
65.0% [49.5, 77.9]
4/40
1/40
0/40
9/40
w/o Task Loss
67.5% [52.0, 79.9]
3/40
3/40
7/40
0/40
Ours
80.0% [65.2, 89.5]
2/40
2/40
3/40
1/40
TABLE I: Ablation experimental results to validate the effectiveness of individual components.
Dataset types
Tested objects
Success rate (Wilson 95% CI)
Robot collision
Early stopping
Over- tracing
Object dropping
Shoelace
\
80.0% [49.0, 94.3]
1/10
0/10
1/10
0/10
Cable
\
90.0% [59.6, 98.2]
1/10
0/10
0/10
0/10
Towel
\
60.0% [31.3, 83.2]
2/10
1/10
1/10
0/10
Cloth
\
80.0% [49.0, 94.3]
0/10
1/10
1/10
0/10
1d
Shoelace Cable
90.0% [59.6, 98.2] 80.0% [49.0, 94.3]
0/10 1/10
0/10 0/10
1/10 1/10
0/10 0/10
2d
Towel Cloth
70.0% [39.7, 89.2] 70.0% [39.7, 89.2]
1/10 0/10
1/10 1/10
0/10 1/10
1/10 1/10
TABLE II: Experimental results using different training datasets.
Unseen objects
Success rate (Wilson 95% CI)
Robot collision
Early stopping
Over- tracing
Object dropping
Rope
70.0% [48.1, 85.5]
0/20
4/20
0/20
2/20
Napkin
60.0% [38.7, 78.1]
2/20
0/20
4/20
2/20
TABLE III: Experimental results on unseen objects.
Department of Robotics, Tohoku University, Sendai, Miyagi, Japan · JC STEM Lab of Robotics for Soft Materials, the Department of Electrical and Computer Engineering, Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China
Department of Industrial Engineering, Purdue University, West Lafayette, IN 47907, USA · Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA