Organizations: Robot Perception Laboratory, Department of Engineering, King’s College London, Strand, London, WC2R 2LS, United Kingdom · Personal Robotics Laboratory, Department of Electrical and Electronic Engineering, Imperial College London, London SW7 2AZ, United Kingdom
Deformable objects often appear in unstructured configurations. Tracing deformable objects helps bringing them into extended states and facilitating the downstream manipulation tasks. Due to the requirements for object-specific modeling or sim-to-real transfer, existing tracing methods either lack generalizability across different categories of deformable objects or struggle to complete tasks reliably in the real world. To address this, we propose a novel visual-tactile imitation learning method to achieve one-dimensional (1D) and two-dimensional (2D) deformable object tracing with a unified model. Our method is designed from both local and global perspectives based on visual and tactile sensing. Locally, we introduce a weighted loss that emphasizes actions maintaining contact near the center of the tactile image, improving fine-grained adjustment. Globally, we propose a tracing task loss that helps the policy to regulate task progression. On the hardware side, to compensate for the limited features extracted from visual information, we integrate tactile sensing into a low-cost teleoperation system considering both the teleoperator and the robot. Extensive ablation and comparative experiments on diverse 1D and 2D deformable objects demonstrate the effectiveness of our approach, achieving an average success rate of 80% on seen objects and 65% on unseen objects.
Figures & tables
Fig. 1: (a) 1d deformable object tracing. (b) 2d deformable object tracing. Using our proposed method, an ABB YuMi traces the objects by sliding a gripper along the 1d deformable object or an edge of the 2d deformable object and transforming them from the unstructured configurations on the left to the extended states on the right.
Fig. 2: Visual-tactile teleoperation system for collecting demonstrations. On the robot side, a top-view camera is installed, and a tactile sensor is mounted on the follower robot’s gripper. On the teleoperator side, visual and tactile images are monitored in real time, and vibration motors are mounted on the leader robot’s gripper.
Fig. 3: Overview of the proposed tracing policy learning framework. The inputs include robot kinematics otK , visual image otV , and tactile image otT collected from the follower robot, while the ground truth consists of action sequence at:t+k recorded from the leader robot. Input features are first extracted using a mlp ( mlp ) and cnn ( cnn ). These features are then concatenated and fed into a Transformer-based policy network, which is trained using a combination of three loss functions: local center loss, global task loss, and regularization loss.
Fig. 4: Tactile images under different contact locations between the object and gripper. (a) Object grasped near the center of the tactile sensing region; (b) Object grasped near the front edge of the tactile sensing region; (c) Object grasped near the rear edge of the tactile sensing region. When the object is grasped near the edges of the sensing region, it is more likely to slip into an imperceptible area.
Fig. 5: 1d and 2d deformable objects used in the experiments. (a) Seen objects; (b) Unseen objects. For each seen object, 25 demonstrations were collected. The trained models were tested on both seen and unseen objects.
Fig. 6: Tactile images providing high-resolution texture information of the tested 1d deformable objects and the hemmed edges of 2d deformable objects. The folded hems of the 2d objects show textures similar to the 1d objects. The tactile textures of the unseen objects also exhibit resemblance to those of the seen objects.
Fig. 7: Success time and completion ratio for the component ablation study. Overall, the tested models showed similar success time. It is observable that all variants demonstrated lower completion ratios, indicating degraded task performance.
Methods
Success rate (Wilson 95% CI)
Robot collision
Early stopping
Over- tracing
Object dropping
Joint Space
70.0% [54.6, 81.9]
1/40
4/40
2/40
5/40
w/o Vision
65.0% [49.5, 77.9]
4/40
2/40
8/40
0/40
w/o Tactile
60.0% [44.6, 73.7]
2/40
5/40
1/40
8/40
w/o Center Loss
65.0% [49.5, 77.9]
4/40
1/40
0/40
9/40
w/o Task Loss
67.5% [52.0, 79.9]
3/40
3/40
7/40
0/40
Ours
80.0% [65.2, 89.5]
2/40
2/40
3/40
1/40
TABLE I: Ablation experimental results to validate the effectiveness of individual components.
Dataset types
Tested objects
Success rate (Wilson 95% CI)
Robot collision
Early stopping
Over- tracing
Object dropping
Shoelace
\
80.0% [49.0, 94.3]
1/10
0/10
1/10
0/10
Cable
\
90.0% [59.6, 98.2]
1/10
0/10
0/10
0/10
Towel
\
60.0% [31.3, 83.2]
2/10
1/10
1/10
0/10
Cloth
\
80.0% [49.0, 94.3]
0/10
1/10
1/10
0/10
1d
Shoelace Cable
90.0% [59.6, 98.2] 80.0% [49.0, 94.3]
0/10 1/10
0/10 0/10
1/10 1/10
0/10 0/10
2d
Towel Cloth
70.0% [39.7, 89.2] 70.0% [39.7, 89.2]
1/10 0/10
1/10 1/10
0/10 1/10
1/10 1/10
TABLE II: Experimental results using different training datasets.
Unseen objects
Success rate (Wilson 95% CI)
Robot collision
Early stopping
Over- tracing
Object dropping
Rope
70.0% [48.1, 85.5]
0/20
4/20
0/20
2/20
Napkin
60.0% [38.7, 78.1]
2/20
0/20
4/20
2/20
TABLE III: Experimental results on unseen objects.
Dexterous manipulation of deformable objects demands continuous fingertip-level regulation of pressure, friction, and incipient slip. We study one of the most challenging cases: dexterous cable tracing, feeding a cable through the hand with repeated pinch-and-curl motions of the thumb and index finger. We introduce Touch2Trace, a tactile-driven imitation-learning system for this task, and provide, to our knowledge, the first systematic real-world characterization of how encoder pretraining, control rate, temporal context, and spatial resolution each shape policy performance. The winning learning recipe combines a tactile encoder pretrained for a custom 32 x 32 piezoresistive sensor (TacV5) via self-supervised learning with a lightweight transformer policy trained on teleoperated demonstrations via behavior cloning, deployed at 60 Hz on a Tesollo DG-5F hand. Tactile feedback without vision or explicit cable-state estimation significantly improves tracing performance versus a proprioception-only baseline: from 0.2 cm to 20.1 cm mean distance and 0% to 93% success rate, with zero-shot transfer to unseen cables and routing conditions. The results quantify the influence of key parameters in tactile-driven systems for reliable dexterous deformable object manipulation.
This paper presents a phase-conditioned, force-aware framework for robust deformable object manipulation. Standard imitation learning policies such as Action Chunking with Transformers (ACT) rely on a Markovian assumption at inference, causing state aliasing when visually similar observations require contradictory actions and preventing autonomous recovery from execution failures. We address this with a closed-loop hierarchical architecture. A FiLM-conditioned ACT encoder modulates feature extraction based on the current task phase, enabling a single unified policy to produce phase-specific behaviors while sharing action dynamics across phases. A multi-modal phase predictor fusing visual, force, and pose feedback estimates the phase in real time, detecting contact failures that are invisible to vision alone and autonomously triggering recovery trajectories. The system is completed by a hybrid impedance controller for compliant execution and a haptic teleoperation interface for force-aware data collection. Ablation studies show that FiLM-based modulation significantly outperforms both unconditioned and token-level conditioned baselines, and t-SNE analysis confirms that FiLM induces well-separated, phase-specific feature representations. Validated on hanging and removing a T-shirt with dual arms, the closed-loop system improves the hanging success rate from 56% to 87% through autonomous error recovery. Code and videos: https://leledeyuan00.github.io/phaser/
Dayuan Chen, Kai Tang, Yukuan Zhang +2
Department of Robotics, Tohoku University, Sendai, Miyagi, Japan · JC STEM Lab of Robotics for Soft Materials, the Department of Electrical and Computer Engineering, Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China
Tactile sensing can substantially improve contact-rich robotic manipulation, yet its practical deployment remains limited by the fragility, calibration requirements, and maintenance burden of tactile hardware. This raises a fundamental question: can robots benefit from tactile knowledge without requiring tactile sensors at deployment? We present TacImag, a tactile imagination framework that predicts tactile observations from vision and proprioception and uses the generated signals to guide manipulation policies. Trained from paired visuotactile demonstrations, TacImag enables touch-informed manipulation using only visual observations at test time. We evaluate TacImag on six simulated and four real-world manipulation tasks. Across simulation and real-world experiments, imagined tactile observations consistently improve manipulation performance without requiring tactile hardware. In real-world experiments, imagined force fields improve contact-sensitive tasks by 44.4% on average, whereas imagined tactile images improve texture-sensitive tasks by 23.3%, revealing that the effectiveness of tactile imagination depends strongly on the relationship between tactile representation and task requirements. Our results further suggest that tactile imagination does not simply recover missing tactile measurements. Instead, it acts as a form of contact-aware supervision that transforms subtle visual interaction cues into representations that are easier for manipulation policies to exploit.
Zhiyuan Zhang, Adeesh Desai, Jyun-Chi Hu +7
Department of Industrial Engineering, Purdue University, West Lafayette, IN 47907, USA · Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA