Learning Bimanual Cloth Manipulation with Vision-based Tactile Sensing via Single Robotic Arm

Dongmyoung Lee; Wei Chen; Xiaoshuai Chen; Rui Zong; Petar Kormushev

cs.ROMar 11, 2026

Learning Bimanual Cloth Manipulation with Vision-based Tactile Sensing via Single Robotic Arm

Authors: Dongmyoung Lee, Wei Chen, Xiaoshuai Chen, Rui Zong, Petar Kormushev

Organizations: Robot Intelligence Lab, Dyson School of Design Engineering, Imperial College London, 25 Exhibition Road, London, SW7 2DB, UK

Abstract

Robotic cloth manipulation remains challenging due to the high-dimensional state space of fabrics, their deformable nature, and frequent occlusions that limit vision-based sensing. Although dual-arm systems can mitigate some of these issues, they increase hardware and control complexity. This paper presents Touch G.O.G., a compact vision-based tactile gripper and perception/control framework for single-arm bimanual cloth manipulation. The proposed framework combines three key components: (1) a novel gripper design and control strategy for in-gripper cloth sliding with a single robot arm, (2) a Vision Foundation Model-backboned Vision Transformer pipeline for cloth part classification (PC-Net) and edge pose estimation (PE-Net) using real and synthetic tactile images, and (3) an encoder-decoder synthetic data generator (SD-Net) that reduces manual annotation by producing high-fidelity tactile images. Experiments show 96% accuracy in distinguishing edges, corners, interior regions, and grasp failures, together with sub-millimeter edge localization and 4.5° orientation error. Real-world results demonstrate reliable cloth unfolding, even for crumpled fabrics, using only a single robotic arm. These results highlight Touch G.O.G. as a compact and cost-effective solution for deformable object manipulation.

Figures & tables

Figure 1: Touch G.O.G. framework enabling single-arm bimanual cloth manipulation. Unlike rigid grippers, our system utilizes a human-inspired sliding strategy: (1) The robot identifies structural regions (e.g., corners), (2) The gripper actively expands and modulates friction to slide along the edge, and (3) Real-time tactile feedback corrects pose errors during sliding until the opposing corner is reached.

Figure 2: Touch G.O.G. overview: (left) the CAD model of the Touch G.O.G. , showing the base, the Decoupled Width Control Gripper (D-WCG), and the two Tactile Variable Friction Grippers (T-VFGs) and (right) the versatility of D-WCG and T-VFG.

Figure 3: Detail of the T-VFG in the Touch G.O.G. system, showing the DIGIT sensor for high-resolution tactile feedback and the DC motor that drives the rack-and-pinion mechanism.

Figure 4: Comparison of workspaces: G.O.G. (top) shows the left finger (blue) and right finger (green) workspaces, exhibiting lower dexterity. Touch G.O.G. (bottom) shows larger coverage due to extra degrees of freedom, with left (green) and right (blue) regions superimposed on the CAD model.

Figure 5: Overview of the cloth manipulation framework: The environment setup shows the Touch G.O.G. system integrated with visuotactile sensors. (1) Cloth Part Classification Network ( PC-Net ) is trained with a SAM backbone and a convolutional head. (2) The SAM-backboned Mask Decoder Network ( SD-Net ) generates realistic synthetic tactile images from edge annotations. (3) The resulting synthetic dataset is used to train the Edge Pose Estimation Network ( PE-Net ), which estimates edge pose for (4) precise cloth sliding.

Figure 6: The architecture of SAM-backboned mask decoder network ( SD-Net ). Network input is annotation information of real-world tactile images and output is generated tactile images compared with tactile image inputs.

Figure 7: Confusion matrix of predicted vs. true labels for four classes—(1) Edge, (2) Corner, (3) In‐fabric, and (4) Grasp Failure. Each cell shows the number of instances classified into each category, with diagonal cells indicating correct predictions.

Method / Metric	Precision (%)				Recall (%)
Method / Metric	Edge	Corner	In-Fabric	Grasp Failure	Edge	Corner	In-Fabric	Grasp Failure
B1: ResNet	0.85	0.89	0.94	0.96	0.85	0.84	0.95	1.00
B2: DenseNet	0.92	0.92	0.95	0.97	0.91	0.89	0.97	1.00
B3: ViT	0.87	0.81	0.79	0.90	0.69	0.83	0.87	0.98
PC-Net (Ours)	0.97	0.99	0.94	0.97	0.93	0.97	0.97	1.00

Table I: Classification evaluation for PC-Net .

Figure 8: The sample outputs of SAM-backboned mask decoder network ( SD-Net ). When generating images, random edge pose is input to create synthetic images for random edge poses. This enables to get unseen images that is not captured when training.

Figure 9: Cloth samples used to evaluate the generalization capability of the proposed cloth manipulation framework.

Figure 10: t-SNE visualization comparing SD-Net -generated data (blue) with the real tactile dataset (green). The synthetic data exhibits broader coverage, indicating improved generalization and robustness.

Method / Metric	Position (mm)			Angle (°)
Method / Metric	X	Y	Distance	Angle (°)
Edge Detection	2.47	2.32	3.38	50.22
ResNet-w/o	0.68	1.58	1.72	14.92
DenseNet-w/o	0.46	1.13	1.22	8.19
ViT-w/o	0.37	0.85	0.93	6.48
PE-Net-w/o	0.30	0.72	0.78	5.52

Table II: Edge pose estimation assessment for PE-Net .

Figure 11: Example results of the PE-Net edge pose estimation on trained (top row) and unseen (bottom row) fabric samples. Each sub-image shows the estimated edge (blue line) and the predicted angle (green text), illustrating consistent boundary detection and orientation estimation under diverse conditions.

Figure 12: Real-world cloth sliding experiment using the Touch G.O.G. system. The rows illustrate sequential sliding steps for the flattened and crumpled fabrics. One T-VFG initially grasps a cloth corner, while the other performs sliding until the opposite corner is detected. During sliding, the gripper adjusts its opening width and abduction angle based on edge pose estimates provided by PE-Net .

Config.	TF1	TF2	TF3	TF4	PT1	LB1	PT2	Average
Flattened	4/5	3/5	3/5	4/5	3/5	4/5	3/5	24/35
Crumpled	3/5	2/5	2/5	4/5	3/5	3/5	3/5	20/35

Table III: Performance of Cloth Sliding

Figure 13: Visuotactile edge-alignment control for cloth sliding. The fixed T-VFG holds a cloth corner, while the moving T-VFG uses the DIGIT image to regulate $e_{y}$ and $e_{\theta}$ during sliding.

Deformable Object Manipulation

Deformable Object Manipulation studies how robots perceive, model, and control flexible objects such as cloth, ropes, cables, and soft tissue.

See all 104 papers

Synthetic Data Generation and Vision-based Wrinkle and Keypoint Detection for Bimanual Cloth Manipulation
Jun 4, 2026Ariel Herrera, Xueyang Kang, Atal Anil KumarBimanual Robotic Manipulation Deformable Object Manipulation
Department of Engineering, University of Luxembourg, Esch-sur-Alzette, Luxembourg · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Universit´e de Lorraine, Arts et Metiers Institute of Technology, LCFC, F-57070 Metz, France
Vision-Based 6-DoF Grasp Pose Estimation for Robot Cloth Unfolding
Sep 25, 2026Domen Tabernik, Peter Nimac, Jan Jerićević +2Robotic Grasping Bimanual Robotic Manipulation
Faculty of Computer and Information Science, University of Ljubljana, Večna pot 113, 1000 Ljubljana, Slovenia · Jožef Stefan International Postgraduate School, Jamova cesta 39, 1000 Ljubljana, Slovenia · Jožef Stefan Institute, Jamova cesta 39, 1000 Ljubljana, Slovenia
Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement
Jun 23, 2026Xin Liu, Yulin Li, Ziming Li +7Robotics Simulation Deformable Object Manipulation
National University of Singapore · Shanghai Jiao Tong University

Jun 4, 2026J/K move · Enter open · S save

Synthetic Data Generation and Vision-based Wrinkle and Keypoint Detection for Bimanual Cloth Manipulation

Ariel Herrera, Xueyang Kang, Atal Anil Kumar

Department of Engineering, University of Luxembourg, Esch-sur-Alzette, Luxembourg · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Universit´e de Lorraine, Arts et Metiers Institute of Technology, LCFC, F-57070 Metz, France

Bimanual Robotic Manipulation Deformable Object Manipulation Synthetic Data Generation for Robotics

Open paper AskPDF arXiv

Robotic manipulation of textiles remains challenging because continuous deformation and self-occlusions hinder the robust visual perception required to estimate the cloth's state. To address the lack of annotated real-world data, we developed a Blender-based synthetic pipeline exporting auto-annotated keypoints, and combined manually labeled renders with real-world data to train a wrinkle detector. We present a perception framework integrating a CNN for permutation-invariant keypoint detection and a YOLOv8-OpenCV pipeline to extract grasping points from structural wrinkles. A proposed bimanual algorithm uses this system to stretch fully folded garments via wrinkles, transitioning to keypoint-based ironing once corners emerge. The keypoint model achieves a Mean Position Error (MPE) of 1.7615 pixels. The perception system transfers to physical fabrics without fine-tuning, outperforming baselines that fail in high-occlusion states or yield false positives on severe folds.

Abstract

Deformable Object Manipulation

Explore similar work

Synthetic Data Generation and Vision-based Wrinkle and Keypoint Detection for Bimanual Cloth Manipulation

Vision-Based 6-DoF Grasp Pose Estimation for Robot Cloth Unfolding

Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement