Vision-Based 6-DoF Grasp Pose Estimation for Robot Cloth Unfolding
Authors: Domen Tabernik, Peter Nimac, Jan Jerićević, Danijel Skočaj, Andrej Gams
Organizations: Faculty of Computer and Information Science, University of Ljubljana, Večna pot 113, 1000 Ljubljana, Slovenia · Jožef Stefan International Postgraduate School, Jamova cesta 39, 1000 Ljubljana, Slovenia · Jožef Stefan Institute, Jamova cesta 39, 1000 Ljubljana, Slovenia
Cloth manipulation is a challenging task due to the deformable and high-dimensional nature of cloth, which leads to complex interaction dynamics and perceptual ambiguity arising from frequent occlusions of critical visual cues such as folds, edges, and grasp points. In this work, we tackle cloth unfolding using a regrasping-in-the-air strategy, where one manipulator holds the cloth while the other grasps it at an optimally selected point to unfold it. To this end, we propose CeDiRNet-6DoF, a deep learning framework that jointly predicts effective grasp points and the complete 6-DoF grasp pose from the observed cloth configuration. By integrating dense 3D grasp regression with segmentation and sine-cosine-encoded Euler angles, the proposed method reliably estimates the grasp configuration that maximizes the unfolded cloth area. We extensively evaluated CeDiRNet-6DoF on a bimanual robotic setup within the ICRA 2024 Cloth Competition framework, achieving state-of-the-art performance. An ablation study further validates the benefits of key design components, including joint segmentation, background randomization, and image cropping. These results establish CeDiRNet-6DoF as a robust and versatile foundation for reliable robotic cloth manipulation in unstructured environments.
Figures & tables
Fig. 1 : CeDiRNet-6DoF architecture. CeDiRNet (top) regresses center directions for 2D grasp localization; its 6-DoF extension (bottom) regresses sine/cosine pairs for roll, pitch, and yaw. Orientation and depth yield the 6-DoF pose.
Fig. 2 : Captured images from three different setups: (i) the competition training dataset provided by the organizers (top row), showing the ground truth data; (ii) our setup (middle row), showing the inference results; and (iii) the competition setup at ICRA 2024 (bottom row), showing the inference results from the competition. The red bounding boxes indicate the cropping boundaries based on the robot’s position, as used in our approach.
Fig. 3 : Cloth evaluation reference items for ICRA 2024 set (top row) and our set (bottom row).
Cloth
Coverage
Grasp Success
Coverage for Successful Grasp
ICRA 2024 Set
cloth-checkered-rag*
0.648
0.70
0.804
cloth-waffle-rag*
0.560
0.50
0.793
tshirt-blue-airo*
0.618
0.80
0.691
tshirt-green-airo*
0.667
0.70
0.808
tshirt-zara-airo
0.639
1.00
0.639
TABLE I : Evaluation results for ICRA 2024 Set and Our Set with reported coverage, grasp success and coverage for only successful grasps. Combined results from RUNS A and RUNS B are reported . Items observed during training are marked with (*).
Fig. 4 : Examples of detected grasp points and gripper poses with the corresponding unfolded item at the bottom left of each image. Supplementary video material provides the full execution sequence for all shown examples, including intermediate states between the predicted grasp and the final unfolded outcome.
Fig. 5 : Evaluation of CeDiRNet-6DoF on the ICRA 2024 and our sets. Coverage for ten repeated executions for each item is depicted as green/red circles, with the box-plot showing median (black line), mean (black diamond), min/max (box whiskers) and 75th/25th percentiles (box). Items observed during training are marked with (*).
DoF
Seg.
Random bg. augment.
Crop. by robot pos.
RGB-D
Coverage
Grasp Success
Coverage with Succ. Grasp
ICRA 2024
Our
Both
ICRA 2024
Our
Both
ICRA 2024
Our
Both
RUNS A
6-DoF
0.455
0.420
0.437
0.388
0.225
0.306
0.678
0.735
0.709
6-DoF
✓
0.490
0.458
0.474
0.475
0.325
0.400
0.660
0.646
0.653
6-DoF
✓
✓
0.604
0.578
0.591
0.675
0.625
0.650
0.698
0.706
0.702
6-DoF
✓
✓
✓
0.635
0.583
0.609
0.750
0.625
0.688
0.750
0.683
0.717
\rowfont RUNS B
3-DoF
0.388
0.379
0.383
0.325
0.150
0.238
0.553
0.646
0.581
TABLE II : Ablation study for CeDiRNet-6DoF with individual components disabled. The best result for each metric is shown in bold for each set of experiment runs. In both sets, the best results were mostly achieved with the 6-DoF version using segmentation, random background augmentation , and robot-position cropping.
512×512
1024×1024
1920×1080
3840×2160
A100 40GB SXM
0.27±0.01
0.34±0.01
0.41±0.01
0.81±0.02
RTX 2080Ti 11GB
0.45±0.04
0.55±0.01
0.77±0.03
1.45±0.06
TABLE III : Computational cost with respect to resolution and GPU model (in seconds).
Fig. 6 : Ablation study for individual CeDiRNet-6DoF components on combined ICRA 2024 and our set, with RUNS A/B left/right; dashed/solid black lines show average coverage/95% CI.
Fig. 7 : Proxy evaluation of depth encodings with human judges. We report ELO-like scores obtained by fitting a Bayesian Bradley–Terry model (Bernoulli likelihood; ties treated as 0.5), together with 95% credible intervals. Models also evaluated on real robots are shown in lighter blue.
Rank
Team Name/Model
Coverage ICRA 2024 setup
Coverage Our setup
1st
AIR-JNU
0.60
-
2nd
CeDiRNet-6DoF (our)
0.57
0.61±0.0595% CI
3rd
EWHA-glab
0.55
-
4th
SCUT-ROBOT
0.53
-
5th
Team Greater Bay
0.53
-
6th
Samsung Research China
0.48
-
TABLE IV : Results from the ICRA 2024 cloth competition [ 9 ] comparing our CeDiRNet-6DoF entry with other participants and with our setup on the ICRA 2024 item set.
Fig. 8 : Proxy evaluation against two related-work baselines (in dark blue) with human judges. Reporting ELO-like score from fitted hierarchical Bayesian Bradley–Terry model and 95% credible intervals.
Institut de Robòtica i Informàtica Industrial, CSIC-UPC, Barcelona, Spain · Departament de Matemàtiques, Universitat Politècnica de Catalunya, Barcelona, Spain