Garment manipulation requires inferring both garment identity and complete configuration from partial, self-occluded observations. This creates a chicken-and-egg problem: completing a partial observation requires a garment-specific deformation prior, while identifying the garment requires a configuration that explains the observation. We introduce ClothAtlas, which jointly identifies garments and estimates their complete configurations from partial point clouds and a library of canonical meshes. A shared graph-conditioned flow-matching model samples configurations for each candidate mesh and ranks candidates by observation fit. The model accommodates different mesh shapes, vertex counts, and connectivities, allowing new garments to be added from canonical meshes alone without retraining. On held-out DexGarmentLab garments, ClothAtlas ranks the correct mesh first among 221 candidates for all 22 test garments from a single flat observation. From a single self-occluded observation, it identifies 15/22 garments, compared with 8/22 for a deterministic graph estimator. Persistent vertex identities support grasp-target localization on real garments, reducing mean error from 15.4 to 6.4 cm overall and from 17.0 to 8.3 cm on occluded targets compared with dense descriptor matching. Project page: https://clothatlas.github.io/
Figures & tables
Fig. 1: (a) ClothAtlas combines a canonical mesh library and partial depth observations to identify the garment and estimate its complete configuration. Red points show the observations while the blue surface shows the estimated configuration, including regions occluded from the camera. (b) Four reconstructed configurations illustrate persistent point tracking through deformation and occlusion. The camera icon indicates the observation viewpoint and the colored tracks connect the estimated positions of three mesh vertices through the lifting action.
Fig. 2: ClothAtlas overview. (a) A shared graph-conditioned generative model samples complete configurations from a partial observation, Gaussian noise, and a candidate canonical mesh, which supplies reference vertex positions and connectivity. The same model is used for every candidate mesh. (b) We score each candidate by the observation fit of its configuration estimate, accumulate evidence across observations, and retain the k highest-weight candidates after the initial identification period. (c) For each retained candidate, we sample a new configuration from the current observation and update its weight using the accumulated evidence. We report the highest-weight candidate’s configuration.
Fig. 3: Representative canonical meshes from the retrieval library’s 26 dresses, 95 tops, and 100 trousers.
Configuration
Retrieval
Condition
Method
CD
Vertex p90
Top-1 / Top-5
Clean
dgnn
12.28
35.5
92 / 130
Ours, S=4
11.21
24.8
128 / 170
90% removal
dgnn
15.36
66.8
118 / 246
Ours, S=1
12.92
42.8
254 / 394
Ours, S=4
12.60
37.7
285 / 424
TABLE I: Configuration estimation and exact-mesh retrieval under corrupted observations. Surface Chamfer distance (CD) and vertex p90 are median errors in mm ( ↓ ), with the correct mesh supplied. Retrieval reports top-1 / top-5 counts ( ↑ ), ranking all 221 meshes separately per observation. Counts are out of 198 for clean observations and 594 for each corruption (three realizations per observation). Bold marks the best values within each condition. Ours: ClothAtlas.
Fig. 5: Top-four candidates for a held-out top at lift frame 200: ClothAtlas above, dgnn below. Observation and ground truth are at left while insets show canonical meshes. The correct mesh (green) ranks first for ClothAtlas and fifth for dgnn , outside the displayed candidates. Obs. RMS: observation-to-reconstructed-surface root mean squared distance in mm.
Single observation
Accumulated
Method
Lifted frame 200
Flat frame 0
First three
All nine
ra
0 / 1
12 / 16
11 / 16
0 / 1
lre
1 / 4
0 / 5
0 / 6
1 / 5
dgnn
8 / 14
21 / 21
20 / 21
13 / 15
ClothAtlas
15 / 19
22 / 22
22 / 22
18 / 22
TABLE II: Exact-mesh identification among 221 candidates. Cells report top-1 / top-5 counts for 22 test garments ( ↑ : higher is better). Accumulated scores use γ=1 without pruning. Bold marks the best counts.
Fig. 6: Four separate real-world rollouts compare ClothAtlas (upper rows) with dense semantic descriptors (DD, lower rows) under temporary occlusion (pants, left) and persistent occlusion (top, right). Columns show the initial ZED image, initial and lifted representations, and final grasps. Green marks G2 and cyan its estimate; orange shows the completed mesh, and the heatmap shows descriptor cosine similarity. Checks mark second grasps that reached the specified target; crosses mark failures. ZED is the sole estimator input; the final RealSense views are used only for visualization and annotation.
Mean ± SD error ↓
Success@10 ↑
Method
All
Vis.
Occ.
All
Vis.
Occ.
sp
18.41±7.54
18.55±7.03
18.22±8.24
2.8
1.6
4.3
BootsTAPIR
15.75±8.25
15.93±7.58
15.64±9.02
1.8
0.0
4.3
ddm
15.44±10.12
14.27±10.05
16.99±10.11
34.9
33.9
36.2
ClothAtlas
6.38±6.71
4.91±3.17
8.33±9.27
91.7
96.8
85.1
TABLE III: Grasp-target localization after lifting real garments. Endpoint error is mean ± sample standard deviation in cm ( ↓ ) over available timestamp-aligned predictions. Success@10 is in % ( ↑ ), with unavailable predictions counted as failures. Vis. and occ. denote the 62 visible and 47 occluded endpoints under the fixed 3 cm ZED-support rule.
Seams encode rich structural information about garments but are frequently partially observable in robotic manipulation scenarios. To robustly leverage seam information, we propose a Seam-to-Graph network based on graph neural networks and attention mechanisms. This network maps unstructured seam observations to a topology-encoded structural skeleton graph for real-time garment state estimation. Using this skeleton-graph-based state estimation, we design a deformation-aware, hierarchical visual servoing controller for garment configuration alignment. We implement this controller on a bimanual robot system to load a garment onto a screen printing platen and to align it to the desired configuration precisely. Real-robot experiments demonstrate that the robot using the proposed method not only achieves human-level alignment accuracy with reduced variance in alignment error but is also robust to different garments. These results demonstrate that the use of seam information is effective for garment manipulation.
Xuzhao Huang, Kai Tang, Fuyuki Tokuda +2
JC STEM Lab of Robotics for Soft Materials, Department of Electrical and Electronic Engineering, Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, 000000, China · Unprecedented-scale Data Analytics Center, Tohoku University, Sendai 980-0845, Japan · Department of Electrical and Electronic Engineering, Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, 000000, China
We introduce ClothCode, a unified generative framework for garment state estimation and action-conditioned dynamics prediction in a shared discrete configuration space. A material-aligned tokenizer encodes garment configuration as residual vector-quantized patch tokens, while task-specific Masked Generative Transformers predict these tokens through parallel confidence-guided refinement. Across three garment templates, ClothCode reduces mean displacement error by 26.6% for state estimation and 37.0% for dynamics prediction relative to the strongest baselines. Dynamics prediction requires only 16.7 ms per window, compared with 556.2 ms for diffusion. Integrated into the same model-predictive controller, ClothCode improves real-world folding and flattening success from 56.7% to 78.9%. Further results and videos are available at: https://anonymous.4open.science/r/ClothCode-DE32
Yingdong Ru, Lipeng Zhuang, Shiyu Fan +3
The authors are with the University of Glasgow, Glasgow, United Kingdom.
Robotic manipulation of textiles remains challenging because continuous deformation and self-occlusions hinder the robust visual perception required to estimate the cloth's state. To address the lack of annotated real-world data, we developed a Blender-based synthetic pipeline exporting auto-annotated keypoints, and combined manually labeled renders with real-world data to train a wrinkle detector. We present a perception framework integrating a CNN for permutation-invariant keypoint detection and a YOLOv8-OpenCV pipeline to extract grasping points from structural wrinkles. A proposed bimanual algorithm uses this system to stretch fully folded garments via wrinkles, transitioning to keypoint-based ironing once corners emerge. The keypoint model achieves a Mean Position Error (MPE) of 1.7615 pixels. The perception system transfers to physical fabrics without fine-tuning, outperforming baselines that fail in high-occlusion states or yield false positives on severe folds.
Ariel Herrera, Xueyang Kang, Atal Anil Kumar
Department of Engineering, University of Luxembourg, Esch-sur-Alzette, Luxembourg · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Universit´e de Lorraine, Arts et Metiers Institute of Technology, LCFC, F-57070 Metz, France