ClothAtlas: Joint Garment Identification and State Estimation under Self-Occlusion

Jens Lundell; Alberta Longhini; Lei Shu; Hemal Arora; Evan Robert; Carlota Parés-Morlans; Marcel Büsching; Mårten Björkman

cs.CVOct 12, 2026

ClothAtlas: Joint Garment Identification and State Estimation under Self-Occlusion

Authors: Jens Lundell, Alberta Longhini, Lei Shu, Hemal Arora, Evan Robert, Carlota Parés-Morlans, Marcel Büsching, Mårten Björkman,

Organizations: University of Turku, Finland. · Stanford University, USA. · KTH Royal Institute of Technology, Sweden.

Abstract

Garment manipulation requires inferring both garment identity and complete configuration from partial, self-occluded observations. This creates a chicken-and-egg problem: completing a partial observation requires a garment-specific deformation prior, while identifying the garment requires a configuration that explains the observation. We introduce ClothAtlas, which jointly identifies garments and estimates their complete configurations from partial point clouds and a library of canonical meshes. A shared graph-conditioned flow-matching model samples configurations for each candidate mesh and ranks candidates by observation fit. The model accommodates different mesh shapes, vertex counts, and connectivities, allowing new garments to be added from canonical meshes alone without retraining. On held-out DexGarmentLab garments, ClothAtlas ranks the correct mesh first among 221 candidates for all 22 test garments from a single flat observation. From a single self-occluded observation, it identifies 15/22 garments, compared with 8/22 for a deterministic graph estimator. Persistent vertex identities support grasp-target localization on real garments, reducing mean error from 15.4 to 6.4 cm overall and from 17.0 to 8.3 cm on occluded targets compared with dense descriptor matching. Project page: https://clothatlas.github.io/

Figures & tables

Panel of Figure 1 — Fig. 1: (a) ClothAtlas combines a canonical mesh library and partial depth observations to identify the garment and estimate its complete configuration. Red points show the observations while the blue surface shows the estimated configuration, including regions occluded from the camera. (b) Four reconstructed configurations illustrate persistent point tracking through deformation and occlusion. The camera icon indicates the observation viewpoint and the colored tracks connect the estimated positions of three mesh vertices through the lifting action.

Panel of Figure 2 — Fig. 2: ClothAtlas overview. (a) A shared graph-conditioned generative model samples complete configurations from a partial observation, Gaussian noise, and a candidate canonical mesh, which supplies reference vertex positions and connectivity. The same model is used for every candidate mesh. (b) We score each candidate by the observation fit of its configuration estimate, accumulate evidence across observations, and retain the $k$ highest-weight candidates after the initial identification period. (c) For each retained candidate, we sample a new configuration from the current observation and update its weight using the accumulated evidence. We report the highest-weight candidate’s configuration.

Panel of Figure 3 — Fig. 3: Representative canonical meshes from the retrieval library’s 26 dresses, 95 tops, and 100 trousers.

		Configuration		Retrieval
Condition	Method	CD	Vertex $p_{90}$	Top-1 / Top-5
Clean	dgnn	12.28	35.5	92 / 130
	Ours, $S=4$	11.21	24.8	128 / 170
90% removal	dgnn	15.36	66.8	118 / 246
	Ours, $S=1$	12.92	42.8	254 / 394
	Ours, $S=4$	12.60	37.7	285 / 424

TABLE I: Configuration estimation and exact-mesh retrieval under corrupted observations. Surface Chamfer distance (CD) and vertex

p_{90}

are median errors in mm (

\downarrow

), with the correct mesh supplied. Retrieval reports top-1 / top-5 counts (

\uparrow

), ranking all 221 meshes separately per observation. Counts are out of 198 for clean observations and 594 for each corruption (three realizations per observation). Bold marks the best values within each condition. Ours: ClothAtlas.

Panel of Figure 5 — Fig. 5: Top-four candidates for a held-out top at lift frame 200: ClothAtlas above, dgnn below. Observation and ground truth are at left while insets show canonical meshes. The correct mesh (green) ranks first for ClothAtlas and fifth for dgnn , outside the displayed candidates. Obs. RMS: observation-to-reconstructed-surface root mean squared distance in mm.

	Single observation		Accumulated
Method	Lifted frame 200	Flat frame 0	First three	All nine
ra	0 / 1	12 / 16	11 / 16	0 / 1
lre	1 / 4	0 / 5	0 / 6	1 / 5
dgnn	8 / 14	21 / 21	20 / 21	13 / 15
ClothAtlas	15 / 19	22 / 22	22 / 22	18 / 22

TABLE II: Exact-mesh identification among 221 candidates. Cells report top-1 / top-5 counts for 22 test garments (

\uparrow

: higher is better). Accumulated scores use

\gamma=1

without pruning. Bold marks the best counts.

Panel of Figure 7 — Fig. 6: Four separate real-world rollouts compare ClothAtlas (upper rows) with dense semantic descriptors (DD, lower rows) under temporary occlusion (pants, left) and persistent occlusion (top, right). Columns show the initial ZED image, initial and lifted representations, and final grasps. Green marks G2 and cyan its estimate; orange shows the completed mesh, and the heatmap shows descriptor cosine similarity. Checks mark second grasps that reached the specified target; crosses mark failures. ZED is the sole estimator input; the final RealSense views are used only for visualization and annotation.

Method	All	Vis.	Occ.	All	Vis.	Occ.
	Mean $\pm$ SD error $\downarrow$			Success@10 $\uparrow$
sp	$18.41\mathbin{\pm}7.54$	$18.55\mathbin{\pm}7.03$	$18.22\mathbin{\pm}8.24$	2.8	1.6	4.3
BootsTAPIR	$15.75\mathbin{\pm}8.25$	$15.93\mathbin{\pm}7.58$	$15.64\mathbin{\pm}9.02$	1.8	0.0	4.3
ddm	$15.44\mathbin{\pm}10.12$	$14.27\mathbin{\pm}10.05$	$16.99\mathbin{\pm}10.11$	34.9	33.9	36.2
ClothAtlas	$6.38\mathbin{\pm}6.71$	$4.91\mathbin{\pm}3.17$	$8.33\mathbin{\pm}9.27$	91.7	96.8	85.1

TABLE III: Grasp-target localization after lifting real garments. Endpoint error is mean

\pm

sample standard deviation in cm (

\downarrow

) over available timestamp-aligned predictions. Success@10 is in % (

\uparrow

), with unavailable predictions counted as failures. Vis. and occ. denote the 62 visible and 47 occluded endpoints under the fixed 3 cm ZED-support rule.

3D Garment Reconstruction

See all 15 papers

Seam-to-Graph Reconstruction for Garment Configuration Alignment
Jun 13, 2026Xuzhao Huang, Kai Tang, Fuyuki Tokuda +2Bimanual Robotic Manipulation Deformable Object Manipulation
JC STEM Lab of Robotics for Soft Materials, Department of Electrical and Electronic Engineering, Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, 000000, China · Unprecedented-scale Data Analytics Center, Tohoku University, Sendai 980-0845, Japan · Department of Electrical and Electronic Engineering, Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, 000000, China
ClothCode: Generative Modeling for Garment State Estimation and Dynamics Prediction
Oct 9, 2026Yingdong Ru, Lipeng Zhuang, Shiyu Fan +3Generative Modeling Conditional Generative Modeling
The authors are with the University of Glasgow, Glasgow, United Kingdom.
Synthetic Data Generation and Vision-based Wrinkle and Keypoint Detection for Bimanual Cloth Manipulation
Jun 4, 2026Ariel Herrera, Xueyang Kang, Atal Anil KumarBimanual Robotic Manipulation Deformable Object Manipulation
Department of Engineering, University of Luxembourg, Esch-sur-Alzette, Luxembourg · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Universit´e de Lorraine, Arts et Metiers Institute of Technology, LCFC, F-57070 Metz, France

Jun 13, 2026J/K move · Enter open · S save

Seam-to-Graph Reconstruction for Garment Configuration Alignment

Xuzhao Huang, Kai Tang, Fuyuki Tokuda, Norman C. Tien+1

JC STEM Lab of Robotics for Soft Materials, Department of Electrical and Electronic Engineering, Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, 000000, China · Unprecedented-scale Data Analytics Center, Tohoku University, Sendai 980-0845, Japan · Department of Electrical and Electronic Engineering, Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, 000000, China

Bimanual Robotic Manipulation Deformable Object Manipulation Vision-Based Robot Control

Open paper AskPDF arXiv

Seams encode rich structural information about garments but are frequently partially observable in robotic manipulation scenarios. To robustly leverage seam information, we propose a Seam-to-Graph network based on graph neural networks and attention mechanisms. This network maps unstructured seam observations to a topology-encoded structural skeleton graph for real-time garment state estimation. Using this skeleton-graph-based state estimation, we design a deformation-aware, hierarchical visual servoing controller for garment configuration alignment. We implement this controller on a bimanual robot system to load a garment onto a screen printing platen and to align it to the desired configuration precisely. Real-robot experiments demonstrate that the robot using the proposed method not only achieves human-level alignment accuracy with reduced variance in alignment error but is also robust to different garments. These results demonstrate that the use of seam information is effective for garment manipulation.

Abstract

3D Garment Reconstruction

Explore similar work

Seam-to-Graph Reconstruction for Garment Configuration Alignment

ClothCode: Generative Modeling for Garment State Estimation and Dynamics Prediction

Synthetic Data Generation and Vision-based Wrinkle and Keypoint Detection for Bimanual Cloth Manipulation