Garment flattening and folding require robots to identify garment structure during manipulation and adapt their actions as the cloth deforms. We present ClothDojo, a simulation platform that generates adaptive manipulation trajectories at scale across diverse garments and initial states and evaluates policies in closed loop. ClothDojo encodes each garment's canonical mesh coordinates as an NOCS texture, providing canonical surface-position cues through the visual interface of a pretrained vision-language-action model. We fine-tune the model on human demonstrations and corrections to obtain a data-generation policy. The policy autonomously generates simulated manipulation trajectories, and we retain rollouts that satisfy task-specific success criteria. Each retained trajectory is re-rendered under a randomly sampled appearance to provide RGB observations for downstream policies that do not receive NOCS. Across both tasks, the policy produces nearly 8,000 successful trajectories over hundreds of garments. RGB policies trained on these generated trajectories outperform those trained directly on the original human demonstrations and corrections across all evaluated garment splits and appearance settings. Evaluations across multiple policy models show that overall performance and appearance robustness vary across models and tasks, while generalization to unseen garments remains a common challenge.
Figures & tables
Fig. 2: Autonomous policies generate long-horizon behaviors with varied action transitions and adjustments during execution. Flattening trajectories combine midair regrasping, dynamic unfolding, and local adjustments; folding includes hand switching, garment rotation, and post-fold adjustments. Each row shows keyframes from an independent rollout, ordered from left to right with nonuniform time intervals. Box labels describe observed behaviors; the images are re-rendered from recorded simulation states.
Fig. 3: Overview of the NOCS-guided data-generation pipeline. Expert demonstrations DE initialize a data-generation policy. For selected failures encountered during autonomous execution, an operator restores a chosen state and provides a correction DH . The two sources of human data update the policy, while successful rollouts are re-rendered under randomized appearances to train an RGB policy.
Fig. 4: The core garment pool spans diverse geometries. The figure shows most of the 500 assets, covering different garment lengths, widths, sleeve types, necklines, and asymmetric cuts. All garments are naturally settled and shown at their relative physical sizes.
Task
Expert
HIL corrections
raw → aug. / garments
raw → aug. / garments
Flatten
890 → 1,780 / 182
201 → 402 / 88
Fold
537 → 1,074 / 156
213 → 426 / 125
TABLE I: Human training data for the data-generation policies.
Method
SR (%) ↑
Seen
Unseen
Def. ↓
Rot. ↓
White
24.8
26.3
23.3
0.081
0.024
+ NOCS
62.9
66.3
59.5
0.080
0.024
+ Mirror
69.6
76.0
63.3
0.080
0.025
+ HIL
76.5
82.3
70.8
0.080
0.024
Expert
–
–
–
0.068
0.019
TABLE II: Data-generation-policy performance on flattening.
Method
SR (%) ↑
Seen
Unseen
Q↑
White
55.1
61.0
44.3
0.809
+ NOCS
57.6
61.2
51.1
0.779
+ Mirror
60.5
64.2
53.6
0.846
+ HIL
70.0
73.7
63.2
0.825
Expert
–
–
–
1.037
TABLE III: Data-generation-policy performance on folding.
Fig. 5: The data-generation policy produces successful terminal states with visual quality close to human demonstrations. Rows show selected released states at P10, P50, and P90 of each source’s eligible subset, ranked jointly by deformation and orientation errors for flattening, and by Q for folding. The two values below each flattening image are ddef and drot , respectively (lower is better); folding images report Q (higher is better). Flattening images use later released states, whereas Table II measures first confirmed success.
Task
Human
Generated
Combined
Flatten
1,091 / 217
3,992 / 400
5,083 / 470
Fold
750 / 242
3,856 / 399
4,606 / 458
TABLE IV: RGB-policy training data (trajectories / garments).
Task
Training set
Seen garments
Unseen garments
Avg.
S app.
U app.
S app.
U app.
Flatten
Human
33.3
30.5
22.3
19.3
26.3
Generated
41.3
33.3
23.5
23.8
30.4
Combined
32.8
28.0
20.3
19.5
25.1
Fold
Human
68.0
61.8
40.0
37.8
51.9
Generated
72.3
69.0
47.8
47.0
59.0
TABLE V: RGB-policy success rates (%) by training set.
Task
Method
Seen garments
Unseen garments
Avg.
S app.
U app.
S app.
U app.
Flatten
π0.5
41.3
33.3
23.5
23.8
30.4
StarVLA
25.0
14.3
12.8
8.3
15.1
GR00T
16.0
17.3
10.8
12.3
14.1
Diffusion
3.5
4.5
1.3
3.0
3.1
Fold
π0.5
72.3
69.0
47.8
47.0
59.0
TABLE VI: RGB-policy success rates (%) by method.
Robot
Flatten → Fold
Fold
Flatten SR
Fold SR
Fold SR
Piper
49.4
20.0
17.8
UR5e
68.9
7.1
27.8
TABLE VII: Task composition and cross-embodiment reuse.
Fig. 6: Potential task extensions of ClothDojo. Left: humanoid dressing. Right: bimanual garment hanging. Both settings introduce more complex cloth–rigid-body contacts than tabletop flattening and folding.
Due to the highly deformable nature of garments, training a generalizable policy for robotic T-shirt folding and unfolding remains a significant challenge. In this work, we present a large-scale synthetic dataset for robotic T-shirt folding and unfolding, covering 6 robotic embodiments, 1K T-shirts, 1K environmental assets, and 120K episodes with rich annotations, which can be used to train a wide range of manipulation policies. We first follow the FoldNet pipeline to generate a large-scale dataset of physically simulatable T-shirts with diverse appearances and annotated semantic keypoints. Based on these semantic keypoints, we then generate manipulation demonstrations for different robotic embodiments through a unified rule-based framework. We use these demonstrations to train visuomotor policies, and experimental results demonstrate that models trained solely on our synthetic data can achieve over 90% end-to-end task success rates when directly deployed to unseen real-world environments and previously unseen T-shirts from arbitrary initial configurations. Project URL: https://pku-epic.github.io/FoldNetXX/.
Although cloth is known to exhibit different outcomes under repeated fast dynamic motions, even when the same trajectory is applied, this variability has not yet been systematically characterized. Quantifying it is essential to assess the reliability of learned manipulation policies and the extent to which simulation can reproduce real-world behavior. To study this, we execute the same trajectory ten times across four dynamic tasks, two of which are novel, each tested with three cloths of very different properties and at up to three execution speeds, with a total of 269 recorded rollouts. For all of them, we record small marker positions on the cloth and synchronized stereo camera. We then formalize different metrics to quantify variability, and our results show how it is significant in every test condition, in most cases one to three orders of magnitude above the repeatability inherent to the robot and sensing noise. Our results also show variability is driven mainly by the cloth physical properties but also grows with speed. By replaying all the rollouts in four calibrated modern cloth simulators, we show that none of them can reproduce the variability magnitude we observed in real cloth, nor its ordering. Together with our analysis, we publish the dataset with synchronized OptiTrack, vision and robot logs and their corresponding simulator twins.
Mahed Dadgostar, Guillem Alenyà, Júlia Borràs
Institut de Robòtica i Informàtica Industrial, CSIC-UPC, Spain
We present FoldBack, a self-correcting masked generative policy for long-horizon garment folding. Existing long-trajectory policies may continue after a missed or slipped grasp even when the garment has not reached the intended configuration. We structure FoldBack's recovery mechanisms around three inference-time decisions: when to refine and verify, how to roll back, and where and how to retry. FoldBack aligns refinement and grasp verification with pick-and-place events, returns the robot to a retryable pre-grasp configuration while preserving successful grasps, and selectively regenerates the failed segment and selected future actions while avoiding previous failed grasp locations. To our knowledge, FoldBack is the first editable full-trajectory policy to unify these decisions, enabling failed interactions to be detected, undone, and repaired before execution continues, without recovery demonstrations or base-policy retraining. Across 33 real garments from six categories, FoldBack achieves 75.2% final folding success and 0.837 final-mask IoU, versus 45.7% and 0.689 for the strongest prior baseline.