Garment manipulation has attracted increasing attention due to its critical role in home-assistant robotics. However, the majority of existing garment manipulation works assume an initial state consisting of only one garment, while piled garments are far more common in real-world settings. To bridge this gap, we propose a novel garment retrieval pipeline that can not only follow language instruction to execute safe and clean retrieval but also guarantee exactly one garment is retrieved per attempt, establishing a robust foundation for the execution of downstream tasks (e.g., folding, hanging, wearing). Our pipeline seamlessly integrates vision-language reasoning with visual affordance perception, fully leveraging the high-level reasoning and planning capabilities of VLMs alongside the generalization power of visual affordance for low-level actions. To enhance the VLM's comprehensive awareness of each garment's state within a garment pile, we employ visual segmentation model (SAM2) to execute object segmentation on the garment pile for aiding VLM-based reasoning with sufficient visual cues. A mask fine-tuning mechanism is further integrated to address scenarios where the initial segmentation results are suboptimal. In addition, a dual-arm cooperation framework is deployed to address cases involving large or long garments, as well as excessive garment sagging caused by incorrect grasping point determination, both of which are strenuous for a single arm to handle. The effectiveness of our pipeline are consistently demonstrated across diverse tasks and varying scenarios in both real-world and simulation environments. Project page: https://garmentpile2.github.io/.
Figures & tables
Fig. 2: GarmentPile++ Pipeline. Given the initial observation, SAM2 segments the garment pile. Due to pile complexity, masks may be inaccurate, thus optionally triggering mask fine tuning procedure (Bottom-Left, red) based on VLM reasoning. The finalized mask and observation are then passed to the VLM to select the target garment under task constraints. Treating the target mask as garment pile point-cloud feature, the Retrieval Affordance Model predicts single-arm grasp points, which is used for single-arm retrieval. Then the post-grasp observation is re-queried by the VLM to decide on dual-arm cooperation; if required, a tracking-selection module (Bottom-Right, green) produces cooperative grasp points to ensure successful retrieval. What’s more, if VLM detects the single-arm grasp of more than one garment, the current retrieval will be terminated.
Fig. 3: Segmentation Performance before and after Mask Fine Tuning Procedure. Initial segmentation result (left-side images of each part) may be flawed : e.g., a mask covering multiple garments (mask 1 in blue part row 1, 2; mask 1 in orange part row 1), fragmented masks (mask 5 in orange part row 2; mask 4,6,7 in green part), or color confusion (mask 7 in cyan part covering two white garments). ✩ denotes the lift point for tuning. After fine tuning, the adjusted masks (right-side images of each part) are clearly more reasonable, with each mask generally corresponding to a single complete garment.
Fig. 4: Reasoning Procedure. For sequential garment retrieval (rows 1 and 3), the VLM infers the most suitable garment to retrieve (e.g., topmost or with simpler entanglement) to maximize overall success. For specific garment retrieval (row 2), it inters to remove obstructing garments so that the target garment is quickly exposed and ready for retrieving.
Fig. 5: Affordance Visualization. The color transition (blue → green → red) indicates an increasing affordance level. Beyond generalization across different garment types, the affordance further captures: garment geometry (row 1, center regions of garments are typically preferable.), subtle structures (row 4, garment wrinkle regions easier to grasp exhibit higher affordance level.), spatial relations (row 3 left, grasping left / right region of glove is equivalent, but the right side is preferable due to less garments, as well as tops side of garment in row 3 right; row 2, central regions far from closed boundaries are preferable for grasp safety.)
Method
Retrieve Garments Sequentially
Retrieve a Piece of Specific Garment
Open Boundary
Closed Boundary
Hat (Open Boundary)
Glove (Closed Boundary)
Green Tops (Open Boundary)
Black Trousers (Closed Boundary)
ASRA↑
AMS
ASRA↑
AMS
ASRB↑
AMS
ASRB↑
AMS
ASRB↑
AMS
ASRB↑
AMS
ThinkGrasp
0.641
╲
0.413
╲
0.517
2.017
0.593
1.439
0.542
1.067
0.585
1.171
GarmentPile
0.819
╲
0.792
╲
0.623
2.798
0.659
3.152
0.729
1.957
0.545
1.394
Qwen (only)
0.829
╲
0.720
╲
0.708
1.858
0.739
1.797
0.787
1.160
0.686
1.081
Our Method
0.904
╲
0.874
╲
0.892
2.196
0.925
2.142
0.877
1.106
0.862
1.147
TABLE I: Comparison of baselines on garment retrieval tasks.
Method
Retrieve a Piece of Green Tops (Open Boundary)
Retrieve Garments Sequentially (Closed Boundary)
ASRB↑
ASRA↑
w/o Tracking Selection
0.669
0.739
w/o Dual Arm
0.613
0.634
w/o Affordance & Dual Arm
0.543
0.564
ASRB↑
PDR ↓
ASRA↑
PDR ↓
w/o Mask Fine Tuning
0.841
0.794
0.857
0.624
TABLE II: Ablation of our methods on two typical tasks
Real-world household robots require Vision-Language-Action (VLA) foundation models that can acquire reusable manipulation skills across diverse objects, task conditions, and household environments. Deformable-object folding is a representative challenge, requiring robots to handle clothing items from random initial states across varying categories, geometries, materials, and scenes. However, existing VLA systems commonly train separate policies for different object categories, while naively mixed multi-task training often suffers from task interference and degraded performance. To move beyond category-specific folding policies, we introduce DeMaVLA, a VLA foundation model for generalizable Deformable Manipulation. DeMaVLA adopts a VLM backbone with an action expert and formulates continuous action generation using flow matching. To improve efficiency, the action expert is constructed by pruning every other transformer layer while preserving layer-wise alignment with the VLM backbone, reducing training and inference cost. DeMaVLA is first pre-trained on approximately 5,000 hours of selected real-world dual-arm demonstrations to acquire general manipulation priors. It is then post-trained on mixed folding data that aggregates self-collected demonstrations and corrective trajectories from real-robot failures across multiple folding tasks through a human-in-the-loop Data Aggregation~(DAgger) pipeline. Experiments show that DeMaVLA achieves competitive performance on RoboTwin 2.0 and strong real-world results on our household folding benchmark. These results highlight the value of scalable real-world data, efficient action generation, and corrective learning for general-purpose VLA policies in deformable-object manipulation.
Robotic garment unfolding is essential for downstream tasks, yet quasi-static methods require repeated actions, while existing dynamic approaches predominantly rely on bimanual flinging. We present RotateIt!, a single-arm framework that uses adaptive axial rotation for dynamic garment unfolding. To the best of our knowledge, it is the first unfolding framework to employ dynamic axial rotation as its primary manipulation primitive. From a randomly initialized tabletop configuration, the robot selects a rotation-effective grasp and rotates the lifted garment about an approximately fixed anchor, generating inertial tension that separates overlapping layers within a compact workspace. A grasp ranker selects the anchor, while an online residual policy adapts the rotation extent and speed, thereby determining the release timing. Across seen and unseen simulated garments and eight unseen real garments, RotateIt! improves success within three attempts by 44.0-61.0 percentage points over quasi-static pick-and-place. The simulation-trained policies transfer zero-shot to the real world, achieving 75.6% success, 41% higher first-attempt coverage, and 26% higher final coverage. The resulting states further enable autonomous robotic folding without manual rearrangement.
Zeqing Zhang, Zuokun Xie, Ao Fang +4
Nanyang Technological University · The University of Hong Kong · Chinese Academy of Sciences
Virtual try-on (VTO) has made significant progress in realistically transferring garments onto a target person. Yet most systems give the user little control over how a garment should be worn -- its size (loose or fitted), style (e.g., tucked in or untucked, open or closed), and spatial placement on the body. We address this gap with two complementary contributions. First, we define and solve Visual-Instance-Prompt Segmentation via VIP-SAM: given a flatlay image of a garment, segment that specific instance in a photograph of a person wearing it. This is an instance-level task, distinct from the typically studied category-level segmentation. Second, we introduce CtrlVTON, a controllable VTO framework that recasts try-on as an image editing problem and adds segmentation masks as pixel-level control over garment layout, including style, size, and spatial placement on the body. VIP-SAM and CtrlVTON each achieve state-of-the-art results on their respective tasks. In particular, CtrlVTON generates images that follow user-provided layouts far more faithfully than the strongest proprietary editing systems while matching them on garment fidelity.
Seungyong Lee, Hyun Jun Jang, Sangoh Kim +1
NXN Labs · Work done during internship at NXN Labs · KAIST