GarmentSegAgent: Autonomous Garment Segmentation for Web Fashion Images via Textual Self-Refinement
Authors: Jia Chen, Chao Huang, Xinrong Hu, Mingdi Huang, Ruyang Zhao, Jun Bai
Organizations: School of Computer Science and Artificial Intelligence, Wuhan Textile University, Wuhan, China · Engineering Research Center of Hubei Province for Clothing Information, Wuhan Textile University, Wuhan, China · Australian Institute for Machine Learning (AIML), Adelaide University, Adelaide, Australia
Web fashion images support applications such as visual product search and consumer-to-shop matching, yet their visual heterogeneity makes reliable garment segmentation challenging. Despite advances in promptable segmentation, autonomously identifying and correcting garment-specific errors without human-provided spatial prompts remains difficult. We propose GarmentSegAgent, an autonomous framework that integrates semantic planning, spatial execution, and visual evaluation through iterative textual self-refinement. At its core, we introduce CAST (Constrained Attribution-guided Selective TextGrad), which connects visual error attribution with targeted textual optimization. CAST identifies instruction components responsible for segmentation errors, selectively updates them through textual gradients, and validates the resulting instructions against garment-specific constraints. This enables image-specific error correction at inference time while keeping the underlying foundation models frozen. Experiments on DeepFashion2, ModaNet, and Fashionpedia demonstrate consistent improvements in region overlap and boundary accuracy, alongside more reliable refinement than conventional TextGrad. By enabling visual feedback to guide controlled instruction updates, GarmentSegAgent offers a promising direction for autonomous garment understanding, with potential applications in scalable fashion content analysis, visual retrieval, and consumer-to-shop matching on the Web.
Figures & tables
Figure 1. Motivation for error-attributed selective textual refinement. (a) Initial segmentation with missing garment regions and background leakage. (b) Naive whole-prompt refinement may disrupt otherwise correct instructions. (c) CAST maps visual errors to relevant instruction fields for selective, garment-constrained textual refinement.
Figure 2. Overview of GarmentSegAgent. The Planner generates a garment-aware semantic plan, the Actor produces candidate masks, and the Evaluator selects predictions and provides visual feedback. CAST performs selective, garment-constrained textual refinement to guide subsequent segmentation iterations.
Dataset
Method
Source
mIoU ↑
Dice ↑
Boundary IoU ↑
Precision ↑
Recall ↑
DeepFashion2
SEEM
NeurIPS’23
0.4587
0.5619
0.2372
0.4777
0.7862
LISA
CVPR’24
0.6913
0.7761
0.4688
0.7340
0.8828
SAM 2 (box-only)
ICLR’25
0.6328
0.7047
0.4124
0.7130
0.7391
Grounding DINO + SAM 2
ECCV’24 / ICLR’25
0.2689
0.2889
0.2122
0.3129
0.2759
READ
CVPR’25
0.6067
0.6866
0.4008
0.7728
0.6761
SegAgent
CVPR’25
0.4638
0.5675
0.2477
0.5871
0.7178
Table 1. Quantitative comparison of upper-garment segmentation on DeepFashion2, ModaNet, and Fashionpedia. The best score for each metric within each dataset is highlighted in bold.
DeepFashion2
ModaNet
Fashionpedia
Method
mIoU ↑
BIoU ↑
mIoU ↑
BIoU ↑
mIoU ↑
BIoU ↑
One-pass
0.6961
0.6148
0.6145
0.5369
0.6982
0.6080
Rule-based Refinement
0.7014
0.6179
0.6399
0.5534
0.7080
0.6079
Vanilla TextGrad
0.7568
0.6242
0.6001
0.5221
0.7235
0.6304
All-field TextGrad
0.7491
0.6218
0.6283
0.5392
0.7128
0.6153
Selective TextGrad
0.7836
0.6287
0.6454
0.5558
0.7339
0.6612
Table 2. Comparison of textual refinement strategies on three fashion segmentation benchmarks. BIoU denotes Boundary IoU. All iterative methods share the same foundation models and maximum refinement budget.
Figure 3. Evolution of mIoU during iterative refinement on (a) DeepFashion2, (b) ModaNet, and (c) Fashionpedia. Iteration 0 denotes the initial prediction. Shaded regions indicate 95% bootstrap confidence intervals. Predictions from early-terminated samples are retained for subsequent iterations.
Refinement Outcomes
Inference Cost
Dataset
Method
Δ IoU ↑
IR (%) ↑
DR (%) ↓
Avg. Steps ↓
LLM Calls ↓
Time (s/img) ↓
DeepFashion2
Vanilla TextGrad
+0.0607
72.4
18.0
2.2
4.4
7.8
Full CAST
+0.1163
88.7
6.4
2.6
6.1
10.9
ModaNet
Vanilla TextGrad
-0.0144
43.8
48.2
2.4
4.8
8.5
Full CAST
+0.0588
79.6
12.0
2.7
6.4
11.8
Fashionpedia
Vanilla TextGrad
+0.0253
61.5
29.2
2.2
4.4
7.9
Table 3. Refinement outcomes and inference cost on three fashion benchmarks. IR and DR denote the percentages of improved and degraded predictions, respectively.
Figure 4. Qualitative comparison of upper-garment segmentation under target ambiguity, occlusion, ambiguous boundaries, and multiple garments (top to bottom). Columns show the input, ground truth (GT), and predictions from SAM 2, LISA, SegZero + SAM 2, and GarmentSegAgent (Ours). GT is used only for evaluation.
Variant
mIoU ↑
BIoU ↑
DR (%) ↓
Full CAST
0.8124
0.6318
6.4
w/o VEA
0.7128
0.4924
14.0
w/o Selective Routing
0.7287
0.5121
14.0
w/o Garment Validation
0.7128
0.4917
18.0
Table 4. Ablation of CAST on DeepFashion2. DR: percentage of samples with decreased final IoU.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5. Visualization of iterative garment segmentation refinement. The top row presents the input image, ground truth (GT), and final prediction, while the bottom row shows the predicted masks across Iterations 0–4. GT is used only for offline comparison.
Figure 6. Qualitative comparison of person segmentation on COCO under target ambiguity, occlusion, and ambiguous boundaries.
Method
mIoU
Dice
BIoU
Prec.
Rec.
READ
0.5820
0.6720
0.4660
0.6460
0.6800
SEEM
0.6110
0.6560
0.4990
0.6610
0.7860
LISA
0.6420
0.7140
0.5250
0.7060
0.8830
SegZero
0.6970
0.7520
0.5770
0.7610
0.8770
Grounding DINO+SAM 2
0.7650
0.8210
0.6450
0.8320
0.8500
GarmentSegAgent
0.7138
0.7826
0.6049
0.8017
0.9045
Appendix
Table 5. Cross-domain person segmentation on COCO. BIoU denotes Boundary IoU.
Domain
Method
mIoU ↑
BIoU ↑
Shop
SegZero + SAM 2
0.808
0.625
GSA (TextGrad)
0.817
0.638
GSA (CAST)
0.826
0.650
Consumer
SegZero + SAM 2
0.784
0.584
GSA (TextGrad)
0.792
0.599
GSA (CAST)
0.799
0.614
Appendix
Table 6. Segmentation performance across shop and consumer images in DeepFashion2 (500 images per domain). GSA denotes GarmentSegAgent, and BIoU denotes Boundary IoU.
Component
Refinement Trace
Visual Errors
Missing jacket regions (torso, sleeves, collar, and hem), with erroneous inclusion of hands, skin, and a fur collar.
Selected Fields
Coverage and exclusion.
Initial Instruction
Preserve visible garment fabric and exclude body parts, other garments, and background.
Textual Feedback
Re-examine incomplete jacket coverage and the inclusion of non-garment regions.
Updated Fields
Coverage : retain the visible torso, sleeves, cuffs, collar, and hem. Exclusion : remove body parts, accessories, unrelated garments, and background.
Preserved Rule
Select the eligible upper-body garment with the largest visible fabric area.
Appendix
Table 7. Condensed CAST refinement trace for Fashionpedia image 10324. Textual entries summarize the recorded feedback and updates; IoU is measured offline.
Dataset
Updates
Direct (%)
Repair (%)
Rollback (%)
Viol. (%)
DeepFashion2
650
74.0
26.0
0.0
0.2
ModaNet
640
71.9
28.1
0.0
0.3
Fashionpedia
530
68.2
31.8
0.0
0.8
Appendix
Table 8. Garment-contract validation outcomes. Direct, Repair, and Rollback denote the final handling of proposed updates; Viol. denotes independently assessed constraint violations.
Guideline-consistent semantic segmentation requires more than category recognition, as real-world labeling policies demand fine-grained, task-specific decisions. Recent multi-agent refinement systems improve compliance with such textual guidelines by detecting and correcting errors. However, they are stateless: feedback from the critiquing agent is discarded, causing the same guideline-specific mistakes to be repeatedly rediscovered and corrected across the dataset at the cost of additional refinement. We introduce InsightSeg, an episodic memory mechanism that converts successful correction episodes into reusable, visually grounded insights. A meta-analyzer distills each qualifying episode into directive natural-language insights and anchors them to the local image regions that caused the error using patch-level visual concept vectors. On subsequent images, these concepts are matched against dense patch embeddings to retrieve relevant insights, which condition the segmenting agent before making its first prediction. This shifts the system from correcting recurring errors to preventing them, improving segmentation quality before any refinement occurs. Across Waymo and Cityscapes, InsightSeg improves both first-pass and final guideline-consistent segmentation performance while requiring fewer refinement steps, demonstrating that multi-agent refinement can become more accurate and efficient by drawing on past correction experience.
We study the problem of inverse garment design, reconstructing simulation-ready 3D outfits with arbitrary layers from a single image, with visual agents. Unlike differentiable optimization that is time-consuming, or feed-forward methods limited by training data, we introduce DressAgent, a hierarchical agentic system that recovers garment sewing patterns through planning and closed-loop refinement grounded in physical simulation. To this end, we first present an enhanced representation based on GarmentCode for diverse styles and arbitrary dressing layers, and build the first multi-layer sewing pattern dataset with over 15K outfits for supervision and benchmarking. Inverse design then proceeds coarse to fine: In layer planning and coarse estimation, agents infer layer order and relations, peel each layer into its own view, and estimate coarse patterns. In fine-grained refinement, an agentic trio of Tailor, Critic, and Orchestrator refines patterns in a closed loop of editing, simulating, and judging against the reference with visual and measured evidence. Carefully designed tools and an evolving memory make this refinement converge on the target geometry. We also design an agentic generation method for high-fidelity textures. DressAgent outperforms feed-forward methods by a large margin, surpasses or matches differentiable optimization in a fraction of the time, and generalizes to complex layering, diverse poses, and body shapes.
Nanxiang Jiang, Yunuo Chen, Zeng Tao +5
University of California, Los Angeles · Envora · Stanford University +2
Recent advances in Virtual Try-On (VTON) and Virtual Try-Off (VTOFF) have greatly improved photo-realistic fashion synthesis and garment reconstruction. However, existing datasets remain static, lacking instruction-driven editing for controllable and interactive fashion generation. In this work, we introduce the Dress Editing Dataset (Dress-ED), the first large-scale benchmark that unifies VTON, VTOFF, and text-guided garment editing within a single framework. Each sample in Dress-ED includes an in-shop garment image, the corresponding person image wearing the garment, their edited counterparts, and a natural-language instruction of the desired modification. Built through a fully automated multimodal pipeline that integrates MLLM-based garment understanding, diffusion-based editing, and LLM-guided verification, Dress-ED comprises over 146k verified quadruplets spanning three garment categories and seven edit types, including both appearance (e.g., color, pattern, material) and structural (e.g., sleeve length, neckline) modifications. Based on this benchmark, we further propose a unified multimodal diffusion framework that jointly reasons over linguistic instructions and visual garment cues, serving as a strong baseline for instruction-driven VTON and VTOFF. Dataset and code available at this link: https://github.com/aimagelab/Dress-ED
Davide Lobba, Fulvio Sanguigni, Bin Ren +3
University of Trento, Italy · University of Modena and Reggio Emilia, Italy · MBZUAI, United Arab Emirates