GarmentSegAgent: Autonomous Garment Segmentation for Web Fashion Images via Textual Self-Refinement
Authors: Jia Chen, Chao Huang, Xinrong Hu, Mingdi Huang, Ruyang Zhao, Jun Bai
Organizations: School of Computer Science and Artificial Intelligence, Wuhan Textile University, Wuhan, China · Engineering Research Center of Hubei Province for Clothing Information, Wuhan Textile University, Wuhan, China · Australian Institute for Machine Learning (AIML), Adelaide University, Adelaide, Australia
Web fashion images support applications such as visual product search and consumer-to-shop matching, yet their visual heterogeneity makes reliable garment segmentation challenging. Despite advances in promptable segmentation, autonomously identifying and correcting garment-specific errors without human-provided spatial prompts remains difficult. We propose GarmentSegAgent, an autonomous framework that integrates semantic planning, spatial execution, and visual evaluation through iterative textual self-refinement. At its core, we introduce CAST (Constrained Attribution-guided Selective TextGrad), which connects visual error attribution with targeted textual optimization. CAST identifies instruction components responsible for segmentation errors, selectively updates them through textual gradients, and validates the resulting instructions against garment-specific constraints. This enables image-specific error correction at inference time while keeping the underlying foundation models frozen. Experiments on DeepFashion2, ModaNet, and Fashionpedia demonstrate consistent improvements in region overlap and boundary accuracy, alongside more reliable refinement than conventional TextGrad. By enabling visual feedback to guide controlled instruction updates, GarmentSegAgent offers a promising direction for autonomous garment understanding, with potential applications in scalable fashion content analysis, visual retrieval, and consumer-to-shop matching on the Web.
Figures & tables
Figure 1. Motivation for error-attributed selective textual refinement. (a) Initial segmentation with missing garment regions and background leakage. (b) Naive whole-prompt refinement may disrupt otherwise correct instructions. (c) CAST maps visual errors to relevant instruction fields for selective, garment-constrained textual refinement.
Figure 2. Overview of GarmentSegAgent. The Planner generates a garment-aware semantic plan, the Actor produces candidate masks, and the Evaluator selects predictions and provides visual feedback. CAST performs selective, garment-constrained textual refinement to guide subsequent segmentation iterations.
Dataset
Method
Source
mIoU ↑
Dice ↑
Boundary IoU ↑
Precision ↑
Recall ↑
DeepFashion2
SEEM
NeurIPS’23
0.4587
0.5619
0.2372
0.4777
0.7862
LISA
CVPR’24
0.6913
0.7761
0.4688
0.7340
0.8828
SAM 2 (box-only)
ICLR’25
0.6328
0.7047
0.4124
0.7130
0.7391
Grounding DINO + SAM 2
ECCV’24 / ICLR’25
0.2689
0.2889
0.2122
0.3129
0.2759
READ
CVPR’25
0.6067
0.6866
0.4008
0.7728
0.6761
SegAgent
CVPR’25
0.4638
0.5675
0.2477
0.5871
0.7178
Table 1. Quantitative comparison of upper-garment segmentation on DeepFashion2, ModaNet, and Fashionpedia. The best score for each metric within each dataset is highlighted in bold.
DeepFashion2
ModaNet
Fashionpedia
Method
mIoU ↑
BIoU ↑
mIoU ↑
BIoU ↑
mIoU ↑
BIoU ↑
One-pass
0.6961
0.6148
0.6145
0.5369
0.6982
0.6080
Rule-based Refinement
0.7014
0.6179
0.6399
0.5534
0.7080
0.6079
Vanilla TextGrad
0.7568
0.6242
0.6001
0.5221
0.7235
0.6304
All-field TextGrad
0.7491
0.6218
0.6283
0.5392
0.7128
0.6153
Selective TextGrad
0.7836
0.6287
0.6454
0.5558
0.7339
0.6612
Table 2. Comparison of textual refinement strategies on three fashion segmentation benchmarks. BIoU denotes Boundary IoU. All iterative methods share the same foundation models and maximum refinement budget.
Figure 3. Evolution of mIoU during iterative refinement on (a) DeepFashion2, (b) ModaNet, and (c) Fashionpedia. Iteration 0 denotes the initial prediction. Shaded regions indicate 95% bootstrap confidence intervals. Predictions from early-terminated samples are retained for subsequent iterations.
Refinement Outcomes
Inference Cost
Dataset
Method
Δ IoU ↑
IR (%) ↑
DR (%) ↓
Avg. Steps ↓
LLM Calls ↓
Time (s/img) ↓
DeepFashion2
Vanilla TextGrad
+0.0607
72.4
18.0
2.2
4.4
7.8
Full CAST
+0.1163
88.7
6.4
2.6
6.1
10.9
ModaNet
Vanilla TextGrad
-0.0144
43.8
48.2
2.4
4.8
8.5
Full CAST
+0.0588
79.6
12.0
2.7
6.4
11.8
Fashionpedia
Vanilla TextGrad
+0.0253
61.5
29.2
2.2
4.4
7.9
Table 3. Refinement outcomes and inference cost on three fashion benchmarks. IR and DR denote the percentages of improved and degraded predictions, respectively.
Figure 4. Qualitative comparison of upper-garment segmentation under target ambiguity, occlusion, ambiguous boundaries, and multiple garments (top to bottom). Columns show the input, ground truth (GT), and predictions from SAM 2, LISA, SegZero + SAM 2, and GarmentSegAgent (Ours). GT is used only for evaluation.
Variant
mIoU ↑
BIoU ↑
DR (%) ↓
Full CAST
0.8124
0.6318
6.4
w/o VEA
0.7128
0.4924
14.0
w/o Selective Routing
0.7287
0.5121
14.0
w/o Garment Validation
0.7128
0.4917
18.0
Table 4. Ablation of CAST on DeepFashion2. DR: percentage of samples with decreased final IoU.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5. Visualization of iterative garment segmentation refinement. The top row presents the input image, ground truth (GT), and final prediction, while the bottom row shows the predicted masks across Iterations 0–4. GT is used only for offline comparison.
Figure 6. Qualitative comparison of person segmentation on COCO under target ambiguity, occlusion, and ambiguous boundaries.
Method
mIoU
Dice
BIoU
Prec.
Rec.
READ
0.5820
0.6720
0.4660
0.6460
0.6800
SEEM
0.6110
0.6560
0.4990
0.6610
0.7860
LISA
0.6420
0.7140
0.5250
0.7060
0.8830
SegZero
0.6970
0.7520
0.5770
0.7610
0.8770
Grounding DINO+SAM 2
0.7650
0.8210
0.6450
0.8320
0.8500
GarmentSegAgent
0.7138
0.7826
0.6049
0.8017
0.9045
Appendix
Table 5. Cross-domain person segmentation on COCO. BIoU denotes Boundary IoU.
Domain
Method
mIoU ↑
BIoU ↑
Shop
SegZero + SAM 2
0.808
0.625
GSA (TextGrad)
0.817
0.638
GSA (CAST)
0.826
0.650
Consumer
SegZero + SAM 2
0.784
0.584
GSA (TextGrad)
0.792
0.599
GSA (CAST)
0.799
0.614
Appendix
Table 6. Segmentation performance across shop and consumer images in DeepFashion2 (500 images per domain). GSA denotes GarmentSegAgent, and BIoU denotes Boundary IoU.
Component
Refinement Trace
Visual Errors
Missing jacket regions (torso, sleeves, collar, and hem), with erroneous inclusion of hands, skin, and a fur collar.
Selected Fields
Coverage and exclusion.
Initial Instruction
Preserve visible garment fabric and exclude body parts, other garments, and background.
Textual Feedback
Re-examine incomplete jacket coverage and the inclusion of non-garment regions.
Updated Fields
Coverage : retain the visible torso, sleeves, cuffs, collar, and hem. Exclusion : remove body parts, accessories, unrelated garments, and background.
Preserved Rule
Select the eligible upper-body garment with the largest visible fabric area.
Appendix
Table 7. Condensed CAST refinement trace for Fashionpedia image 10324. Textual entries summarize the recorded feedback and updates; IoU is measured offline.
Dataset
Updates
Direct (%)
Repair (%)
Rollback (%)
Viol. (%)
DeepFashion2
650
74.0
26.0
0.0
0.2
ModaNet
640
71.9
28.1
0.0
0.3
Fashionpedia
530
68.2
31.8
0.0
0.8
Appendix
Table 8. Garment-contract validation outcomes. Direct, Repair, and Rollback denote the final handling of proposed updates; Viol. denotes independently assessed constraint violations.