Inverse Garment Design with Visual Agents

Nanxiang Jiang; Yunuo Chen; Zeng Tao; Zizhang Li; Qi Guo; Jiajun Wu; Yin Yang; Chenfanfu Jiang

cs.AIOct 11, 2026

Inverse Garment Design with Visual Agents

Authors: Nanxiang Jiang, Yunuo Chen, Zeng Tao, Zizhang Li, Qi Guo, Jiajun Wu, Yin Yang, Chenfanfu Jiang

Organizations: University of California, Los Angeles · Envora · Stanford University · Independent Researcher · University of Utah

Abstract

We study the problem of inverse garment design, reconstructing simulation-ready 3D outfits with arbitrary layers from a single image, with visual agents. Unlike differentiable optimization that is time-consuming, or feed-forward methods limited by training data, we introduce DressAgent, a hierarchical agentic system that recovers garment sewing patterns through planning and closed-loop refinement grounded in physical simulation. To this end, we first present an enhanced representation based on GarmentCode for diverse styles and arbitrary dressing layers, and build the first multi-layer sewing pattern dataset with over 15K outfits for supervision and benchmarking. Inverse design then proceeds coarse to fine: In layer planning and coarse estimation, agents infer layer order and relations, peel each layer into its own view, and estimate coarse patterns. In fine-grained refinement, an agentic trio of Tailor, Critic, and Orchestrator refines patterns in a closed loop of editing, simulating, and judging against the reference with visual and measured evidence. Carefully designed tools and an evolving memory make this refinement converge on the target geometry. We also design an agentic generation method for high-fidelity textures. DressAgent outperforms feed-forward methods by a large margin, surpasses or matches differentiable optimization in a fraction of the time, and generalizes to complex layering, diverse poses, and body shapes.

Figures & tables

Figure 1: We introduce DressAgent for inverse garment design with agentic planning and refinement. (a) Results on in-the-wild photos with arbitrary layering, diverse poses, and body shapes. (b) The outfits drape and animate under physics. (c) Every garment is a sewing pattern with high-fidelity texture. (d) DressAgent pushes the Pareto frontier of quality and speed.

Figure 2: Enhanced Garment Representation and Dataset. Top : Examples of enhanced expressivity and layered representation in GarmentCode-XL. Bottom : Examples from GarmentCodeData-XL. Its $15{,}205$ outfits contain $4{,}640$ one-layer, $9{,}282$ two-layer, $721$ three-layer, and $562$ four-layer examples. From left to right, colors indicate one to four layers. For more details, see Appendix B .

Figure 3: Layer Planning and Coarse Pattern Estimation. The Layer Planner reasons ordered layers and peels each into view. The Visual Reader generates a Garment Style Template per layer, then an encoder-decoder with distilled GarmentCode-XL semantics maps it to a coarse program.

Figure 4: Fine-Grained Refinement with an Agentic Trio. Top : illustration of one refinement round (from Iter 1 to Iter 2 of the inner layer). The Critic scores each garment and identifies discrepancies, the Orchestrator saves a checkpoint and instructs the Tailor, and the Tailor applies the requested edits and simulates the pattern. Bottom : examples of rollout trajectories. For agent implementation, detailed prompts, and the complete record of this example, see Appendix D .

Figure 5: Texture Generation and Simulation. We design an agentic generative approach to recover textures for the sewing patterns, and use an XPBD-based cloth solver for dynamic simulation.

Figure 6: Qualitative comparison on in-the-wild images. Top five : reconstructing outfits with one to three layers (left to right). Bottom five : generalization to diverse poses and body shapes. ChatGarment, ISP, and GarmentRecovery lack texture generation, so we show only their geometry.

	Structure Acc. $\uparrow$			Geometry (CloSe)		Geometry (4D-Dress)
Method	Panel	Edge	Stitch	CD $\downarrow$	IoU $\uparrow$	CD $\downarrow$	IoU $\uparrow$	Avg. Time $\downarrow$ (min)
SewFormer	18.67	28.80	7.24	7.40	0.348	7.58	0.496	2.0
ChatGarment	51.84	57.15	70.18	2.39	0.580	1.94	0.763	4.2
GarmentGPT	43.62	38.54	65.43	5.50	0.381	10.46	0.204	0.4
DressCode	19.67	26.47	30.97	2.77	0.542	2.56	0.685	2.2
ISP	-	-	-	1.79	0.687	1.16	0.859	90.0

Table 1: Quantitative comparison. We report the structure accuracy of sewing patterns and the geometry accuracy of draped garments. ISP and GarmentRecovery are excluded from structure accuracy, as they do not produce explicit sewing patterns with panel geometry. Our method achieves the best quality on most metrics, and is significantly faster than differentiable optimization.

	GarmentCodeData-XL		DeepFashion2
Comparison	Struct. Acc. $\uparrow$	IoU $\uparrow$	IoU $\uparrow$
w/o Refinement	63.25	0.623	0.562
w/o GarmentCode-XL	57.99	0.574	0.650
w/o Critic	69.56	0.822	0.765
w/o Measure	67.21	0.804	0.772
Full	73.59	0.839	0.783

Table 3: Ablation on representation and the agentic trio. DeepFashion2 IoU is measured against the annotated clothing masks.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Garment Representation
Symbol	Meaning
$S=P(B)$	Sewing pattern of one garment: its flat panels and the stitches that join them, drafted by $P$ on body $B$ .
$P=\langle\mathcal{F},\mathcal{D}\rangle$	Garment program: panel constructors with their design parameters.
$\mathcal{F}=\{f_{i}\}$	Panel constructors, one per component such as a sleeve or a bodice.
$f_{i}$	The $i$ -th constructor, drafting a component as parametric 2D curves controlled by $d_{i}$ and $B$ .
$\mathcal{D}=\{d_{i}\}$	Design parameters, $131$ in GarmentCode-XL.

Appendix

Table 4: Notations. Summary of the main symbols and notations used in the paper.

Figure 10: Examples of Construction details of GarmentCode-XL. New panel constructors and outfit fields are marked $+$ , and expanded ranges of existing parameters are marked $\sim$ . The first seven widen the design space to real-world styles, and the last adds the layered outfit structure with dressing order and relations.

Figure 11: Example of a two-layer GarmentCode-XL outfit. The tee is tucked into the skirt within the inner layer $L_{1}$ . The cardigan forms the outer layer $L_{2}$ , with $L_{1}$ worn under $L_{2}$ .

Parameter type	Count	Options	Answer format
Length (float)	$83$	$308$	3 to 6 ordered visual states
Count (int)	$15$	$55$	ordered states, or one option per admissible count
Boolean	$14$	$28$	the two geometric states of the flag
Categorical	$11$	$56$	one option per legal value
Categorical with absence	$8$	$34$	one option per legal value, plus absence
Total	$131$	$481$

Appendix

Table 5: The frozen answer space. Here, every design parameter of GarmentCode-XL becomes exactly one multiple choice question.

	Symmetric		Asymmetric
Garment type	Questions	Numeric	Questions	Numeric
Top	$48$	$34$	$79$	$56$
Skirt	$42$	$33$	$42$	$33$
Pants	$15$	$12$	$15$	$12$
Dress	$90$	$67$	$121$	$89$
Jumpsuit	$63$	$46$	$94$	$68$

Appendix

Table 6: Layer-specific questionnaire size. Number of questions retained from

\mathcal{Q}

for each garment type and symmetry state.

Figure 12: Architecture of the decoder

\Psi

. The frozen encoder

E

embeds each question with its selected answer. A projection MLP maps the embeddings to the decoder width, and a causal Transformer with

12

blocks decodes the numeric program parameters.

Tool	Access	Arguments	Description
Observation
Render	T	pictures	Views of any step: the photograph, the target and drape pair of either view, the before/after strip, per-garment mask diffs, and the flat sewing pattern.
Measure	T	step	Measurement tables of one simulated step: every garment edge on the drape against the target, and the same sizes read off the drafted pattern, in percent of body height.
Modification
Draft	T	edits	Drafts the current program with the edits and returns the garment sizes of the flat pattern, with the change against the current step.
Fit	T	part , channel , lever , target	Value of one numeric lever that lands one measured size on its target, found through drafting probes (Appendix D.4 ). Costs no simulation.

Appendix

Table 7: The complete tool interface of Stage 2. Access: T = Tailor, C = Critic, O = Orchestrator.

Panel of Figure 13 — Figure 13: The observation $O^{t}$ given to the agents in the recorded round. Left : the paired front view of step 1 (Iter 1 of the inner layer in Fig. 4 ), which the Critic and the Tailor read. The skirt has been corrected, and the tee is still wide, deep at the neck, and long in the sleeve. Right : the same observation at step 2, after the five fitted edits below. Both panels carry the same landmark lines and body ruler in percent of body height.

Figure 14: Examples of the five texture classes. From left to right: a solid fabric, a knit without an exact repeat, placed artwork over a base fabric, an exact repeating lattice, and a non-repeating print, each handled by the tool named below it.

	Canonical A-pose			Target Pose
Method	$\Delta E$ $\downarrow$	CLIP-I $\uparrow$	DINO $\uparrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	$\Delta E$ $\downarrow$	CLIP-I $\uparrow$	DINO $\uparrow$
SewFormer	20.61	0.661	0.252	8.39	0.543	0.433	14.05	0.681	0.324
DressCode	9.67	0.633	0.305	8.68	0.565	0.407	12.93	0.652	0.275
Dress 1-to-3	-	-	-	8.26	0.531	0.410	21.26	0.650	0.229
Ours	9.57	0.821	0.658	9.94	0.615	0.352	8.43	0.800	0.597

Appendix

Table 8: Quantitative comparison on texture generation. We evaluate the texture fidelity of each method against the input photo on different metrics. Dress 1-to-3 reconstructs the garment directly in the target pose, so we do not report its results in the canonical A-pose.

	GarmentCodeData-XL (max 4 layers)		DeepFashion2 (max 3 layers)
Method	Count Acc. $\uparrow$	Struct. Acc. $\mid$ Count $\uparrow$	Count Acc. $\uparrow$	Struct. Acc. $\mid$ Count $\uparrow$
DressCode	34.0%	76.5%	31.7%	52.6%
ISP	50.0%	87.0%	65.0%	84.6%
GarmentRecovery	49.5%	85.9%	61.7%	83.8%
Ours	95.5%	100.0%	96.7%	98.3%

Appendix

Table 9: Quantitative comparison on layer structure. We report the accuracy of the layer count, and the accuracy of the layer structure conditional on a correct count, on GarmentCodeData-XL and DeepFashion2. Only the methods that can model more than one layer are included.

	CloSe			4D-Dress
Comparison	CLIP-I $\uparrow$	DINO $\uparrow$	$\Delta E$ $\downarrow$	CLIP-I $\uparrow$	DINO $\uparrow$	$\Delta E$ $\downarrow$
Other outfits vs. photo	0.869	0.678	7.55	0.705	0.391	23.29
Second photo vs. photo	-	-	-	0.917	0.776	3.46
Canonical view vs. photo	0.907	0.845	5.49	0.806	0.729	4.31

Appendix

Table 10: Faithfulness of the canonical view. We compare garment crops with the input photograph. Second photo is another frame of the same 4D-Dress capture, a real view of the same outfit. Other outfits compares the canonical view with the photographs of all other outfits in the dataset.

Figure 15: From one T-shirt to any garment. We show four examples of refinement runs that start from the same plain T-shirt (left). The trio first intelligently modifies the construction, then refines the sizes to match the photo.

Figure 16: Refinement from a plain T-shirt. Silhouette IoU on CloSe and 4D-Dress over the refinement iterations, starting from the Stage-1 program (solid) or from a plain T-shirt (dashed). Bands show the 95% bootstrap confidence interval.

	Geometry (CloSe)		Geometry (4D-Dress)
Starting point	CD $\downarrow$	IoU $\uparrow$	CD $\downarrow$	IoU $\uparrow$
From Plain T-shirt	1.75	0.699	1.31	0.871
From Stage-1 Program (Ours)	1.20	0.764	1.21	0.880

Appendix

Table 11: Refinement from a plain T-shirt. Comparison of the geometry of the reposed garment against the scan, with the protocol of Table 1 .

	CloSe		4D-Dress
Setting	CD $\downarrow$	IoU $\uparrow$	CD $\downarrow$	IoU $\uparrow$
Input only	1.36	0.743	2.28	0.782
Front only	1.29	0.750	1.73	0.833
No peeling	-	-	1.64	0.846
Full	1.20	0.764	1.21	0.880

Appendix

Table 12: Ablation on synthesized views. We report the geometry of the reposed garment against the scan, graded as in Table 1 . Since CloSe has few multi-layer outfits, while 4D-Dress captures outfits with an outer garment and labels it as its own class, we test no peeling only on 4D-Dress.

Comparison	Struct. Acc. $\uparrow$	IoU $\uparrow$	Self-Intersection (‰) $\downarrow$
	GarmentCodeData-XL	DeepFashion2
w/o Rollback	58.70	0.736	0.16
Full	73.59	0.839	0.04

Appendix

Table 13: Ablation on Rollback. We ablate the Rollback tool of the agentic trio. Self-intersection is the share of cloth edges that intersect the garment itself on the delivered drape.

Figure 17: Ablation on single-agent refinement. Per-garment silhouette IoU and measurement error of the delivered garment after each simulation on CloSe and 4D-Dress. Bands show the 95% bootstrap confidence interval. The single agent oscillates and plateaus early, while our agentic trio converges progressively.

Method	IoU $\uparrow$	Meas. Err. $\downarrow$	Edits / Iter.	Tokens (k) $\downarrow$	Time (s) $\downarrow$
Single agent	0.820	1.80%	6.2	38.7	748
Single agent + self-selection	0.833	1.69%	6.2	38.7	748
Agentic trio (ours)	0.846	1.65%	2.8	15.0	412

Appendix

Table 14: Ablation on single-agent refinement. All methods use the same coarse pattern, backbone, tools, memory, and 10 simulations per layer. Measurement error is in percent of body height. Tokens and time are the output tokens and VLM time per case. We find that the single agent is worse in quality, biased in self-evaluation, and higher in cost.

Figure 18: Ablation on texture generation. (a) Naive UV mapping without Grounded SAM mixes garments: the stripes of the tee bleed onto the pants. (b) With Grounded SAM, the garments separate, but unobserved cloth admits no extrapolation and the observed print lands warped. (c) Our agentic generation method continues the stripes at their measured repeat, restore the logo, and keep the pants a uniform black.

Figure 19: Screenshot of the user study question on geometry.

Figure 20: Screenshot of the user study question on plausibility.

Figure 21: Screenshot of the user study question on texture.

Figure 22: Failure case. The sash hanging free at the waist cannot be recovered by the current representation. Future work may let the agent write code to extend the representation on its own.

Figure 23: Additional results on in-the-wild photos with different layering, poses, and body shapes.

Explore similar work

InverseDraping: Recovering Sewing Patterns from 3D Garment Surfaces via BoxMesh Bridging
Apr 3, 2026Leyang Jin, Zirong Jin, Zisheng Ye +43D Reconstruction Deformable Shape Reconstruction
Mohamed bin Zayed University of Artificial Intelligence · The Chinese University of Hong Kong, Shenzhen · School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, FNii-Shenzhen, and Guangdong Provincial Key Laboratory of Future Networks of Intelligence +1
DressWild: Feed-Forward Pose-Agnostic Garment Sewing Pattern Generation from In-the-Wild Images
Feb 18, 2026Zeng Tao, Ying Jiang, Yunuo Chen +73D Reconstruction Feed-Forward 3D Reconstruction
UCLA · Fudan University · Style3D +2
Stitched Embeddings: A Unified Latent Space for 3D Garments and 2D Patterns
Jul 1, 2026Andrea Sanchietti, Riccardo Marin, Bharat Lal Bhatnagar +23D Reconstruction Multimodal Embedding
University of Tübingen, Germany · Tübingen AI Center · Technical University of Munich, Germany +3

Apr 3, 2026J/K move · Enter open · S save

InverseDraping: Recovering Sewing Patterns from 3D Garment Surfaces via BoxMesh Bridging

Leyang Jin, Zirong Jin, Zisheng Ye, Haokai Pang+3

Mohamed bin Zayed University of Artificial Intelligence · The Chinese University of Hong Kong, Shenzhen · School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, FNii-Shenzhen, and Guangdong Provincial Key Laboratory of Future Networks of Intelligence+1

3D Reconstruction Deformable Shape Reconstruction

Open paper AskPDF arXiv

Recovering sewing patterns from draped 3D garments is a challenging problem in human digitization research. In contrast to the well-studied forward process of draping designed sewing patterns using mature physical simulation engines, the inverse process of recovering parametric 2D patterns from deformed garment geometry remains fundamentally ill-posed for existing methods. We propose a two-stage framework that centers on a structured intermediate representation, BoxMesh, which serves as the key to bridging the gap between 3D garment geometry and parametric sewing patterns. BoxMesh encodes both garment-level geometry and panel-level structure in 3D, while explicitly disentangling intrinsic panel geometry and stitching topology from draping-induced deformations. This representation imposes a physically grounded structure on the problem, significantly reducing ambiguity. In Stage I, a geometry-driven autoregressive model infers BoxMesh from the input 3D garment. In Stage II, a semantics-aware autoregressive model parses BoxMesh into parametric sewing patterns. We adopt autoregressive modeling to naturally handle the variable-length and structured nature of panel configurations and stitching relationships. This decomposition separates geometric inversion from structured pattern inference, leading to more accurate and robust recovery. Extensive experiments demonstrate that our method achieves state-of-the-art performance on the GarmentCodeData benchmark and generalizes effectively to real-world scans and single-view images.

Abstract

Explore similar work

InverseDraping: Recovering Sewing Patterns from 3D Garment Surfaces via BoxMesh Bridging

DressWild: Feed-Forward Pose-Agnostic Garment Sewing Pattern Generation from In-the-Wild Images

Stitched Embeddings: A Unified Latent Space for 3D Garments and 2D Patterns