What Matters in Virtual Try-Off? Dual-UNet Diffusion Model For Garment Reconstruction

Loc-Phat Truong; Meysam Madadi; Sergio Escalera

cs.CVApr 9, 2026

What Matters in Virtual Try-Off? Dual-UNet Diffusion Model For Garment Reconstruction

Authors: Loc-Phat Truong, Meysam Madadi, Sergio Escalera

Organizations: Computer Vision Center, Spain · Universitat de Barcelona, Spain

Abstract

Virtual Try-On (VTON) has seen rapid advancements, providing a strong foundation for generative fashion tasks. However, the inverse problem, Virtual Try-Off (VTOFF)-aimed at reconstructing the canonical garment from a draped-on image-remains a less understood domain, distinct from the heavily researched field of VTON. In this work, we seek to establish a robust architectural foundation for VTOFF by studying and adapting various diffusion-based strategies from VTON and general Latent Diffusion Models (LDMs). We focus our investigation on the Dual-UNet Diffusion Model architecture and analyze three axes of design: (i) Generation Backbone: comparing Stable Diffusion variants; (ii) Conditioning: ablating different mask designs, masked/unmasked inputs for image conditioning, and the utility of high-level semantic features; and (iii) Losses and Training Strategies: evaluating the impact of the auxiliary attention-based loss, perceptual objectives and multi-stage curriculum schedules. Extensive experiments reveal trade-offs across various configuration options. Evaluated on VITON-HD and DressCode datasets, our framework achieves state-of-the-art performance with a drop of 9.5% on the primary metric DISTS and competitive performance on LPIPS, FID, KID, and SSIM, providing both stronger baselines and insights to guide future Virtual Try-Off research.

Figures & tables

Figure 1: State-of-the-art single-UNet, TryOffDiff [ 30 ] and Try-Off-Anyone [ 34 ] , vs. our adapted Dual-UNet try-off results. Our Dual-UNet architecture achieves high-quality and realistic garment generation while precisely preserving fine-grained details across diverse datasets.

Figure 2: Overview of the Dual-UNet Diffusion Model framework for VTOFF . The framework has two branches: (1) Generation branch: CreationNet, a denoising UNet, generates the garment image; (2) Conditioning branch: includes IP-Adapter [ 37 ] and a text encoder for high-level features, and ConditionNet for low-level features. VTON styled input, the channel concatenation of VAE-encoded latents of the person, mask, and cloth-agnostic image, goes into the ConditionNet, while isolated-garment image is fed into the IP-Adapter. High-level (garment description [ $D$ ], IP-Adapter tokens) and low-level (ConditionNet) features fuse via self-attention and cross-attention to condition CreationNet.

Architecture Choice	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	FID $\downarrow$	KID 1 1 1 KID is multiplied by $10^{3}$ to bring it to a similar scale as other metrics and is the mean value over 10 evaluation runs. $\downarrow$
inpainting	73.77	31.83	22.18	11.32	1.99
base	73.94	30.14	20.90	10.17	1.89
SDv1.5	69.16	39.89	25.06	12.55	2.66
SDXL	73.94	30.14	20.90	10.17	1.89
Truncated	69.70	38.11	30.60	18.40	4.09
IP-Adapter + Text	73.94	30.14	20.90	10.17	1.89

Table 1: Comparison of the Dual-UNet architectural design ablations as presented in Sec. 3.1 . Note bold indicates the best value

Masking Operation	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	FID $\downarrow$	KID 1 1 1 KID is multiplied by $10^{3}$ to bring it to a similar scale as other metrics and is the mean value over 10 evaluation runs. $\downarrow$
Fit mask	74.20	30.96	21.30	10.58	2.11
Dilated mask	73.94	30.14	20.90	10.17	1.89
Unmasked IP	73.57	30.00	20.96	10.31	2.12
Masked IP	73.94	30.14	20.90	10.17	1.89

Table 2: Comparisons related to masking ablations, including the type of mask and IP-Adapter input masking strategies.

Curriculum	Color Aug?	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	FID $\downarrow$	KID 1 1 1 KID is multiplied by $10^{3}$ to bring it to a similar scale as other metrics and is the mean value over 10 evaluation runs. $\downarrow$
Curr 1	all	70.70	34.02	22.40	10.95	2.26
Curr 2	all	75.47	27.34	20.26	10.01	2.05
Curr 3	all	74.50	28.45	20.99	10.20	2.09
Curr 4	all	73.65	29.99	21.04	10.13	1.88
Curr 5	none	75.72	27.46	20.23	9.78	1.76
Curr 6	stage 1	74.70	28.75	20.10	9.73	1.82

Table 3: Comparison of training curricula. Curr 6, with Color Jitter in Stage 1 only , achieves the best performance on DISTS and FID.

Figure 3: Visualization of Leffa loss impact on guiding attentions toward correct regions and implicitly prevent fine-grained details distortion. (a) with Leffa, (b) without Leffa

Auxiliary Loss	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	FID $\downarrow$	KID 1 1 1 KID is multiplied by $10^{3}$ to bring it to a similar scale as other metrics and is the mean value over 10 evaluation runs. $\downarrow$
w/o Leffa	74.51	30.37	20.96	10.62	2.22
w/ Leffa	75.65	28.15	20.48	10.65	2.27
None	75.47	27.34	20.26	10.01	2.05
ELatent-LPIPS	74.54	28.35	20.45	9.96	2.00
LPL	74.93	28.79	20.35	10.15	2.04

Table 4: Impact of auxiliary losses on VTOFF performance.

Method	Resolution	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	FID $\downarrow$	KID 1 1 1 KID is multiplied by $10^{3}$ to bring it to a similar scale as other metrics and is the mean value over 10 evaluation runs. $\downarrow$
TryOffDiff [ 30 ] (HD)	$512\times 384$	75.02	28.52	22.32	24.56	9.52
Try-Off-Anyone [ 34 ] (HD)	$512\times 384$	72.35	34.08	22.11	11.57	2.01
Ours (HD)	$512\times 384$	74.70	28.75	20.10	9.73	1.82
IGR [ 27 ] (HD)	$1024\times 768$	78.95	29.46	20.45	13.14	2.97
Ours (HD)	$1024\times 768$	76.04	31.41	19.59	10.69	2.31
TryOffDiff [ 30 ] (DC) $\star$	$512\times 384$	80.8	31.6	21.6	17.1	4.7

Table 5: Quantitative results comparison with other methods on the VITON-HD (HD) dataset and DressCode (DC) for virtual try-off.

\star

DISTS is computed at

341\times 256

resolution following TryOffDiff [ 30 ] evaluation protocol.

Figure 4: Qualitative comparison on VITON-HD dataset between our approach and previous works. Zoom in for better inspection. More examples are given in Supp. Sec. 6 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Failure case in VTOFF due to wrong and flat attention similar to VTON [ 42 ] . Left: Reference image used in ConditionNet; Center: VTOFF result, which fails to present the olive green stripe; Right: The corresponding attention map of the red region in the result with respect to the reference image.

Resolution	Stage	Trainable params	Batch Size	GPU
$512\times 384$	cre_ip	3.074B	4	L40S
	cre_ip + Leffa	3.075B	2	L40S
	cond + Leffa	2.651B	4	L40S
	joint	5.642B	12	H100
	joint + Leffa	5.642B	12	H100
$1024\times 768$	cre_ip	3.074B	12	H100

Appendix

Table 6: Trainable parameters, batch size and corresponding GPU usage for each stage in the training curriculum.

Resolution	Memory	Inference Time (sec/im)
$512\times 384$	$\sim$ 16 GB	2
$1024\times 768$	$\sim$ 23 GB	5

Appendix

Table 7: Inference Resource Consumption using float16 mixed precision on a consumer NVIDIA 4090 24GB GPU with batch size as 1.

Figure 6: Disentangling transformations applied to the ground truth images of the VITON-HD test set, showing in each pair of images: the left image is ground truth and the right one is the modified image. From top to bottom: color modifications (background, cloth), cloth shape distortions (rotation & scaling, TPS warping), and mild/drastic texture alterations (random copy-paste, blurring). The reflection on VTOFF metrics of these transformations are provided in Tab. 8 and Sec. 4.1 .

Method	SSIM $\uparrow$	LPIPS $\downarrow$	DISTS $\downarrow$	FID $\downarrow$	CLIP-FID $\downarrow$	KID 1 1 1 KID is multiplied by $10^{3}$ to bring it to a similar scale as other metrics and is the mean value over 10 evaluation runs. $\downarrow$
Background brightness	99.49	1.74	3.29	1.10	1.25	-0.12
Cloth color	99.04	2.87	4.70	1.83	0.44	-0.11
Cloth shape	87.09	13.79	8.45	3.40	0.43	0.18
Cloth texture (mild)	96.15	8.46	9.56	7.30	0.94	2.3
Cloth texture (drastic)	93.99	11.82	12.54	15.95	2.62	7.08

Appendix

Table 8: Effect of Background, Color, Shape, and Texture Changes on VTOFF Metrics. Texture changes strongly affect DISTS, FID, and KID, while shape alterations drastically decrease LPIPS and SSIM-based metrics. CLIP-FID is less suitable for VTOFF.

Figure 7: Qualitative comparison of architectural ablations for the proposed Dual-UNet Diffusion Framework on VTOFF. (1) the person wearing the garment, (2) ground-truth canonical garment, (3) the result of the base architecture described in Sec 4.1.3 (main paper), and (4–8) samples of different variants illustrating for Sec 4.2&4.3 in the main paper: (4)-Inpainting CreationNet in Sec 4.2.1, (5)-SD v1.5 in Sec 4.2.2, (6)-High-level features truncated model in Sec 4.3.1, (7)-Fit mask type in Sec 4.3.2, and (8)-Unmasked input for IP-Adapter in Sec 4.3.3. This illustration clear shows the most suitable architecture for Dual-UNet Framework is the base model as it produces natural high quality image and the identity of cloth is well-preserved.

Figure 8: Color fading effect caused by Color Jitter augmentation in later stages of training.

Figure 9: Quantitative comparison of random vs. fixed prompt training across different prompt complexities (ascending from level 0 to 3). Random prompt augmentation yields more consistent performance across prompt levels. Especially, it maintains competitive performance on the shortest prompt - level 0, indicating slightly improved robustness. However, its best performance remains lower than the case where the model is trained with a fixed prompt and evaluated with the corresponding prompt.

Figure 10: More qualitative comparison with previous works on at $512\times 384$ resolution. Qualitative comparison on Unsplash Real-World dataset between our approach and previous works, which are trained on VITON-HD dataset.

Figure 11: More qualitative comparison with previous works on VITON-HD at $512\times 384$ resolution.

Figure 12: More qualitative examples on VITON-HD at $1024\times 768$ resolution produced by our method.

Figure 13: More qualitative examples on DressCode at $512\times 384$ resolution produced by our method.

Figure 14: Illustration of Failure Mode. Comparison showing the challenge of maintaining accurate generation for small garment types (top row), and intricate and small text (bottom row).

Diffusion Models

Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.

See all 1,176 papers

Rethinking Garment Conditioning in Diffusion-based Virtual Try-On: Decouple, Don't Denoise
Nov 24, 2025Kihyun Na, Jinyoung Choi, Injung KimVirtual Try-On Diffusion Model Fine-Tuning
Advanced AI Talent Education and Research Group for Industrial Innovation, Handong Global University, Pohang 37554, Republic of Korea · Department of Artificial Intelligence, Computer and Electrical Engineering, Handong Global University, Pohang 37554, Republic of Korea
OmniDiT: Extending Diffusion Transformer to Omni-VTON Framework
Mar 20, 2026Weixuan Zeng, Pengcheng Wei, Huaiqing Wang +8Virtual Try-On Diffusion Transformer
The Chinese University of Hong Kong, ShenZhen · Beihang University · KuaiShou
FDM-MFVT: Few-step Sampling Diffusion Model for Mask-Free Virtual Try-On
Jun 28, 2026Jiaxin Liu, Xiaoye Liang, Lai Jiang +2Virtual Try-On Few-Step Diffusion Sampling
School of Electronic Information Engineering, Beihang University, Beijing, China · Zhongguancun Academy, Beijing, China · State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China

Nov 24, 2025J/K move · Enter open · S save

Rethinking Garment Conditioning in Diffusion-based Virtual Try-On: Decouple, Don't Denoise

Kihyun Na, Jinyoung Choi, Injung Kim

Advanced AI Talent Education and Research Group for Industrial Innovation, Handong Global University, Pohang 37554, Republic of Korea · Department of Artificial Intelligence, Computer and Electrical Engineering, Handong Global University, Pohang 37554, Republic of Korea

Virtual Try-On Diffusion Model Fine-Tuning Conditional Diffusion Models

Open paper AskPDF arXiv

Virtual Try-On (VTON) synthesizes realistic images of a person wearing a target garment, with broad applications in e-commerce and fashion. Diffusion-based dual-UNet methods achieve strong results but double the parameters by dedicating a separate network to garment conditioning. Spatial concatenation offers a simpler single-network alternative, yet both UNet- and DiT-based instantiations report that full fine-tuning is ineffective, and the community has settled for attention-only training. We ask: why does full fine-tuning fail, and can this be resolved? Through what is, to our knowledge, the first visualization study of dual-UNet reference network behavior, we identify a unifying insight: garment conditioning must be decoupled from the denoising process. Spatial concatenation violates this by embedding the garment within the denoising target, causing three conflicts: guidance leakage, gradient competition, and train-test discrepancy. We derive three design principles to restore this decoupling and implement them as a pure recipe atop a standard architecture with no modification. The resulting model, DeCo-VTON (860M params), achieves single-network state of the art, matching the dual-UNet state of the art at half the cost while being preferred in human evaluation.

Abstract

Diffusion Models

Explore similar work

Rethinking Garment Conditioning in Diffusion-based Virtual Try-On: Decouple, Don't Denoise

OmniDiT: Extending Diffusion Transformer to Omni-VTON Framework

FDM-MFVT: Few-step Sampling Diffusion Model for Mask-Free Virtual Try-On