ShoeFit: A New Dataset and Dual-image-stream DiT Framework for Virtual Footwear Try-On

Yuhan Li; Zhiyu Jin; Yifan Tong; Wenxiang Shang; Benlei Cui; Xuanhong Chen; Hangcheng Zhu; Bingbing Ni

cs.CVOct 13, 2026

ShoeFit: A New Dataset and Dual-image-stream DiT Framework for Virtual Footwear Try-On

Authors: Yuhan Li, Zhiyu Jin, Yifan Tong, Wenxiang Shang, Benlei Cui, Xuanhong Chen, Hangcheng Zhu, Bingbing Ni

Organizations: Shanghai Jiao Tong University · Alibaba Group

Abstract

Virtual footwear try-on (VFTON), a critical yet underexplored area in virtual try-on (VTON), aims to synthesize faithful try-on results given diverse footwear and model images while maintaining 3D consistency and texture authenticity. Unlike conventional garment-focused VTON methods, VFTON presents unique challenges due to (1) Data Scarcity, which arises from the difficulty of perfectly matching product shoes with models wearing the identical ones, (2) Viewpoint Misalignment, where the target foot pose and source shoe views are always misaligned, leading to incomplete texture information and detail distortion, and (3) Background-induced Color Distortion, where complex material of footwear interacts with environmental lighting, causing unintended color contamination. To address these challenges, we introduce MVShoes, a multi-view shoe try-on dataset consisting of 7305 well-annotated image triplets, covering diverse footwear categories and challenging try-on scenarios. Furthermore, we propose a dual-stream DiT architecture, ShoeFit, designed to mitigate viewpoint misalignment through Multi-View Conditioning with 3D Rotary Position Embedding, and alleviate background-induced distortion using the LayeredRefAttention which leverages background features to modulate footwear latents. The proposed framework effectively decouples shoe appearance from environmental interferences while preserving high-quality texture detail through decoupled denoising and conditioning branches. Extensive quantitative and qualitative experiments demonstrate that our method substantially improves rendering fidelity and robustness under challenging real-world product shoes, establishing a new benchmark in high-fidelity footwear try-on synthesis.

Figures & tables

Figure 1: VFTON presents three challenges, while ShoeFit gives a successful solution.

Figure 2: We illustrate examples of the MVShoes dataset in (a) and data processing pipeline in (b).

Figure 3: The category distribution of MVShoes.

Figure 4: We illustrate existing and our DiT framework comparison in (a)-(b), and heatmaps visualization for component validation of Multi-view Conditioning and LayeredRefAttention in (c)-(d).

Figure 5: Comparisons on the hard scenes, where ShoeFit delivers high-fidelity and rubost results.

View Num.	Method	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	KID $\downarrow$	DISTS $\downarrow$
1-View	FLUX.1-Fill [ 6 ]	0.725	0.201	24.44	2.221	0.1293
	FLUX.1-Fill-Redux [ 7 ]	0.724	0.199	24.67	2.086	0.1274
	OOTDiffusion-SDXL [ 65 ]	0.724	0.194	22.60	2.345	0.1114
	CatVTON-Flux-Lite [ 13 ]	0.745	0.185	22.71	3.327	0.1179
	ShoeFit-1V	0.772	0.159	21.07	0.944	0.1059
2-View	CatVTON-Flux-Lite-2V [ 13 ]	0.756	0.171	22.64	3.318	0.1040

Table 1: Quantitative comparison on MVshoes. ShoeFit significantly surpasses all baselines.

Figure 6: Ablation study. We highlight the improvements by all three components of method in red.

Method	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	KID $\downarrow$	DISTS $\downarrow$
Vanilla Model	0.745	0.185	22.71	3.327	0.1179
+ D-i-s	0.761	0.174	21.57	1.493	0.1038
+ D-i-s + Standard Attn (w/ mask)	0.756	0.181	21.70	1.698	0.1056
+ D-i-s + LRA (w/ mask)	0.772	0.159	21.07	0.944	0.1059
+ D-i-s + LRA + MC (w/o 3D RoPE)	0.778	0.152	20.18	0.695	0.0971
+ D-i-s + LRA + MC (Full)	0.780	0.149	20.18	0.485	0.0941

Table 2: Quantitative ablation study of each component in Section of Method.

Method	Inference Time per Image	GPU Memory Usage
Cat-VTON (image + text streams)	22.27 seconds	35.5 GB
ShoeFit (Dual-image-stream, D-i-s)	17.44 seconds	22.0 GB
ShoeFit + D-i-s + LayeredRefAttention	21.15 seconds	28.5 GB

Table 3: Comparison of inference time and GPU memory usage across different architectures.

Figure 7: Similar to the common constraints faced by existing generative models, ShoeFit occasionally struggles to accurately render small logos and intricate text due to their small portion in images and high-frequency variations.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Primary Categories	Subcategory
Casual Shoes : 3158 (43.23%)	Lifestyle Casual Shoes: 803 (10.99%)
	Sneakers: 1428 (19.55%)
	Canvas Shoes: 141 (1.93%)
	Children Casual Shoes: 99 (1.36%)
	Men Casual Shoes: 687 (9.40%)
Athletic Shoes : 558 (7.64%)	Running Shoes: 322 (4.41%)

Appendix

Table 4: Shoe Category Statistics. We report the sample count for each subcategory, with its proportion indicated in parentheses.

Scene Types	Number of samples	Percentage
Top-down Foot	1719	23.53%
Horizontal Foot	4330	59.27%
Half-body Model	833	11.40%
Full-body Model	423	5.79%

Appendix

Table 5: Human scene statistics.

Figure 8: We provide more visual samples from MVShoes datasets.

Figure 9: More visual results of the ablation study. We highlight the improvements by all three components of the method in red. Best viewed when zoomed in.

Figure 10: We provide how LayeredRefAttention is used within a single-stream DiT block.

Figure 11: More visual comparisons on the MVShoes by ShoeFit. Best viewed when zoomed in.

Figure 12: More visual results on the MVShoes by ShoeFit. Best viewed when zoomed in.

Figure 13: More visual results on two primary failure modes of our method: (a) structural failure observed during early training stages, and (b) artifacts arising from inaccurate masks in long footwear scenarios.

Figure 14: More visual results on out-of-distribution (OOD) samples not present in the MVShoes dataset. The high-quality try-on results demonstrate the model’s notable stability and strong generalization capabilities.

Virtual Try-On

Virtual try-on generates images or videos showing people wearing selected garments while preserving appearance, fit, pose, and identity.

See all 55 papers

FIT: A Large-Scale Dataset for Fit-Aware Virtual Try-On
Apr 9, 2026Johanna Karras, Yuanhao Wang, Yingwei Li +1Virtual Try-On Synthetic Data Generation
Paul G. Allen School for Computer Science and Engineering, University of Washington, Seattle, WA, USA · Google Research, Seattle, CA, USA · Google Research, Mountain View, CA, USA
What Matters in Virtual Try-Off? Dual-UNet Diffusion Model For Garment Reconstruction
Apr 9, 2026Loc-Phat Truong, Meysam Madadi, Sergio EscaleraDiffusion Models Diffusion-Based Image Generation
Computer Vision Center, Spain · Universitat de Barcelona, Spain
FitVTON: Fit-aware Virtual Try-On via Body-Garment Size Control
Jun 10, 2026Yiqun Ning, Ao Shen, Chenhang He +1Virtual Try-On
Department of Computing, The Hong Kong Polytechnic University, Hong Kong · Nuvatech

Apr 9, 2026J/K move · Enter open · S save

FIT: A Large-Scale Dataset for Fit-Aware Virtual Try-On

Johanna Karras, Yuanhao Wang, Yingwei Li, Ira Kemelmacher-Shlizerman

Paul G. Allen School for Computer Science and Engineering, University of Washington, Seattle, WA, USA · Google Research, Seattle, CA, USA · Google Research, Mountain View, CA, USA

Virtual Try-On Synthetic Data Generation

Open paper AskPDF arXiv

Given a person and a garment image, virtual try-on (VTO) aims to synthesize a realistic image of the person wearing the garment, while preserving their original pose and identity. Although recent VTO methods excel at visualizing garment appearance, they largely overlook a crucial aspect of the try-on experience: the accuracy of garment fit -- for example, depicting how an extra-large shirt looks on an extra-small person. A key obstacle is the absence of datasets that provide precise garment and body size information, particularly for "ill-fit" cases, where garments are significantly too large or too small. Consequently, current VTO methods default to generating well-fitted results regardless of the garment or person size. In this paper, we take the first steps towards solving this open problem. We introduce FIT (Fit-Inclusive Try-on), a large-scale VTO dataset comprising over 1.13M try-on image triplets accompanied by precise body and garment measurements. We overcome the challenges of data collection via a scalable synthetic strategy: (1) We programmatically generate 3D garments using GarmentCode and drape them via physics simulation to capture realistic garment fit. (2) We employ a novel re-texturing framework to transform synthetic renderings into photorealistic images while strictly preserving geometry. (3) We introduce person identity preservation into our re-texturing model to generate paired person images (same person, different garments) for supervised training. Finally, we leverage our FIT dataset to train a baseline fit-aware virtual try-on model. Our data and results set the new state-of-the-art for fit-aware virtual try-on, as well as offer a robust benchmark for future research. We will make all data and code publicly available on our project page: https://johannakarras.github.io/FIT.

Abstract

Virtual Try-On

Explore similar work

FIT: A Large-Scale Dataset for Fit-Aware Virtual Try-On

What Matters in Virtual Try-Off? Dual-UNet Diffusion Model For Garment Reconstruction

FitVTON: Fit-aware Virtual Try-On via Body-Garment Size Control