OmniFashion: Towards Generalist Fashion Intelligence via Multi-Task Vision-Language Learning

Zhengwei Yang; Andi Long; Hao Li; Zechao Hu; Kui Jiang; Zheng Wang

cs.CVMar 3, 2026

OmniFashion: Towards Generalist Fashion Intelligence via Multi-Task Vision-Language Learning

Authors: Zhengwei Yang, Andi Long, Hao Li, Zechao Hu, Kui Jiang, Zheng Wang

Organizations: National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science, Wuhan University · Harbin Institute of Technology

Abstract

Fashion intelligence spans multiple tasks, i.e., retrieval, recommendation, recognition, and dialogue, yet remains hindered by fragmented supervision and incomplete fashion annotations. These limitations jointly restrict the formation of consistent visual-semantic structures, preventing recent vision-language models (VLMs) from serving as a generalist fashion brain that unifies understanding and reasoning across tasks. Therefore, we construct FashionX, a million-scale dataset that exhaustively annotates visible fashion items within an outfit and organizes attributes from global to part-level. Built upon this foundation, we propose OmniFashion, a unified vision-language framework that bridges diverse fashion tasks under a unified fashion dialogue paradigm, enabling both multi-task reasoning and interactive dialogue. Experiments on multi-subtasks and retrieval benchmarks show that OmniFashion achieves strong task-level accuracy and cross-task generalization, highlighting its offering of a scalable path toward universal, dialogue-oriented fashion intelligence.

Figures & tables

Figure 1: Users interacting with fashion systems require various abilities. Traditional systems address these tasks separately, while general-purpose VLMs offer generic yet shallow responses. OmniFashion unifies multi-task learning with fashion-aware perception and interactive reasoning.

Figure 2: Comparison between existing fashion datasets and FashionX . Prior datasets show incomplete and inconsistent annotations (marked in red), while FashionX offers a unified head-to-toe coverage with hierarchical annotation structure spanning description, global- and part-level semantics.

Figure 3: Overview of the FashionX annotation pipeline.

Figure 4: Pipeline of OmniFashion. OmniFashion builds on FashionX datasets that first construct data with garment and corresponding description/attribute. The VLM output will be penalized by the constructed dialogue as an answer.

Figure 5: Illustration of the progressive learning task design in OmniFashion. Different color stands for different tasks.

	O	T	B	F	A	Outfits
Num.	261,389	952,229	729,562	593,448	797,970	1,032,710
Attri. Num.	2,656,329	7,855,390	5,099,191	3,964,328	4,533,838	6,135,520
Total Items	3,334,598
Total Attri.	24,109,076

Table 1: Statistical Detail of FashionX. Abbreviations: One-piece (O), top (T), bottom (B), footwear (F), accessory (A).

Methods	Size	Overall Style		Part Style		Occasion		Color		I2T	T2I	I2I	CIR	Mean Acc
Methods	Size	A@1	Acc	A@1	Acc	A@1	Acc	A@1	Acc	Acc	Acc	Acc	Acc	Mean Acc
Close-sourced VLMs
Grok-4-Fast	-	38.0	66.6	11.6	20.3	87.8	91.5	67.1	69.3	90.4	20.6	10.7	15.4	48.1
Gemini-2.5-Flash	-	75.3	89.8	10.7	10.9	92.0	94.9	78.8	81.2	99.7	99.3	60.9	19.3	69.5
Claude-4.5-Sonnet	-	74.6	91.2	32.4	40.5	89.7	99.2	67.9	70.6	98.5	95.8	58.6	23.2	72.2
Open-sourced VLMs

Table 2: Comparison with the SOTA methods across eight fashion dialogue subtasks, including overall style, part style, occasion, color recognition, and four retrieval dialogue settings (I2T, T2I, I2I, CIR). The table reports both A@1 and Acc metrics. OmniFashion shows consistently strong performance across all subtasks and achieves the highest mean accuracy among the open- and close-source models.

Methods	InShop
Methods	R@1	R@10	R@20	mAP
Hyper-DINO [ 71 ]	92.6	98.4	99.0	-
Hyper-ViT [ 71 ]	92.7	98.4	98.9	-
IBC [ 72 ]	92.8	98.5	99.1	-
DADA [ 73 ]	93.0	98.5	98.9	-
MGA [ 74 ]	94.3	98.8	99.1	81.1

Table 3: Comparison on the InShop retrieval benchmark.

Methods	Consumer2Shop
Methods	R@1	R@10	R@20	mAP
Ensemble [ 75 ]	23.1	48.0	-	-
RTS [ 76 ]	37.8	71.1	77.2	43.0
CTL [ 77 ]	37.6	71.1	77.6	43.1
MGA [ 74 ]	38.4	72.3	78.3	44.2
Hyper-DINO [ 71 ]	42.5	65.9	-	-

Table 4: Comparison on the Consumer2Shop retrieval benchmark.

Figure 6: Ablation on the task variants. Training without assistance tasks (blue, w/o assis) or multi-image recommendation tasks (red, w/o multiqa) are shown with accuracy on all subtasks. The gray trend lines upper the bars in each group indicate performance changes relative to the baseline.

Figure 7: Ablation on the two-stage training. The radar chart compares accuracy across tasks for models trained with s1 only, s2 only, and both stages (ours); the bar chart summarizes overall gains vs. the baseline. Red (solid) indicates the full model, yellow (dashed) and green (dotted) denote s1-only and s2-only variants, respectively, with blue showing the baseline.

Figure 8: Qualitative comparison across subtasks. Each block shows input queries and model responses. Colored text highlights correct ( green ), incorrect ( red ), and uncertain ( orange ) predictions, while underlined phrases mark key visual cues.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

FashionStylist: An Expert Knowledge-enhanced Multimodal Dataset for Fashion Understanding
Apr 10, 2026Kaidong Feng, Zhuoxuan Huang, Huizhong Guo +7Multimodal Understanding
Yanshan University Qinhuangdao, China · Central South University Changsha, China · Zhejiang University Hangzhou, China +3
FashionLens: Toward Versatile Fashion Image Retrieval via Task-Adaptive Learning
May 21, 2026Haokun Wen, Xuemeng Song, Xinghao Xie +3Multi-Task Learning Image-Text Retrieval
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen 518055, China · Department of Data Science, City University of Hong Kong, Hong Kong, China · Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China +4
Diverse-Intent Multi-Turn Fashion Image Retrieval
Jul 22, 2026Mingqiang Tang, Haokun Wen, Meng Liu +3Multimodal IR Cross-Modal Alignment
1Southern University of Science and Technology, Shenzhen, China · 2Harbin Institute of Technology (Shenzhen), Shenzhen, China · 3Shandong University, Jinan, China +1

Apr 10, 2026J/K move · Enter open · S save

FashionStylist: An Expert Knowledge-enhanced Multimodal Dataset for Fashion Understanding

Kaidong Feng, Zhuoxuan Huang, Huizhong Guo, Yuting Jin+6

Yanshan University Qinhuangdao, China · Central South University Changsha, China · Zhejiang University Hangzhou, China+3

Multimodal Understanding

Open paper AskPDF arXiv

Fashion understanding requires both visual perception and expert-level reasoning about style, occasion, compatibility, and outfit rationale. However, existing fashion datasets remain fragmented and task-specific, often focusing on item attributes, outfit co-occurrence, or weak textual supervision, and thus provide limited support for holistic outfit understanding. In this paper, we introduce FashionStylist, an expert-annotated benchmark for holistic and expert-level fashion understanding. Constructed through a dedicated fashion-expert annotation pipeline, FashionStylist provides professionally grounded annotations at both the item and outfit levels. It supports three representative tasks: outfit-to-item grounding, outfit completion, and outfit evaluation. These tasks cover realistic item recovery from complex outfits with layering and accessories, compatibility-aware composition beyond co-occurrence matching, and expert-level assessment of style, season, occasion, and overall coherence. Experimental results show that FashionStylist serves not only as a unified benchmark for multiple fashion tasks, but also as an effective training resource for improving grounding, completion, and outfit-level semantic evaluation in MLLM-based fashion systems.

Abstract

Vision-Language Models

Explore similar work

FashionStylist: An Expert Knowledge-enhanced Multimodal Dataset for Fashion Understanding

FashionLens: Toward Versatile Fashion Image Retrieval via Task-Adaptive Learning

Diverse-Intent Multi-Turn Fashion Image Retrieval