FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval

Tripti Shukla; Shwetha Somasundaram; Akshat Gupta; Rohit Garg; Paul Duff; Ian Anderson

cs.IROct 13, 2026

FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval

Authors: Tripti Shukla, Shwetha Somasundaram, Akshat Gupta, Rohit Garg, Paul Duff, Ian Anderson

Organizations: Glance Bengaluru, India · Glance Vilnius, Lithuania · Glance London, United Kingdom

Abstract

Evaluating fashion retrieval systems remains challenging due to the limited availability of benchmarks that capture the fine-grained, open-ended queries encountered in real-world search and recommendation. Existing benchmarks often rely on manual curation, predefined taxonomies, or synthetic prompts, limiting scalability and semantic coverage. We introduce FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval, an automated framework that derives user-type queries and relevance sets directly from catalog structure. Our key insight is that hierarchical discrete representations learned by Residual Quantized Variational Autoencoders (RQ-VAE) organize products into semantically coherent clusters at progressively finer granularities. We leverage this structure to construct FASHION-RQBENCH, a scalable benchmark without manual query annotation or predefined taxonomies. We propose four complementary tasks for evaluating ranking, relevance classification, fine-grained sibling discrimination, and paraphrase robustness, alongside query- and attribute-level relevance metrics. Experiments across recent retrieval models demonstrate that FASHION-RQBENCH differentiates model capabilities across query types and attributes. A human study further shows that benchmark queries achieve comparable naturalness and higher intent clarity than real user queries, supporting their validity for real-world evaluation. The code and FASHION-RQBENCH dataset will be publicly released upon acceptance.

Figures & tables

Figure 1: Overview of FASHION-RQ. The framework leverages an RQ-VAE hierarchy to automatically generate natural-language queries and graded relevance sets, with Exact, Sibling, Related, and Irrelevant relationships. The resulting benchmark supports four complementary retrieval tasks, while weighted query relevance and attribute-level metrics evaluate overall intent satisfaction and fine-grained attribute matching.

Figure 2: Attribute Exclusivity Loss. Per-level supervised contrastive supervision encourages progressively finer attribute semantics to form compact, distinct regions in the RQ-VAE codebooks, reducing the scattering of identical attribute combinations across codes.

Statistic	Value	Details
Master categories	5	Apparel, Accessories, Footwear, Personal Care, Free Items
Subcategories	45	–
Article types	143	–
Base colors	47	–
Genders	5	Men, Women, Unisex, Boys, Girls
Benchmark clusters	588	17 / 135 / 436 at levels 1 / 2 / 3

Table 1: Overview of the Fashion-RQBENCH evaluation set.

	T1: Ranking		T2: Relevance Classification		T3: Sibling Discrimination		T4: Paraphrase Robustness
Model	nDCG@10	Recall@10	Accuracy	Macro-F1	nDCG@3	Recall@3	Consistency@5	Consistency@10
BM25 [ 36 ]	0.7736	0.7376	0.4023	0.3919	0.6142	0.7723	0.5067	0.7419
RoBERTa-large [ 21 ]	0.7391	0.7252	0.3749	0.3481	0.5491	0.7260	0.4462	0.7135
SimCSE-RoBERTa-large [ 52 ]	0.8655	0.8242	0.5046	0.5047	0.6421	0.8145	0.5914	0.8197
Sentence-T5-large [ 29 ]	0.8235	0.7845	0.4328	0.4227	0.6610	0.8318	0.5879	0.8115
Qwen3-Embedding-4B [ 42 ]	0.8719	0.8279	0.5145	0.5146	0.6641	0.8232	0.6041	0.8373

Table 2: Performance on the four FASHION-RQ tasks. Best results are shown in bold and second-best results are underlined.

Model	Rel.	Age	Brand	Color	Gender	Mat.	Occ.	Pattern	Price	Prod.	Style
Qwen3-Emb.-8B [ 42 ]	0.9274	0.9907	0.8560	0.9125	0.9750	0.5367	0.9632	0.8521	0.6484	0.9409	0.4291
E5-Mistral-7B [ 48 ]	0.9158	0.9900	0.8914	0.8782	0.9676	0.5232	0.9618	0.8465	0.6543	0.9395	0.4265
Qwen3-Emb.-4B [ 42 ]	0.9254	0.9895	0.8221	0.8965	0.9721	0.5217	0.9624	0.8327	0.6426	0.9438	0.4204
Sentence-T5-L [ 29 ]	0.9018	0.9918	0.7587	0.9159	0.9911	0.5310	0.9628	0.8395	0.6562	0.9392	0.4229
Sup-SimCSE-RoBERTa-L [ 52 ]	0.8923	0.9922	0.6613	0.9008	0.9873	0.5003	0.9616	0.8296	0.6247	0.9368	0.4292
BM25 [ 36 ]	0.8753	0.9826	0.7927	0.8451	0.8889	0.4987	0.9585	0.7934	0.6169	0.8929	0.4211

Table 3: Weighted query-relevance and attribute-level matching scores across retrieval models. Attribute weights

w_{a}

are derived from the cross-encoder and used to compute the overall weighted relevance score, Rel.

Source	Naturalness	Intent Clarity
CHAT	3.52	2.83
LLM	2.84	3.76
RQBENCH (Ours)	3.62	3.91

Table 4: Mean ratings from the user study on a 5-point Likert scale. Higher scores indicate greater perceived naturalness and intent clarity.

Attribute	64	128	256
Master Category	99.8%	50.4%	1.0%
Subcategory	99.4%	50.2%	1.0%
Article Type	98.3%	49.7%	1.0%
Gender	98.1%	49.6%	1.0%
Base Color	87.9%	44.5%	1.0%
Usage	97.7%	49.4%	1.0%

Table 5: Effect of codebook size on weighted attribute purity for 64, 128 and 256 size.

	64		128		256
Level	Clusters	Items/Cl.	Clusters	Items/Cl.	Clusters	Items/Cl.
D1	44	823.9	156	285.0	267	166.5
D2	3,378	219.2	10,378	48.28	17,378	3.56
D3	23,956	79.5	30,045	13.48	36,122	1.23

Table 6: Cluster fragmentation as a function of codebook size.

Task	# Queries	# Pairs/Cands	Cands per Query	Leaf Clusters	Query Length
T1: Ranking	2,477	36,782	12–15 (mean 14.8)	416	3–26 words (mean 10.1)
T2: Relevance Classification	2,477	19,731	7–8 (mean 8.0)	416	3–26 words (mean 10.1)
T3: Sibling Discrimination	1,968	9,801	4–5 (mean 5.0)	331	3–26 words (mean 10.1)
T4: Query Robustness (Paraphrase)	416 groups	2,477 queries	12–15 (mean 14.9)	416	2–10 paraphrases/group (mean 6.0)

Table 7: Per-task composition of Fashion-RQBENCH. All queries are anchored at level-3 (leaf) clusters; Task 1 and Task 4 share the same 2,477 anchor queries (Task 4 paraphrases each one 2–10

\times

Task	Exact	Sibling	Related	Irrelevant
T1: Ranking (36,782 pairs)	20.2%	25.9%	26.9%	26.9%
T2: Classification (19,731 pairs)	25.1%	24.7%	25.1%	25.1%

Table 8: Relevance-label distribution over (query, candidate) pairs.

Attribute	$N$	$\overline{\Delta}$	95% CI	Sign Consistency
Product Type	561	0.579	[0.440, 0.713]	0.78
Gender	730	0.730	[0.625, 0.838]	0.80
Age Group	129	0.737	[0.537, 0.951]	0.83
Color	786	0.926	[0.828, 1.025]	0.86
Material	259	0.589	[0.384, 0.796]	0.75
Occasion	952	0.072	[ $-$ 0.010, 0.153]	0.53

Table 9: Controlled matched-twin-pair ablation using the cross-encoder relevance oracle.

N

denotes the number of query–twin-pair observations.

\overline{\Delta}

is the mean cross-encoder logit difference, and the confidence interval is obtained by resampling twin pairs with replacement. Sign consistency denotes the fraction of observations having the same sign as

\overline{\Delta}

Attribute	Ridge Coef.	VIF	Final $w_{a}$
Product Type	0.467	1.10	0.290
Gender	0.125	1.06	0.127
Age Group	0.006	1.00	0.066
Color	0.114	1.01	0.138
Material	0.026	1.02	0.063
Occasion	0.004	1.02	0.008

Table 10: Regression-based attribute importance and final blended weights. Ridge coefficients are estimated from cross-encoder logits. VIF measures multicollinearity among attribute features.

Model	Scoring	Key Hyperparameters
BM25 [ 36 ]	Sparse lexical	$k_{1}=1.2$ , $b=0.75$ ; English stopwords; Porter stemmer
SPLADE++ EnsembleDistil [ 10 ]	Learned sparse	max_seq_len =512; batch size=64; $\max(\log(1+\mathrm{ReLU}(\cdot)))$ over MLM logits; vocabulary size=30,522
ColBERTv2 [ 37 ]	Multi-vector MaxSim	128 dimensions/token; query_maxlen =32; doc_maxlen =180; FP16 with FP32 MaxSim accumulation; batch size=128

Table 11: Lexical, learned-sparse, and multi-vector retrieval baselines.

Model	Dim.	Pooling	Max Seq. Len
Contriever-MSMARCO [ 14 ]	768	Mean	512
RoBERTa-large [ 21 ]	1024	Mean(first + last layer)	512
SimCSE-sup (RoBERTa-large) [ 52 ]	1024	pooler_output (CLS)	512
Sentence-T5-large [ 29 ]	768 †	Mean + Dense( $1024\!\rightarrow\!768$ , tanh)	256

Table 12: Dense single-vector retrieval baselines with non-LLM backbones.

Model	Params.	Dim.	Max Seq. Len	EOS Policy
Qwen3-Embedding-0.6B [ 42 ]	0.6B	1024	8192	auto ‡
Qwen3-Embedding-4B [ 42 ]	4B	2560	8192	auto ‡
Qwen3-Embedding-8B [ 42 ]	8B	4096	8192	auto ‡
E5-Mistral-7B-Instruct [ 48 ]	7B	4096	4096	manual §
SFR-Embedding-Mistral [ 27 ]	7B	4096	4096	auto

Table 13: LLM-backbone embedding baselines. All models use last-token pooling, bfloat16 , and

\ell_{2}

normalization. Queries are prefixed with “Given a fashion product search query, retrieve relevant product listings that match the query” , while documents are unprefixed.

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 396 papers

May 21, 2026

FashionLens: Toward Versatile Fashion Image Retrieval via Task-Adaptive Learning

Fashion image retrieval is a cornerstone of modern e-commerce systems. A unified framework that supports diverse query formats and search intentions is highly desired in practice. However, existing approaches focus on narrow retrieval tasks and do not fully capture such diversity. Therefore, in this work, we aim to develop a unified framework capable of handling diverse realistic fashion retrieval scenarios, achieving truly versatile fashion image retrieval. To establish a data foundation, we first introduce U-FIRE, a comprehensive benchmark that consolidates fragmented fashion datasets into a unified collection, supplemented by two manually curated datasets for testing generalization. Building upon this, we propose FashionLens, a unified framework based on Multimodal Large Language Models. To handle divergent matching objectives, we design a Proposal-Guided Spherical Query Calibrator that dynamically shifts query representations into task-aligned metric spaces via adaptive spherical linear interpolation. Additionally, to mitigate the optimization imbalance caused by varying task complexities and data scales, we develop a Gradient-Guided Adaptive Sampling strategy that automatically re-weights tasks based on realtime learning difficulty and the data scale prior. Experiments on U-FIRE show that FashionLens achieves state-of-the-art performance across diverse retrieval scenarios and generalizes robustly to unseen tasks. The data and code are publicly released at https://github.com/haokunwen/FashionLens.

Haokun Wen, Xuemeng Song, Xinghao Xie +3

School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen 518055, China · Department of Data Science, City University of Hong Kong, Hong Kong, China · Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China +4

Multi-Task Learning Image-Text Retrieval Cross-Modal Retrieval

Ask PDF

Jul 22, 2026

Diverse-Intent Multi-Turn Fashion Image Retrieval

Real-world fashion search involves interactive retrieval across multiple turns. However, existing multi-turn retrieval methods are built on a restrictive assumption that every interaction follows the same attribute-editing paradigm, leaving heterogeneous intent transitions unexplored. Moreover, existing approaches often rely on textification to bridge multimodal queries and visual retrieval, which may lose fine-grained visual cues. To address these gaps, we introduce DIM-Fashion, a benchmark of 26K multi-turn sessions constructed from 13 fashion retrieval datasets across 7 tasks, featuring diverse intent transitions and rollback behaviors. We further propose FashionAM, an MLLM-VLP framework that directly aligns multimodal conversational queries with a fashion-oriented gallery embedding space, avoiding intermediate textification. Extensive experiments demonstrate the effectiveness of FashionAM over existing approaches. The dataset and code will be made publicly available upon acceptance.

Mingqiang Tang, Haokun Wen, Meng Liu +3

1Southern University of Science and Technology, Shenzhen, China · 2Harbin Institute of Technology (Shenzhen), Shenzhen, China · 3Shandong University, Jinan, China +1

Multimodal IR Cross-Modal Alignment Cross-Modal Retrieval

Ask PDF

Jun 11, 2026

VietFashion: Benchmarking Sketch-Text Composed Image Retrieval for Cultural Outfits

Cultural garments pose a unique challenge for visual retrieval systems, as their identity often depends on subtle structural and symbolic details that are poorly captured by standard AI models. We introduce VietFashion, a new benchmark for sketch-text composed image retrieval centered on the Ao Dai, a traditional Vietnamese garment. VietFashion enables designers and researchers to retrieve culturally meaningful outfits using a combination of hand-drawn sketches, which convey garment structure, and textual descriptions, which encode cultural semantics. The dataset is initialized with 650 sketches and expanded using generative models to produce over 21,000 photorealistic images with aligned captions. Textual prompts that describe detailed outfit attributes, which are extracted from fashion magazines to ensure authenticity and diversity. To better reflect the inherent ambiguity of design intent, VietFashion adopts a multi-target retrieval setting, where a single query may correspond to multiple valid results. We establish standardized evaluation protocols and benchmark state-of-the-art composed image retrieval methods. Experimental results reveal significant performance gaps in modeling fine-grained cultural semantics and multi-modal composition, positioning VietFashion as a challenging benchmark for fine-grained fashion retrieval. The dataset is publicly available at: https://hng0303.github.io/VietFashion.

Hoang-Nguyen Cao, Le-Hoang Bui, Dinh-Khoi Vo +2

University of Science · Ho Chi Minh City, Vietnam · Vietnam National University

Fine-Grained Image Retrieval Image-Text Retrieval Image Retrieval Composed Image Retrieval

Ask PDF

Abstract

Automated Evaluation

Explore similar work

FashionLens: Toward Versatile Fashion Image Retrieval via Task-Adaptive Learning

Diverse-Intent Multi-Turn Fashion Image Retrieval

VietFashion: Benchmarking Sketch-Text Composed Image Retrieval for Cultural Outfits