FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval

Tripti Shukla; Shwetha Somasundaram; Akshat Gupta; Rohit Garg; Paul Duff; Ian Anderson

cs.IROct 13, 2026

FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval

Authors: Tripti Shukla, Shwetha Somasundaram, Akshat Gupta, Rohit Garg, Paul Duff, Ian Anderson

Organizations: Glance Bengaluru, India · Glance Vilnius, Lithuania · Glance London, United Kingdom

Abstract

Evaluating fashion retrieval systems remains challenging due to the limited availability of benchmarks that capture the fine-grained, open-ended queries encountered in real-world search and recommendation. Existing benchmarks often rely on manual curation, predefined taxonomies, or synthetic prompts, limiting scalability and semantic coverage. We introduce FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval, an automated framework that derives user-type queries and relevance sets directly from catalog structure. Our key insight is that hierarchical discrete representations learned by Residual Quantized Variational Autoencoders (RQ-VAE) organize products into semantically coherent clusters at progressively finer granularities. We leverage this structure to construct FASHION-RQBENCH, a scalable benchmark without manual query annotation or predefined taxonomies. We propose four complementary tasks for evaluating ranking, relevance classification, fine-grained sibling discrimination, and paraphrase robustness, alongside query- and attribute-level relevance metrics. Experiments across recent retrieval models demonstrate that FASHION-RQBENCH differentiates model capabilities across query types and attributes. A human study further shows that benchmark queries achieve comparable naturalness and higher intent clarity than real user queries, supporting their validity for real-world evaluation. The code and FASHION-RQBENCH dataset will be publicly released upon acceptance.

Figures & tables

Figure 1: Overview of FASHION-RQ. The framework leverages an RQ-VAE hierarchy to automatically generate natural-language queries and graded relevance sets, with Exact, Sibling, Related, and Irrelevant relationships. The resulting benchmark supports four complementary retrieval tasks, while weighted query relevance and attribute-level metrics evaluate overall intent satisfaction and fine-grained attribute matching.

Figure 2: Attribute Exclusivity Loss. Per-level supervised contrastive supervision encourages progressively finer attribute semantics to form compact, distinct regions in the RQ-VAE codebooks, reducing the scattering of identical attribute combinations across codes.

Statistic	Value	Details
Master categories	5	Apparel, Accessories, Footwear, Personal Care, Free Items
Subcategories	45	–
Article types	143	–
Base colors	47	–
Genders	5	Men, Women, Unisex, Boys, Girls
Benchmark clusters	588	17 / 135 / 436 at levels 1 / 2 / 3

Table 1: Overview of the Fashion-RQBENCH evaluation set.

	T1: Ranking		T2: Relevance Classification		T3: Sibling Discrimination		T4: Paraphrase Robustness
Model	nDCG@10	Recall@10	Accuracy	Macro-F1	nDCG@3	Recall@3	Consistency@5	Consistency@10
BM25 [ 36 ]	0.7736	0.7376	0.4023	0.3919	0.6142	0.7723	0.5067	0.7419
RoBERTa-large [ 21 ]	0.7391	0.7252	0.3749	0.3481	0.5491	0.7260	0.4462	0.7135
SimCSE-RoBERTa-large [ 52 ]	0.8655	0.8242	0.5046	0.5047	0.6421	0.8145	0.5914	0.8197
Sentence-T5-large [ 29 ]	0.8235	0.7845	0.4328	0.4227	0.6610	0.8318	0.5879	0.8115
Qwen3-Embedding-4B [ 42 ]	0.8719	0.8279	0.5145	0.5146	0.6641	0.8232	0.6041	0.8373

Table 2: Performance on the four FASHION-RQ tasks. Best results are shown in bold and second-best results are underlined.

Model	Rel.	Age	Brand	Color	Gender	Mat.	Occ.	Pattern	Price	Prod.	Style
Qwen3-Emb.-8B [ 42 ]	0.9274	0.9907	0.8560	0.9125	0.9750	0.5367	0.9632	0.8521	0.6484	0.9409	0.4291
E5-Mistral-7B [ 48 ]	0.9158	0.9900	0.8914	0.8782	0.9676	0.5232	0.9618	0.8465	0.6543	0.9395	0.4265
Qwen3-Emb.-4B [ 42 ]	0.9254	0.9895	0.8221	0.8965	0.9721	0.5217	0.9624	0.8327	0.6426	0.9438	0.4204
Sentence-T5-L [ 29 ]	0.9018	0.9918	0.7587	0.9159	0.9911	0.5310	0.9628	0.8395	0.6562	0.9392	0.4229
Sup-SimCSE-RoBERTa-L [ 52 ]	0.8923	0.9922	0.6613	0.9008	0.9873	0.5003	0.9616	0.8296	0.6247	0.9368	0.4292
BM25 [ 36 ]	0.8753	0.9826	0.7927	0.8451	0.8889	0.4987	0.9585	0.7934	0.6169	0.8929	0.4211

Table 3: Weighted query-relevance and attribute-level matching scores across retrieval models. Attribute weights

w_{a}

are derived from the cross-encoder and used to compute the overall weighted relevance score, Rel.

Source	Naturalness	Intent Clarity
CHAT	3.52	2.83
LLM	2.84	3.76
RQBENCH (Ours)	3.62	3.91

Table 4: Mean ratings from the user study on a 5-point Likert scale. Higher scores indicate greater perceived naturalness and intent clarity.

Attribute	64	128	256
Master Category	99.8%	50.4%	1.0%
Subcategory	99.4%	50.2%	1.0%
Article Type	98.3%	49.7%	1.0%
Gender	98.1%	49.6%	1.0%
Base Color	87.9%	44.5%	1.0%
Usage	97.7%	49.4%	1.0%

Table 5: Effect of codebook size on weighted attribute purity for 64, 128 and 256 size.

	64		128		256
Level	Clusters	Items/Cl.	Clusters	Items/Cl.	Clusters	Items/Cl.
D1	44	823.9	156	285.0	267	166.5
D2	3,378	219.2	10,378	48.28	17,378	3.56
D3	23,956	79.5	30,045	13.48	36,122	1.23

Table 6: Cluster fragmentation as a function of codebook size.

Task	# Queries	# Pairs/Cands	Cands per Query	Leaf Clusters	Query Length
T1: Ranking	2,477	36,782	12–15 (mean 14.8)	416	3–26 words (mean 10.1)
T2: Relevance Classification	2,477	19,731	7–8 (mean 8.0)	416	3–26 words (mean 10.1)
T3: Sibling Discrimination	1,968	9,801	4–5 (mean 5.0)	331	3–26 words (mean 10.1)
T4: Query Robustness (Paraphrase)	416 groups	2,477 queries	12–15 (mean 14.9)	416	2–10 paraphrases/group (mean 6.0)

Table 7: Per-task composition of Fashion-RQBENCH. All queries are anchored at level-3 (leaf) clusters; Task 1 and Task 4 share the same 2,477 anchor queries (Task 4 paraphrases each one 2–10

\times

Task	Exact	Sibling	Related	Irrelevant
T1: Ranking (36,782 pairs)	20.2%	25.9%	26.9%	26.9%
T2: Classification (19,731 pairs)	25.1%	24.7%	25.1%	25.1%

Table 8: Relevance-label distribution over (query, candidate) pairs.

Attribute	$N$	$\overline{\Delta}$	95% CI	Sign Consistency
Product Type	561	0.579	[0.440, 0.713]	0.78
Gender	730	0.730	[0.625, 0.838]	0.80
Age Group	129	0.737	[0.537, 0.951]	0.83
Color	786	0.926	[0.828, 1.025]	0.86
Material	259	0.589	[0.384, 0.796]	0.75
Occasion	952	0.072	[ $-$ 0.010, 0.153]	0.53

Table 9: Controlled matched-twin-pair ablation using the cross-encoder relevance oracle.

N

denotes the number of query–twin-pair observations.

\overline{\Delta}

is the mean cross-encoder logit difference, and the confidence interval is obtained by resampling twin pairs with replacement. Sign consistency denotes the fraction of observations having the same sign as

\overline{\Delta}

Attribute	Ridge Coef.	VIF	Final $w_{a}$
Product Type	0.467	1.10	0.290
Gender	0.125	1.06	0.127
Age Group	0.006	1.00	0.066
Color	0.114	1.01	0.138
Material	0.026	1.02	0.063
Occasion	0.004	1.02	0.008

Table 10: Regression-based attribute importance and final blended weights. Ridge coefficients are estimated from cross-encoder logits. VIF measures multicollinearity among attribute features.

Model	Scoring	Key Hyperparameters
BM25 [ 36 ]	Sparse lexical	$k_{1}=1.2$ , $b=0.75$ ; English stopwords; Porter stemmer
SPLADE++ EnsembleDistil [ 10 ]	Learned sparse	max_seq_len =512; batch size=64; $\max(\log(1+\mathrm{ReLU}(\cdot)))$ over MLM logits; vocabulary size=30,522
ColBERTv2 [ 37 ]	Multi-vector MaxSim	128 dimensions/token; query_maxlen =32; doc_maxlen =180; FP16 with FP32 MaxSim accumulation; batch size=128

Table 11: Lexical, learned-sparse, and multi-vector retrieval baselines.

Model	Dim.	Pooling	Max Seq. Len
Contriever-MSMARCO [ 14 ]	768	Mean	512
RoBERTa-large [ 21 ]	1024	Mean(first + last layer)	512
SimCSE-sup (RoBERTa-large) [ 52 ]	1024	pooler_output (CLS)	512
Sentence-T5-large [ 29 ]	768 †	Mean + Dense( $1024\!\rightarrow\!768$ , tanh)	256

Table 12: Dense single-vector retrieval baselines with non-LLM backbones.

Model	Params.	Dim.	Max Seq. Len	EOS Policy
Qwen3-Embedding-0.6B [ 42 ]	0.6B	1024	8192	auto ‡
Qwen3-Embedding-4B [ 42 ]	4B	2560	8192	auto ‡
Qwen3-Embedding-8B [ 42 ]	8B	4096	8192	auto ‡
E5-Mistral-7B-Instruct [ 48 ]	7B	4096	4096	manual §
SFR-Embedding-Mistral [ 27 ]	7B	4096	4096	auto

Table 13: LLM-backbone embedding baselines. All models use last-token pooling, bfloat16 , and

\ell_{2}

normalization. Queries are prefixed with “Given a fashion product search query, retrieve relevant product listings that match the query” , while documents are unprefixed.

Automated Evaluation

Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.

See all 396 papers

FashionLens: Toward Versatile Fashion Image Retrieval via Task-Adaptive Learning
May 21, 2026Haokun Wen, Xuemeng Song, Xinghao Xie +3Multi-Task Learning Image-Text Retrieval
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen 518055, China · Department of Data Science, City University of Hong Kong, Hong Kong, China · Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China +4
Diverse-Intent Multi-Turn Fashion Image Retrieval
Jul 22, 2026Mingqiang Tang, Haokun Wen, Meng Liu +3Multimodal IR Cross-Modal Alignment
1Southern University of Science and Technology, Shenzhen, China · 2Harbin Institute of Technology (Shenzhen), Shenzhen, China · 3Shandong University, Jinan, China +1
VietFashion: Benchmarking Sketch-Text Composed Image Retrieval for Cultural Outfits
Jun 11, 2026Hoang-Nguyen Cao, Le-Hoang Bui, Dinh-Khoi Vo +2Fine-Grained Image Retrieval Image-Text Retrieval
University of Science · Ho Chi Minh City, Vietnam · Vietnam National University

May 21, 2026J/K move · Enter open · S save

FashionLens: Toward Versatile Fashion Image Retrieval via Task-Adaptive Learning

Haokun Wen, Xuemeng Song, Xinghao Xie, Xiaolin Chen+2

School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen 518055, China · Department of Data Science, City University of Hong Kong, Hong Kong, China · Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China+4

Multi-Task Learning Image-Text Retrieval Cross-Modal Retrieval

Open paper AskPDF arXiv

Fashion image retrieval is a cornerstone of modern e-commerce systems. A unified framework that supports diverse query formats and search intentions is highly desired in practice. However, existing approaches focus on narrow retrieval tasks and do not fully capture such diversity. Therefore, in this work, we aim to develop a unified framework capable of handling diverse realistic fashion retrieval scenarios, achieving truly versatile fashion image retrieval. To establish a data foundation, we first introduce U-FIRE, a comprehensive benchmark that consolidates fragmented fashion datasets into a unified collection, supplemented by two manually curated datasets for testing generalization. Building upon this, we propose FashionLens, a unified framework based on Multimodal Large Language Models. To handle divergent matching objectives, we design a Proposal-Guided Spherical Query Calibrator that dynamically shifts query representations into task-aligned metric spaces via adaptive spherical linear interpolation. Additionally, to mitigate the optimization imbalance caused by varying task complexities and data scales, we develop a Gradient-Guided Adaptive Sampling strategy that automatically re-weights tasks based on realtime learning difficulty and the data scale prior. Experiments on U-FIRE show that FashionLens achieves state-of-the-art performance across diverse retrieval scenarios and generalizes robustly to unseen tasks. The data and code are publicly released at https://github.com/haokunwen/FashionLens.

Abstract

Automated Evaluation

Explore similar work

FashionLens: Toward Versatile Fashion Image Retrieval via Task-Adaptive Learning

Diverse-Intent Multi-Turn Fashion Image Retrieval

VietFashion: Benchmarking Sketch-Text Composed Image Retrieval for Cultural Outfits