Sketch2Inspire: Structure-Sensitive Evaluation for Product Retrieval

Ge Kong

cs.CVAug 29, 2026

Sketch2Inspire: Structure-Sensitive Evaluation for Product Retrieval

Authors: Ge Kong

Organizations: Beihang University

Abstract

Early-stage product design retrieval often requires more than category recognition: designers may need reference examples that match both a short semantic intent and a rough structural cue. Existing product-image resources and generic image--text retrieval benchmarks rarely separate category retrieval from within-category structural fit. We present Sketch2Inspire, built from a curated subset of Amazon Berkeley Objects with aligned text queries, edge-based sketch-proxy queries, and fused text--sketch queries. The resource separates broad category-level retrieval from structure-sensitive within-category retrieval and includes a human-graded reference protocol for calibration. We evaluate a lightweight reference system based on pretrained CLIP-family encoders, comparing text-only retrieval, sketch-only retrieval, weighted late fusion, and text-first reranking without updating model weights. Under broad relevance, late fusion obtains the highest score (nDCG = 0.9962). Under automatic structure-sensitive relevance, late fusion again obtains the highest score (nDCG = 0.7015), exceeding text-only retrieval (nDCG = 0.5912). In the human-graded results, late fusion obtains the highest nDCG@10 (0.9133), while text-only retrieval ranks second (0.9030). These results show that the retrieval gain from multimodal input depends on how relevance is defined. Sketch2Inspire therefore provides a diagnostic resource for evaluating modality contribution and supports the development of structure-aware product-retrieval protocols with independent human annotation.

Figures & tables

Figure 1: Sketch2Inspire overview. The evaluation resource pairs controlled text and sketch-proxy queries with a curated product gallery, then evaluates text-only, sketch-only, late-fusion, and reranking retrieval under broad category-level, structure-sensitive, and human-graded protocols.

Setting	Cats.	Img./cat.	Total	Query coverage
Expanded Setting	300	30	9000	text / sketch / fusion
Structure-Sensitive Setting	260	15	3900	text / sketch / fusion

Table 1: Evaluation resources used in Sketch2Inspire. The human-graded protocol is reported separately from the automatic structure-sensitive setting.

Method	Recall@5	Recall@10	mAP	nDCG
text-only	0.2500	0.4983	0.9839	0.9698
sketch-only	0.2117	0.3983	0.7686	0.8984
late fusion ( $\alpha=0.6$ )	0.2417	0.4683	0.9047	0.9962

Table 2: Automatic results on Expanded Setting under broad category-level relevance.

Setting	Method	Recall@5	Recall@10	mAP	nDCG
category-level	text-only	0.3500	0.6881	0.9606	0.9866
category-level	sketch-only	0.3048	0.5810	0.8132	0.9117
category-level	late fusion	0.3548	0.6810	0.9548	0.9858
category-level	text-first reranking	0.3452	0.6571	0.9160	0.9581
fine-grained structural	text-only	0.3778	0.7667	0.3902	0.5912
fine-grained structural	sketch-only	0.5889	0.7333	0.5070	0.6785

Table 3: Automatic results on Structure-Sensitive Setting. The late-fusion row uses the highest-scoring fine-grained structural setting (

\alpha=0.40

), and text-first reranking uses a text candidate pool of 20 images.

Variant	Recall@5	Recall@10	mAP	nDCG
$\alpha=0.05$	0.4111	0.8000	0.4092	0.6035
$\alpha=0.10$	0.4556	0.8444	0.4374	0.6276
$\alpha=0.20$	0.5111	0.9000	0.4519	0.6333
$\alpha=0.30$	0.5778	0.9222	0.4821	0.6613
$\alpha=0.40$	0.6111	0.9000	0.5290	0.7015
$\alpha=0.50$	0.6222	0.8778	0.5252	0.6962

Table 4: Alpha sweep on Structure-Sensitive Setting under fine-grained structural relevance.

Method	manual nDCG@10	manual mAP@10	manual Recall@5
text-only	0.9030	0.9479	0.5336
sketch-only	0.8888	0.8945	0.4722
late fusion	0.9133	0.9196	0.5206

Table 5: Results on Human-Graded Reference Protocol. This human-graded evaluation is reported as a separate manual check and is not part of the automatic structure-sensitive protocol.

Figure 2: Qualitative retrieval comparison on representative cases from Structure-Sensitive Setting. Each row shows a sketch-proxy query and text prompt, followed by top retrieved examples from text-only retrieval, sketch-only retrieval, late fusion, and text-first reranking. Borders indicate stored proxy annotations used for visualization. The figure illustrates why text can dominate broad category matching while sketch cues can help with within-category structural drift.

Explore similar work

Apr 17, 2026cs.CV

Sketch and Text Synergy: Fusing Structural Contours and Descriptive Attributes for Fine-Grained Image Retrieval

Fine-grained image retrieval via hand-drawn sketches or textual descriptions remains a critical challenge due to inherent modality gaps. While hand-drawn sketches capture complex structural contours, they lack color and texture, which text effectively provides despite omitting spatial contours. Motivated by the complementary nature of these modalities, we propose the Sketch and Text Based Image Retrieval (STBIR) framework. By synergizing the rich color and texture cues from text with the structural outlines provided by sketches, STBIR achieves superior fine-grained retrieval performance. First, a curriculum learning driven robustness enhancement module is proposed to enhance the model's robustness when handling queries of varying quality. Second, we introduce a category-knowledge-based feature space optimization module, thereby significantly boosting the model's representational power. Finally, we design a multi-stage cross-modal feature alignment mechanism to effectively mitigate the challenges of cross modal feature alignment. Furthermore, we curate the fine-grained STBIR benchmark dataset to rigorously validate the efficacy of our proposed framework and to provide data support as a reference for subsequent related research. Extensive experiments demonstrate that the proposed STBIR framework significantly outperforms state of the art methods.

Siyuan Wang, Hanchen Gao, Guangming Zhu +5

Xidian University · First Aircraft Design Institute

Cross-Modal Representation Learning Fine-Grained Image Retrieval Cross-Modal Feature Matching Composed Image Retrieval

Ask PDF

Aug 21, 2026cs.CV

EviRank: Structured Relevance Evidence for Multimodal Image Re-ranking

Real-world image search queries are multimodal and compositional: ``find this shirt in pink'' specifies an entity to retain, an attribute to modify, and context to ignore. Yet existing re-rankers either compress such multifaceted relevance into an opaque embedding or rely on free-form chain-of-thought that easily omits or hallucinates fine-grained constraints. Drawing on rubric- and checklist-based evaluation from NLP, we recast multimodal image re-ranking as a semantic constraint satisfaction problem and propose EviRank, which parses any query - text-only, image-only, or composed - into a unified evidence package: typed criteria across six semantic slots (e.g., entities, attributes, relations), each labelled required, forbidden, or ignorable. Re-ranking then reduces to evidence-conditioned verification, combining deterministic rubric scoring and evidence-grounded listwise comparison in a single training-free procedure. The explicit evidence can further serve as structured supervision for optionally distilling a lightweight student. Across five benchmarks spanning text-to-image, image-to-image, and composed image retrieval, EviRank achieves state-of-the-art performance, and the distilled student preserves over 90% of the teacher's capability at substantially lower cost.

Enjun Du, Siyi Liu, Zirong Chen +8

The Hong Kong University of Science and Technology (Guangzhou) · Tencent Yuanbao · The University of Hong Kong

Multimodal Reranking Multimodal IR Fine-Grained Image Retrieval Composed Image Retrieval

Ask PDF

May 18, 2026cs.CV

What Matters for Grocery Product Retrieval with Open Source Vision Language Models

Multimodal product retrieval (MPR) underpins checkout-free retail and automated inventory systems, yet it demands fine-grained SKU discrimination that standard vision-language benchmarks fail to capture. We present the first systematic zero-shot evaluation of 190 open-source VLMs on the MPR task of the GroceryVision Challenge, isolating pre-training data, architecture, and input resolution. Our analysis yields three actionable findings. \textbf{(1) Data quality trumps scale.} Switching from raw web-scrapes to filtered datasets delivers up to 16.6% accuracy gains, exceeding the benefit of doubling model parameters. \textbf{(2) Efficient models can win.} MobileCLIP-B (150M parameters) outperforms 351M counterparts trained on noisy data. We introduce \textit{semantic power density} (

φ

), an efficiency metric that penalizes sub-threshold accuracy. \textbf{(3) A precision gap persists.} State-of-the-art models achieve 94.5% Recall@5 but suffer a 17.5% drop at Recall@1, revealing that contrastive embeddings cluster categories effectively but fail to rank visually similar SKUs. Code and evaluation scripts are available at https://github.com/upeee/openmpr.

Emmanuel G. Maminta, Rowel O. Atienza

AI Graduate Program, University of the Philippines, Diliman, Quezon City · EEEI, University of the Philippines, Diliman, Quezon City

VLM Evaluation Fine-Grained Image Retrieval Information Retrieval Vision-Language Retrieval Image Retrieval Multimodal Understanding

Ask PDF