Visual Product Search Benchmark

Karthik Sulthanpete Govindappa

cs.CVMar 17, 2026

Visual Product Search Benchmark

Authors: Karthik Sulthanpete Govindappa

Organizations: nyris GmbH, Berlin, Germany

Abstract

Reliable product identification from images is a critical requirement in industrial and commercial applications, particularly in maintenance, procurement, and operational workflows where incorrect matches can lead to costly downstream failures. At the core of such systems lies the visual search component, which must retrieve and rank the exact object instance from large and continuously evolving catalogs under diverse imaging conditions. This report presents a structured benchmark of modern visual embedding models for instance-level image retrieval, with a focus on industrial applications. A curated set of open-source foundation embedding models, proprietary multi-modal embedding systems, and domain-specific vision-only models are evaluated under a unified image-to-image retrieval protocol. The benchmark includes curated datasets, which includes industrial datasets derived from production deployments in Manufacturing, Automotive, DIY, and Retail, as well as established public benchmarks. Evaluation is conducted without post-processing, isolating the retrieval capability of each model. The results provide insight into how well contemporary foundation and unified embedding models transfer to fine-grained instance retrieval tasks, and how they compare to models explicitly trained for industrial applications. By emphasizing realistic constraints, heterogeneous image conditions, and exact instance matching requirements, this benchmark aims to inform both practitioners and researchers about the strengths and limitations of current visual embedding approaches in production-level product identification systems. An interactive companion website presenting the benchmark results, evaluation details, and additional visualizations is available at https://benchmark.nyris.io.

Figures & tables

	Dataset	Retrieval Setup	Gallery Type	#Images (Query)	#Images (Gallery)	#Products (Query)	#Products (Gallery)
Private	Clips-and-Connectors v1	Inter	Closed	3,624	200,496	453	12,531
	Furniture v1	Inter	Closed	371	85,165	347	50,641
	DIY v1	Inter	Closed	5,008	221,376	5,008	221,376
	Automotive v1	Inter	Closed	2,062	19,610	379	10,379
Public	SOP	Intra	Closed	60,502	60,502	11,316	11,316
	ILIAS	Inter	Open	1,232	504,715	2,500	1,000 †

Table 1: Overview of the datasets used for evaluating instance-level image retrieval. For intra-retrieval datasets, the same split is used as query and gallery, with self-matches excluded during evaluation. † In the ILIAS dataset, the gallery contains 1,000 unique products, while the remaining images correspond to distractors that do not belong to any query product.

Model	Provider	Arch	Input Size	Retrieval Expertise	Embedding Dimension
Open-Source
DINOv2-Large	Meta	ViT	224 $\times$ 224	Generic	1024
DINOv3-ViT-L/16	Meta	ViT	224 $\times$ 224	Generic	1024
Jina-Embeddings-v4	Jina	VLM	Adaptive †	Generic	2048
Nomic-Embed-Multi-Modal-3B	Nomic	VLM	Adaptive †	Generic	2048
PE-Core-L/14	Meta	ViT	336 $\times$ 336	Generic	1024

Table 2: Overview of evaluated embedding models grouped by availability. † Adaptive input size indicates that the model accepts variable-resolution inputs without requiring fixed resizing.

General
Model	Proprietary Datasets												Public Datasets										Average
	Clips-Connectors v1			DIY v1			Furniture v1			Automotive v1			ILIAS				Products-10K			SOP			Avg
	R@1	R@5	mAP@20	R@1	R@5	mAP@20	R@1	R@5	mAP@20	R@1	R@5	mAP@20	R@1	R@5	mAP@20	mAP@1000	R@1	R@5	mAP@20	R@1	R@5	mAP@20	R@1	R@5	mAP@20
Nomic Embed MM 3B	2.5	6.2	0.4	12.0	24.5	17.2	26.1	43.1	14.5	10.2	20.2	9.7	–	–	–	–	54.3	72.2	33.7	56.9	69.3	32.6	27.0	39.3	18.0
Jina Embeddings v4	1.7	4.7	0.3	12.0	24.0	17.6	32.9	51.2	18.5	8.3	17.9	8.1	–	–	–	–	56.4	74.6	38.1	59.5	72.2	35.1	28.5	40.8	19.6
DINOv2 Large	13.7	26.4	2.7	18.2	35.1	24.9	40.2	59.8	24.6	19.3	31.8	18.8	–	–	–	–	50.2	66.4	29.0	56.3	67.7	31.9	33.0	47.9	22.0

Table 3: Image-to-image instance retrieval results (R@1, R@5, mAP@20) grouped by proprietary and public datasets. Best score in each column is shown in bold. Models sorted by overall average. † Averages for DINOv3 ViT-L/16 and GEM v5.1 exclude ILIAS.

Visual Search

Visual search studies how people and AI systems locate, inspect, and retrieve relevant objects or evidence in images and visual environments.

See all 36 papers

Jun 12, 2026

IndustryBench-MIPU: Benchmarking Multi-Image Attribute Value Extraction for Industrial Products

Industrial products such as valves and circuit breakers are defined by dense technical specifications that govern procurement, compatibility, and safety across supply chains. These specifications are scattered across multiple heterogeneous product images, including specification tables, nameplates, and technical drawings, yet whether Multimodal Large Language Models (MLLMs) can reliably recover them remains underexplored. To fill this gap, we introduce IndustryBench-MIPU, the first large-scale benchmark for multi-image industrial product understanding, built around structured attribute extraction -- recovering property-value pairs from product images. This task jointly probes text recognition on specification tables and nameplates, visual reasoning over technical drawings, domain knowledge to decode industrial terminology, and cross-image evidence integration to assemble scattered specifications. Concretely, the benchmark comprises 4,559 products across 27,652 images with 103,703 annotations spanning 18 industrial categories, constructed through multi-model consensus and three-tier quality assurance. Evaluating nine MLLMs under both single-image and product-level multi-image settings reveals a stark completeness gap: models achieve high precision (86--94%) but the best recovers only 49.9% of product-level attributes; moving from single-image to multi-image extraction costs 15--34 percentage points of recall. Multi-image completeness, not single-image accuracy, is the core bottleneck. Dataset and code are publicly available.

Haonan Qi, Jin Cao, Yongqi Zhang +9

Multimodal and Industrial AI TeamTaobao&Tmall, Alibaba Group

Multimodal Large Language Models Multimodal Model Evaluation Document Image Understanding Product Attribute Extraction

Ask PDF

Jun 6, 2026

Unification of Closed-Open Industrial Detection Scenarios: New Large-Scale Benchmarks,Challenges and Baselines

Large-scale Visual-Language Models (LVLMs) have achieved remarkable success in natural visual tasks, yet their application to industrial defect detection remains challenging due to two fundamental limitations: (i) the scarcity of large-scale industrial datasets that cover diverse defect categories across multiple domains, and (ii) the reliance on manual prompts (points, boxes, masks) that introduce subjective noise and lack text-visual interaction for fine-grained understanding. To address these challenges, we introduce a Large-Scale Multi-Modal Industrial Open-Closed benchmark (MMIOC-1M) containing over one million samples across

14

super-categories,

29

industrial scenes, and

351

defect subcategories. To our knowledge, MMIOC-1M is the first unified largest benchmark supporting both open-vocabulary and closed-set industrial detection, providing valuable pre-training data for LVLMs in industrial scenarios. Furthermore, we propose a Refined Text-Visual Prompt Network (RTVPNet) that incorporates three key innovations: (1) an expert-assisted domain projection mechanism that enables rapid adaptation of general vision models to industrial domains, (2) an energy-based sparse sampling strategy that automatically generates refined visual prompts without manual intervention, and (3) a bidirectional text-visual interaction module that enhances cross-modal semantic alignment and understanding. Extensive experiments demonstrate that RTVPNet achieves state-of-the-art performance on MMIOC-1M, LVIS, and COCO benchmarks while maintaining computational efficiency. The dataset and code are available at https://github.com/hellozzk/MMIO.

Zekai Zhang, Jinglin Zhang, Qinghui Chen +8

School of Control Science and Engineering, Shandong University, Jinan 250061, China · Shandong Computer Science Center, Qilu University of Technology, Jinan, China · CEREMADE, University Paris Dauphine, PSL Research University, CNRS, UMR 7534, 75775 Paris, France +4

Open-Vocabulary Object Detection VLM Adaptation Industrial Anomaly Detection Visual Prompt Learning

Ask PDF

Jun 2, 2026

VistaHop: Benchmarking Multi-hop Visual Reasoning for Visual DeepSearch

Visual DeepSearch requires multimodal large reasoning model (MLRM) agents to answer complex visual queries by repeatedly inspecting image regions, grounding intermediate reasoning in visual evidence, and connecting fine-grained clues across long reasoning chains. However, existing benchmarks mainly focus on single-step visual understanding or static image-question answering, offering limited evaluation of iterative image inspection, visual-anchor grounding, and multi-hop evidence integration. In this work, we introduce VistaHop, a benchmark for evaluating vision-centric search and multi-hop visual reasoning in Visual DeepSearch. VistaHop contains 300 high-resolution images, 25 visual search scenarios, and 350 multi-hop QA tasks that require models to follow evidence chains from visual anchors or fuse information across multiple image-grounded reasoning paths. We further develop VistaArena, a unified evaluation environment that supports tool-augmented reasoning with text search, image search, image cropping, and evidence-based answer validation. Experiments on seven representative MLRMs show that current models remain far from solving VistaHop: the best model, SenseNova-MARS-32B, achieves only 24.31% Pass@1. These results reveal persistent limitations in visual grounding, evidence revisiting, long-chain reasoning, and multi-anchor information fusion, highlighting the need for stronger benchmarks and training methods for Visual DeepSearch.

Hang He, Chuhuai Yue, Chengqi Dong +6

1East China Normal University · 2Meituan · 3Shanghai Innovation Institute

VLM Evaluation Visual Question Answering Visual Reasoning Tool-Using Vision-Language Agents Visually Grounded Reasoning Visual Search

Ask PDF

Abstract

Visual Search

Explore similar work

IndustryBench-MIPU: Benchmarking Multi-Image Attribute Value Extraction for Industrial Products

Unification of Closed-Open Industrial Detection Scenarios: New Large-Scale Benchmarks,Challenges and Baselines

VistaHop: Benchmarking Multi-hop Visual Reasoning for Visual DeepSearch