Product linking, the entity-resolution task of mapping merchant product records to canonical catalog products, consolidates fragmented listings so downstream search, recommendation, and advertising see one clean entry per product. At marketplace scale, billions of noisy, multi-category records must be resolved against tens of millions of canonical products, where scoring every candidate with a single model is either too weak for the hard cases or too costly for the easy ones. We present a production retrieve-then-match cascade that spends computation in proportion to difficulty: retrieval surfaces plausible matches, a lightweight text cross-encoder auto-resolves the high-confidence majority, and an agentic multimodal vision-language model settles the ambiguous remainder by inspecting product images and issuing web searches for evidence that is in neither record. The cross-encoder is distilled from millions of dual-VLM-consensus labels, retiring human annotation from the training set, and is calibrated to auto-accept links at a 98% precision bar validated against a smaller operator-certified audit. The agent is a self-hosted open-weight model that reaches a closed frontier VLM's precision at a four-point recall cost (88% versus 92%) for roughly one-seventh the per-pair cost, with no fine-tuning. Per-pair cost spans nearly five orders of magnitude from the cheap cross-encoder to the frontier VLM, so escalating only the hard tail to the agent raises end-to-end link coverage from the cheap stage's 68% to 77%.
Figures & tables
Figure 1. The adaptive-routing cascade: retrieval narrows each merchant record to K=20 candidates, a cheap text-only cross-encoder routes each pair by confidence (auto-accepting high , auto-rejecting low ), and only the ambiguous medium band escalates to the agentic multimodal VLM. Per-pair cost is relative to the cross-encoder (Table 1 ).
Model
Rel. cost
Modality
Cross-encoder matcher
1×
text
Open-weight agentic VLM
∼7,000×
text+image+search
Closed frontier VLM
∼50,000×
text+image+search
Table 1. Relative per-pair cost across the three matching models, normalized to the cheap cross-encoder ( 1× ); values are relative multipliers only.
Table 2. Retrieval recall on the deduplication slice (the primary, independent-ground-truth metric; Section 4.1 ), under the production configuration (image+text+GTIN ANN, RRF fusion, K=20 ) and single-factor changes to it.
Recipe
R@P98 (stress audit)
Raw barcode digits, no mitigation
0.200
+ barcode dropout + name-similarity guardrail
0.692
+ distillation from the guardrailed teacher, 1 epoch
0.743
+ distillation from the guardrailed teacher, 2 epochs
0.768
Table 3. Recall at the 98% precision operating point (R@P98) on a stress audit oversampled for barcode-collision pairs (same barcode, conflicting name), across the mitigation progression. Single-seed for the distillation rows (Section 4.2.2 ).
System
Accuracy (%)
Precision (%)
Recall (%)
Human operators
83.0
94.4
77.0
Closed VLM (GPT-5.4)
96.7
99.1
95.5
Δ (VLM − operators)
+13.7
+4.7
+18.5
Table 4. Agentic VLM versus trained human operators on the hardest medium-confidence match tasks. Differences are significant at p<0.0001 ; the 95% confidence intervals on the deltas exclude zero.
Dimension
Open (Qwen 3.6, self-hosted)
Closed (GPT-5.4, API)
Precision
98.0%
98.0%
Recall
88%
92%
Relative cost/pair
∼ 1 ×
∼ 7 ×
Search backend
self-hosted web search (MCP)
OpenAI-hosted web search
Fine-tuning / weights
full control
none
Table 5. Open-weight self-hosted agent versus the closed frontier agent on the hard-tail matching stage, at equal precision. Cost is per pair, normalized to the open-weight stack (open =1× here; Table 1 gives the absolute cost ladder); other rows are relative or architectural.
Entity resolution links records that refer to the same real-world entity across datasets despite differences in names, attributes, formats, and languages.
Associating user search queries with the correct brand entity is critical for e-commerce product retrieval, yet remains challenging due to the brevity of queries (three to four words on average), their lack of grammatical structure, and a catalog of hundreds of thousands of distinct brands. We formulate this as a brand entity linking task and develop two complementary solutions deployed at scale: (1) a cascaded pipeline that first detects brand mentions via sequence labeling and then disambiguates against a brand knowledge base, and (2) a single-stage approach that frames linking as extreme multiclass classification, directly mapping queries to brand identifiers. Through extensive multilingual evaluation (11 languages) and a controlled online experiment, we demonstrate that the proposed methods substantially improve brand recall while maintaining high precision, leading to measurable gains in customer engagement.
Dong Liu, Sreyashi Nag
Amazon Luxembourg, Luxembourg · Amazon Palo Alto, California, United States
Multimodal product retrieval (MPR) underpins checkout-free retail and automated inventory systems, yet it demands fine-grained SKU discrimination that standard vision-language benchmarks fail to capture. We present the first systematic zero-shot evaluation of 190 open-source VLMs on the MPR task of the GroceryVision Challenge, isolating pre-training data, architecture, and input resolution. Our analysis yields three actionable findings. (1) Data quality trumps scale. Switching from raw web-scrapes to filtered datasets delivers up to 16.6% accuracy gains, exceeding the benefit of doubling model parameters. (2) Efficient models can win. MobileCLIP-B (150M parameters) outperforms 351M counterparts trained on noisy data. We introduce semantic power density (φ), an efficiency metric that penalizes sub-threshold accuracy. (3) A precision gap persists. State-of-the-art models achieve 94.5% Recall@5 but suffer a 17.5% drop at Recall@1, revealing that contrastive embeddings cluster categories effectively but fail to rank visually similar SKUs. Code and evaluation scripts are available at https://github.com/upeee/openmpr.
Emmanuel G. Maminta, Rowel O. Atienza
AI Graduate Program, University of the Philippines, Diliman, Quezon City · EEEI, University of the Philippines, Diliman, Quezon City
E-commerce search requires distinguishing products that are merely related to a query from those that directly satisfy the user's shopping intent. We augment query-product pairs with structured LLM-generated query and product attributes and human-validated relevance, explanations, and centrality judgments, and evaluate these signals using a simple dual-encoder retriever and MLP re-ranker. On an augmented subset of ESCI, a human-feature oracle reaches 0.9382 nDCG@10, while a human-free trained Q+P configuration reaches 0.9258. Synthetic approximations of the human signals reach 0.9150 overall but provide substantial gains for difficult, low-performing queries. Ablations show that most of the oracle improvement comes from post-edited explanations and annotator comments rather than the scalar centrality feature, suggesting that LLMs are most useful for exposing and approximating structured semantic supervision rather than replacing human judgment directly.
Girish A. Koushik, Swapnil Bhosale, Samarth Agrawal +4
Computer Science, University of Surrey, Guildford, United Kingdom · People-Centred AI, University of Surrey, Guildford, United Kingdom · eBay USA +2