Retail product catalogs assembled from web image search are cheap to build and pervasively noisy: queries return packaging shots instead of products, and entries proliferate under names that cannot be distinguished visually. Consolidating such a catalog can look like an alternative to scaling the model. We show this is not a comparison between two ways of improving a recognizer, but between improving a recognizer and changing what counts as correct. We study a public archive of 165,769 cheese images across 1,824 query-named entries. A packaging filter and near-duplicate removal reduce it to a 998-class recognition set on one fixed, group-aware split. Canonicalizing entries to their base type (844 classes) raises top-1 accuracy by 4.0 points. Scoring the unchanged model against the merged label space, by summing its predicted probability within each canonical group, recovers +4.27 points with no retraining (+4.25 over six seeds). Retraining on the consolidated catalog then adds -0.10±0.53, +0.08±0.65 and -0.10±0.65 points at 28, 50 and 88 M parameters, indistinguishable from zero at every capacity (± is the across-seed standard deviation). That component is specific, not mechanical: random merges with the identical size profile yield only +0.68 points, and the three backbones agree to within 0.1 points. But it is overwhelmingly category coarsening, not alias correction: across the 37 merged groups, type coarsenings supply 94% of the gain and alias collapses only 7%. We also detect no interaction between consolidation and backbone size: tripling backbone parameters is worth +1.50±0.57 points on the raw catalog and +1.42±00.44 on the canonical one, a within-seed interaction of -0.08 [-0.79,+0.63]. On this archive, the reported consolidation gain primarily reflects the evaluation label space, and separating that out costs one evaluation
Figures & tables
Figure 1: The two defects. Top: the Stage-1 probe scores packaging shots near zero and product shots near one. The leftmost image shows a packaged snack rather than unpackaged cheese, returned by “cheddar cheese” . Bottom: four distinct photographs from folders the scrape treats as four classes and Stage 3 maps to cheddar . This is why no per-image relabeling fixes a defect of this kind.
Quantity
Value
Folders (cheese names)
1,824
Total readable images
165,769
Images kept after packaging filter
62,174 (37.5%)
Distinct photographs after dedup
33,583 (54.0% of kept)
Cheeses with ≥20 distinct images
998
Distinct images/class, median / mean
35 / 36.1
Table 1: Archive and label-set statistics. The recognition subset exceeds the distinct-photograph count because a duplicate group spanning several folders contributes one image to each (Section 4.2 ).
Figure 2: The pipeline. Near-duplicate grouping reduces the filtered images to 33,583 distinct photographs before any split is drawn. Stage 3 changes the label space, not the weights, so Stage 2’s frozen model can be read out over it without retraining. The two branches illustrate this comparison, decomposed in Figure 3 . Six-seed ConvNeXt-Small means (Table 3 ).
Figure 3: (a) Raw-label evaluation, two canonical rescoring rules, and canonical-label retraining, across three backbone sizes (six-seed means). The filled marker shows the accuracy obtained from the frozen model’s posterior over canonical labels, without retraining. It matches the retrained model at every size. (b) Leave-one-out sensitivity: the gain is concentrated in blue , which coarsens 43 entries into a broad type rather than collapsing aliases.
Method
top-1
top-5
Uniform random baseline
0.10%
0.50%
Zero-shot CLIP (text-name match)
5.4%
11.1%
CLIP linear probe (frozen)
20.4%
37.7%
ConvNeXt-Tiny fine-tuned, 25 ep
25.6%
44.1%
ConvNeXt-Small fine-tuned, 25 ep
26.3%
44.7%
ConvNeXt-Base fine-tuned, 25 ep
26.6%
45.4%
Table 2: Stage 2: cheese-type recognition ( 998 classes). Every row is scored on the same held-out test split ( N=5,317 ) of the deduplicated 80 -capped subset. The probe is fit on that split’s 25,325 training images. ConvNeXt-Small remains the reference model for Sections 6.3 to 6.4 , with six-seed results in Table 3 . ConvNeXt rows here use seed 42 . No row is bolded: no single capacity step separates from its neighbor (Table 4 ).
Figure 4: Validation top-1 per epoch, seed 42 . The three raw-label backbones each peak before epoch 25 and their final accuracies differ by under two points, so Table 2 compares them at a fixed epoch budget. The two 844 -class runs are plotted for reference. Inset: epochs 15 to 25 .
Model
raw
marg.
retr.
defin.
resid.
Tiny ( 28 M)
25.30
29.60
29.51
+4.30±.24
−0.10±.53
Small ( 50 M)
26.11
30.36
30.44
+4.25±.09
+0.08±.65
Base ( 88 M)
26.80
31.03
30.93
+4.23±.20
−0.10±.65
CLIP probe, frozen
20.44
24.21
23.34
+3.76
−0.87
Small, filter 0.7
25.66
30.75
30.70
+5.09
−0.05
Table 3: Decomposing the merge gain: six-seed mean top-1 for the three ConvNeXt backbones. Accuracies are percentages; components are percentage points with six-seed standard deviations. Each row’s own raw model is rescored under the same canonical map by the marginal rule. The last two rows are single runs, and the 0.7 -filter row rebuilds the split, so only its decomposition is comparable.
Comparison
Δ
b / c
McNemar p
Tiny → Small (capacity)
+0.8
322 / 363
0.13
Small → Base (capacity)
+0.2
317 / 328
0.69
Tiny → Base (capacity)
+1.0
302 / 354
0.046
Small → Base, merged labels
+1.0
279 / 333
0.03
Merged → merged + class-bal.
−4.1
429 / 213
1×10−17
Raw → rescored, marginal
+4.3
32 / 259
3×10−45
Table 4: Exact McNemar tests on paired per-image outcomes, shared test split ( N=5,317 ), with the discordant counts used in each test: b correct before and wrong after, c wrong before and correct after.
Automated product recognition is a cornerstone of modern retail intelligence; however, accurately matching real-world, in-store images against extensive corporate catalogs remains a major scalability bottleneck for large-scale applications. In this work, we address this challenge by reformulating the task as an embedding-based cross-domain retrieval problem rather than a standard closed-set classification task. Specifically, we define the objective as retrieving the most corresponding catalog reference image for a given real-world product query crop from an expansive inventory. To bridge the severe domain gap between pristine studio packshots and noisy in-store queries, we introduce a novel catalog-to-real multi-stage contrastive learning paradigm (Cat2Real). This framework fine-tunes a vision backbone by systematically exploiting both item-level and image-level similarities to drive targeted hard negative mining. Extensive empirical evaluations demonstrate that our paradigm scales seamlessly to unseen products and categories, yielding outstanding zero-shot generalization performance even in the complete absence of real-world training images for novel inventory.
Product linking, the entity-resolution task of mapping merchant product records to canonical catalog products, consolidates fragmented listings so downstream search, recommendation, and advertising see one clean entry per product. At marketplace scale, billions of noisy, multi-category records must be resolved against tens of millions of canonical products, where scoring every candidate with a single model is either too weak for the hard cases or too costly for the easy ones. We present a production retrieve-then-match cascade that spends computation in proportion to difficulty: retrieval surfaces plausible matches, a lightweight text cross-encoder auto-resolves the high-confidence majority, and an agentic multimodal vision-language model settles the ambiguous remainder by inspecting product images and issuing web searches for evidence that is in neither record. The cross-encoder is distilled from millions of dual-VLM-consensus labels, retiring human annotation from the training set, and is calibrated to auto-accept links at a 98% precision bar validated against a smaller operator-certified audit. The agent is a self-hosted open-weight model that reaches a closed frontier VLM's precision at a four-point recall cost (88% versus 92%) for roughly one-seventh the per-pair cost, with no fine-tuning. Per-pair cost spans nearly five orders of magnitude from the cheap cross-encoder to the frontier VLM, so escalating only the hard tail to the agent raises end-to-end link coverage from the cheap stage's 68% to 77%.
Multimodal product retrieval (MPR) underpins checkout-free retail and automated inventory systems, yet it demands fine-grained SKU discrimination that standard vision-language benchmarks fail to capture. We present the first systematic zero-shot evaluation of 190 open-source VLMs on the MPR task of the GroceryVision Challenge, isolating pre-training data, architecture, and input resolution. Our analysis yields three actionable findings. (1) Data quality trumps scale. Switching from raw web-scrapes to filtered datasets delivers up to 16.6% accuracy gains, exceeding the benefit of doubling model parameters. (2) Efficient models can win. MobileCLIP-B (150M parameters) outperforms 351M counterparts trained on noisy data. We introduce semantic power density (φ), an efficiency metric that penalizes sub-threshold accuracy. (3) A precision gap persists. State-of-the-art models achieve 94.5% Recall@5 but suffer a 17.5% drop at Recall@1, revealing that contrastive embeddings cluster categories effectively but fail to rank visually similar SKUs. Code and evaluation scripts are available at https://github.com/upeee/openmpr.
Emmanuel G. Maminta, Rowel O. Atienza
AI Graduate Program, University of the Philippines, Diliman, Quezon City · EEEI, University of the Philippines, Diliman, Quezon City