cs.CVSep 8, 2026

The MODA General Attribute Suite: A Four-Track Evaluation Benchmark for Fashion Attribute Extraction

Authors: Arkid Mitra

Organizations: Hopit AI

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Fashion attribute extraction is evaluated inconsistently: results are reported as single aggregate numbers across image types that pose different problems, fields that are not visible in an image are scored as ordinary negatives, and the effect of vocabulary mismatch between datasets is acknowledged but not measured. We release the MODA General Attribute Suite, a four-track benchmark that keeps these problems separate by construction. Each track (localized garment crops, catalogue product images, applicability-aware full-body photographs, and product text) carries its own frozen test set, input contract, metric, and leakage unit, and the tracks are never averaged. The protocol requires label-blind prediction, SHA-256 commitment of prediction files before any label is opened, a fail-closed scorer, and a 10,000-sample paired cluster bootstrap at each track's natural leakage unit; promotion requires a positive interval on every track rather than a favourable mean. We release the scorers, the split builders, our own prediction files with their hashes for every image-track system in the main results and for runs we lose, and the MODA_NER(V) model checkpoints for three of the four tracks. The text model is not distributed; its benchmark is. We report baseline results, six interventions that did not improve them, and the limitations of the evidence. On the garment-crop track we also measure how far the headline depends on the benchmark's own design: the order of the two strongest systems depends on whether fields are micro- or macro-averaged, two thirds of the cells the protocol judges carry no annotation, and disabling the applicability decision cuts attribute micro-F1 from 0.63 to 0.27, so most of that score rests on absence decisions the source labels cannot adjudicate. This version also corrects one figure in an external comparison that an inference preprocessing defect had affected.

Figures & tables

Figure 1: The evaluation protocol as a seven-stage gated pipeline. Stages 1 and 5 are hard gates: no model output exists before the manifest is frozen, and no label is opened before predictions are hash-committed. Fashionpedia [1] crops share source images; DeepFashion-MultiModal [2] images share product groups, splitting on rows instead would leak, which is why stage 3 fixes the unit that stage 7’s bootstrap later reuses.
Figure 2: The margin is distributed, not carried by outliers: 27 of 28 fields are positive, and the median margin is around +0.10. The two tracks are scored separately and never averaged; they appear on one axis here only to show the spread. The single negative field, lower_fabric , is one where neither system is usable.
Figure 3: Four held-out images, with each system’s actual committed prediction. The composite answers in the Shopping100k vocabulary, and its dominant prediction, square , returned on 1,242 of the 1,856 images, has no slot in this evaluation’s two-way partition and so no correct answer to be scored against. 315 held-out images show exactly this pattern: gold is a scored neckline value, the composite says square , and the text-prototype route is right.
More in this topic

Product Attribute Extraction

See all 15 papers

Explore similar work

CardsList
May 11, 2026

Fashion Florence: Fine-Tuning Florence-2 for Structured Fashion Attribute Extraction

We present Fashion Florence, a Florence-2 vision-language model fine-tuned with LoRA to extract structured fashion attributes from clothing images. Given a single photograph, the model generates a JSON object containing category, color, material, style tags, and occasion tags, structured output suitable for direct programmatic consumption by downstream recommendation and retrieval systems. Fine-tuning data is derived from the iMaterialist Fashion dataset (228 labels), where we collapse fine-grained annotations into a compact 6-category, 16-color, 19-style schema via rule-based label engineering. We apply LoRA (r=16, alpha=32) to all decoder linear layers, training for 3 epochs on 3,688 examples. On a held-out test set of 461 images, Fashion Florence achieves 94.6% category accuracy and 63.0% material accuracy, compared to 89.3% / 43.3% for GPT-4o-mini and 87.4% for Gemini 2.5 Flash. Fashion Florence produces valid JSON in 99.8% of outputs while running at 0.77B parameters on a single GPU at zero marginal inference cost. Style tag F1 reaches 0.753 vs. 0.612 (Gemini) and 0.398 (GPT-4o-mini). The model is deployed as a Hugging Face Space and integrated into Loom, an open-source outfit recommendation system.

Anushree Berlia

VLM AdaptationMultimodal Foundation ModelsProduct Attribute Extraction
AskPDF
Aug 12, 2026

Attribute-Conditioned Multimodal Slot Factorization for Controllable Fashion Retrieval

Fashion retrieval often requires satisfying multiple attributes at once, such as category, color, pattern, and demographic. Monolithic embeddings mix these signals into a single vector, making attribute-specific control difficult at retrieval time. Many existing semantic-ID methods provide discrete item codes, but these codes are typically optimized as item-level or residual addresses and do not expose named, independently controllable attribute slots. We introduce MM-slotgate, a multimodal slot encoder that factorizes Fashion-CLIP text and image embeddings into four named attribute slots. Each slot learns its own text-image gate, so visually grounded attributes such as color and pattern can rely more on image evidence, while taxonomy-oriented attributes such as category and demographic can remain more text-driven. On H&M, using a combined slot-similarity and slot-logit retrieval score, MM-slotgate achieves 0.7566 macro ConstraintSatisfied@10, outperforming equal-weight multimodal fusion (0.7142) and fCLIP text-only retrieval (0.4755). The largest gain is on color, which improves from 0.321 to 0.889 (+0.568 absolute), as the learned color gate assigns 57.4% weight to image evidence. The learned gates are interpretable without modality supervision: color is image-leaning, category is text-leaning, and pattern and demographic lie near the middle. The resulting slots also remain controllable: linear probes show no measured excess leakage beyond the label-correlation baseline, and quantized slot codes support targeted intervention, including a 15.3x lift for color. These results suggest that controllable fashion retrieval benefits from typed, attribute-conditioned multimodal slots rather than either a single global embedding or opaque item-level semantic IDs.

Najmeh Forouzandehmehr, Topojoy Biswas, Evren Korpeoglu +1

Walmart Global Tech Sunnyvale, California, USA

Multimodal IRMultimodal Disentangled Representation LearningMultimodal Embedding
AskPDF
Jul 8, 2026

SynthAVE: Scalable Synthetic Labeling for E-Commerce with LLM-Arena Validation

Fine-tuning large language models (LLMs) for e-commerce attribute extraction requires labeled data representative across thousands of product types, attributes, and multiple languages. This combinatorial scale translates to millions of annotations, rendering human labeling prohibitively costly. While recent work has demonstrated synthetic label generation using LLMs, deploying such approaches at industrial scale requires integrated quality control mechanisms. We present SynthAVE, a large-scale benchmark for attribute-value verification spanning 12,726 products across 229 product types, 792 attributes, and 4 languages (Spanish, French, Italian, German). To validate synthetic labels at scale, we introduce a multi-LLM arena framework where each sample is evaluated by 21 judge configurations (7 model families ×\times 3 prompts), with final labels determined via majority voting; disagreements with the synthetic label are expert-adjudicated and agreement cases are audited on a stratified sample. The majority vote ensemble agrees with expert labels at Cohen's κ=0.92κ= 0.92 (95.0% agreement), while individual judges agree with one another only moderately (Fleiss' κ=0.76κ= 0.76)--by design, since we select for diversity. This demonstrates that diverse models with varying individual judgments aggregate into highly reliable predictions, enabling cost-effective validation at scale while concentrating expert effort on the cases where it changes the label. We estimate the resulting label quality at 97.9%.

Andrea Scarinci, Virginia Negri, Brayan Impata +3

Amazon

LLM-as-a-JudgeLLM-Assisted AnnotationDocument Information ExtractionLLM Information ExtractionSynthetic Data EvaluationProduct Attribute Extraction
AskPDF