Mine and Refine: Optimizing Graded Relevance in E-commerce Semantic Search Retrieval
Authors: Jiaqi Xi, Raghav Saboo, Luming Chen, Johny Rufus, Aditya Dodda, Ved Sampath, Kenny Chi, Elyse Winer, Akshad Viswanathan, Martin Wang, Sudeep Das
Organizations: DoorDash Inc. New York City, New York, USA · DoorDash Inc. Sunnyvale, California, USA · DoorDash Inc. USA · DoorDash Inc. San Francisco, California, USA
Embedding-based retrieval (EBR) for large-scale e-commerce search faces three intertwined challenges: graded (non-binary) relevance where engagement signals are noisy and intent-varying while business relevance guidelines admit acceptable-but-not-exact matches, false negatives in hard sample mining, and unstable similarity score separability across relevance levels, the last of which complicates hybrid search score fusion and downstream ranking. We propose Mine and Refine, a two-stage contrastive training framework that addresses all three. A lightweight LLM, fine-tuned with engagement-driven audit, serves as a guideline-aligned scalable labeler throughout training. Stage 1 establishes a robust global embedding space via label-aware supervised contrastive learning; Stage 2 mines hard samples, re-annotates them with the LLM labeler to mitigate spurious negatives, and refines the model through a multi-level extension of circle loss that enforces margin-controlled separation across relevance levels. Deployed in production e-commerce search across multiple product verticals, the approach delivers statistically significant lifts in user engagement and gross order value, and substantially improves retrieval and end-to-end relevance metrics.
Figures & tables
Figure 1. Mine and Refine Overview: Stage 1 trains a shared encoder on the original dataset using supervised contrastive (SupCon) loss. Stage 2 continues training on a curriculum dataset using circle loss with self-pacing sample weighting and a definite convergence target induced by circular decision boundary.
Figure 2. Score separability improvement from Stage 1 to Stage 2. Top row: Violin plots of similarity score distributions by relevance level. Bottom row: Distributions of query-level average margin: Ed(2)⟨q,d(j)⟩−Ed(0)⟨q,d(i)⟩,i>j,i,j∈{0,1,2} .
Recall@K
Precision@K
NDCG@K
Model
Size
Dim
10
50
100
10
50
100
10
50
100
multilingual-E5-small ( Wang et al., 2024 )
0.1B
384
-
-
-
-
-
-
-
-
-
GTE-multilingual-base dense ( Zhang et al., 2024 )
0.3B
768
4.16%
5.51%
5.35%
4.21%
5.88%
6.23%
4.06%
5.43%
5.68%
multilingual-E5-large ( Wang et al., 2024 )
0.6B
1024
4.10%
4.78%
4.58%
4.11%
4.90%
5.01%
4.13%
4.97%
5.09%
BGE-m3 dense ( Chen et al., 2024 )
0.6B
1024
-0.86%
0.98%
1.37%
-0.96%
0.88%
1.50%
-1.10%
0.52%
1.02%
Qwen3-Embedding-0.6B ( Zhang et al., 2025 )
0.6B
1024
3.28%
5.44%
5.46%
3.36%
5.71%
6.23%
3.17%
5.17%
5.59%
Table 1. Offline Model Evaluation across Metrics
System
Relevance NDCG@ 10
ATCR
CVR
GOV
Lexical baseline
-
-
-
-
Hybrid baseline
1.66%
0.23% (0.00)
0.10% (0.01)
0.17% (0.04)
Hybrid w/ Mine and Refine
1.30%
1.04% (0.00)
1.27% (0.02)
0.44% (0.03)
Table 2. Side by Side and Online A/B test results: relative improvement (%) of Mine and Refine over the baseline system
Recall@K
Precision@K
(Structure, Loss)
Cat.
10
50
100
10
50
100
(A, TL)
-
-
-
-
-
-
-
(A, CL)
-
-11.7%
-11.0%
-10.3%
-11.4%
-10.7%
-10.3%
(A, SC)
4
1.1%
2.4%
2.5%
0.8%
1.9%
2.0%
(A, SC)
-
2.6%
2.8%
2.3%
2.4%
2.5%
2.0%
(S, SC)
4
6.5%
5.5%
4.7%
6.3%
5.5%
4.7%
Table 3. Ablation Studies on Model Architecture and Item Category Enrichment
Recall@K
Precision@K
Spelling Variations
K=10
K=50
K=100
K=10
K=50
K=100
None
–
–
–
–
–
–
Regularization
-2.3%
-1.6%
-1.5%
-2.3%
-1.7%
-1.5%
In-place Substitution
-7.0%
-5.9%
-5.3%
-6.9%
-5.9%
-5.5%
Additive Augmentation
3.1%
1.3%
0.4%
3.2%
1.8%
0.9%
Table 4. Ablation Study on Spelling Augmentation
Recall@K
Precision@K
Margin
Synthetic Queries
10
50
100
10
50
100
Avg*
Worst*
None
–
–
–
–
–
–
–
–
Random
6%
3%
2%
6%
3%
3%
-13%
-38%
Pos-missing + Expanded
14%
13%
12%
14%
13%
12%
34%
-4%
Table 5. Ablation Study on Synthetic Query Enrichment
Semantic retrieval in e-commerce must handle short, noisy, and colloquial queries over large product catalogs with fine-grained attribute distinctions. We present a Siamese LLM dual-encoder trained through a two-stage pipeline: contrastive learning with a false-negative margin mask to prevent penalization of near-duplicate products, followed by Relative Odds Alignment for Retrieval (ROAR), a preference optimization objective that extends Bradley-Terry to variable-sized graded relevance groups via consecutive odds-ratio margins. The training corpus mirrors this progression - substitute query-product pairs provide coarse semantic supervision in Stage 1 and graded relevance annotations drive fine-grained ranking in Stage 2. The resulting system accurately retrieves exact matches while correctly ordering substitutes and complementary products, with gains confirmed across query-frequency strata and business verticals, and statistical significance validated through live A/B deployment at scale.
Semantic Search on LinkedIn must retrieve relevant profiles from a corpus of hundreds of millions in response to natural-language queries such as "a fintech founder in Berlin who worked in payments." The deployed relevance policy is bottleneck-oriented: every active non-negotiable facet must be satisfied, and a pre-existing LLM Graded Relevance (GR) judge operationalizes this through a fixed min/median aggregation over facet grades. Cosine similarity instead averages evidence, letting a strong match on one facet mask failure on another, capping the recall of the first-stage (L0) retriever. We present a policy-aligned retrieval framework: embeddings are partitioned into eight category-supervised segments whose scores follow the same min/median rule at serving time; for multi-vector retrieval, this segment score is computed independently per tagged document slot and maximized across slots. A lightweight single-slot Stage-1 scorer generates high-recall candidates, while scale-invariant relative-norm gating keeps category activation consistent across training, evaluation, and serving. On 21K held-out queries, this representation improves offline relevance over a matched-capacity baseline, with gains broadly distributed across facet combinations. We serve this framework with a two-stage GPU architecture: an FP8 coarse ranker scores the full corpus, increasing per-shard capacity by 71% and Stage-1 matmul throughput by 36%, then an FP16 stage exactly re-ranks an oversampled candidate set, recovering 99.6-99.8% of full-FP16 recall at over 500 QPS per shard replica. In a member-randomized A/B test, exploratory-query Precision@10 under the unchanged GR judge rises from 63.7% to 79.0% and navigational Precision@1 from 65.5% to 74.7%, with a blinded human evaluation independently confirming the Precision@10 gain.
E-commerce search requires distinguishing products that are merely related to a query from those that directly satisfy the user's shopping intent. We augment query-product pairs with structured LLM-generated query and product attributes and human-validated relevance, explanations, and centrality judgments, and evaluate these signals using a simple dual-encoder retriever and MLP re-ranker. On an augmented subset of ESCI, a human-feature oracle reaches 0.9382 nDCG@10, while a human-free trained Q+P configuration reaches 0.9258. Synthetic approximations of the human signals reach 0.9150 overall but provide substantial gains for difficult, low-performing queries. Ablations show that most of the oracle improvement comes from post-edited explanations and annotator comments rather than the scalar centrality feature, suggesting that LLMs are most useful for exposing and approximating structured semantic supervision rather than replacing human judgment directly.
Girish A. Koushik, Swapnil Bhosale, Samarth Agrawal +4
Computer Science, University of Surrey, Guildford, United Kingdom · People-Centred AI, University of Surrey, Guildford, United Kingdom · eBay USA +2