Mine and Refine: Optimizing Graded Relevance in E-commerce Semantic Search Retrieval
Authors: Jiaqi Xi, Raghav Saboo, Luming Chen, Johny Rufus, Aditya Dodda, Ved Sampath, Kenny Chi, Elyse Winer, Akshad Viswanathan, Martin Wang, Sudeep Das
Organizations: DoorDash Inc. New York City, New York, USA · DoorDash Inc. Sunnyvale, California, USA · DoorDash Inc. USA · DoorDash Inc. San Francisco, California, USA
Embedding-based retrieval (EBR) for large-scale e-commerce search faces three intertwined challenges: graded (non-binary) relevance where engagement signals are noisy and intent-varying while business relevance guidelines admit acceptable-but-not-exact matches, false negatives in hard sample mining, and unstable similarity score separability across relevance levels, the last of which complicates hybrid search score fusion and downstream ranking. We propose Mine and Refine, a two-stage contrastive training framework that addresses all three. A lightweight LLM, fine-tuned with engagement-driven audit, serves as a guideline-aligned scalable labeler throughout training. Stage 1 establishes a robust global embedding space via label-aware supervised contrastive learning; Stage 2 mines hard samples, re-annotates them with the LLM labeler to mitigate spurious negatives, and refines the model through a multi-level extension of circle loss that enforces margin-controlled separation across relevance levels. Deployed in production e-commerce search across multiple product verticals, the approach delivers statistically significant lifts in user engagement and gross order value, and substantially improves retrieval and end-to-end relevance metrics.
Figures & tables
Figure 1. Mine and Refine Overview: Stage 1 trains a shared encoder on the original dataset using supervised contrastive (SupCon) loss. Stage 2 continues training on a curriculum dataset using circle loss with self-pacing sample weighting and a definite convergence target induced by circular decision boundary.
Figure 2. Score separability improvement from Stage 1 to Stage 2. Top row: Violin plots of similarity score distributions by relevance level. Bottom row: Distributions of query-level average margin: Ed(2)⟨q,d(j)⟩−Ed(0)⟨q,d(i)⟩,i>j,i,j∈{0,1,2} .
Recall@K
Precision@K
NDCG@K
Model
Size
Dim
10
50
100
10
50
100
10
50
100
multilingual-E5-small ( Wang et al., 2024 )
0.1B
384
-
-
-
-
-
-
-
-
-
GTE-multilingual-base dense ( Zhang et al., 2024 )
0.3B
768
4.16%
5.51%
5.35%
4.21%
5.88%
6.23%
4.06%
5.43%
5.68%
multilingual-E5-large ( Wang et al., 2024 )
0.6B
1024
4.10%
4.78%
4.58%
4.11%
4.90%
5.01%
4.13%
4.97%
5.09%
BGE-m3 dense ( Chen et al., 2024 )
0.6B
1024
-0.86%
0.98%
1.37%
-0.96%
0.88%
1.50%
-1.10%
0.52%
1.02%
Qwen3-Embedding-0.6B ( Zhang et al., 2025 )
0.6B
1024
3.28%
5.44%
5.46%
3.36%
5.71%
6.23%
3.17%
5.17%
5.59%
Table 1. Offline Model Evaluation across Metrics
System
Relevance NDCG@ 10
ATCR
CVR
GOV
Lexical baseline
-
-
-
-
Hybrid baseline
1.66%
0.23% (0.00)
0.10% (0.01)
0.17% (0.04)
Hybrid w/ Mine and Refine
1.30%
1.04% (0.00)
1.27% (0.02)
0.44% (0.03)
Table 2. Side by Side and Online A/B test results: relative improvement (%) of Mine and Refine over the baseline system
Recall@K
Precision@K
(Structure, Loss)
Cat.
10
50
100
10
50
100
(A, TL)
-
-
-
-
-
-
-
(A, CL)
-
-11.7%
-11.0%
-10.3%
-11.4%
-10.7%
-10.3%
(A, SC)
4
1.1%
2.4%
2.5%
0.8%
1.9%
2.0%
(A, SC)
-
2.6%
2.8%
2.3%
2.4%
2.5%
2.0%
(S, SC)
4
6.5%
5.5%
4.7%
6.3%
5.5%
4.7%
Table 3. Ablation Studies on Model Architecture and Item Category Enrichment
Recall@K
Precision@K
Spelling Variations
K=10
K=50
K=100
K=10
K=50
K=100
None
–
–
–
–
–
–
Regularization
-2.3%
-1.6%
-1.5%
-2.3%
-1.7%
-1.5%
In-place Substitution
-7.0%
-5.9%
-5.3%
-6.9%
-5.9%
-5.5%
Additive Augmentation
3.1%
1.3%
0.4%
3.2%
1.8%
0.9%
Table 4. Ablation Study on Spelling Augmentation
Recall@K
Precision@K
Margin
Synthetic Queries
10
50
100
10
50
100
Avg*
Worst*
None
–
–
–
–
–
–
–
–
Random
6%
3%
2%
6%
3%
3%
-13%
-38%
Pos-missing + Expanded
14%
13%
12%
14%
13%
12%
34%
-4%
Table 5. Ablation Study on Synthetic Query Enrichment