From a Large-Scale Teacher to Efficient Multimodal Students for Search Relevance Prediction

Tomasz Gniazdowski; Mohammadreza Momenifar; Nadia Vase; Shahram Khadivi; Wiktor Walentynowicz; Nicola Ueffing; Jan Pawlowski; Piotr Szymanski

cs.IROct 12, 2026

From a Large-Scale Teacher to Efficient Multimodal Students for Search Relevance Prediction

Authors: Tomasz Gniazdowski, Mohammadreza Momenifar, Nadia Vase, Shahram Khadivi, Wiktor Walentynowicz, Nicola Ueffing, Jan Pawlowski, Piotr Szymanski,

Organizations: eBay Tooploox Wroclaw, Poland · eBay San Jose, USA · eBay Aachen, Germany

Abstract

Relevance prediction between user queries and items is a fundamental component of e-commerce search systems. Traditionally, high-quality relevance labels are obtained through human judgment, which is reliable but expensive, slow, and difficult to scale. Recent advances in large language models (LLMs) enable automated relevance judgment at scale, achieving performance comparable to human annotators. However, such systems often rely on large commercial LLMs with complex prompts, resulting in high inference costs and limited throughput. In this work, we propose a framework that bridges this gap by distilling a high-performing LLM-based relevance judgment system into efficient multimodal student models. We first curate a small, high-quality human-annotated dataset to serve as ground truth. Using this dataset, we develop a reliable LLM-as-a-judge pipeline through careful prompt engineering and iterative refinement. This pipeline is then used to generate large-scale relevance labels, which serve as training data for compact multimodal models. Our distilled models achieve performance close to the teacher system while significantly improving latency, cost, and scalability. This approach enables practical deployment of high-quality relevance prediction systems in production environments.

Figures & tables

Dataset	Description
Train (483,527)	Training dataset generated with GPT-4.1 . Samples are balanced across class labels and categories.
Validation 1 (1,500)	Validation set generated with GPT-4.1 and balanced across class labels. This split contains queries that also appear in the training set (of course for different items), resulting in broader category coverage and serving as an in-distribution validation benchmark.
Validation 2 (2,466)	Validation set generated with GPT-4.1 and balanced across class labels and category names. Model selection is based on performance on this split. In contrast to Validation 1, it contains queries that do not appear in the training set, making it more challenging despite its narrower category coverage.
Test 1 (239)	Test set annotated independently by three human annotators. This split primarily contains edge cases and particularly challenging examples. This set is used in the process of optimizing prompt for the LLM-as-a-judge pipeline.
Test 2 (233)	Test set annotated by a single human annotator. This dataset is completely independent of the prompt engineering process for LLM-as-a-judge .

Table 1. Summary of the datasets used in our experiments, including split sizes, construction protocol, and annotation procedure. Validation 1 and Validation 2 differ in their degree of overlap with the training dataset queries, while Test 1 and Test 2 are human-annotated evaluation sets. Number of samples in each dataset is provided inside brackets.

	GPT-4.1		modernBERT-large-RLV		Gemma-3-IT-RLV (our)
	Teacher		Encoder Baseline		4B		12B		27B
Dataset	acc [%]	F1 [%]	acc [%]	F1 [%]	acc [%]	F1 [%]	acc [%]	F1 [%]	acc [%]	F1 [%]
Validation 1	—	—	69.93	69.80	79.33	79.22	82.33	82.39	81.47	81.50
Validation 2	—	—	72.30	71.74	79.52	79.11	82.20	81.82	83.33	82.96
Test 1	83.68	81.30	74.06	61.29	80.34	76.19	81.59	77.78	81.59	78.71
Test 2	64.38	56.09	57.08	47.40	63.95	56.47	65.24	57.56	62.23	54.37

Table 2. Accuracy and F1 macro obtained for the finetuned models for search relevance classification for the Gemma-3-IT-RLV family models compared to the teacher GPT-4.1 model on all evaluation datasets.

Dataset	Model	Bad [%]	Fair [%]	Good [%]	Excellent [%]
Test 1	GPT-4.1	100.00	80.70	53.85	90.63
	Gemma-3-4B-IT-RLV (our)	92.31	76.00	47.62	88.82
	Gemma-3-12B-IT-RLV (our)	83.33	84.62	54.35	88.82
	Gemma-3-27B-IT-RLV (our)	92.31	82.35	51.16	89.02
Test 2	GPT-4.1	57.78	47.19	36.56	82.85
	Gemma-3-4B-IT-RLV (our)	53.33	50.00	41.18	81.39

Table 3. F1 metric values for search relevance classification for the Gemma-3-IT-RLV family models compared to the teacher GPT-4.1 model per class on Test 1 and Test 2 datasets.

Panel of Figure 1 — Figure 1. Confusion matrices for teacher and Gemma-3-12B-IT-RLV models on Test 1 dataset.

Panel of Figure 2 — Figure 2. Confusion matrices for teacher and Gemma-3-12B-IT-RLV models on Test 2 dataset.

Model	Accuracy
Gemma-3-4B-IT-RLV	(80.57 $\pm$ 0.21)%
Gemma-3-12B-IT-RLV	(81.63 $\pm$ 0.29)%
Gemma-3-27B-IT-RLV	(81.38 $\pm$ 0.25)%

Table 4. Mean accuracy and standard deviation values for all finetuned models for 20 independent model runs for search relevance classification.

Dataset	Model	chrf [%]	BS F1 [%]
Val 1	Gemma-3-4B-IT-RLV	67.15	93.78
	Gemma-3-12B-IT-RLV	67.97	93.89
	Gemma-3-27B-IT-RLV	68.42	93.98
Val 2	Gemma-3-4B-IT-RLV	66.22	93.45
	Gemma-3-12B-IT-RLV	67.48	93.70
	Gemma-3-27B-IT-RLV	67.68	93.79

Table 5. Models explanation quality—chrf and BERTScore F1 mean metric values (marked as BS F1 in Table) for all finetuned models from the Gemma-3-IT-RLV family calculated on Validation 1 (Val 1) and Validation 2 (Val 2) datasets.

Dataset	Gemma-3-12B-IT-RLV
	FP16	FP8
Validation 1 [%]	82.33	82.00 [-0.33]
Validation 2 [%]	82.20	81.87 [-0.33]
Test 1 [%]	81.59	81.17 [-0.42]
Test 2 [%]	65.24	65.67 [+0.43]
TP w/o explan. [req/s]	194.52	271.54 [+77.02]

Table 6. Accuracy [%] and throughput for the Gemma-3-12B-IT-RLV models quantized to FP16 and FP8. Throughput (marked in Table as TP) is reported in requests per second [req/s]. Experiments are conducted on a single NVIDIA H100 GPU.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Label	Bad	Fair	Good	Excellent
Count	6	26	43	164

Appendix

Table 7. Label distribution for the multi-annotator dataset (239 samples, 3 judges per sample).

Label	Bad	Fair	Good	Excellent
Count	25	44	53	111

Appendix

Table 8. Label distribution for the single-annotator dataset (233 samples, 1 judge per sample).

Weights	Model	Test 1 [%]
closed-source	GPT-5	80.33
	GPT-4.1	83.68
	GPT-4o	79.92
	GPT-4.1-mini	74.06
	GPT-4.1-nano	55.65
open-source	Gemma-3-27B-IT	68.20

Appendix

Table 9. Accuracy scores [%] on the Test 1 dataset for the commercial and open-source LLMs. All models are prompted using the prompt described in Section 3.2 .

Dataset	chrf	BS F1	LLM judge
Dataset	chrf	BS F1	same	similar	different
Val 1	75.07	95.25	875 (58.3%)	590 (39.3%)	35 (2.3%)
Val 2	74.81	95.06	1532 (62.1%)	852 (34.6%)	82 (3.3%)
Test 1	75.27	95.17	142 (59.4%)	91 (38.1%)	6 (2.5%)
Test 2	73.27	94.79	125 (53.7%)	99 (42.5%)	9 (3.9%)

Appendix

Table 10. Models explanation quality—chrf, BERTScore F1 mean metric (marked in Table as BS F1) values and LLM-as-a-judge results. Here the lower-case Gemma-3-27B-IT-RLV predictions are set as labels and lower-case predictions of Gemma-3-12B-IT-RLV as predictions on Validation 1 and Validation 2 datasets (marked as Val 1 and Val 2 in Table) and Test 1 and Test 2 datasets.

Hyperparameter	Gemma-3-IT-RLV
Hyperparameter	4B	12B	27B
Learning rate	$2e^{-5}$	$2e^{-5}$	$2e^{-5}$
LR scheduler	cosine	cosine	cosine
Weight decay	0.1	0.1	0.1
Gradient clip	1.0	1.0	1.0
Optimizer	AdamW	AdamW	AdamW

Appendix

Table 11. The best hyperparameter sets found for all analyzed models. Hyperparameters for AdamW optimizer

\beta_{1}

and

\beta_{2}

are presented in the brackets.

Model name	Metric	Gemma-3-4B-IT-RLV		Gemma-3-12B-IT-RLV		Gemma-3-27B-IT-RLV
Train dataset version	Metric	v1	v2	v1	v2	v1	v2
Validation 1	acc [%]	78.00	79.00 [+1.00 $\uparrow$ ]	80.13	81.27 [+1.14 $\uparrow$ ]	80.67	81.20 [+0.53 $\uparrow$ ]
Test 1	acc [%]	76.15	79.50 [+3.35 $\uparrow$ ]	75.73	79.92 [+4.19 $\uparrow$ ]	78.24	79.50 [+1.26 $\uparrow$ ]
Test 1	F1 [%]	68.92	71.03 [+2.11 $\uparrow$ ]	64.20	76.96 [+12.76 $\uparrow$ ]	69.85	76.82 [+6.97 $\uparrow$ ]
Test 2	acc [%]	61.80	61.80 [-]	65.24	62.66 [-2.58 $\downarrow$ ]	63.09	64.38 [+1.29 $\uparrow$ ]

Appendix

Table 12. The influence of the dataset version (number of datapoints and their diversity) on the search relevance classification quality.

Model name	Metric	Gemma-3-4B-IT-RLV		Gemma-3-12B-IT-RLV		Gemma-3-27B-IT-RLV
Max image edge length	Metric	300 px	900 px	300 px	900 px	300 px	900 px
Validation 1	acc [%]	79.00	79.33 [+0.33 $\uparrow$ ]	81.27	82.33 [+1.06 $\uparrow$ ]	81.20	81.47 [+0.27 $\uparrow$ ]
Validation 2	acc [%]	79.40	79.52 [+0.12 $\uparrow$ ]	81.96	82.20 [+0.24 $\uparrow$ ]	81.83	83.33 [+1.50 $\uparrow$ ]
Test 1	acc [%]	79.50	80.34 [+0.84 $\uparrow$ ]	79.92	81.59 [+1.67 $\uparrow$ ]	79.50	81.59 [+2.09 $\uparrow$ ]
Test 1	F1 [%]	71.03	76.19 [+5.16 $\uparrow$ ]	76.96	77.78 [+0.82 $\uparrow$ ]	76.82	78.71 [+1.89 $\uparrow$ ]
Test 2	acc [%]	61.80	63.95 [+2.15 $\uparrow$ ]	62.66	65.24 [+2.58 $\uparrow$ ]	64.38	62.23 [-2.15 $\downarrow$ ]

Appendix

Table 13. The influence of the dataset’s maximum image edge length (300 pixels and 900 pixels) on the search relevance classification quality.

Model name	Metric	Gemma-3-4B-IT-RLV		Gemma-3-12B-IT-RLV		Gemma-3-27B-IT-RLV
Explanation generation	Metric	w/o	w/	w/o	w/	w/o	w/
Validation 1	acc [%]	78.20	79.33 [+1.13 $\uparrow$ ]	78.67	82.33 [+3.66 $\uparrow$ ]	80.60	81.47 [+0.87 $\uparrow$ ]
Validation 2	acc [%]	77.45	79.52 [+2.07 $\uparrow$ ]	80.45	82.20 [+1.75 $\uparrow$ ]	80.82	83.33 [+2.51 $\uparrow$ ]
Test 1	acc [%]	79.08	80.34 [+1.26 $\uparrow$ ]	80.33	81.59 [+1.26 $\uparrow$ ]	80.33	81.59 [+1.26 $\uparrow$ ]
Test 1	F1 [%]	68.32	76.19 [+7.87 $\uparrow$ ]	72.62	77.78 [+5.16 $\uparrow$ ]	70.47	78.71 [+8.24 $\uparrow$ ]
Test 2	acc [%]	60.09	63.95 [+3.86 $\uparrow$ ]	65.24	65.24 [-]	63.95	62.23 [-1.72 $\downarrow$ ]

Appendix

Table 14. The influence of finetuning the models on the examples without and with explanations on the search relevance classification quality. Models trained without explanations are trained only on the relevance scores.

Model name	Metric	Gemma-3-4B-IT-RLV		Gemma-3-12B-IT-RLV		Gemma-3-27B-IT-RLV
Image usage	Metric	w/o	w/	w/o	w/	w/o	w/
Validation 1	acc [%]	71.80	78.20 [+6.4 $\uparrow$ ]	77.60	78.67 [+1.07 $\uparrow$ ]	79.13	80.60 [+1.47 $\uparrow$ ]
Validation 2	acc [%]	74.17	77.45 [+3.28 $\uparrow$ ]	78.26	80.45 [+2.19 $\uparrow$ ]	79.76	80.82 [+1.06 $\uparrow$ ]
Test 1	acc [%]	72.38	79.08 [+6.7 $\uparrow$ ]	78.66	80.33 [+1.67 $\uparrow$ ]	77.41	80.33 [+2.92 $\uparrow$ ]
Test 1	F1 [%]	62.24	68.32 [+6.08 $\uparrow$ ]	65.33	72.62 [+7.29 $\uparrow$ ]	70.09	70.47 [+0.38 $\uparrow$ ]
Test 2	acc [%]	61.37	60.09 [-1.28 $\downarrow$ ]	63.09	65.24 [+2.15 $\uparrow$ ]	61.80	63.95 [+2.15 $\uparrow$ ]

Appendix

Table 15. The influence of finetuning the models without and with item images on the search relevance classification quality. Since the explanation contains information from the item images, both models are trained without explanations.

Model name	GPT-4.1	Gemma-3-12B-IT-RLV-FP8
Setup	—	w/o explan.	w/ explan.
1M in. tokens	$2	$0.0065-$ 0.0108	$0.0234-$ 0.0391
1M out. tokens	$8	$0.0065-$ 0.0108	$0.0234-$ 0.0391

Appendix

Table 16. Comparison of cost per 1 million input and 1 million output tokens for GPT-4.1 and hosted Gemma-3-12B-IT-RLV-FP8 on an external GPU provider in the label-only- and explanation-generation setups.

Information Retrieval

Information Retrieval studies how to find, rank, and organize relevant information in large collections of data.

See all 476 papers

Advancing Relevance Measurement with Vision-Language Models for Web-Scale Search
Aug 3, 2026Han Wang, Alex Whitworth, Pak Ming Cheung +5VLM Evaluation Automated Evaluation
Pinterest · San Francisco, CA, USA
AutoRelAnnotator: Calibrated Model Cascades for Cost-Efficient Relevance Evaluation in Sponsored Search
Jun 24, 2026Md Omar Faruk Rokon, Shasvat Desai, Hong Yao +1Online Advertising Cost-Aware Inference
Walmart Global Tech, Sunnyvale, CA, USA
TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance
Oct 9, 2025Jianhui Yang, Yiming Jin, Pengkun Jiao +6RL for Language Models Reward Shaping
Tsinghua University Beijing, China · Taobao & Tmall Group of Alibaba Beijing, China · Taobao & Tmall Group of Alibaba Hangzhou, China +1

Aug 3, 2026J/K move · Enter open · S save

Advancing Relevance Measurement with Vision-Language Models for Web-Scale Search

Han Wang, Alex Whitworth, Pak Ming Cheung, Zhenjie Zhang+4

Pinterest · San Francisco, CA, USA

VLM Evaluation Automated Evaluation IR Evaluation

Open paper AskPDF arXiv

Relevance evaluation plays a crucial role in personalized search systems, serving as a guardrail alongside user engagement metrics to ensure that search results align with user queries and intent. While human annotation is the traditional method for relevance evaluation, its high cost and long turnaround time limit its scalability. In this work, we present a VLM-based automated relevance evaluation pipeline deployed within Pinterest Search for online A/B experiments. We rigorously validate the alignment between VLM-generated judgments and human annotations, demonstrating that VLMs can provide reliable relevance measurement for experiments while greatly improving the evaluation efficiency. Leveraging VLM-based labeling further unlocks opportunities to expand the query set, optimize sampling design, and efficiently assess a wider range of search experiences at scale. This approach leads to higher-quality relevance metrics and significantly reduces the Minimum Detectable Effects (MDEs) in online experiment measurements.

Abstract

Information Retrieval

Explore similar work

Advancing Relevance Measurement with Vision-Language Models for Web-Scale Search

AutoRelAnnotator: Calibrated Model Cascades for Cost-Efficient Relevance Evaluation in Sponsored Search

TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance