SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generation with LLMs

Haotian Chen; Qingqing Long; Meng Xiao; Xiao Luo; Wei Ju; Chengrui Wang; Yuanchun Zhou; Xuezhi Wang

cs.CLAug 12, 2025

SciRerankBench: Benchmarking Rerankers Towards Scientific Retrieval-Augmented Generation with LLMs

Authors: Haotian Chen, Qingqing Long, Meng Xiao, Xiao Luo, Wei Ju, Chengrui Wang, Yuanchun Zhou, Xuezhi Wang,

Organizations: Computer Network Information Center, Chinese Academy of Sciences · University of California, Los Angeles · Peking University

Abstract

Scientific literature question answering is a pivotal step towards new scientific discoveries. Recently, two-stage retrieval-augmented generated large language models (RAG-LLMs) have shown impressive advancements in this domain. Such a two-stage framework, especially the second stage (reranker), is particularly essential in the scientific domain, where subtle differences in terminology may have a greatly negative impact on the final factual-oriented or knowledge-intensive answers. Despite this significant progress, the potential and limitations of these works remain unexplored. In this work, we present a Scientific Rerank-oriented RAG Benchmark (SciRerankBench), for evaluating rerankers within RAG-LLMs systems, spanning five scientific subjects. To rigorously assess the reranker performance in terms of noise resilience, relevance disambiguation, and factual consistency, we develop three types of question-context-answer (Q-C-A) pairs, i.e., Noisy Contexts (NC), Semantically Similar but Logically Irrelevant Contexts (SSLI), and Counterfactual Contexts (CC). Through systematic evaluation of 13 widely used rerankers on five families of LLMs, we provide detailed insights into their relative strengths and limitations. To the best of our knowledge, SciRerankBench is the first benchmark specifically developed to evaluate rerankers within RAG-LLMs, which provides valuable observations and guidance for their future development.

Figures & tables

Figure 1: Overall architecture of the two-stage RAG-LLM pipeline. The first stage conducts fast but low-precision ( embedding model ) retrievals, while the second stage ( rerankers ) emphasizes high-precision but higher computational costs for a trade-off between effectiveness and efficiency in RAG-LLMs. This two-stage setup first narrows the candidate set efficiently, then reranks results for higher quality generation.

Type	Work	Source Data	Scientific Field	Knowledge Level	Rerank Oriented	Dataset Size
Dataset	NQ Kwiatkowski et al. (2019)	Google search log	✗	General	✗	307k (train), 7.8k (dev/test)
	HotpotQA Yang et al. (2018)	Wikipedia	✗	General	✗	113k
	PubMedQA Jin et al. (2019)	PubMed abstracts	Bio.	Expert	✗	1k expert, 61k unlabeled, 211k generated
Toolkit	FlashRAG Jin et al. (2025)	Public benchmarks	✗	General	✓	unknown
Toolkit	rankify Abdallah et al. (2026)	Public benchmarks	✗	General	✓	unknown
Bench.	CRAG Yang et al. (2024)	Web search log	✗	General	✗	4.4K

Table 1: Comparison of SciRerankBench among related datasets, toolkits and benchmarks.

Figure 2: Overview of the synthetic dataset construction process with multi-hop reasoning and diverse distractors.

Task	Sem. Sim.	Fluency	Coherence	Distract.	Fact. Acc.
NC	–	4.81	4.91	–	–
SSLI	0.83	5.00	4.77	4.34	2.08
CC	–	4.94	4.38	4.75	1.17
Multi-Hop	–	4.63	4.77	–	–

Table 2: Human evaluation results for four task settings. Metrics not applicable to a given task are marked as “-”.

Figure 3: Evaluation results under 3 different settings, averaged across all evaluated rerankers: without any context, with retrieved contexts, and with contexts after applying the reranking stage by recall scores. Error bars represent standard deviation across rerankers.

Figure 4: Evaluation of rerankers at generation stage on five tasks using Recall metric.

Figure 5: Evaluation of rerankers at rerank stage on five tasks using Recall@10 metric.

Figure 6: Running efficiency analysis of rerankers.

Figure 7: Visualization of question-context relevance (before and after rerankering) under 4 evaluation tasks.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Subject	Task Type	Number
Biology	NC	2499
	Base	2499
	CC	2496
	Multi-Hop	1246
	SSLI	2497
Math	NC	2494

Appendix

Table 3: QA Counts for different subjects and tasks.

Reranker	Subject	Multi-Hop	NC	CC	SSLI	Base
LLM2Vec	Bio.	33.72 $\pm$ 0.36	32.77 $\pm$ 0.45	31.78 $\pm$ 0.52	31.34 $\pm$ 0.61	31.46 $\pm$ 0.50
	Geo.	44.34 $\pm$ 3.09	41.84 $\pm$ 2.70	38.45 $\pm$ 2.10	37.06 $\pm$ 1.85	35.99 $\pm$ 1.75
	Chem.	55.28 $\pm$ 0.84	53.75 $\pm$ 0.88	51.35 $\pm$ 1.10	49.39 $\pm$ 1.05	49.45 $\pm$ 0.90
	Phy.	50.18 $\pm$ 0.83	47.82 $\pm$ 0.75	44.42 $\pm$ 0.82	43.96 $\pm$ 0.65	45.69 $\pm$ 0.88
	Math.	33.04 $\pm$ 0.66	29.49 $\pm$ 1.10	31.05 $\pm$ 0.92	29.26 $\pm$ 1.15	28.99 $\pm$ 1.90
Rearank	Bio.	49.98 $\pm$ 0.79	48.57 $\pm$ 1.27	47.11 $\pm$ 0.81	46.48 $\pm$ 2.16	46.69 $\pm$ 0.94

Appendix

Table 4: Performance of LLM-based and Agent-based rerankers across various evaluation tasks (Recall)

Model	Family	Size
Mistral-7B-v0.2	Mistral	7B
Mistral-24B	Mistral	24B
LLaMA-7B	Meta (LLaMA)	7B
LLaMA2-70B	Meta (LLaMA2)	70B
DeepSeek-V3-7B	DeepSeek	7B
DeepSeek-V3-671B	DeepSeek	671B

Appendix

Table 5: LLMs evaluated in this paper.

Model	Category	Reranking Strategy
BGE	Transformer	Cross-Encoder
Jina	Transformer	Cross-Encoder
BCE	Transformer	Cross-Encoder
MXBAI	Transformer	Cross-Encoder
MiniLM	Sentence Transformer	distilled self-attention
ColBert	BERT	late interaction mechanism

Appendix

Table 6: RAG reranking methods evaluated in this paper.

Reranker	Subject	Multi-Hop	NC	CC	SSLI	Base
LLM2Vec	Bio.	85.87 $\pm$ 0.82	70.18 $\pm$ 2.05	40.39 $\pm$ 2.09	40.09 $\pm$ 1.89	76.20 $\pm$ 1.09
	Geo.	83.80 $\pm$ 0.75	68.00 $\pm$ 1.85	38.50 $\pm$ 1.90	38.20 $\pm$ 1.60	74.00 $\pm$ 1.10
	Chem.	84.90 $\pm$ 0.70	69.05 $\pm$ 1.95	39.20 $\pm$ 1.85	39.10 $\pm$ 1.70	75.10 $\pm$ 1.05
	Phy.	81.60 $\pm$ 0.85	66.50 $\pm$ 1.50	37.50 $\pm$ 1.60	37.40 $\pm$ 1.45	72.20 $\pm$ 1.25
	Math.	82.70 $\pm$ 0.80	67.20 $\pm$ 1.65	37.90 $\pm$ 1.70	37.80 $\pm$ 1.50	73.10 $\pm$ 1.20
Rearank	Bio.	70.35 $\pm$ 0.43	72.68 $\pm$ 0.31	68.42 $\pm$ 0.64	71.43 $\pm$ 0.85	70.33 $\pm$ 1.92

Appendix

Table 7: Performance of LLM-based and Agent-based rerankers across various evaluation tasks (Recall@10).

Reranker	Subject	Multi-Hop	NC	CC	SSLI	Base
LLM2Vec	Bio.	68.49 $\pm$ 7.41	66.67 $\pm$ 6.55	21.43 $\pm$ 4.21	15.78 $\pm$ 3.28	64.62 $\pm$ 6.21
	Geo.	69.29 $\pm$ 7.41	63.80 $\pm$ 6.55	20.78 $\pm$ 4.21	14.93 $\pm$ 3.28	59.84 $\pm$ 6.21
	Chem.	66.66 $\pm$ 7.41	62.82 $\pm$ 6.55	20.26 $\pm$ 4.21	14.63 $\pm$ 3.28	59.99 $\pm$ 6.21
	Phy.	66.62 $\pm$ 7.41	63.31 $\pm$ 6.55	20.38 $\pm$ 4.21	14.78 $\pm$ 3.28	61.17 $\pm$ 6.21
	Math.	67.56 $\pm$ 7.41	62.62 $\pm$ 6.55	19.92 $\pm$ 4.21	14.52 $\pm$ 3.28	62.00 $\pm$ 6.21
Rearank	Bio.	38.65 $\pm$ 3.09	63.26 $\pm$ 0.96	54.85 $\pm$ 0.67	55.01 $\pm$ 0.84	51.81 $\pm$ 2.01

Appendix

Table 8: Performance of LLM-based and Agent-based rerankers across various evaluation tasks (Recall@5).

Reranker	Subject	Multi-Hop	NC	CC	SSLI	Base
LLM2Vec	Bio.	51.60 $\pm$ 7.85	55.55 $\pm$ 5.61	18.85 $\pm$ 3.36	14.83 $\pm$ 1.91	58.55 $\pm$ 5.56
	Geo.	50.86 $\pm$ 7.85	54.03 $\pm$ 5.61	18.14 $\pm$ 3.36	13.94 $\pm$ 1.91	53.93 $\pm$ 5.56
	Chem.	49.96 $\pm$ 7.85	53.27 $\pm$ 5.61	18.32 $\pm$ 3.36	13.94 $\pm$ 1.91	55.07 $\pm$ 5.56
	Phy.	50.22 $\pm$ 7.85	53.71 $\pm$ 5.61	18.71 $\pm$ 3.36	14.13 $\pm$ 1.91	55.37 $\pm$ 5.56
	Math.	51.10 $\pm$ 7.85	53.46 $\pm$ 5.61	18.12 $\pm$ 3.36	13.80 $\pm$ 1.91	56.48 $\pm$ 5.56
Rearank	Bio.	36.57 $\pm$ 2.69	48.28 $\pm$ 1.07	51.16 $\pm$ 0.62	51.82 $\pm$ 0.82	48.91 $\pm$ 2.01

Appendix

Table 9: Performance of LLM-based and Agent-based rerankers across various evaluation tasks (AP).

Reranker	Subject	Multi-Hop	NC	CC	SSLI	Base
LLM2Vec	Bio.	0.52 $\pm$ 0.08	0.55 $\pm$ 0.06	0.19 $\pm$ 0.05	0.14 $\pm$ 0.01	0.58 $\pm$ 0.06
	Geo.	0.51 $\pm$ 0.08	0.54 $\pm$ 0.06	0.18 $\pm$ 0.05	0.13 $\pm$ 0.01	0.54 $\pm$ 0.06
	Chem.	0.49 $\pm$ 0.08	0.53 $\pm$ 0.06	0.18 $\pm$ 0.05	0.13 $\pm$ 0.01	0.54 $\pm$ 0.06
	Phy.	0.50 $\pm$ 0.08	0.53 $\pm$ 0.06	0.18 $\pm$ 0.05	0.13 $\pm$ 0.01	0.55 $\pm$ 0.06
	Math.	0.51 $\pm$ 0.08	0.53 $\pm$ 0.06	0.18 $\pm$ 0.05	0.13 $\pm$ 0.01	0.56 $\pm$ 0.06
Rearank	Bio.	0.38 $\pm$ 0.03	0.54 $\pm$ 0.01	0.52 $\pm$ 0.01	0.53 $\pm$ 0.01	0.50 $\pm$ 0.02

Appendix

Table 10: Performance of LLM-based and Agent-based rerankers across various evaluation tasks (MRR).

Panel of Figure 8 — Figure 8: Prompt used in the Rerank Evaluation Task

Reranker	Subject	Multi-Hop	NC	CC	SSLI	Base
BCE	Bio.	55.08 $\pm$ 2.62	53.57 $\pm$ 2.20	51.66 $\pm$ 0.48	51.89 $\pm$ 1.40	63.48 $\pm$ 0.57
	Geo.	52.44 $\pm$ 1.23	51.84 $\pm$ 1.53	48.94 $\pm$ 1.01	48.14 $\pm$ 1.73	60.20 $\pm$ 0.42
	Chem.	53.62 $\pm$ 2.08	52.65 $\pm$ 0.81	49.74 $\pm$ 1.22	49.44 $\pm$ 2.50	59.61 $\pm$ 0.75
	Phy.	49.77 $\pm$ 0.90	51.93 $\pm$ 1.66	45.32 $\pm$ 1.12	46.05 $\pm$ 1.45	61.94 $\pm$ 0.74
	Math.	52.51 $\pm$ 1.48	52.38 $\pm$ 0.18	49.26 $\pm$ 1.47	47.84 $\pm$ 1.52	62.14 $\pm$ 2.30
BGE	Bio.	54.55 $\pm$ 1.70	53.56 $\pm$ 2.00	55.52 $\pm$ 0.79	54.56 $\pm$ 1.16	64.74 $\pm$ 0.71

Appendix

Table 11: Performance of different rerankers across various evaluation tasks (Qwen-70B,Recall)

Reranker	Subject	Multi-Hop	NC	CC	SSLI	Base
BCE	Bio.	48.97 $\pm$ 1.83	51.82 $\pm$ 0.48	49.73 $\pm$ 0.89	50.50 $\pm$ 1.35	60.57 $\pm$ 0.48
	Geo.	50.32 $\pm$ 1.20	51.94 $\pm$ 0.70	47.85 $\pm$ 0.95	48.30 $\pm$ 1.40	59.12 $\pm$ 0.65
	Chem.	50.40 $\pm$ 1.15	51.40 $\pm$ 0.60	48.92 $\pm$ 1.05	49.40 $\pm$ 1.20	58.84 $\pm$ 0.73
	Phy.	48.36 $\pm$ 1.32	51.55 $\pm$ 0.88	45.21 $\pm$ 0.90	46.82 $\pm$ 1.48	61.78 $\pm$ 1.12
	Math.	49.55 $\pm$ 1.18	51.68 $\pm$ 1.12	49.50 $\pm$ 1.00	48.96 $\pm$ 1.33	60.88 $\pm$ 1.40
BGE	Bio.	52.08 $\pm$ 1.66	52.84 $\pm$ 0.36	52.32 $\pm$ 1.25	53.63 $\pm$ 1.10	60.95 $\pm$ 0.74

Appendix

Table 12: Performance of different rerankers across various evaluation tasks (LLaMA2-70B,Recall)

Reranker	Subject	Multi-Hop	NC	CC	SSLI	Base
BCE	Bio.	92.09 $\pm$ 0.50	73.17 $\pm$ 0.45	74.57 $\pm$ 0.58	74.77 $\pm$ 0.49	86.64 $\pm$ 0.72
	Geo.	88.83 $\pm$ 1.49	68.72 $\pm$ 1.32	70.48 $\pm$ 1.92	70.44 $\pm$ 0.69	81.05 $\pm$ 1.47
	Chem.	90.56 $\pm$ 0.23	69.93 $\pm$ 1.50	71.62 $\pm$ 1.00	71.67 $\pm$ 1.84	82.42 $\pm$ 1.86
	Phy.	86.20 $\pm$ 0.95	66.72 $\pm$ 0.37	68.40 $\pm$ 0.75	68.12 $\pm$ 0.50	78.95 $\pm$ 1.20
	Math.	87.17 $\pm$ 1.31	67.77 $\pm$ 1.12	69.12 $\pm$ 1.18	69.20 $\pm$ 0.75	79.67 $\pm$ 1.32
BGE	Bio.	92.75 $\pm$ 0.52	73.55 $\pm$ 0.76	79.33 $\pm$ 0.70	80.53 $\pm$ 0.43	88.03 $\pm$ 0.96

Appendix

Table 13: Performance of different rerankers across various evaluation tasks Recall@10

Reranker	Subject	Multi-Hop	NC	CC	SSLI	Base
BCE	Bio.	69.41 $\pm$ 1.42	62.68 $\pm$ 0.65	52.73 $\pm$ 0.50	52.01 $\pm$ 1.57	71.76 $\pm$ 1.39
	Geo.	70.05 $\pm$ 1.35	61.89 $\pm$ 0.99	52.79 $\pm$ 1.16	53.03 $\pm$ 2.45	67.81 $\pm$ 1.09
	Chem.	67.62 $\pm$ 1.96	60.37 $\pm$ 1.56	49.18 $\pm$ 0.52	50.17 $\pm$ 1.86	68.47 $\pm$ 1.48
	Phy.	68.47 $\pm$ 3.12	60.83 $\pm$ 1.01	48.63 $\pm$ 0.63	50.56 $\pm$ 0.52	70.04 $\pm$ 2.28
	Math.	68.45 $\pm$ 0.84	60.52 $\pm$ 2.11	46.52 $\pm$ 1.28	45.62 $\pm$ 2.95	69.61 $\pm$ 2.18
BGE	Bio.	72.45 $\pm$ 1.19	62.98 $\pm$ 0.92	60.40 $\pm$ 1.27	60.50 $\pm$ 1.63	73.88 $\pm$ 1.62

Appendix

Table 14: Performance of different rerankers across various evaluation tasks (Recall@5)

Reranker	Subject	Multi-Hop	NC	CC	SSLI	Base
BCE	Bio.	61.40 $\pm$ 1.08	59.29 $\pm$ 0.39	45.74 $\pm$ 1.20	44.97 $\pm$ 1.71	69.50 $\pm$ 2.02
	Geo.	60.27 $\pm$ 0.98	59.95 $\pm$ 0.85	44.67 $\pm$ 0.61	45.82 $\pm$ 1.57	66.07 $\pm$ 0.50
	Chem.	58.74 $\pm$ 1.62	58.10 $\pm$ 1.69	43.67 $\pm$ 1.15	43.67 $\pm$ 2.20	67.43 $\pm$ 1.46
	Phy.	59.06 $\pm$ 4.33	58.53 $\pm$ 0.72	43.55 $\pm$ 0.66	44.79 $\pm$ 1.08	68.55 $\pm$ 1.90
	Math.	59.58 $\pm$ 0.32	58.62 $\pm$ 2.08	41.63 $\pm$ 0.62	41.48 $\pm$ 1.28	68.50 $\pm$ 1.87
BGE	Bio.	62.01 $\pm$ 0.67	60.47 $\pm$ 0.31	54.96 $\pm$ 0.96	55.52 $\pm$ 1.09	73.85 $\pm$ 2.06

Appendix

Table 15: Performance of different rerankers across various evaluation tasks (AP)

Reranker	Subject	Multi-Hop	NC	CC	SSLI	Base
BCE	Bio.	0.63 $\pm$ 0.01	0.60 $\pm$ 0.01	0.46 $\pm$ 0.01	0.45 $\pm$ 0.02	0.71 $\pm$ 0.02
	Geo.	0.63 $\pm$ 0.01	0.61 $\pm$ 0.01	0.45 $\pm$ 0.01	0.46 $\pm$ 0.02	0.68 $\pm$ 0.01
	Chem.	0.59 $\pm$ 0.02	0.59 $\pm$ 0.02	0.43 $\pm$ 0.01	0.43 $\pm$ 0.02	0.69 $\pm$ 0.01
	Phy.	0.60 $\pm$ 0.04	0.59 $\pm$ 0.01	0.43 $\pm$ 0.00	0.44 $\pm$ 0.01	0.70 $\pm$ 0.01
	Math.	0.62 $\pm$ 0.01	0.59 $\pm$ 0.02	0.42 $\pm$ 0.01	0.41 $\pm$ 0.01	0.70 $\pm$ 0.01
BGE	Bio.	0.64 $\pm$ 0.00	0.61 $\pm$ 0.00	0.56 $\pm$ 0.01	0.56 $\pm$ 0.01	0.76 $\pm$ 0.02

Appendix

Table 16: Performance of different rerankers across various evaluation tasks (MRR)

Panel of Figure 9 — Figure 9: A Counterfactual synthetic (Question, Context, Answer) Sample

Panel of Figure 10 — Figure 10: A SSLI synthetic (Question, Context, Answer) Sample

Panel of Figure 11 — Figure 11: A multi-hop synthetic (Question, Context, Answer) sample.

Panel of Figure 12 — Figure 12: A random mix synthetic (Question, Context, Answer) Sample (NC)

LLM Reranking

Large language model reranking uses LLMs to reorder retrieved documents or candidate outputs by relevance, usefulness, or task-specific criteria.

See all 132 papers

Re-ranking and Late Interaction Drive Retrieval Quality: A Controlled Comparison of RAG Strategies for Scientific Question Answering
Sep 29, 2026Bhagyesh Rathi, Eshan Chawla, William B. AndreopoulosLate Interaction Retrieval Scientific QA
San Jose State University, Department of Computer Science
SciRet: A Compute-Aware Empirical Study of Retrieval and Reranking for Scientific RAG
Aug 4, 2026Kaysarul Anas Apurba, Md. Hasibul Hasan, Rofiqul Alam Shehab +1Cross-Encoder Reranking Scientific QA
Laurentian University · North South University
SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
Jun 11, 2026Pierre Beckmann, Marco Valentino, Andre FreitasLLM Evaluation Scientific Reasoning
Idiap Research Institute, Switzerland · EPFL, Switzerland · School of Computer Science, University of Sheffield, UK +2

Sep 29, 2026J/K move · Enter open · S save

Re-ranking and Late Interaction Drive Retrieval Quality: A Controlled Comparison of RAG Strategies for Scientific Question Answering

Bhagyesh Rathi, Eshan Chawla, William B. Andreopoulos

San Jose State University, Department of Computer Science

Late Interaction Retrieval Scientific QA LLM Reranking Agentic Retrieval RAG Evaluation

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) is now the standard way to ground Large Language Models (LLMs) in external knowledge, yet the design space of retrieval pipelines is large and the trade-offs between variants are not well understood, especially on domain-specific corpora at realistic scale. In this work, we present a controlled comparison of six retrieval strategies for scientific question answering: (i) classic top-k dense retrieval, (ii) LLM-based query rephrasing, (iii) query rephrasing followed by LLM-based reranking, (iv) multi-query fusion via Reciprocal Rank Fusion (RRF), (v) an agentic tool-call pipeline in which the generator decides for itself whether to retrieve, and (vi) late-interaction retrieval with ColBERTv2. All six pipelines share the same generator (Meta-Llama/Llama-3.1-8B-Instruct), prompt, and evaluation protocol; the five single-vector pipelines additionally share SPECTER2 embeddings and a Chroma vector store; and all six retrieve from the full corpus of 463,971 arXiv papers dated 2024-2025. To support reproducible, large-scale evaluation, we also release a synthetic question dataset of 19,484 problem-statement and methodology questions generated by Llama-3.1-8B-Instruct from a random sample of 10,000 papers across academic domains (query generation succeeded for 9,742 of them), and every strategy is evaluated on this same query set. We describe the architecture and implementation of each pipeline, release the code and the synthetic question dataset, and evaluate each strategy with an LLM-as-a-judge protocol along multiple quality dimensions, together with direct gold-paper retrieval metrics. The result is an open testbed for studying the cost and quality trade-offs of RAG design choices on a research-literature corpus, and a basis for future work on faithfulness, retrieval robustness, and agentic retrieval.

Abstract

LLM Reranking

Explore similar work

Re-ranking and Late Interaction Drive Retrieval Quality: A Controlled Comparison of RAG Strategies for Scientific Question Answering

SciRet: A Compute-Aware Empirical Study of Retrieval and Reranking for Scientific RAG

SciR: A Controllable Benchmark for Scientific Reasoning in LLMs