Large-Scale Biomedical RAG: Corpus Scaling, Document Representation, and Efficiency Trade-offs

Adrián Sánchez-Freire; Gonzalo Pérez-Gómez; Aitziber Atutxa; Ander Barrena

cs.CLOct 19, 2026

Large-Scale Biomedical RAG: Corpus Scaling, Document Representation, and Efficiency Trade-offs

Authors: Adrián Sánchez-Freire, Gonzalo Pérez-Gómez, Aitziber Atutxa, Ander Barrena

Organizations: HiTZ Basque Center for Language Technology, University of the Basque Country (EHU), P.º Rafael Moreno "Pitxitxi", 2, Bilbao, 48013, Biscay, Spain

Abstract

We present a controlled, large-scale evaluation of how corpus scale, document representation, and reranking affect the effectiveness and efficiency of biomedical Retrieval-Augmented Generation (RAG) systems. Retrieval collections were progressively expanded while retaining the relevant source documents, enabling consistent comparisons across corpus sizes. We compared abstract, flat full-text, and hierarchical full-text representations across first-stage retrieval, reranking, and end-to-end multiple-choice question answering. Configurations selected on development data were evaluated on five external biomedical benchmarks and a held-out test set, while retrieval latency and index storage were measured to quantify operational cost. Performance patterns observed with small or controlled collections did not consistently persist as collections grew, and the usefulness of reranking depended on the corpus and retrieval setting. RAG improved three of four model configurations across the external benchmarks, with mean gains of up to 9.6 percentage points, and improved all four configurations on the held-out test set by 10.2-30.3 points. Abstract retrieval performed best when reference evidence was abstract-aligned, whereas full-text representations were generally more effective when richer evidence was needed. Hierarchical representation matched or outperformed flat full-text retrieval in most end-to-end comparisons while requiring 67% fewer indexed units and 44-56% less storage. These findings show that corpus scale should be treated as an explicit evaluation variable in biomedical RAG, and that the most appropriate design depends on evidence format, model behavior, and operational constraints.

Figures & tables

Reference	Max. index	Biomed.	Scaling	Doc. repr.	Efficiency
Borgeaud et al. [ 4 ]	28B units	–	$\sim$	–	–
Izacard et al. [ 13 ]	387M units	–	–	–	–
Ning et al. [ 29 ]	264M units	–	$\checkmark$	–	$\checkmark$
Tao et al. [ 42 ]	n.r.	$\sim$	–	$\checkmark$	–
Otegi et al. [ 31 ]	400K units	$\checkmark$	–	$\checkmark$	–
Xiong et al. [ 48 ]	54.2M units	$\checkmark$	–	–	–

Table 1: Comparison with representative retrieval-augmented studies. Max. index reports the largest retrieval collection evaluated by each study in its reported indexing unit. The remaining columns indicate biomedical or medical evaluation, controlled corpus scaling, document-representation comparison, and efficiency analysis.

\checkmark

denotes explicit coverage,

\sim

partial or indirect coverage, – not addressed, and n.r. not reported.

				Retrieval units
Corpus	Scale	Factor ( $\times$ )	Source docs	Abstract	Full-text	TreeRAG
PubMed	Control	1.0	0.3K	0.3K	11.2K	2.6K
	Tiny	6.6	1.8K	1.8K	53.4K	15.9K
	Small	44.1	11.8K	11.8K	330.6K	105.7K
	Medium	292.6	78.4K	78.4K	2.2M	700.2K
	Large	1942.2	520.5K	520.5K	14.4M	5.2M

Table 2: Corpus scales and representation-dependent retrieval-index sizes. Source docs denotes the number of documents in each retrieval pool, whereas Retrieval units reports the number of indexed abstracts, full-text chunks, or TreeRAG units. Factors are defined relative to the corpus-specific control setting and applied independently to both corpora. Values are abbreviated for readability; K denotes thousands and M denotes millions. The full setting is available only for PubMed.

Figure 1: Retriever evaluation on DEV using Macro Recall@100. Panel (a) shows MedAESQA across PubMed corpus scales, and panel (b) shows MIRIAD DEV across MIRIAD corpus scales. Colors denote the document representation: Abstract, Full-text, or TreeRAG. Diagonal split fills indicate ties between representations. Retriever labels correspond to BM25, BGE-base-en-v1.5 (BGE), and Qwen3-Embedding-8B (Qwen3). The bold label and black star indicate the highest-scoring retriever at the largest available scale for each dataset.

Figure 2: Reranker evaluation on DEV using Macro Recall@5. Panel (a) shows MedAESQA across PubMed corpus scales, and panel (b) shows MIRIAD DEV across MIRIAD corpus scales. All configurations use Qwen3-Embedding-8B as the first-stage retriever. Colors denote the document representation: Abstract, Full-text, or TreeRAG. Diagonal split fills indicate ties between representations. Labels correspond to No-Reranker, BGE-Reranker-v2-M3 (BGE), Jina-Reranker-v3 (Jina), and Qwen3-Reranker-8B (Qwen3). The bold label and black star indicate the highest-scoring reranking configuration at the largest available scale for each dataset.

Figure 3: End-to-end MCQA accuracy gains over each generator’s LLM-only baseline for (a) MedAESQA and (b) MIRIAD DEV, reported in percentage points across corpus scales and document representations. Horizontal reference lines indicate the LLM-only baseline at zero and the dataset-specific Oracle gain. MedAESQA uses Qwen3-Embedding-8B without reranking, whereas MIRIAD DEV uses Qwen3-Embedding-8B followed by Qwen3-Reranker-8B.

				RAG $\Delta$
Model	Corpus	Dataset	LLM-only	Abstract	Full-text	TreeRAG
Gemma-3	PubMed	MMLU-Med	57.4	+7.2	+7.8	+8.5
		MedQA-US	46.1	+11.0	+13.4	+10.8
		MedMCQA	44.9	+1.8	+5.0	+7.4
		PubMedQA	51.4	+0.6	+5.4	+3.4
		BioASQ-Y/N	70.2	+8.3	+4.7	+8.3

Table 3: End-to-end MCQA performance on external benchmarks and MIRIAD TEST. LLM-only reports absolute accuracy (%), while RAG

\Delta

reports the percentage-point change relative to the corresponding baseline for each document representation. PubMed denotes retrieval over the full PubMed corpus, whereas MIRIAD denotes retrieval over the large MIRIAD corpus. Shaded cells show the average across the five external benchmarks. Bold marks the largest improvement in each row, or the LLM-only baseline when no RAG configuration improves upon it; ties are retained. Negative deltas are shown in red.

Figure 4: Efficiency–effectiveness trade-off on MedAESQA using the full PubMed Abstract index. The x-axis reports mean per-query latency on a logarithmic scale, and the y-axis reports Macro Recall@5. Open circles denote first-stage retrievers, while filled markers denote the corresponding configurations after reranking the top 100 candidates. Colors identify the first-stage retriever, marker shapes identify the reranker, and arrows connect each retriever to its reranked variants.

Retrieval-Augmented Generation

Retrieval-augmented generation connects language models to external knowledge so answers can use current, domain-specific evidence.

See all 792 papers

Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study
May 4, 2026Devi Prasad Bal, Subhashree PuhanCross-Encoder Reranking Retrieval-Augmented Generation
The University of Texas at Austin
BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms
Jul 29, 2026Pengyu Wang, Benfeng Xu, Shaohan Wang +5Retrieval-Augmented Generation Graph Retrieval-Augmented Generation
University of Science and Technology of China, Hefei, China · Metastone Technology, Beijing, China · Information Technology Research Center, Beijing Academy of Agriculture and Forestry Sciences, Beijing, China
When Retrieval Doesn't Help: A Large-Scale Study of Biomedical RAG
Jun 2, 2026Erfan Nourbakhsh, Rocky Slavin, Ke Yang +1Retrieval-Augmented Generation LLM Grounding
The University of Texas at San Antonio

May 4, 2026J/K move · Enter open · S save

Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study

Devi Prasad Bal, Subhashree Puhan

The University of Texas at Austin

Cross-Encoder Reranking Retrieval-Augmented Generation Biomedical QA Biomedical IR RAG Evaluation Hybrid Retrieval

Open paper AskPDF arXiv

Retrieval-Augmented Generation (RAG) offers a well-established path to grounding large language model (LLM) outputs in external knowledge, yet the question of which retrieval strategy works best in a high-stakes domain such as biomedicine has not received the controlled, multi-metric treatment it deserves. This paper presents a systematic empirical comparison of five retrieval strategies -- Dense Vector Search, Hybrid BM25 + Dense retrieval, Cross-Encoder Reranking, Multi-Query Expansion, and Maximal Marginal Relevance (MMR) -- within a biomedical question-answering RAG pipeline. All strategies share a fixed generation model (GPT-4o-mini), a common vector store (ChromaDB), and OpenAI's text-embedding-3-small embeddings, ensuring that observed differences are attributable to retrieval alone. Evaluation is conducted on 250 question-answer pairs drawn from a preprocessed subset of the BioASQ benchmark (rag-mini-bioasq) using four DeepEval metrics: contextual precision, contextual recall, faithfulness, and answer relevancy, each reported with 95% confidence intervals. A no-context ablation is included as a lower bound. Cross-Encoder Reranking achieves the best composite score (0.827) and highest contextual precision (0.852), confirming that query-document interaction yields measurable retrieval gains. Multi-Query Expansion, despite its recall-oriented design, produces the weakest contextual precision (0.671), suggesting naive query diversification introduces retrieval noise. MMR sacrifices answer relevancy for diversity, while the Dense baseline (composite 0.822) falls within 0.005 points of the top strategy. All RAG conditions dramatically outperform the no-context ablation on answer relevancy (0.658-0.701 vs. 0.287), confirming the practical value of retrieval. The full pipeline, hyperparameters, and evaluation code are publicly available.

Abstract

Retrieval-Augmented Generation

Explore similar work

Benchmarking Retrieval Strategies for Biomedical Retrieval-Augmented Generation: A Controlled Empirical Study

BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms

When Retrieval Doesn't Help: A Large-Scale Study of Biomedical RAG