Retrieval quality in RAG systems is commonly reported as a single aggregate score, which can hide large differences across query types and language conditions. We study this problem in Bengali agricultural advisory, where farmer queries are often colloquial while official advisory documents use formal scientific terminology. We construct a test collection of 1,000 queries and 2,882 knowledge nodes extracted from 284 official Bangladeshi agricultural publications, and use it to evaluate five retrieval architectures and six embedding models under three controlled language conditions. The results show that no single retrieval method is consistently best. For native Bengali queries, BM25 is the strongest single retriever (R@10 = 0.506) while Hybrid RRF reaches the highest overall R@10 of 0.539. However, dense retrieval performance varies sharply by query type: R@10 is 0.093 on colloquial farmer queries and 0.970 on formal safety queries. Across language conditions, BM25 R@10 drops from 0.506 on Bengali queries to 0.004 when English queries are matched against the Bengali corpus, while dense retrieval falls only from 0.464 to 0.425. We also find that embedding task configuration and passage length can each change reported R@10 by a factor of seven, independent of architecture. These results show why low-resource RAG evaluation should report performance by language condition and query type rather than relying on aggregate scores alone. The dataset and evaluation scripts are available at https://huggingface.co/datasets/RaiyanKhaan/AgriTrust-RAG.
Figures & tables
Resource
Low-res.
Arch.
Retr.
Cross-
Prov.
lang.
isolated
only
lingual
ground.
BEIR ( Thakur et al., 2021 )
✗
✓
✓
✗
✗
CRAG ( Yang et al., 2024 )
✗
✗
✗
✗
✗
MIRACL ( Zhang et al., 2023 )
✓
✗
✓
✓
✗
Mr. TyDi ( Zhang et al., 2021 )
✓
✗
✓
✓
✗
Amharic study ( Yeshambel et al., 2025 )
✓
✗
✓
✗
✗
Table 1. Positioning of our benchmark relative to closest prior benchmarks and systems.
Figure 1. Canonical knowledge node construction pipeline. Source documents are parsed, chunked by topic, and processed into structured, provenance-grounded JSON nodes. Flowchart illustrating the benchmark construction pipeline: 284 Bengali PDFs parsed to 2,680 section passages, yielding 2,882 canonical knowledge nodes (1,022 image-linked), 19,768 entities, 17,501 factual triples, and 900 verified benchmark queries.
Property
Value
Source PDFs
284
Organizations
5 (BRRI, IRRI, DAE, SRDI, MoA)
KG nodes
2,882
Image-linked nodes
1,022 (35.5%)
Unique entities
19,768
Factual triples
17,501
Table 2. Benchmark dataset statistics.
Architecture
R@1
R@5
R@10
MRR
nDCG@10
Hybrid RRF (Gemini+BM25)
.291
.466
.539
.551
.461
BM25 (Sparse)
.205
.405
.506
.481
.407
Dense (Gemini-001)
.320
.431
.464
.514
.431
Dense (BGE-M3 Native)
.281
.369
.408
.432
.370
ColBERT (BGE-M3) ‡
.255
.414
.487
.324
.416
Table 3. L1 Retrieval on 900 Answerable Bengali Queries (BN → BN). Bold = best overall; underline = best single-method.
Setting
Dense (Gemini)
BM25
Winner
Bengali (BN → BN)
.464 [.43, .50]
.506 [.47, .54]
BM25 (+9%)
Cross-lingual (EN → BN)
.425 [.39, .46]
.004 [.00, .01]
Dense ( ≈ 100 × )
English (EN → EN)
.442 [.41, .47]
.384 [.35, .42]
Dense (+15%)
Table 4. Cross-Lingual Comparison (95% BCa CIs for R@10 in brackets; 900 queries).
Architecture
Farmer
Safety
KG-Ground.
Farm.+Safe.
(colloquial)
(formal)
(formal)
(combined)
Dense (Gemini)
.093 [.06,.13]
.970 [.94,.99]
.489 [.44,.54]
.444 [.40,.49]
BM25
.523 [.47,.58]
.539 [.47,.60]
.478 [.43,.53]
.529 [.48,.57]
ColBERT (BGE-M3) ‡
.210 [.17,.26]
.675 [.60,.74]
.600 [.55,.65]
.396 [.35,.44]
Table 5. R@10 by query category (95% BCa CIs in brackets; 900 verified queries; Farmer=300, Safety=200, KG-grounded=400).
Figure 2. R@10 by query register, plotted from Table 5 . Dense retrieval is sharply bimodal — it nearly fails on colloquial farmer queries and saturates on formal safety queries — while BM25 stays comparatively stable across registers. Bar chart showing R@10 across Farmer, Safety, and KG-grounded registers for Dense Gemini, BM25, and ColBERT.
Model
Dim
Separation
R@10
Gemini-embedding-001
3,072
+0.127
.464 [.43,.50]
BGE-M3 (native proj.)
1,024
+0.133
.408 [.38,.44]
E5-Large (multilingual)
1,024
+0.037
.284 [.26,.32]
Qwen3 Embedding 8B
4,096
+0.108
.241 [.21,.27]
MPNet (paraphrase)
768
−0.010
.009 [.00,.02]
MiniLM (paraphrase)
384
−0.039
.005 [.00,.01]
Table 6. Multi-Embedding Analysis (95% BCa CIs for R@10 in brackets; 900 queries, BN → BN). Random baseline: separation = 0.
Farmer query (colloquial): [My chickpea plants’ leaves are turning yellow and drooping, what should I do?]
Gold node title: [Chickpea crop disease control and remedies] ( B7_CH2_CHHOLA_0006_N1 )
Gold node entity: wilt disease (formal pathological term; not present in query)
Retrieval-augmented generation (RAG) is a common way to ground language models in external documents and up-to-date information. Classical retrieval systems relied on lexical methods such as BM25, which rank documents by term overlap with corpus-level weighting. End-to-end multimodal retrievers trained on large query-document datasets claim substantial improvements over these approaches, especially for multilingual documents with complex visual layouts. We demonstrate that better document representation is the primary driver of benchmark improvements. By systematically varying transcription and preprocessing methods while holding the retrieval mechanism fixed, we demonstrate that BM25 can recover large gaps on multilingual and visual benchmarks. Our findings call for decomposed evaluation benchmarks that separately measure transcription and retrieval capabilities, enabling the field to correctly attribute progress and focus effort where it matters.
Martin Asenov, Kenza Benkirane, Dan Goldwater +1
Parexel AI Labs London, United Kingdom · Parexel AI Labs San Francisco, United-States
Retrieval-Augmented Generation (RAG) systems rely critically on the retriever module to surface relevant context for large language models. Although numerous retrievers have recently been proposed, each built on different ranking principles such as lexical matching, dense embeddings, or graph citations, there remains a lack of systematic understanding of how these mechanisms differ and overlap. Existing benchmarks primarily compare entire RAG pipelines or introduce new datasets, providing little guidance on selecting or combining retrievers themselves. Those that do compare retrievers directly use a limited set of evaluation tools which fail to capture complementary and overlapping strengths. This work presents MIGRASCOPE, a Mutual Information based RAG Retriever Analysis Scope. We revisit state-of-the-art retrievers and introduce principled metrics grounded in information and statistical estimation theory to quantify retrieval quality, redundancy, synergy, and marginal contribution. We further show that if chosen carefully, an ensemble of retrievers outperforms any single retriever. We leverage the developed tools over major RAG corpora to provide unique insights on contribution levels of the state-of-the-art retrievers. Our findings provide a fresh perspective on the structure of modern retrieval techniques and actionable guidance for designing robust and efficient RAG systems.
Retrieval Augmented Generation (RAG) grounds large language models in external knowledge. Modern pipelines combine embedding, chunking, retrieval, reranking, query augmentation, context compression, and agentic verification, yet these techniques are typically evaluated one at a time. It remains unclear whether their isolated gains survive composition, whether retrieval quality and answer faithfulness favor the same design choices, and whether conclusions from small corpora hold at large scale. We pre-declare three hypotheses and introduce LASAGNA, an open-source framework for controlled RAG evaluation. We test 450 configurations across eight benchmarks and corpora ranging from 5K to 200K documents. Three findings emerge. First, individually strong techniques from different phases of a RAG pipeline do not necessarily compose: three retrieval techniques predict a +47% combined improvement but deliver only +17% when used together. Second, component gains stack differently for retrieval and faithfulness, and optimizing one can hurt the other. Third, the best composition changes as the corpus grows: the combination that wins on a small corpus may not remain best at large scale. These findings show that gains measured in isolation are not reliable predictors of realistic performance, and that new RAG techniques should be tested with the full surrounding stack. We call on researchers to do so and supply LASAGNA to simplify these experiments.