Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG

Martin Asenov; Kenza Benkirane; Dan Goldwater; Aneiss Ghodsi

cs.CLMar 4, 2026

Retrieval or Representation? Reassessing Benchmark Gaps in Multilingual and Visually Rich RAG

Authors: Martin Asenov, Kenza Benkirane, Dan Goldwater, Aneiss Ghodsi

Organizations: Parexel AI Labs London, United Kingdom · Parexel AI Labs San Francisco, United-States

Abstract

Retrieval-augmented generation (RAG) is a common way to ground language models in external documents and up-to-date information. Classical retrieval systems relied on lexical methods such as BM25, which rank documents by term overlap with corpus-level weighting. End-to-end multimodal retrievers trained on large query-document datasets claim substantial improvements over these approaches, especially for multilingual documents with complex visual layouts. We demonstrate that better document representation is the primary driver of benchmark improvements. By systematically varying transcription and preprocessing methods while holding the retrieval mechanism fixed, we demonstrate that BM25 can recover large gaps on multilingual and visual benchmarks. Our findings call for decomposed evaluation benchmarks that separately measure transcription and retrieval capabilities, enabling the field to correctly attribute progress and focus effort where it matters.

Figures & tables

Figure 1: Multilingual benchmark results across 15 languages. Average Top-5 retrieval accuracy for different methods. Methods are sorted by performance (highest to lowest). Lexical retrievers (BM25) are shown with diagonal hatching. BM25+OCR indexes text produced by state-of-the-art OCR models and preprocessing techniques per different languages. Release years are shown in parentheses.

Panel of Figure 2 — Figure 2: Language-specific BM25 optimization. Top-5 retrieval accuracy for BM25 under different OCR/transcription pipelines (Adobe, EasyOCR, Ministral 3B, Mistral OCR 3) and language-specific text processing (lemmatization, stemming, morphological analysis, and segmentation).

Panel of Figure 3 — Figure 3: OCR impact on text retrieval methods. a) BM25 retrieval performance for different Top-K values b) Impact on different retrieval for different combinations of transcription OCR models and text retrieval models.

Figure 4: Figure-heavy focused QA benchmark results. Average Top-5 retrieval accuracy for different methods. Methods are sorted by performance (highest to lowest). Lexical retrievers (BM25) are shown with diagonal hatching. BM25+OCR uses a small visual language model Mistral (2025a) for transcription. Release years are shown in parentheses.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Text-based Methods
	Spanish		Italian		German		French		Dutch		Arabic		Croatian		Japanese
Method	top1	top5	top1	top5	top1	top5	top1	top5	top1	top5	top1	top5	top1	top5	top1	top5
BM25 Robertson and Walker (1994)	60.23	82.55	59.14	82.14	65.79	86.87	54.05	77.84	59.83	84.94	7.25	20.73	52.95	72.90	11.84	39.35
SBERT Reimers and Gurevych (2019)	22.77	41.83	21.82	41.12	25.74	48.54	27.43	51.33	27.99	52.25	4.02	17.29	17.72	36.67	13.06	41.24
BGE-large Xiao et al. (2023)	34.55	60.41	30.27	56.24	39.75	66.82	41.34	67.42	39.14	67.53	6.15	19.53	32.67	58.14	31.92	64.97
BGE-M3 Chen et al. (2024b)	58.16	83.13	52.94	77.96	67.64	88.94	60.68	82.10	63.62	87.73	10.55	26.26	59.07	81.46	58.38	84.33

Appendix

Table 1: Multilingual retrieval accuracy (%) on VisR-Bench. Top-1 and Top-5 accuracy across 15 languages.

Language	Best OCR	Top-1 (%)	Top-5 (%)	Features
Spanish	EasyOCR	61.07	84.98	Stemming
Italian	EasyOCR	60.30	84.96	None
German	Ministral 3B	66.87	89.36	Stemming
French	Ministral 3B	58.43	84.14	None
Dutch	EasyOCR	62.18	87.13	None
Arabic	Mistral OCR 3	25.92	47.20	Morphology

Appendix

Table 2: Best configuration per language based on Top-5 accuracy. We report the optimal OCR model and linguistic processing features for each of the 15 languages tested.

		Adobe Text Extract			EasyOCR			Ministral 3B			Mistral OCR 3
Language	Config	MSLS	T-1	T-5	MSLS	T-1	T-5	MSLS	T-1	T-5	MSLS	T-1	T-5
Spanish	1	×✓××	57.65	82.00	×✓××	61.07	84.98	×✓××	56.98	82.92	×✓××	56.69	79.82
Spanish	2	××××	60.23	82.55	××××	58.49	81.70	××××	60.07	83.87	××××	59.03	80.18
Italian	1	×✓××	56.57	80.70	×✓××	57.54	83.41	×✓××	55.56	81.33	×✓××	55.05	79.03
Italian	2	××××	59.14	82.14	××××	60.30	84.96	××××	58.74	83.35	××××	57.42	80.09
German	1	×✓××	66.09	87.82	×✓××	67.47	89.19	×✓××	66.87	89.36	×✓××	66.87	87.34

Appendix

Table 3: Complete BM25 ablation study across all languages and OCR models on multilingual VisR-Bench. ✓=enabled, ×=disabled. Top-1 / Top-5 accuracy (%).

A. Text-Based Retrieval (Default OCR)
		Figure		Table		Text		Macro Avg.
Retriever	OCR / Transcription	top1	top5	top1	top5	top1	top5	top1	top5
BM25 Baeza-Yates et al. (1999)	Adobe Text Extract	24.27	46.12	38.66	66.54	64.74	89.10	42.56	67.25
SBERT Reimers and Gurevych (2019)	Adobe Text Extract	25.24	49.27	26.31	52.68	49.96	76.97	33.84	59.64
BGE-large Xiao et al. (2023)	Adobe Text Extract	31.55	56.07	40.36	70.14	57.00	82.68	42.97	69.63
BGE-M3 Chen et al. (2024b)	Adobe Text Extract	31.07	56.80	51.11	78.51	67.68	89.89	49.96	73.95

Appendix

Table 4: Retrieval accuracy (%) on VisR-Bench across document types. A: Text-based retrieval with default OCR. B: Controlled OCR ablation (retriever fixed; only transcription varies). C: Multimodal retrievers operating directly on document images. Macro Avg. is the unweighted mean over Figure/Table/Text.

RAG Evaluation

Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.

See all 354 papers

Loud and Clear: Making Visual Evidence Count in Multimodal Document RAG
Oct 15, 2026Eunchan Jo, Seounghyun Jang, Sangwoo MoMultimodal RAG Multimodal QA
POSTECH
Revisiting RAG Retrievers: An Information Theoretic Benchmark
Feb 25, 2026Wenqing Zheng, Dmitri Kalaev, Noah Fatsi +5RAG Evaluation IR Evaluation
CapitalOne.
MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generation
Jun 24, 2026Xiaochen Wang, Bao Hoang, Han Liu +2Multimodal RAG Retrieval-Augmented Generation
The Pennsylvania State University University Park, Pennsylvania, USA · Michigan State University East Lansing, Michigan, USA · Dalian University of Technology Dalian, Liaoning, China +1

Oct 15, 2026J/K move · Enter open · S save

Loud and Clear: Making Visual Evidence Count in Multimodal Document RAG

Eunchan Jo, Seounghyun Jang, Sangwoo Mo

POSTECH

Multimodal RAG Multimodal QA

Open paper AskPDF arXiv

Multimodal document retrieval-augmented generation (RAG) has focused largely on retrieval, yet its answers depend on how well a vision-language model reads the retrieved pages. We find that evidence in retrieved pages remains underused even when the correct pages are retrieved. On SciEGQA oracle pages, five prior methods give the generator 0.7K to 2.7K visual tokens per query on average under a common image processor, yet raising the page-only budget from 1K to 32K improves accuracy by 15.22 points. At a fixed 16K budget, adding ground-truth crops improves accuracy by 3.75 points over page-only input. We study what the generator needs from retrieved pages through two requirements: Loud, making full use of visual token capacity, and Clear, refining the page so that its evidence becomes prominent. For Loud, increasing either the visual token budget or visual fidelity, the image resolution those tokens preserve, improves accuracy with the other held fixed, and full-page scaling outperforms repetition and tiling at matched budgets. For Clear, suppressing surrounding content improves accuracy without enlarging selected regions, and adding OCR of these regions to page images outperforms either input alone. We implement both as Loud and Clear (LoC), which combines higher-resolution pages with OCR from query-relevant regions without ground-truth annotations or additional training. LoC outperforms eight prior methods on SciEGQA, ViDoRe-v3, and MMLongBench-Doc, improving over the strongest, ColQwen3, by 8.88, 3.09, and 4.22 points with the same retriever and generator. It also outperforms its page-only variant under the same visual token budget. These results locate a bottleneck beyond retrieval: how the retrieved pages are given to the generator.

Abstract

RAG Evaluation

Explore similar work

Loud and Clear: Making Visual Evidence Count in Multimodal Document RAG

Revisiting RAG Retrievers: An Information Theoretic Benchmark

MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generation