Defining Page-Centric Visual Document Retrieval and Evaluating Existing Benchmarks
Organizations: Nanyang Technological University Singapore, Singapore
Abstract
In modern information retrieval (IR), documents are increasingly encountered as structured sequences of visually rich, paginated units, rather than as flat text. Example documents encountered in our daily life include Word files, PDFs, and slide decks. Free text-centric retrieval systems either ignore this structure entirely or destroy it through arbitrary passage chunking. In this paper, we formulate page-centric Visual Document Retrieval (VDR) with pages as the fundamental indexing units and document context available for assessing page relevance. The formulation distinguishes two retrieval granularities, page and document, and three retrieval paradigms: independent-item, evidence-set, and iterative retrieval. We then assess 42 benchmarks using eight criteria covering corpus characteristics, query construction, relevance annotation, and ranking support. Our analysis shows two main findings on current VDR research. First, evaluation remains concentrated on page-level independent-item retrieval, with fewer resources supporting document-level, evidence-set, and iterative settings. Second, queries originating from real user information needs independently of target corpus content and graded relevance judgments are uncommon, while the coverage and verification of relevance judgments vary substantially. Both findings reveal gaps between the capabilities applications require and those existing evaluation protocols can establish. We provide benchmark selection guidance and identify priorities for future benchmark design, emphasizing alignment between task requirements, relevance semantics, and evaluation protocols.
Figures & tables
| Sources and Structural Traceability (SS&T) | |
| 1 | The released data contain only isolated page images, images, figures, tables, page regions, or reassembled text blocks, and neither their source documents nor page order can be recovered. |
| 2 | The document to which each page belongs can be inferred, but document identifiers, page identifiers, or page order are incomplete. Pages have been extracted, removed, translated, re-rendered, or reorganized without adequate documentation. |
| 3 | Complete document boundaries are preserved, documents and pages have stable identifiers, and page order is explicit. The benchmark documents operations such as re-rendering, optical character recognition, translation, deduplication, filtering, and cropping. |
| Corpus Size (CS): total number of pages across all documents | |
| 1 | The underlying corpus contains fewer than 10,000 pages in total. |
| 2 | The underlying corpus contains at least 10,000 pages in total. |
| Granularity | Paradigm | Benchmarks |
| Items | ViDoRe V3 (21), MultiDocR (19), KoViDoRe (19), MMDocRAG (18), LitTraceQA (17), ViDoRe V2 (16) | |
| Page | Set | Double-Bench (20) |
| Iterative | FinMRAGBench (16), VisDocAgentBench (16), M4DocBench (15), MADQA (15) | |
| Items | DocViRe (18), LitTraceQA (17) | |
| Document | Set | M3DocVQA (15) |
| Iterative | PaperScope (18), M4DocBench (15) |
| Application | Granularity | Paradigm | Selection rule | Benchmarks |
| Enterprise and knowledge-base search | Page; document for holistic requests | Independent items | SS=3, CS=2, DL=2, RC 3, RL 2 | ViDoRe V3, MultiDocR |
| RAG over visually-rich documents | Page; Document | All three paradigms | DL=2, RC 3, RQ 2 | ViDoRe V3, Double-Bench, MultiDocR, MMDocRAG, DocViRe, PaperScope |
| Financial and legal analysis | Page; Document | All three paradigms | SS=3, DL=2, RC 2, domain-specific | ViDoRe V3, MultiDocR, DocViRe, FinMRAGBench, M4DocBench |
| Multi-document, multi-hop QA | Both, with cross-document aggregation | Evidence sets; iterative | DL=2, RC 3, RS=2 | Double-Bench, PaperScope, VisDocAgentBench |
| Benchmarks | Query | Document |
| SDS KoPub VDR ( Lee et al., 2025 ) | ko | ko |
| KoViDoRe ( Choi et al., 2026 ) | ko | ko |
| CHaystack ( Li, 2026 ) | zh | zh |
| ViDoRe V1 ( Faysse et al., 2025 ) | en , fr | en , fr |
| DocQIR ( Ho et al., 2026 ) | en , hi, ja, ko, pt | en |
| JinaVDR ( Günther et al., 2025 ) | en & 19 langs. † | en & 19 langs. † |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Dimensions | Document source and characteristics | Query and Annotation | Results and Ranking | Total | Retrieval | |||||
| Benchmark | SS&T | CS | DL | AP | RC | RQ | RL | RS | Score | Paradigm |
| ViDoRe V3 ( Loison et al., 2026 ) | 3 | 2 | 2 | 3 | 4 | 4 | 2 | 1 | 21 | Items |
| Double-Bench ( Shen et al., 2026 ) | 3 | 2 | 2 | 2 | 4 | 4 | 1 | 2 | 20 | Set |
| MultiDocR ( Hu et al., 2026b ) | 3 | 2 | 2 | 1 | 3 | 4 | 3 | 1 | 19 | Items |
| KoViDoRe ( Choi et al., 2026 ) | 3 | 1 | 2 | 3 | 3 | 4 | 2 | 1 | 19 | Items |
| MMDocRAG ( Dong et al., 2025 ) | 3 | 2 | 2 | 1 | 4 | 4 | 1 | 1 | 18 | Items |
Visual Document Retrieval
Visual document retrieval finds relevant pages or regions in image-based documents using visual and multimodal representations.
See all 69 papers