VL-DocIR: A Benchmark for Vision-Based Long Document Information Retrieval

Simon Ott; Yufan Chen; Ruiping Liu; Junwei Zheng; Jiale Wei; Di Wen; Kunyu Peng; Jiaming Zhang

cs.AIOct 12, 2026

VL-DocIR: A Benchmark for Vision-Based Long Document Information Retrieval

Authors: Simon Ott, Yufan Chen, Ruiping Liu, Junwei Zheng, Jiale Wei, Di Wen, Kunyu Peng, Jiaming Zhang,

Organizations: Karlsruhe Institute of Technology · Hunan University

Abstract

Vision-based document retrieval has improved rapidly, but current evaluation settings still emphasize short documents and single relevant pages. This leaves an important gap between leaderboard performance and realistic long-document retrieval, where the evidence needed to answer a query may span multiple pages or even multiple documents. We present VL-DocIR, a page-level benchmark for evidence-complete visual long-document retrieval. VL-DocIR is built from 29,641 HTML-born documents rendered into 388,548 page images from Wikipedia, arXiv, PubMed, and SEC proxy statements. It contains 271,760 queries across 23 domains and six evidence structures, covering single-page, within-document multi-page, and cross-document evidence configurations. Evidence annotations are grounded to rendered pages and HTML element identifiers, and queries are filtered with a cleaning pipeline targeting clarity, correctness, closedness, and the absence of explicit layout references. We evaluate leading single-vector and multi-vector visual retrievers together with diagnostic retrieval settings such as document-context scoring, query splitting, reranking, and round-robin ensembles. The strongest evaluated configuration reaches 69.3 nDCGAll@10 and 80.6 RecallAll@10 under structure-balanced Macro evaluation, but source- and evidence-structure breakdowns show that cross-document retrieval remains substantially harder than single-page retrieval, and that document-context scoring is especially helpful for within-document multi-page queries. VL-DocIR exposes these failure modes and provides a more demanding target for visual long-document retrieval. The new dataset and code will be made publicly available at https://yufanchen96.github.io/VL-DocIR.

Figures & tables

Source	Docs	Avg. Pages
Wikipedia	11,988	6.49
arXiv	10,846	15.50
PubMed	4,815	12.26
SEC Proxy	1,992	41.97
Total	29,641	13.11

Table 1: Benchmark statistics.

Figure 1: Benchmark query creation process overview.

	Data Source
Evidence Structure	Wiki	arXiv	PubMed	SEC	Total
Page-Single	26,306	14,027	14,128	31,735	86,196
Page-Multi	34,222	26,990	29,620	29,792	120,624
Doc-Adjacent	3,283	2,556	3,360	2,773	11,972
Doc-Near	1,172	634	1,000	586	3,392
Doc-Far	255	461	613	1,234	2,563

Table 2: Benchmark Query Distribution Overview.

Model	Aggregate	nDCG@k			Recall@k			nDCGAll@k			RecallAll@k
Model	Aggregate	@3	@5	@10	@3	@5	@10	@3	@5	@10	@3	@5	@10
ColSmol 500	Macro	49.43	53.36	56.26	52.89	61.04	68.62	35.30	42.11	47.33	39.01	49.10	58.15
ColSmol 500	Micro	54.98	57.76	59.95	61.35	67.70	73.97	50.63	53.87	56.58	57.03	63.45	69.81
ColModernVBERT	Macro	47.33	50.91	53.72	50.48	57.93	65.30	33.28	39.31	44.29	36.73	45.70	54.35
ColModernVBERT	Micro	51.69	54.39	56.55	57.73	63.92	70.11	47.45	50.57	53.20	53.51	59.73	65.96
GME Qwen2-VL	Macro	55.27	59.51	62.72	59.35	68.10	76.42	41.08	48.82	55.10	45.33	56.70	67.34

Table 3: Average Retrieval Results on our VL-DocIR benchmark. Each model family is reported with the best variants under structure-balanced Macro (primary) and query-averaged Micro (secondary). SV/MV denotes single-vector and multi-vector retrieval. Ensemble denotes a combined setting. Bold indicates the best score within each aggregation protocol. The more detailed results and analysis are reported in the Appendix.

Family	Aggregation	Base	Q-Split	Ctx	B+Ctx	Rerank
ColPali	Macro	39.41/49.13	–	50.87/ 65.69	48.69/62.55	51.12 /57.16
ColPali	Micro	47.85/60.26	–	53.48/ 69.90	53.14/69.12	61.33 /67.17
ColQwen2.5	Macro	56.72/67.23	57.48/69.19	62.80/ 75.78	62.74/75.68	62.88 /71.62
ColQwen2.5	Micro	65.51/77.04	65.64/78.63	66.85/80.10	67.81/ 81.41	71.94 /80.36
ColNomic	Macro	61.82/71.76	–	66.03/78.19	67.06 / 79.60	65.00/74.62
ColNomic	Micro	70.68/81.24	–	70.07/82.26	71.96/ 84.81	73.76 /83.11

Table 4: Retrieval-stage setting effects measured by nDCGAll@10 and RecallAll@10. Ctx denotes document-context scoring, B+Ctx combines the ranked lists from base and document-context retrieval, Rerank denotes reranking, and Q-Split denotes query splitting. Bold indicates the best value for each metric.

Panel of Figure 2 — Figure 2: Source and evidence-structure-specific RecallAll@5 for the best evaluated configuration. Each cell shows the RecallAll@5 for a given source and evidence structure. Darker cells indicate higher RecallAll@5.

Evidence structure	#Queries	Macro RAll@5
Page-Single	86,196	93.4
Page-Multi	120,624	94.4
Doc-Adjacent	11,972	71.4
Doc-Near	3,392	69.0
Doc-Far	2,563	66.3
Cross-Doc	47,013	27.6

Table 5: Evidence-structure aggregate view for ColNomic B+Ctx+Rank. RAll@5 denotes RecallAll@5 macro-averaged over the four data sources.

Figure 3: Cross-Doc query top-5 retrieval results of ColQwen2.5 and ColQwen2.5 with reranking. The blue rectangles indicate the evidence bounding-boxes. The green rectangles highlight correctly retrieved pages.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Bounding-box annotation visualization for the Wikipedia article about the Portugal national football team.

Panel of Figure 5 — Figure 5: Domain distribution in the document dataset.

Panel of Figure 6 — Figure 6: Page-count distributions with 5-page bins for every data source. The plots show that Wikipedia articles tend to be shorter documents, whereas SEC Proxy Statements are much longer. The page count distribution of arXiv and PubMed articles is similar, with the most articles having between 10 and 15 pages.

Table 6: Confusion matrices for different question LLM-evaluation removal classification thresholds (total samples = 1438). Positive P stands for a question being removed. Negative N stands for a question not being removed. FOR is the false omission rate, where omission means non-removal.

	Predicted P	Predicted N
Actual P	237	203
Actual N	62	936

Appendix

Table 7: Confusion matrix for LLM query removal classification strategy (total samples = 1438, false omission rate = 0.1782).

	Predicted P	Predicted N
Actual P	305	135
Actual N	140	858

Appendix

Table 8: Confusion matrix for combination of LLM classifier and Query LLM rating removal methods (total samples = 1438, false omission rate = 0.1360).

Table 9: Confusion matrices for the Logistic Regression and XGBoost classifier for query removal. Positive P stands for a query being removed. Negative N stands for a query not being removed. FOR is the false omission rate, where omission means non-removal.

	Predicted P	Predicted N
Actual P	116	85
Actual N	92	345

Appendix

Table 10: Confusion matrix for five nearest neighbor question removal classification strategy (total samples = 638, FOR = 0.1977).

	Page-Single	Page-Multi	Doc-Adjacent	Doc-Near	Doc-Far	Cross-Doc	Total
Financial Reports	31,735	29,792	2,773	586	1,234	13,671	79,791
Medical Reports	7,835	17,006	1,902	523	416	6,324	34,006
Sports	5,891	10,017	1,067	436	69	7,026	24,506
Astrophysics	4,359	8,420	941	205	167	7,829	21,921
Science	6,006	7,080	632	212	41	2,566	16,537
Geography	6,064	6,957	442	206	85	2,267	16,021

Appendix

Table 11: Distribution of dataset questions by question type and domain type.

Panel of Figure 7 — Figure 7: Co-occurrence of query evidence types and evidence type distribution. The most common occurring combinations are text, table row, and image with text.

Source	Evidence structure	Correct/Total	F	E	U	N
Wikipedia	Page-Single	29/30	1	0	0	0
	Page-Multi	30/30	0	0	0	0
	Doc-Adjacent	26/30	3	1	0	0
	Doc-Near	29/30	1	0	0	0
	Doc-Far	28/30	0	0	2	0
	Cross-Doc	45/50	4	0	1	0

Appendix

Table 12: Stratified annotation audit . F: incorrectly formulated or insufficiently identifying query; E: incorrect evidence; U: unnecessary evidence page; N: non-presence query without directly retrievable positive evidence.

Model	Aggregate	nDCG@k			Recall@k			nDCGAll@k			RecallAll@k
Model	Aggregate	@3	@5	@10	@3	@5	@10	@3	@5	@10	@3	@5	@10
ColSmol 500	Macro	49.43	53.36	56.26	52.89	61.04	68.62	35.30	42.11	47.33	39.01	49.10	58.15
ColSmol 500	Micro	54.98	57.76	59.95	61.35	67.70	73.97	50.63	53.87	56.58	57.03	63.45	69.81
ColModernVBERT	Macro	47.33	50.91	53.72	50.48	57.93	65.30	33.28	39.31	44.29	36.73	45.70	54.35
ColModernVBERT	Micro	51.69	54.39	56.55	57.73	63.92	70.11	47.45	50.57	53.20	53.51	59.73	65.96
GME Qwen2-VL	Macro	55.27	59.51	62.72	59.35	68.10	76.42	41.08	48.82	55.10	45.33	56.70	67.34

Appendix

Table 13: Overall retrieval benchmark results with the nDCG@k Järvelin and Kekäläinen (2002) , Recall@k, nDCGAll@k Järvelin and Kekäläinen (2002) , and RecallAll@k as evaluation metrics. The best models are highlighted with blue , the second best with green , and the third best with bold text. The numbers below the evaluation metrics represent the values of parameter k.

Model / Setting	nDCG@k			Recall@k			nDCGAll@k			RecallAll@k
Model / Setting	@3	@5	@10	@3	@5	@10	@3	@5	@10	@3	@5	@10
ColNomic	64.68	68.47	70.89	67.65	75.36	81.58	49.33	57.13	62.63	52.99	63.85	72.56
ColNomic (Ctx)	65.11	69.58	72.74	68.45	77.58	85.71	49.55	58.77	66.07	53.51	66.34	78.01
ColNomic (Rerank)	67.61	71.33	73.59	70.44	77.97	83.72	52.82	60.59	65.82	56.31	67.09	75.39
ColNomic (Ensemble)	69.02	73.22	75.97	72.43	80.95	87.93	54.10	63.08	69.78	57.88	70.37	80.85
OCR-BM25	52.04	56.18	59.29	55.69	64.28	72.39	38.00	45.14	50.95	41.92	52.49	62.45

Appendix

Table 14: Source-averaged Macro retrieval results (%). Each score is the equal-weight mean of the four source-specific six-structure Macro scores. Bold indicates the highest score in each column.

Model / Setting	nDCG@k			Recall@k			nDCGAll@k			RecallAll@k
Model / Setting	@3	@5	@10	@3	@5	@10	@3	@5	@10	@3	@5	@10
ColNomic	70.06	74.15	76.58	73.02	81.32	87.51	54.93	63.96	69.97	58.58	70.98	80.31
ColNomic (Ctx)	67.87	72.58	75.65	70.71	80.28	88.15	52.05	62.50	70.11	55.70	69.86	81.76
ColNomic (Rerank)	72.22	76.21	78.55	74.95	83.00	88.91	57.74	66.77	72.47	61.15	73.40	82.34
ColNomic (Ensemble)	72.83	77.18	79.82	75.83	84.58	91.22	58.28	68.50	75.26	61.85	75.56	85.93
OCR-BM25	58.06	62.23	65.51	61.86	70.47	78.91	43.78	51.68	58.38	47.87	59.16	70.42

Appendix

Table 15: Domain-averaged Macro retrieval results (%). Each score is the equal-weight mean of the 23 domain-specific six-structure Macro scores. Bold indicates the highest score in each column.

Panel of Figure 8 — Figure 8: Overall RecallAll@k results grouped by ColPali Faysse et al. (2025) , ColQwen2.5 Faysse et al. (2025) , and ColNomic Team (2025) variants for $k\in\{3,5,10\}$ . B denotes base model and Ctx stands for document context embeddings

	Model	Page-Single	Page-Multi	Doc-Adjacent	Doc-Near	Doc-Far	Cross-Doc
Wikipedia	ColQwen3 Huang and Tan (2025)	90.5	90.3	57.1	49.7	49.8	45.2
	ColQwen2.5 Faysse et al. (2025)	84.9	85.6	48.9	44.3	46.7	41.0
	ColQwen2.5 (Ctx)	83.8	85.0	62.0	59.9	57.3	32.9
	ColQwen2.5 (Q-Split)	84.3	84.5	48.2	45.1	47.5	45.8
	ColQwen2.5 (Rerank)	89.9	90.4	56.0	49.8	58.4	50.9
	ColNomic Team (2025)	89.4	89.7	54.3	50.0	53.3	48.5

Appendix

Table 16: Retrieval results by data source and query evidence structure type with the RecallAll@5 evaluation metric. The best models are highlighted with blue , the second best with green , and the third best has bold text.

Evidence Type	Model	nDCGAll@k Järvelin and Kekäläinen (2002)			RecallAll@k
		3	5	10	3	5	10
Text	ColQwen3 Huang and Tan (2025)	68.5	71.1	73.3	73.5	78.1	82.3
	ColQwen2.5 Faysse et al. (2025)	59.3	62.4	65.0	65.6	71.6	77.4
	ColQwen2.5 (Ctx)	59.0	62.4	65.3	65.8	72.4	78.9
	ColQwen2.5 (Q-Split)	58.3	61.8	64.8	65.4	72.2	78.9
	ColQwen2.5 (Rerank)	67.1	69.7	71.8	72.1	76.8	80.8

Appendix

Table 17: Retrieval results by evidence type with the nDCGAll@k Järvelin and Kekäläinen (2002) and RecallAll@k as evaluation metrics. The best models are highlighted with blue , the second best with green , and the third best has bold text. The numbers below the evaluation metrics represent the values of parameter k. The three evidence groups text, table, and image refer to the cases that only this evidence is the ground truth. Combinations are not shown to remove the impact of inter-dependencies.

Panel of Figure 9 — Figure 9: Best cross-document retrieval RecallAll@5 for domains with at least 100 Cross-Doc queries. For each domain, the value is the maximum over all evaluated models.

Panel of Figure 10 — Figure 10: Analysis of vectors count impact in document context embeddings measured by RecallAll@10 for each query evidence structure type for ColQwen2.5 Faysse et al. (2025) as base model. We omit the cross-document line for reasons of clarity since the low RecallAll@10 values would distort the axis.

Evidence Structure	Model	Wikipedia	SEC Proxy Reports	arXiv	PubMed
Page-Single	ColPali	63.9	41.5	70.4	79.4
	ColQwen	84.9	71.0	90.5	92.7
	ColNomic	89.4	79.5	93.0	94.3
	ColPali Reduced	78.8	53.6	85.0	89.4
	ColQwen Reduced	92.7	81.1	96.9	97.2
	ColNomic Reduced	95.5	86.8	98.1	97.9

Appendix

Table 18: RecallAll@5 by data source and query evidence structure type comparing the the retrieval on the full and reduced dataset. The model names with suffix ’Reduced’ are the models which are evaluated on the reduced dataset, whereas the other models are evaluated on the whole document dataset. We evaluate the three base models ColPali Faysse et al. (2025) , ColQwen2.5 Faysse et al. (2025) , and ColNomic Team (2025) . The best models are highlighted with blue , the second best with green , and the third best has bold text.

Panel of Figure 11 — Figure 11: Analysis of Cross-Doc query retrieval results for each data source and the models ColQwen2.5 Faysse et al. (2025) , ColQwen2.5 (Doc Ctx), ColQwen2.5 (Q-Split), and ColQwen2.5 (Rerank). Instead of just deciding whether a retrieval is successful, we breakdown the results into page-level (filled bars) and document-level (dotted bars) evaluation. Additionally, each level is divided into the three classes that all evidences could be retrieved, partially, or none which is indicated by the colors.

Panel of Figure 12 — Figure 12: Comparison of structure-balanced Macro RecallAll@10 between queries generated by GPT-4.1-mini and Qwen3-VL-32B-Instruct. Each point represents a retrieval configuration evaluated on the same full corpus. Colors indicate the model family, and the dashed line denotes equal scores.

Model / Setting	Aggregate	nDCG@k			Recall@k			nDCGAll@k			RecallAll@k
Model / Setting	Aggregate	@3	@5	@10	@3	@5	@10	@3	@5	@10	@3	@5	@10
ColNomic (Ensemble)	Macro	70.58	74.42	76.89	74.01	81.76	88.02	57.31	65.67	71.51	61.16	72.66	81.74
ColNomic (Ensemble)	Micro	77.58	79.53	80.87	83.24	87.57	91.22	73.31	75.94	78.05	79.02	83.67	87.80
ColNomic	Macro	67.41	70.93	73.39	70.64	77.83	84.12	53.60	60.91	66.34	57.39	67.55	76.19
ColNomic	Micro	73.27	75.35	76.85	78.92	83.60	87.78	68.95	71.61	73.85	74.68	79.63	84.23
ColNomic (Ctx)	Macro	65.72	70.13	73.18	68.99	77.99	85.78	51.46	60.75	67.74	55.26	68.07	79.20

Appendix

Table 19: Retrieval results on GPT-4.1-mini queries (%; 59,983 queries). Macro gives equal weight to the six evidence structures within this subset, and Micro averages over its queries. All nine configurations retrieve from the same 388,548-page corpus.

Model / Setting	Aggregate	nDCG@k			Recall@k			nDCGAll@k			RecallAll@k
Model / Setting	Aggregate	@3	@5	@10	@3	@5	@10	@3	@5	@10	@3	@5	@10
ColNomic (Ensemble)	Macro	67.97	72.32	75.15	71.63	80.46	87.68	52.85	62.02	68.85	56.71	69.57	80.46
ColNomic (Ensemble)	Micro	74.26	76.79	78.51	79.84	85.37	90.04	68.32	72.03	74.96	73.88	80.17	85.78
ColNomic	Macro	63.15	66.96	69.39	66.12	73.90	80.13	47.82	55.53	60.95	51.43	62.24	70.93
ColNomic	Micro	69.67	72.02	73.64	75.11	80.32	84.80	63.96	67.19	69.78	69.59	75.28	80.39
ColNomic (Ctx)	Macro	64.64	69.19	72.42	68.08	77.40	85.73	48.87	58.20	65.69	52.87	65.97	78.03

Appendix

Table 20: Retrieval results on Qwen3-VL-32B-Instruct queries (%; 211,777 queries). Macro gives equal weight to the six evidence structures within this subset, and Micro averages over its queries. All nine configurations retrieve from the same 388,548-page corpus.

Visual Document Retrieval

Visual document retrieval finds relevant pages or regions in image-based documents using visual and multimodal representations.

See all 64 papers

VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents
Oct 14, 2026Zhivar Sourati, Marianne Menglin Liu, Rongguang Wang +12Long-Document VQA Document QA
University of Southern California · Oracle AI · Turing Enterprise Inc +1
One Fact, Many Places: Redundant Evidence in Long Visual Documents Breaks Single-Gold Evaluation of Retrieval, Citation and Attribution
Oct 19, 2026Xiaokang Yuan, Binhui Yao, Luwei Yang +2IR Evaluation Evidence Retrieval
Zhejiang University, Hangzhou, China · Zhejiang Geely Holding Group Co., Ltd, Hangzhou, China
DocMemo: Dynamic Evidence Discovery via Probabilistic Memory-Guided Retrieval for Multi-Modal Document Understanding
Aug 7, 2026Hanshu Yao, Janfeng Zhong, Niu Lian +1Multimodal Memory Document QA
1Harbin Institute of Technology, Shenzhen · 2Tsinghua Shenzhen International Graduate School, Tsinghua University

Oct 14, 2026J/K move · Enter open · S save

VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents

Zhivar Sourati, Marianne Menglin Liu, Rongguang Wang, Zheng Wang+11

University of Southern California · Oracle AI · Turing Enterprise Inc+1

Long-Document VQA Document QA Visual Question Answering VLM Evaluation Evidence Retrieval

Open paper AskPDF arXiv

Existing long-document QA benchmarks rarely combine large-scale human-authored questions with detailed visual evidence annotations, limiting diagnosis of evidence localization and interpretation failures. We present VELD-QA, a large-scale benchmark for Visual Evidence grounding and question answering in Long Documents. It contains 15,000 fully human-authored question-answer pairs across 5,859 real-world PDFs totaling 372,445 pages, balancing chart, table, and complex-layout tasks. Human annotations identify supporting pages and evidence elements and describe cited visual content, enabling separate retrieval and interpretation evaluation. We evaluate nine vision-language models under four evidence settings and seven retrieval-augmented and agentic systems. Our analysis identifies bottlenecks in evidence localization and interpretation. Retrieval-based methods substantially underperform the oracle setting with ground-truth supporting pages; full-document access does not close the gap. Localization alone is insufficient: oracle accuracy peaks at 81.7%, highlighting persistent visual interpretation and reasoning challenges. Enriching inputs with our visual-element annotations narrows the performance gap by 17-24%, demonstrating value beyond evaluation and motivating data enrichment for multimodal document QA. Our code is available at https://github.com/oracle-samples/veld-qa.

Abstract

Visual Document Retrieval

Explore similar work

VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents

One Fact, Many Places: Redundant Evidence in Long Visual Documents Breaks Single-Gold Evaluation of Retrieval, Citation and Attribution

DocMemo: Dynamic Evidence Discovery via Probabilistic Memory-Guided Retrieval for Multi-Modal Document Understanding