Vision-based document retrieval has improved rapidly, but current evaluation settings still emphasize short documents and single relevant pages. This leaves an important gap between leaderboard performance and realistic long-document retrieval, where the evidence needed to answer a query may span multiple pages or even multiple documents. We present VL-DocIR, a page-level benchmark for evidence-complete visual long-document retrieval. VL-DocIR is built from 29,641 HTML-born documents rendered into 388,548 page images from Wikipedia, arXiv, PubMed, and SEC proxy statements. It contains 271,760 queries across 23 domains and six evidence structures, covering single-page, within-document multi-page, and cross-document evidence configurations. Evidence annotations are grounded to rendered pages and HTML element identifiers, and queries are filtered with a cleaning pipeline targeting clarity, correctness, closedness, and the absence of explicit layout references. We evaluate leading single-vector and multi-vector visual retrievers together with diagnostic retrieval settings such as document-context scoring, query splitting, reranking, and round-robin ensembles. The strongest evaluated configuration reaches 69.3 nDCGAll@10 and 80.6 RecallAll@10 under structure-balanced Macro evaluation, but source- and evidence-structure breakdowns show that cross-document retrieval remains substantially harder than single-page retrieval, and that document-context scoring is especially helpful for within-document multi-page queries. VL-DocIR exposes these failure modes and provides a more demanding target for visual long-document retrieval. The new dataset and code will be made publicly available at https://yufanchen96.github.io/VL-DocIR.
Figures & tables
Source
Docs
Avg. Pages
Wikipedia
11,988
6.49
arXiv
10,846
15.50
PubMed
4,815
12.26
SEC Proxy
1,992
41.97
Total
29,641
13.11
Table 1: Benchmark statistics.
Figure 1: Benchmark query creation process overview.
Data Source
Evidence Structure
Wiki
arXiv
PubMed
SEC
Total
Page-Single
26,306
14,027
14,128
31,735
86,196
Page-Multi
34,222
26,990
29,620
29,792
120,624
Doc-Adjacent
3,283
2,556
3,360
2,773
11,972
Doc-Near
1,172
634
1,000
586
3,392
Doc-Far
255
461
613
1,234
2,563
Table 2: Benchmark Query Distribution Overview.
Model
Aggregate
nDCG@k
Recall@k
nDCGAll@k
RecallAll@k
@3
@5
@10
@3
@5
@10
@3
@5
@10
@3
@5
@10
ColSmol 500
Macro
49.43
53.36
56.26
52.89
61.04
68.62
35.30
42.11
47.33
39.01
49.10
58.15
Micro
54.98
57.76
59.95
61.35
67.70
73.97
50.63
53.87
56.58
57.03
63.45
69.81
ColModernVBERT
Macro
47.33
50.91
53.72
50.48
57.93
65.30
33.28
39.31
44.29
36.73
45.70
54.35
Micro
51.69
54.39
56.55
57.73
63.92
70.11
47.45
50.57
53.20
53.51
59.73
65.96
GME Qwen2-VL
Macro
55.27
59.51
62.72
59.35
68.10
76.42
41.08
48.82
55.10
45.33
56.70
67.34
Table 3: Average Retrieval Results on our VL-DocIR benchmark. Each model family is reported with the best variants under structure-balanced Macro (primary) and query-averaged Micro (secondary). SV/MV denotes single-vector and multi-vector retrieval. Ensemble denotes a combined setting. Bold indicates the best score within each aggregation protocol. The more detailed results and analysis are reported in the Appendix.
Family
Aggregation
Base
Q-Split
Ctx
B+Ctx
Rerank
ColPali
Macro
39.41/49.13
–
50.87/ 65.69
48.69/62.55
51.12 /57.16
Micro
47.85/60.26
–
53.48/ 69.90
53.14/69.12
61.33 /67.17
ColQwen2.5
Macro
56.72/67.23
57.48/69.19
62.80/ 75.78
62.74/75.68
62.88 /71.62
Micro
65.51/77.04
65.64/78.63
66.85/80.10
67.81/ 81.41
71.94 /80.36
ColNomic
Macro
61.82/71.76
–
66.03/78.19
67.06 / 79.60
65.00/74.62
Micro
70.68/81.24
–
70.07/82.26
71.96/ 84.81
73.76 /83.11
Table 4: Retrieval-stage setting effects measured by nDCGAll@10 and RecallAll@10. Ctx denotes document-context scoring, B+Ctx combines the ranked lists from base and document-context retrieval, Rerank denotes reranking, and Q-Split denotes query splitting. Bold indicates the best value for each metric.
Figure 2: Source and evidence-structure-specific RecallAll@5 for the best evaluated configuration. Each cell shows the RecallAll@5 for a given source and evidence structure. Darker cells indicate higher RecallAll@5.
Evidence structure
#Queries
Macro RAll@5
Page-Single
86,196
93.4
Page-Multi
120,624
94.4
Doc-Adjacent
11,972
71.4
Doc-Near
3,392
69.0
Doc-Far
2,563
66.3
Cross-Doc
47,013
27.6
Table 5: Evidence-structure aggregate view for ColNomic B+Ctx+Rank. RAll@5 denotes RecallAll@5 macro-averaged over the four data sources.
Figure 3: Cross-Doc query top-5 retrieval results of ColQwen2.5 and ColQwen2.5 with reranking. The blue rectangles indicate the evidence bounding-boxes. The green rectangles highlight correctly retrieved pages.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Bounding-box annotation visualization for the Wikipedia article about the Portugal national football team.
Figure 5: Domain distribution in the document dataset.
Figure 6: Page-count distributions with 5-page bins for every data source. The plots show that Wikipedia articles tend to be shorter documents, whereas SEC Proxy Statements are much longer. The page count distribution of arXiv and PubMed articles is similar, with the most articles having between 10 and 15 pages.
Table 6: Confusion matrices for different question LLM-evaluation removal classification thresholds (total samples = 1438). Positive P stands for a question being removed. Negative N stands for a question not being removed. FOR is the false omission rate, where omission means non-removal.
Table 8: Confusion matrix for combination of LLM classifier and Query LLM rating removal methods (total samples = 1438, false omission rate = 0.1360).
Table 9: Confusion matrices for the Logistic Regression and XGBoost classifier for query removal. Positive P stands for a query being removed. Negative N stands for a query not being removed. FOR is the false omission rate, where omission means non-removal.
Predicted P
Predicted N
Actual P
116
85
Actual N
92
345
Appendix
Table 10: Confusion matrix for five nearest neighbor question removal classification strategy (total samples = 638, FOR = 0.1977).
Page-Single
Page-Multi
Doc-Adjacent
Doc-Near
Doc-Far
Cross-Doc
Total
Financial Reports
31,735
29,792
2,773
586
1,234
13,671
79,791
Medical Reports
7,835
17,006
1,902
523
416
6,324
34,006
Sports
5,891
10,017
1,067
436
69
7,026
24,506
Astrophysics
4,359
8,420
941
205
167
7,829
21,921
Science
6,006
7,080
632
212
41
2,566
16,537
Geography
6,064
6,957
442
206
85
2,267
16,021
Appendix
Table 11: Distribution of dataset questions by question type and domain type.
Figure 7: Co-occurrence of query evidence types and evidence type distribution. The most common occurring combinations are text, table row, and image with text.
Table 13: Overall retrieval benchmark results with the nDCG@k Järvelin and Kekäläinen (2002) , Recall@k, nDCGAll@k Järvelin and Kekäläinen (2002) , and RecallAll@k as evaluation metrics. The best models are highlighted with blue , the second best with green , and the third best with bold text. The numbers below the evaluation metrics represent the values of parameter k.
Model / Setting
nDCG@k
Recall@k
nDCGAll@k
RecallAll@k
@3
@5
@10
@3
@5
@10
@3
@5
@10
@3
@5
@10
ColNomic
64.68
68.47
70.89
67.65
75.36
81.58
49.33
57.13
62.63
52.99
63.85
72.56
ColNomic (Ctx)
65.11
69.58
72.74
68.45
77.58
85.71
49.55
58.77
66.07
53.51
66.34
78.01
ColNomic (Rerank)
67.61
71.33
73.59
70.44
77.97
83.72
52.82
60.59
65.82
56.31
67.09
75.39
ColNomic (Ensemble)
69.02
73.22
75.97
72.43
80.95
87.93
54.10
63.08
69.78
57.88
70.37
80.85
OCR-BM25
52.04
56.18
59.29
55.69
64.28
72.39
38.00
45.14
50.95
41.92
52.49
62.45
Appendix
Table 14: Source-averaged Macro retrieval results (%). Each score is the equal-weight mean of the four source-specific six-structure Macro scores. Bold indicates the highest score in each column.
Model / Setting
nDCG@k
Recall@k
nDCGAll@k
RecallAll@k
@3
@5
@10
@3
@5
@10
@3
@5
@10
@3
@5
@10
ColNomic
70.06
74.15
76.58
73.02
81.32
87.51
54.93
63.96
69.97
58.58
70.98
80.31
ColNomic (Ctx)
67.87
72.58
75.65
70.71
80.28
88.15
52.05
62.50
70.11
55.70
69.86
81.76
ColNomic (Rerank)
72.22
76.21
78.55
74.95
83.00
88.91
57.74
66.77
72.47
61.15
73.40
82.34
ColNomic (Ensemble)
72.83
77.18
79.82
75.83
84.58
91.22
58.28
68.50
75.26
61.85
75.56
85.93
OCR-BM25
58.06
62.23
65.51
61.86
70.47
78.91
43.78
51.68
58.38
47.87
59.16
70.42
Appendix
Table 15: Domain-averaged Macro retrieval results (%). Each score is the equal-weight mean of the 23 domain-specific six-structure Macro scores. Bold indicates the highest score in each column.
Figure 8: Overall RecallAll@k results grouped by ColPali Faysse et al. (2025) , ColQwen2.5 Faysse et al. (2025) , and ColNomic Team (2025) variants for k∈{3,5,10} . B denotes base model and Ctx stands for document context embeddings
Model
Page-Single
Page-Multi
Doc-Adjacent
Doc-Near
Doc-Far
Cross-Doc
Wikipedia
ColQwen3 Huang and Tan (2025)
90.5
90.3
57.1
49.7
49.8
45.2
ColQwen2.5 Faysse et al. (2025)
84.9
85.6
48.9
44.3
46.7
41.0
ColQwen2.5 (Ctx)
83.8
85.0
62.0
59.9
57.3
32.9
ColQwen2.5 (Q-Split)
84.3
84.5
48.2
45.1
47.5
45.8
ColQwen2.5 (Rerank)
89.9
90.4
56.0
49.8
58.4
50.9
ColNomic Team (2025)
89.4
89.7
54.3
50.0
53.3
48.5
Appendix
Table 16: Retrieval results by data source and query evidence structure type with the RecallAll@5 evaluation metric. The best models are highlighted with blue , the second best with green , and the third best has bold text.
Evidence Type
Model
nDCGAll@k Järvelin and Kekäläinen (2002)
RecallAll@k
3
5
10
3
5
10
Text
ColQwen3 Huang and Tan (2025)
68.5
71.1
73.3
73.5
78.1
82.3
ColQwen2.5 Faysse et al. (2025)
59.3
62.4
65.0
65.6
71.6
77.4
ColQwen2.5 (Ctx)
59.0
62.4
65.3
65.8
72.4
78.9
ColQwen2.5 (Q-Split)
58.3
61.8
64.8
65.4
72.2
78.9
ColQwen2.5 (Rerank)
67.1
69.7
71.8
72.1
76.8
80.8
Appendix
Table 17: Retrieval results by evidence type with the nDCGAll@k Järvelin and Kekäläinen (2002) and RecallAll@k as evaluation metrics. The best models are highlighted with blue , the second best with green , and the third best has bold text. The numbers below the evaluation metrics represent the values of parameter k. The three evidence groups text, table, and image refer to the cases that only this evidence is the ground truth. Combinations are not shown to remove the impact of inter-dependencies.
Figure 9: Best cross-document retrieval RecallAll@5 for domains with at least 100 Cross-Doc queries. For each domain, the value is the maximum over all evaluated models.
Figure 10: Analysis of vectors count impact in document context embeddings measured by RecallAll@10 for each query evidence structure type for ColQwen2.5 Faysse et al. (2025) as base model. We omit the cross-document line for reasons of clarity since the low RecallAll@10 values would distort the axis.
Evidence Structure
Model
Wikipedia
SEC Proxy Reports
arXiv
PubMed
Page-Single
ColPali
63.9
41.5
70.4
79.4
ColQwen
84.9
71.0
90.5
92.7
ColNomic
89.4
79.5
93.0
94.3
ColPali Reduced
78.8
53.6
85.0
89.4
ColQwen Reduced
92.7
81.1
96.9
97.2
ColNomic Reduced
95.5
86.8
98.1
97.9
Appendix
Table 18: RecallAll@5 by data source and query evidence structure type comparing the the retrieval on the full and reduced dataset. The model names with suffix ’Reduced’ are the models which are evaluated on the reduced dataset, whereas the other models are evaluated on the whole document dataset. We evaluate the three base models ColPali Faysse et al. (2025) , ColQwen2.5 Faysse et al. (2025) , and ColNomic Team (2025) . The best models are highlighted with blue , the second best with green , and the third best has bold text.
Figure 11: Analysis of Cross-Doc query retrieval results for each data source and the models ColQwen2.5 Faysse et al. (2025) , ColQwen2.5 (Doc Ctx), ColQwen2.5 (Q-Split), and ColQwen2.5 (Rerank). Instead of just deciding whether a retrieval is successful, we breakdown the results into page-level (filled bars) and document-level (dotted bars) evaluation. Additionally, each level is divided into the three classes that all evidences could be retrieved, partially, or none which is indicated by the colors.
Figure 12: Comparison of structure-balanced Macro RecallAll@10 between queries generated by GPT-4.1-mini and Qwen3-VL-32B-Instruct. Each point represents a retrieval configuration evaluated on the same full corpus. Colors indicate the model family, and the dashed line denotes equal scores.
Model / Setting
Aggregate
nDCG@k
Recall@k
nDCGAll@k
RecallAll@k
@3
@5
@10
@3
@5
@10
@3
@5
@10
@3
@5
@10
ColNomic (Ensemble)
Macro
70.58
74.42
76.89
74.01
81.76
88.02
57.31
65.67
71.51
61.16
72.66
81.74
Micro
77.58
79.53
80.87
83.24
87.57
91.22
73.31
75.94
78.05
79.02
83.67
87.80
ColNomic
Macro
67.41
70.93
73.39
70.64
77.83
84.12
53.60
60.91
66.34
57.39
67.55
76.19
Micro
73.27
75.35
76.85
78.92
83.60
87.78
68.95
71.61
73.85
74.68
79.63
84.23
ColNomic (Ctx)
Macro
65.72
70.13
73.18
68.99
77.99
85.78
51.46
60.75
67.74
55.26
68.07
79.20
Appendix
Table 19: Retrieval results on GPT-4.1-mini queries (%; 59,983 queries). Macro gives equal weight to the six evidence structures within this subset, and Micro averages over its queries. All nine configurations retrieve from the same 388,548-page corpus.
Model / Setting
Aggregate
nDCG@k
Recall@k
nDCGAll@k
RecallAll@k
@3
@5
@10
@3
@5
@10
@3
@5
@10
@3
@5
@10
ColNomic (Ensemble)
Macro
67.97
72.32
75.15
71.63
80.46
87.68
52.85
62.02
68.85
56.71
69.57
80.46
Micro
74.26
76.79
78.51
79.84
85.37
90.04
68.32
72.03
74.96
73.88
80.17
85.78
ColNomic
Macro
63.15
66.96
69.39
66.12
73.90
80.13
47.82
55.53
60.95
51.43
62.24
70.93
Micro
69.67
72.02
73.64
75.11
80.32
84.80
63.96
67.19
69.78
69.59
75.28
80.39
ColNomic (Ctx)
Macro
64.64
69.19
72.42
68.08
77.40
85.73
48.87
58.20
65.69
52.87
65.97
78.03
Appendix
Table 20: Retrieval results on Qwen3-VL-32B-Instruct queries (%; 211,777 queries). Macro gives equal weight to the six evidence structures within this subset, and Micro averages over its queries. All nine configurations retrieve from the same 388,548-page corpus.
Existing long-document QA benchmarks rarely combine large-scale human-authored questions with detailed visual evidence annotations, limiting diagnosis of evidence localization and interpretation failures. We present VELD-QA, a large-scale benchmark for Visual Evidence grounding and question answering in Long Documents. It contains 15,000 fully human-authored question-answer pairs across 5,859 real-world PDFs totaling 372,445 pages, balancing chart, table, and complex-layout tasks. Human annotations identify supporting pages and evidence elements and describe cited visual content, enabling separate retrieval and interpretation evaluation. We evaluate nine vision-language models under four evidence settings and seven retrieval-augmented and agentic systems. Our analysis identifies bottlenecks in evidence localization and interpretation. Retrieval-based methods substantially underperform the oracle setting with ground-truth supporting pages; full-document access does not close the gap. Localization alone is insufficient: oracle accuracy peaks at 81.7%, highlighting persistent visual interpretation and reasoning challenges. Enriching inputs with our visual-element annotations narrows the performance gap by 17-24%, demonstrating value beyond evaluation and motivating data enrichment for multimodal document QA. Our code is available at https://github.com/oracle-samples/veld-qa.
Zhivar Sourati, Marianne Menglin Liu, Rongguang Wang +12
University of Southern California · Oracle AI · Turing Enterprise Inc +1
Document understanding is increasingly judged on where an answer comes from, scored against annotated gold locations as if each fact lived in one place. In long visual documents, a fact often lives in several places: a chart value is restated in the text, a key figure reappears in the summary. We ask how often such copies occur in standard benchmarks and what they do to evaluation. We annotate 420 questions from MMLongBench-Doc and MMDocRAG with the locations that independently establish each required fact, checked by two independent reviewers. Copies are common and often cross-modal: 19.5% of MMLongBench-Doc test questions have a strict copy outside the gold, and in 28% of MMDocRAG questions a non-gold candidate alone supplies a required fact. Single-gold scoring then fails in three predictable ways. A retriever that finds a copy is scored as a miss: page recall@1 of text and visual retrievers is underestimated by 13-24 points on questions with a cross-page copy. A model that cites a copy is scored as citing wrongly: crediting equivalent evidence reverses 11% of pairwise orderings among 65 competitive MMDocRAG systems. Masking the gold region leaves the fact available elsewhere: with a copy in the input, 41-56% of correct answers across four models survive it yet fail once every annotated instance is masked. Set-level masking with automatically found copies closes 78% of the gap to masking every annotated instance at matched granularity (three models); an unverified key-string rule lowers survival at least as far but masks three times the area. We release the annotations and redundancy-aware metrics.
Xiaokang Yuan, Binhui Yao, Luwei Yang +2
Zhejiang University, Hangzhou, China · Zhejiang Geely Holding Group Co., Ltd, Hangzhou, China
Long-document understanding requires locating sparse and heterogeneous evidence across hundreds of pages, yet existing systems remain limited by static retrieval and fragile cross-round memory. Mainstream single-round methods commit to a fixed top-k page set at the outset and struggle to recover from early retrieval errors; recent iterative approaches allow multi-round evidence acquisition, but they do not investigate the propagation mechanism of cross-round states, making it difficult to track the dynamic changes in page relevance. To address these limitations, we propose DocMemo, a memory-guided framework that formulates long-document reasoning as dynamic evidence exploration. DocMemo maintains a tri-level retrieval state consisting of Document Schema Memory, Page Belief Memory, and Question Episodic Memory, which respectively capture structural priors, dynamic relevance estimation, and query-specific reasoning trajectories. During reasoning, DocMemo continuously refines cross-round page selection through Bayesian page belief updating with Thompson sampling, spatial proximity propagation, and structure-aware adaptive-granularity evidence access, while supplementing page-level evidence with fine-grained visual regions. Experiments on 3 benchmarks show that DocMemo achieves state-of-the-art performance and validate the efficacy of structured memory and dynamic page belief updating. Code is available at https://github.com/Harrygof/DocMemo.
Hanshu Yao, Janfeng Zhong, Niu Lian +1
1Harbin Institute of Technology, Shenzhen · 2Tsinghua Shenzhen International Graduate School, Tsinghua University