VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents

Zhivar Sourati; Marianne Menglin Liu; Rongguang Wang; Zheng Wang; Yazhe Hu; Lan Guo; Yuying Wang; Mengqing Guo

cs.CVOct 14, 2026

VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents

Authors: Zhivar Sourati, Marianne Menglin Liu, Rongguang Wang, Zheng Wang, Yazhe Hu, Lan Guo, Yuying Wang, Mengqing Guo,

Organizations: University of Southern California · Oracle AI · Turing Enterprise Inc · University of Pennsylvania

Abstract

Existing long-document QA benchmarks rarely combine large-scale human-authored questions with detailed visual evidence annotations, limiting diagnosis of evidence localization and interpretation failures. We present VELD-QA, a large-scale benchmark for Visual Evidence grounding and question answering in Long Documents. It contains 15,000 fully human-authored question-answer pairs across 5,859 real-world PDFs totaling 372,445 pages, balancing chart, table, and complex-layout tasks. Human annotations identify supporting pages and evidence elements and describe cited visual content, enabling separate retrieval and interpretation evaluation. We evaluate nine vision-language models under four evidence settings and seven retrieval-augmented and agentic systems. Our analysis identifies bottlenecks in evidence localization and interpretation. Retrieval-based methods substantially underperform the oracle setting with ground-truth supporting pages; full-document access does not close the gap. Localization alone is insufficient: oracle accuracy peaks at 81.7%, highlighting persistent visual interpretation and reasoning challenges. Enriching inputs with our visual-element annotations narrows the performance gap by 17-24%, demonstrating value beyond evaluation and motivating data enrichment for multimodal document QA. Our code is available at https://github.com/oracle-samples/veld-qa.

Figures & tables

Figure 1: VELD-QA comprises 15,000 human-authored QA examples across three task categories: Chart QA , Table Understanding , and Complex Layout . Each example shows, from top to bottom: page-tagged evidence excerpts with answer-relevant content highlighted; the question, answer, and reasoning trace; and descriptions of the cited visual elements.

Benchmark	Release	Dataset Scale		Document Scale		Answer Evidence		Annotation
		#Docs	#QA	Avg. Pages	Avg. Tokens	Cross-page (%)	Cross-element (%)	Fully Human QA	Human Visual Element Descriptions
SlideVQA ( Tanaka et al., 2023 )	2023-01	2,619	14,484	20	2,030.5	—	—	3	7
MMLongBench-Doc ( Ma et al., 2024 )	2024-07	135	1,082	47.5	21,214.1	33.7	22.6	3	7
SPIQA ( Pramanick et al., 2024 )	2024-07	118	666	10	5,000	—	—	7	7
M-LongDoc ( Chia et al., 2025 )	2024-11	180	851	210.8	120,988.0	0	—	7	7
LongDocURL ( Deng et al., 2025 )	2024-12	396	2,325	85.6	43,622.6	52.9	37.1	7	7

Table 1: Comparison of VELD-QA with long-document QA benchmarks. Human QA indicates fully human-authored questions. Element descriptions are annotations linked to cited visual elements, and cross-element measures questions requiring evidence from multiple element types.

Document
Statistic	Number
PDF files	5,859
Distinct PDF byte contents	5,585
Total PDF pages	372,445
Avg. pages per PDF file	63.6
Spreadsheet workbooks	1,392

Table 2: Statistics for VELD-QA benchmark.

Figure 2: VELD-QA construction. Real PDFs from five genres are collected and screened (1); each task then follows its own lane, where annotators prepare the content that questions will cite (2) and write each question and answer with its supporting evidence (3). All items pass a shared review with calibration, complete first-level and sampled second-level review, and quality checks, with failed items returned for revision (4), and the data are split by document (5). Right: an example annotation record from the test set. Every question, answer, and evidence citation is written by a human.

Figure 3: Question diversity across document-QA benchmarks. (a) Estimated percentage of questions with a semantically similar question among others from the same benchmark (MPNet embeddings, cosine similarity

\geq

0.8). (b) Distinct combinations of reasoning operations, constraints, and requested answer forms in each benchmark’s annotated questions; these counts grow with the number of questions (size-matched comparison in the text).

Model	Question only	Gold pages	Top-3 RAG	All pages	Gold $-$ RAG	All $-$ RAG
Open-weight
Gemma 3 12B IT	15.1	39.1	37.1 Q	28.2	+2.1	$-$ 8.8
MUSE Glimmer 30B	13.2	73.9	66.4 Q	53.2	+7.5	$-$ 13.3
Proprietary
Gemini 2.5 Flash	11.6	70.5	51.7 Q	46.1	+18.7	$-$ 5.6
Gemini 2.5 Pro	14.2	74.0	59.4 Q	56.0	+14.6	$-$ 3.4

Table 3: Accuracy (%) on the VELD-QA test set under four evidence settings. Top-3 RAG retrieves pages with ColQwen2 ( Q ) or ColPali ( P ); the other settings use no retriever.

System	Accuracy	$\Delta$ vs. RAG [95% CI]	$\Delta$ Chart	$\Delta$ Table	$\Delta$ Layout
Gold pages (oracle)	78.8	+4.5 [+2.7, +6.2]	$-$ 1.6	+10.2	+5.2
All pages	72.3	$-$ 2.0 [ $-$ 4.2, +0.1]	$-$ 6.6	+3.8	$-$ 2.9
Top-3 RAG (ColPali)	74.4	–	–	–	–
LAD-RAG ( Sourati et al., 2026a )	75.7	+1.3 [ $-$ 0.6, +3.1]	$-$ 1.0	+4.8	+0.3
MDocAgent ( Han et al., 2025 )	76.6	+2.2 [+0.0, +4.2]	$-$ 1.3	+8.4	$-$ 0.1
SimpleDoc ( Jain et al., 2025 )	77.9	+3.5 [+1.4, +5.7]	+1.9	+8.0	+1.0

Table 4: Retrieval-augmented and agentic systems with a shared GPT-5.4 backbone. We report accuracy (%) and paired differences from top-3 RAG in percentage points, with 95% confidence intervals. Top-3 RAG and gold-page input use identical answer prompts and page rendering.

	Recall@ $k$				Gold pages		Document pages
Retriever	1	3	5	10	1	$\geq$ 2	$\leq$ 20	21–100	101–300
ColQwen2	68.6	88.3	93.0	96.0	90.9	73.3	92.4	87.8	85.4
ColPali	63.1	82.9	90.7	94.8	85.2	70.0	88.2	82.9	78.9
E5 (text)	40.3	63.3	72.9	83.0	66.2	46.8	76.5	63.5	53.0

Table 5: Page retrieval against the human-annotated evidence pages on non-chart test questions (chart documents are single pages). Recall@

k

: share of gold pages in the top

k

Figure 4: Accuracy by question type and evidence setting, averaged over the nine models, for chart and table questions (number of test questions under each label). Comparative and analytical questions are much harder than descriptive ones only on tables.

Figure 5: Gain in accuracy from adding VELD-QA’s annotations of the cited elements, as text, to the same gold pages (paired). Error bars: 95% CIs. The annotations are question-specific gold evidence: a diagnostic of reading, not a deployable system.

Figure 6: Top-3 RAG accuracy (%) by document genre and task, averaged over the nine models, with the number of test questions per cell. The 22 source categories are grouped into five genres.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Question type	Chart QA	Table QA
Descriptive	1,446	1,370
Comparative	1,530	1,721
Analytical	2,024	1,909

Appendix

Table 6: Retained question-type annotations. Complex-layout records do not retain this taxonomy. Labels characterize question types rather than calibrated difficulty.

System (GPT-5.4)	Calls	Prompt tok.	Output tok.
Question only	1.0	105	28
Gold pages	1.0	2,931	38
MDocAgent	5.0	14,071	639
SimpleDoc	2.0	9,454	590
DocAgent	4.3	45,056	236
MADQA BM25-MLLM	4.7	9,582	137

Appendix

Table 7: Mean model calls and tokens per test question for systems with a shared GPT-5.4 backbone.

Figure 7: All pages minus top-3 RAG accuracy by document length (non-chart questions). Above zero, reading every page beats retrieval; this holds only for the three frontier models (highlighted) and shrinks for long documents.

Figure 8: Gold

-

RAG accuracy gap on non-chart questions, split by whether all gold pages are in the top three (blue,

n

= 886) or at least one is missed (orange,

n

= 153). Strong models lose accuracy only when retrieval misses; Gemini loses accuracy even when every gold page is retrieved. Error bars: 95% CIs.

Figure 9: Accuracy versus cost per question for GPT-5.4-based systems on the questions answered by every system (log scale; GPT-5.4 list prices). Circles: fixed evidence settings; diamonds: agentic systems.

Long-Document VQA

See all 17 papers

DocAttriBench: Benchmarking Answer Grounding in Document Visual Question Answering
Sep 17, 2026Luca De Grandis, Silvia Cappelletti, William Raccagni +3Visual Question Answering Vision-Language Grounding
University of Modena and Reggio Emilia, Modena, Italy · University of Modena and Reggio Emilia Modena, Italy · University of Pisa, Pisa, Italy +1
VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA
Jun 15, 2026Young Rok Jang, Hyesoo Kong, Kyunghwan An +3Multimodal QA Multimodal Understanding
LG AI Research
DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding
May 9, 2026Xiang Feng, Jiawei Zhou, Zhangfeng Huang +6Multimodal Understanding LLM Answer Verification
School of Computer Science, National Engineering Research Center for Multimedia Software and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University, China · Alibaba Group, Hangzhou, China · Independent Researcher +1

Sep 17, 2026J/K move · Enter open · S save

DocAttriBench: Benchmarking Answer Grounding in Document Visual Question Answering

Luca De Grandis, Silvia Cappelletti, William Raccagni, Marcella Cornia+2

University of Modena and Reggio Emilia, Modena, Italy · University of Modena and Reggio Emilia Modena, Italy · University of Pisa, Pisa, Italy+1

Visual Question Answering Vision-Language Grounding

Open paper AskPDF arXiv

Answer grounding in document visual question answering remains an open challenge: most benchmarks lack grounding annotations or provide limited-quality labels, while constructing grounded datasets still requires costly manual effort. We introduce DocAttriBench (DAB), a large-scale benchmark for fine-grained, element-level source attribution in Document VQA, grounding answers to specific layout elements such as text blocks, tables, and images. To build DAB, we propose a Mask-based Perplexity-Derived Attribution method (MAPPET) that combines document layout and language modeling to identify the most informative element for each answer. MAPPET measures the increase in perplexity after masking candidate elements and attributes the answer to the element contributing most to model confidence. Applying MAPPET to multiple existing Document VQA datasets yields DAB, with 237k documents and 296k question-answer pairs with element-level grounding. We benchmark grounding-capable multimodal LLMs on DAB, evaluating answer accuracy, attribution accuracy, and overall answer quality. Results show that while larger models generally achieve higher answer accuracy, even the strongest models often fail to localize the supporting elements. DAB provides a scalable benchmark for developing grounded, verifiable, and trustworthy Document VQA models. Dataset and code are available at https://aimagelab.github.io/DocAttriBench/.

Abstract

Long-Document VQA

Explore similar work

DocAttriBench: Benchmarking Answer Grounding in Document Visual Question Answering

VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA

DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding