Found but Not Read: When Extracted Text Closes the Retrieval-Reading Gap in Document Vision-Language Models

Qingtao Xia; Siyao Cheng; Jiahua Bao; Jiaxing Du; Jie Liu

cs.CVOct 2, 2026

Found but Not Read: When Extracted Text Closes the Retrieval-Reading Gap in Document Vision-Language Models

Authors: Qingtao Xia, Siyao Cheng, Jiahua Bao, Jiaxing Du, Jie Liu

Organizations: Harbin Institute of Technology, Harbin, China

Abstract

Retrieval-augmented document question answering assumes that once the right page is found, a vision-language model (VLM) can read it. We show that this assumption often fails, leaving a retrieval-reading gap: evidence found but not used. A paired protocol isolates this gap by comparing answers from the retrieved page images alone with answers from the same images plus their extracted text. On FoveDoc-Bench, our benchmark with traceable evidence, retrieval finds nearly every evidence page, yet adding CPU-OCR text raises strict accuracy by 13 to 16 points. An exact text layer roughly doubles the gain, which appears across six VLMs from three families and, within one family, narrows with scale without closing. The reader can read this evidence but cannot find it: crops of it recover most of the text gain, boxes around it on the page none. The same protocol identifies two boundaries. Extracted text helps on textual evidence but is neutral or harmful on charts and figures. Its advantage shrinks as retrieval degrades, and unrelated text of the same form adds nothing detectable. Extracted text is an amplifier of retrieval that works, not a substitute for retrieval that does not. Our code is available at https://github.com/atoz03/fovedoc-sup.

Figures & tables

Figure 1: The paired protocol and its headline result. Preparation: ColQwen2 retrieves $16$ page images per question at saturated recall; one V-NIAH and one V-MQAR sample are shown verbatim. Step 1: one frozen VLM answers from the images (arm A) and from the same images plus the $16$ CPU-OCR blocks scoring highest against the question (arm B), so $\Delta$ is evidence the reader did not use from pixels already in its context. Step 2: strict accuracy (Qwen3-VL-2B, $n{=}1173$ per task; pp $=$ percentage points) and one verbatim evidence case per direction.

benchmark	questions	pages/doc	gold pages	recall@16
V-NIAH	1173	16–24	1	1.000
V-MQAR	1173	16–24	2	0.998
MMLongBench-Doc	$1091^{\ast}$	$\leq 468$	$0$ – $n$	0.844

Table 1: Benchmarks. Recall@16 is the fraction of questions whose every gold page is among the

16

ColQwen2 retrieves: FoveDoc-Bench is saturated, MMLongBench-Doc is not.

task	images	+OCR	+PDF	$\Delta_{\text{OCR}}$	$\Delta_{\text{PDF}}$	retained
V-MQAR	0.153	0.287	0.432	$+0.134$	$+0.279$	48%
V-NIAH	0.350	0.506	0.691	$+0.157$	$+0.342$	46%

Table 2: The retrieval–reading gap. Strict accuracy, Qwen3-VL-2B,

n{=}1173

per task, paired on identical pages. Both memory arms add packed text to the

16

page images. “Retained” is the OCR gain as a share of the text-layer gain. OCR contrasts:

95\%

CIs

[+0.109,+0.158]

and

[+0.127,+0.187]

, both

p<10^{-23}

Figure 2: Text-layer gains for six VLMs from three families. (a) Strict accuracy from images (grey) to images plus text-layer memory (blue) for six stock readers on V-MQAR (circles) and V-NIAH (squares), first

400

samples per task; labels give the paired gain in pp, every cell positive at

p<10^{-13}

. † Image splitting disabled or pixels capped to fit

16

pages. (b) Within Qwen3-VL at identical settings the gap narrows monotonically with scale and is still

+0.23

at 8B.

stratum	$n$	images	+mem.	$\Delta$	$p$
answerable, annotated	838	0.220	0.223	$+0.004$	0.86
full recall@16	706	0.227	0.242	$+0.016$	0.32
partial recall	132	0.182	0.121	$-0.061$	0.099
unanswerable	244	0.041	0.090	$+0.049$	0.014

Table 3: MMLongBench-Doc, paired, strict accuracy, same reader, retriever, memory builder and judge as Table 2 ; Bonferroni

\alpha{=}0.0125

over the four strata.

Figure 3: Retrieval and modality. (a) ColQwen2 full recall@

k

: V-NIAH and V-MQAR (

1173

each), and MMLongBench-Doc (

845

questions with gold pages). (b) MMLongBench-Doc memory

-

images by evidence tag (tags overlap,

95\%

CIs) and, below the rule, on the

414

single-tag questions: text against chart/figure differs by

+13.1

pp.

Figure 4: Three controls on the memory advantage, all with OCR memory, each on a real sample; results in Fig. 5 . Control 1 varies recall at a fixed $16$ -page budget on the same samples. Control 2 drops zero-recall samples whose packed memory contains the answer string. Control 3 builds the memory from another document’s text under the same rule.

Figure 5: Results of the controls in Fig. 4 . (a, b) The memory advantage (labelled) falls monotonically as gold pages are withdrawn; foreign-document text tracks the image arm at both endpoints. (c) At zero recall the leak-free residual survives, but foreign-document memory of identical form is indistinguishable from the image arm: what remains is same-document context, not format. Error bars are

95\%

CIs.

Document QA

Document question answering uses AI to answer questions from text, tables, images, and structured content in documents.

See all 92 papers

VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents
Oct 14, 2026Zhivar Sourati, Marianne Menglin Liu, Rongguang Wang +12Long-Document VQA Document QA
University of Southern California · Oracle AI · Turing Enterprise Inc +1
Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA
Apr 15, 2026Yuanlei Zheng, Pei Fu, Hang Li +9Document QA Long-Document VQA
School of Software Engineering, Huazhong University of Science and Technology · MiLM Plus, Xiaomi Inc. · School of Computer Science and Technology, Huazhong University of Science and Technology +1
An Empirical Study of VLM Pipelines for Long-Document QA
Sep 24, 2026Kenan E. Ak, Jay Mohta, Gwang Gook Lee +2VLM Evaluation Tool Use in VLMs

Oct 14, 2026J/K move · Enter open · S save

VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents

Zhivar Sourati, Marianne Menglin Liu, Rongguang Wang, Zheng Wang+11

University of Southern California · Oracle AI · Turing Enterprise Inc+1

Long-Document VQA Document QA Evidence-Grounded QA Visual Question Answering VLM Evaluation Evidence Retrieval

Open paper AskPDF arXiv

Existing long-document QA benchmarks rarely combine large-scale human-authored questions with detailed visual evidence annotations, limiting diagnosis of evidence localization and interpretation failures. We present VELD-QA, a large-scale benchmark for Visual Evidence grounding and question answering in Long Documents. It contains 15,000 fully human-authored question-answer pairs across 5,859 real-world PDFs totaling 372,445 pages, balancing chart, table, and complex-layout tasks. Human annotations identify supporting pages and evidence elements and describe cited visual content, enabling separate retrieval and interpretation evaluation. We evaluate nine vision-language models under four evidence settings and seven retrieval-augmented and agentic systems. Our analysis identifies bottlenecks in evidence localization and interpretation. Retrieval-based methods substantially underperform the oracle setting with ground-truth supporting pages; full-document access does not close the gap. Localization alone is insufficient: oracle accuracy peaks at 81.7%, highlighting persistent visual interpretation and reasoning challenges. Enriching inputs with our visual-element annotations narrows the performance gap by 17-24%, demonstrating value beyond evaluation and motivating data enrichment for multimodal document QA. Our code is available at https://github.com/oracle-samples/veld-qa.

Abstract

Document QA

Explore similar work

VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents

Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA

An Empirical Study of VLM Pipelines for Long-Document QA