VisProve: Learning Typed Visual Provenance for Scientific Document Question Answering

Jingxuan Wei; Zhengyang Wang; Zheng Guo; Xiaoming Wu; Fuqiang Wang; Xin Wang; Bihui Yu; Yining Wang

cs.CVNov 7, 2026

VisProve: Learning Typed Visual Provenance for Scientific Document Question Answering

Authors: Jingxuan Wei, Zhengyang Wang, Zheng Guo, Xiaoming Wu, Fuqiang Wang, Xin Wang, Bihui Yu, Yining Wang,

Organizations: University of Chinese Academy of Sciences · Qilu University of Technology · Shandong Computer Science Center · Unisound AI ·

Abstract

Visual citations show where an answer is supported, but not how the cited content contributes to it. In scientific document question answering, a model may read a value directly, compress several observations, or combine evidence to infer a conclusion. We study typed visual provenance, where a system generates an answer together with source regions and explicit support relations. We introduce ReViE, a dataset of 3,941 questions over complete scientific documents, with page, section, and region annotations labeled as Quotation, Compression, or Inference. We further propose VisProve, which augments joint answer and provenance generation with differentiable Spatial Evidence Reading and Evidence-Answer Relation Feedback. VisProve conditions next-token prediction on localized visual evidence and its estimated support relation through a gated norm-bounded residual. On the document-disjoint ReViE test set, VisProve improves Typed-F1 from 3.52 to 5.48 and Section-F1 from 45.99 to 53.24 over joint supervised fine-tuning with the same backbone, while Page-F1 changes only marginally. These results suggest that coupling visual grounding with support relation modeling mainly improves fine-grained provenance, while the low absolute Typed-F1 reveals a substantial gap between answer generation and precise visual evidence attribution.

Figures & tables

Figure 1: Overview of fine-grained typed visual provenance. The model links a complete answer to supporting visual regions and specifies how each region contributes to the answer.

Figure 2: ReViE construction. Source QA pairs and newly generated, verified multi-hop QA pairs undergo fixed-QA evidence refinement. A later branch constructs additional QA pairs from contiguous evidence across adjacent pages. Supporting regions receive relation annotations, and complete ordered document pages are restored for every retained instance.

Figure 3: Evidence-region distribution by spatial structure and support relation. Percentages are over 8,523 annotated regions.

Figure 4: VisProve. Joint SFT learns answer–provenance generation. SER reads localized visual features, ERF encodes their support relation, and a gated residual combines the two representations before the original language head.

Model	Answer		Localization				Typed Prov.		Summary
	R-L $\uparrow$	B-F1 $\uparrow$	P-F1 $\uparrow$	S-F1 $\uparrow$	E-IoU $\uparrow$	R-F1 $\uparrow$	Rel-F1 $\uparrow$	T-F1 $\uparrow$	Overall $\uparrow$	Judge $\uparrow$
Open-source models
Qwen3-VL-8B	48.27	86.08	66.55	31.90	1.32	1.60	0.73	0.18	30.99	3.04
InternVL3.5-8B	26.63	84.26	21.81	14.35	0.54	0.76	0.66	0.00	21.71	2.66
MiniCPM-V4.5-8B	10.73	80.99	1.88	4.63	0.00	0.00	0.00	0.00	15.83	1.88
Qwen3-VL-32B	33.24	84.70	66.16	37.01	1.77	2.20	0.93	0.87	28.89	3.34

Table 1: Main results on ReViE. E-IoU, R-F1, and Rel-F1 condition on exact page-set prediction; T-F1 uses all 409 test instances. Overall equally averages the three metric categories. Judge uses 50 shared questions. Light orange and light green mark the best and second-best values, respectively.

Model	Answer		Localization				Typed Prov.		Summary
	R-L $\uparrow$	B-F1 $\uparrow$	P-F1 $\uparrow$	S-F1 $\uparrow$	E-IoU $\uparrow$	R-F1 $\uparrow$	Rel-F1 $\uparrow$	T-F1 $\uparrow$	Overall $\uparrow$	Judge $\uparrow$
VisProve (Ours)	67.08	94.18	75.73	53.24	8.58	9.68	9.22	5.48	41.60	3.46
w/o ERF	66.85	93.92	75.21	45.06	6.66	7.60	7.03	3.73	39.80	3.56
w/o SER	67.82	93.96	74.69	48.61	6.86	7.94	7.29	4.11	40.37	3.54
Default SFT	64.24	93.85	75.00	45.99	5.07	5.91	5.22	3.52	38.80	3.52

Table 2: Ablation study on ReViE. Metric definitions, evaluation conditions, and best/second-best highlighting follow Table 1 .

Method	Answer		Localization			Summary
	R-L $\uparrow$	B-F1 $\uparrow$	P-F1 $\uparrow$	E-IoU $\uparrow$	R-F1 $\uparrow$	Overall ${}_{\mathrm{shared}}\!\uparrow$
VISA ( Ma et al., 2025 )	11.65	85.53	7.50	0.02	0.02	25.55
CoE ( Liu et al., 2026a )	10.45	84.57	30.42	1.37	1.55	29.31
UniDoc-RL-7B ( Wang et al., 2026 )	28.37	87.10	65.31	0.60	0.77	39.98
VisProve (Ours)	67.08	94.18	75.73	8.58	9.68	55.98

Table 3: Comparison in the shared answer–page–region output space.

\mathrm{Overall}_{\mathrm{shared}}

uses two categories and differs from Overall in Table 1 . E-IoU and R-F1 retain the page-exact conditioning. Light orange and light green mark the best and second-best values, respectively.

Figure 5: Consistency between LLM-as-Judge scores and human ratings across 11 evaluated models.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Split	Instances	Documents	Pages	Evidence regions
Train	3,115	1,296	25,439	6,738
Validation	417	161	3,106	921
Test	409	155	3,436	864
Total	3,941	1,612	31,981	8,523

Appendix

Table 4: Split-level statistics of ReViE. Page counts refer to unique document pages within each split.

Domain	Instances	Instance %	Documents	Document %
Computer Science	493	12.51	200	12.41
Economics	330	8.37	169	10.48
Electrical Engineering and Systems Science	662	16.80	232	14.39
Mathematics	107	2.72	59	3.66
Physics	480	12.18	184	11.41
Quantitative Biology	555	14.08	273	16.94

Appendix

Table 5: Domain distribution of ReViE at the instance and document levels.

Evidence modality			Evidence–answer relation
Modality	Instances	%	Relation	Regions	%
Text	2,803	71.12	Quotation	6,555	76.91
Figure	632	16.04	Compression	242	2.84
Table	212	5.38	Inference	1,726	20.25
Mixed	294	7.46
Total	3,941	100.00	Total	8,523	100.00

Appendix

Table 6: Evidence modality and evidence–answer relation distributions in ReViE. Modality statistics are computed over 3,941 instances, whereas relation statistics are computed over 8,523 evidence regions.

	Document		Visual evidence				Supervision
Benchmark	Sci.	Input	Locator	MR	MP	Contig.	Hop unit	Rel.
DocVQA	—	1	—	—	—	—	—	—
MP-DocVQA	—	Cap	P	—	—	—	—	—
SlideVQA	—	Cap	P	—	$\checkmark$	—	Slide	—
MMLongBench-Doc	$\circ$	Full	P	—	$\checkmark$	—	—	—
XL-DocBench	$\circ$	Full	P+T	—	$\checkmark$	—	Sect.	—

Appendix

Table 7: Comparison of ReViE with seven visual document benchmarks. Columns summarize document context, visual evidence, and reasoning and relation supervision.

Configuration	Trainable components	Auxiliary supervision	Field CE values
w/o ERF	Spatial reader, geometric fusion, gate, and up-projection	Page and box at $\tau_{b}$ , with two loss contributions	rel_bbox
w/o SER	Relation classifier and relation embeddings only	Relation at $\tau_{r}$	section , relation
VisProve	All provenance-enhancement components	Page, box, and relation at $\tau_{r}$ ; page and box at $\tau_{s}$	rel_bbox , section , relation

Appendix

Table 8: Configuration-specific training of provenance-enhancement components. The SFT backbone remains frozen in all variants.

Figure 9: Representative prediction errors. Left: a correct chart-reading answer paired with a misplaced evidence box. Middle: incomplete cross-page evidence coverage accompanied by an incorrect identification. Right: incomplete evidence coverage, an underspecified answer, and a relation discrepancy.

Evidence-Grounded QA

See all 56 papers

VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents
Oct 14, 2026Zhivar Sourati, Marianne Menglin Liu, Rongguang Wang +12Long-Document Visual Question Answering Document QA
University of Southern California · Oracle AI · Turing Enterprise Inc +1
ProvenAI: Provenance-Native Traces of Evidence in Generated Answers
Jun 24, 2026Mohammad Faizan, Dalal AlharthiMulti-Hop QA Data Provenance
College of Information Science University of Arizona
CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence
May 13, 2026Dongsheng Ma, Jiayu Li, Zhengren Wang +9VLM Evaluation Citation Verification
Peking University · Shanghai Artificial Intelligence Laboratory

Oct 14, 2026J/K move · Enter open · S save

VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents

Zhivar Sourati, Marianne Menglin Liu, Rongguang Wang, Zheng Wang+11

University of Southern California · Oracle AI · Turing Enterprise Inc+1

Long-Document Visual Question Answering Document QA Evidence-Grounded QA Visual Question Answering VLM Evaluation Evidence Retrieval

Open paper AskPDF arXiv

Existing long-document QA benchmarks rarely combine large-scale human-authored questions with detailed visual evidence annotations, limiting diagnosis of evidence localization and interpretation failures. We present VELD-QA, a large-scale benchmark for Visual Evidence grounding and question answering in Long Documents. It contains 15,000 fully human-authored question-answer pairs across 5,859 real-world PDFs totaling 372,445 pages, balancing chart, table, and complex-layout tasks. Human annotations identify supporting pages and evidence elements and describe cited visual content, enabling separate retrieval and interpretation evaluation. We evaluate nine vision-language models under four evidence settings and seven retrieval-augmented and agentic systems. Our analysis identifies bottlenecks in evidence localization and interpretation. Retrieval-based methods substantially underperform the oracle setting with ground-truth supporting pages; full-document access does not close the gap. Localization alone is insufficient: oracle accuracy peaks at 81.7%, highlighting persistent visual interpretation and reasoning challenges. Enriching inputs with our visual-element annotations narrows the performance gap by 17-24%, demonstrating value beyond evaluation and motivating data enrichment for multimodal document QA. Our code is available at https://github.com/oracle-samples/veld-qa.

Abstract

Evidence-Grounded QA

Explore similar work

VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents

ProvenAI: Provenance-Native Traces of Evidence in Generated Answers

CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence