Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA

Yuanlei Zheng; Pei Fu; Hang Li; Ziyang Wang; Yuyi Zhang; Wenyu Ruan; Xiaojin Zhang; Zhongyu Wei

cs.CLApr 15, 2026

Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA

Authors: Yuanlei Zheng, Pei Fu, Hang Li, Ziyang Wang, Yuyi Zhang, Wenyu Ruan, Xiaojin Zhang, Zhongyu Wei, +4 more

Organizations: School of Software Engineering, Huazhong University of Science and Technology · MiLM Plus, Xiaomi Inc. · School of Computer Science and Technology, Huazhong University of Science and Technology · School of Data Science, Fudan University

Abstract

Multi-page Document Visual Question Answering requires reasoning over semantics, layouts, and visual elements in long, visually dense documents. Existing OCR-free methods face a trade-off between capacity and precision: end-to-end models scale poorly with document length, while visual retrieval-based pipelines are brittle and passive. We propose Doc- $V^*$ , an OCR-free agentic framework that casts multi-page DocVQA as sequential evidence aggregation. Doc- $V^*$ begins with a thumbnail overview, then actively navigates via semantic retrieval and targeted page fetching, and aggregates evidence in a structured working memory for grounded reasoning. Trained by imitation learning from expert trajectories and further optimized with Group Relative Policy Optimization, Doc- $V^*$ balances answer accuracy with evidence-seeking efficiency. Across five benchmarks, Doc- $V^*$ outperforms open-source baselines and approaches proprietary models, improving out-of-domain performance by up to 47.9% over RAG baseline. Other results reveal effective evidence aggregation with selective attention, not increased input pages.

Figures & tables

Figure 1: The Doc- $V^{*}$ agent workflow for multi-page document VQA. It adopts an active perception paradigm by planning from a global thumbnail view and iteratively deciding when to fetch high-resolution pages or perform semantic searches, aggregating evidence in a structured working memory for grounded answering.

Figure 2: Overview of the training pipeline for Doc- $V^{*}$ . (a) Training data construction. Documents and queries are paired to generate thumbnail-guided reasoning trajectories, followed by quality filtering. (b) Supervised fine-tuning (SFT). (c) Reinforcement learning with GRPO.

Method	Backbone	Param	Paradigm	DUDE (ANLS)	MPDocVQA (ANLS)	SlideVQA (F1)	MMLong. (Acc)	LongDoc. (Acc)
Closed Source
Gemini-1.5-Pro	-	-	E2E	46.0	-	-	28.2	50.9
GPT-4o mini	-	-	E2E	46.5	-	60.7	28.6	-
GPT-4o	-	-	E2E	54.1	67.4	65.8	42.8	64.5
GPT-4.1	-	-	E2E	50.2	-	74.7	45.6	-
Claude-3.7-Sonnet	-	-	E2E	58.1	-	76.3	33.9	-

Table 1: Comparison of different methods on five long-context and multi-page document understanding benchmarks. The results are reported on DUDE (ANLS), MPDocVQA (ANLS), SlideVQA (F1), MMLongBench-Doc (Acc), and LongDocURL (Acc). “Param.” denotes the parameter scale (referring specifically to the Generator for RAG methods). “Backbone” indicates the underlying LLM or LVLM used. “Paradigm” categorizes methods into End-to-End ( E2E ), Retrieval-Augmented Generation ( RAG ), or Agent . The best and second-best results among Open Source methods are highlighted in bold and underlined , respectively. Scores marked with an asterisk ( ∗ ) indicate that the method’s backbone was supervised fine-tuned on that specific benchmark’s training set. Red subscripts in parentheses indicate the absolute performance gain over the baseline (Qwen2.5-VL).

Figure 3: Efficiency–effectiveness trade-off across SlideVQA , LongDocURL , and MMLongBench-Doc .The top row reports Page-F1, measuring the quality of page selection under different input budgets, while the bottom row shows downstream task performance. For Doc-

V^{*}

, Page-F1 is computed based on the pages that the model explicitly predicts as relevant, i.e., the model outputs a set of relevant_pages , which are then compared against the ground-truth evidence pages to compute F1.

Retriever	Model	Avg. Pages	Page-F1	Overall	SIN	MUL	UNA
ColQwen	Qwen2.5-VL	6.0	30.9	35.5	37.0	13.4	70.4
ColQwen	Doc- $V^{*}$	5.6	49.7	42.1	54.6	23.5	45.7
BGE-Large	Qwen2.5-VL	9.0	17.6	33.0	31.2	9.8	77.1
BGE-Large	Doc- $V^{*}$	8.4	34.0	36.3	45.7	18.5	45.7
BM25	Qwen2.5-VL	10.0	20.5	32.9	32.7	11.3	71.3
BM25	Doc- $V^{*}$	9.2	36.8	37.5	48.4	20.4	43.0

Table 2: Comparison of different retrievers on MMLongBench-Doc.

Figure 4: Accuracy vs. document length under different methods (RAG uses top-k = 5 retrieval).

Figure 5: (a) : Comparison of average inference latency per sample across different methods. (b) : Comparison of average peak GPU memory consumption per sample under different methods.

Cognitive Modules			MMLong.	LongDoc.	SlideVQA
T	A	M	(Acc)	(Acc)	(F1)
✓	✓	✓	39.8	53.0	73.8
✗	✓	✓	$34.9_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}4.9)}}}$	$46.3_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}6.7)}}}$	$68.3_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}5.5)}}}$
✓	✗	✓	$35.9_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}4.7)}}}$	$49.5_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}3.5)}}}$	$71.8_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}2.0)}}}$
✓	✓	✗	$36.4_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}3.4)}}}$	$47.1_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}5.9)}}}$	$69.8_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}4.0)}}}$

Table 3: Ablation study on the cognitive modules of the Doc-

V^{*}

agent. T : Global Thumbnail Overview; A : Page-by-page content analyis; M : Memory.

Setting	Acc $\uparrow$	Avg. Pages $\downarrow$
Doc- $V^{*}$	39.8	6.4
w/o Retrieval	34.9	14.2
w/o Fetch	35.2	5.9

Table 4: Action ablation study on MMLongBench-Doc. Removing either retrieval or fetch leads to clear performance degradation.

Metric	SlideVQA		LongDoc.		MMLong.
Metric	RP	FP	RP	FP	RP	FP
Ratio	97.6	4.1	99.8	3.6	94.0	14.7
Recall	95.7	70.9	83.4	37.3	75.4	55.9
Precision	39.0	81.2	32.7	36.6	33.1	49.9
F1	54.1	72.9	44.4	31.9	42.1	49.6

Table 5: Page-level analysis of agent tool usage and retrieval quality across three benchmarks. RP denotes pages retrieved by the retrieval_page , while FP denotes pages obtained via the fetch_page . Ratio indicates the proportion of samples in which the corresponding tool is invoked. Recall , Precision , and F1 are computed at the page level.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Average document length across datasets. The figure reports the average number of pages per document for MP-DocVQA, DUDE, SlideVQA, LongDocURL, and MMLongBench-Doc, illustrating the increasing document length and context complexity from standard document QA benchmarks to long-context multi-modal settings.

Method	Retriever	Backbone	Param	OCR-Free	Paradigm	Training Dataset
Method	Retriever	Backbone	Param	OCR-Free	Paradigm	DUDE	MPDocVQA	SlideVQA
HiVT5 (PR)	-	DiT / T5	0.3B	$\times$	E2E	$\times$	$\checkmark$	$\times$
CREAM (ACM MM’24)	bge-large	Pix2Struct / LLaMa2	7B	$\times$	RAG	$\checkmark$	$\checkmark$	$\times$
mPLUG-DocOwl2 (ACL’25)	-	ViT / LLaMa	8B	$\checkmark$	E2E	$\checkmark$	$\checkmark$	$\times$
M3DocRAG (arXiv’24)	Colpali	Qwen2-VL	7B	$\checkmark$	RAG	$\times$	$\times$	$\times$
VisRAG (ICLR’25)	VisRAG-Ret	MiniCPM-V 2.6	8B	$\checkmark$	RAG	$\times$	$\times$	$\times$

Appendix

Table 6: Detailed configurations of Open Source baselines. “Retriever” denotes the model used for page retrieval. “Param” refers to the parameter size of the LLM backbone. “Paradigm” categorizes methods into End-to-End ( E2E ), Retrieval-Augmented Generation ( RAG ), or Agent . The columns under “Training Dataset” indicate whether the backbone was fine-tuned (

\checkmark

) on the corresponding benchmark’s training set or evaluated in a zero-shot setting (

\times

K	Avg. Pages	Overall	Breakdown
K	Avg. Pages	Overall	SIN	MUL	UNA
Adaptive	5.6	42.1	54.6	23.5	45.7
1	3.0	40.5	51.6	17.0	56.1
2	4.3	39.7	54.4	20.7	39.9
3	5.4	40.1	53.3	23.0	40.8
4	6.5	41.1	53.3	24.0	43.5

Appendix

Table 7: Impact of K on MMLongBench-Doc. “Adaptive” denotes the document-adaptive setting

K=\min(\lceil N/10\rceil,4)

, where

N

is the total number of pages.

Iteration	Avg. Pages	Overall	Breakdown
Iteration	Avg. Pages	Overall	SIN	MUL	UNA
3	4.2	41.1	54.0	22.5	44.4
4	4.6	41.2	54.3	23.0	43.5
5	4.9	41.4	54.3	23.4	43.5
6	5.2	41.5	54.4	24.0	44.0
7	5.6	42.1	54.6	23.5	45.7

Appendix

Table 8: Impact of maximum interaction steps on MMLongBench-Doc.

Figure 7: Case 1 in SlideVQA between different methods

Figure 8: Case 2 in SlideVQA between different methods

Figure 9: Case 3 in SlideVQA between different methods

Explore similar work

InSight-doc: Agentic Visual Perception for Long-Document Understanding
Aug 11, 2026Kaican Li, Weiyan Xie, Lewei Yao +4Document Understanding Visual Question Answering
1The Hong Kong University of Science and Technology · 2Huawei
Capability-Routed Visual Retrieval and Evidence Threading for Long-Context Document Question Answering
Sep 7, 2026Amirul Rahman, Aisha Karim, Kenji Nakamura +1Document QA Multimodal RAG
University of Malaya
HierDoc: Hierarchical Page-to-Region Evidence Routing for Long-Document Visual Question Answering
Jul 31, 2026Rongjian Gu, Wengang Zhou, Junyu Xiong +4Visual Question Answering Document QA
University of Science and Technology of China · 2iFLYTEK Research · 3Anhui Transport Consulting & Design Institute Co. Ltd., Hefei, China

Aug 11, 2026J/K move · Enter open · S save

InSight-doc: Agentic Visual Perception for Long-Document Understanding

Kaican Li, Weiyan Xie, Lewei Yao, Jiannan Wu+3

1The Hong Kong University of Science and Technology · 2Huawei

Document Understanding Visual Question Answering Tool-Using Vision-Language Agents Active Visual Perception Document Image Understanding

Open paper AskPDF arXiv

Long-document understanding often requires reasoning over many visually rich pages, making inference costly and prone to context rot. In this work, we propose InSight-doc, an agentic visual perception framework that treats visual resolution as an adaptive reasoning-time resource. InSight-doc starts from low resolution and selectively zooms into high-resolution regions for finer evidence, without relying on any external retriever. To train such an agent, we construct an active-perception corpus of 17.9K high-quality SFT examples with region-level zoom-in trajectories, accompanied by 19.2K hard RL examples. Through SFT+RL, InSight-doc-8B improves the baseline by 4.3--16.4 accuracy points over document VQA benchmarks. On long documents, it reduces hallucination by more than 40% and inference latency by 41%--68% while maintaining an accuracy lead. Our code, datasets, and model are released at https://github.com/m-Just/InSight-doc .

Abstract

Explore similar work

InSight-doc: Agentic Visual Perception for Long-Document Understanding

Capability-Routed Visual Retrieval and Evidence Threading for Long-Context Document Question Answering

HierDoc: Hierarchical Page-to-Region Evidence Routing for Long-Document Visual Question Answering