Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA

Yuanlei Zheng; Pei Fu; Hang Li; Ziyang Wang; Yuyi Zhang; Wenyu Ruan; Xiaojin Zhang; Zhongyu Wei

cs.CLApr 15, 2026

Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA

Authors: Yuanlei Zheng, Pei Fu, Hang Li, Ziyang Wang, Yuyi Zhang, Wenyu Ruan, Xiaojin Zhang, Zhongyu Wei, +4 more

Organizations: School of Software Engineering, Huazhong University of Science and Technology · MiLM Plus, Xiaomi Inc. · School of Computer Science and Technology, Huazhong University of Science and Technology · School of Data Science, Fudan University

Abstract

Multi-page Document Visual Question Answering requires reasoning over semantics, layouts, and visual elements in long, visually dense documents. Existing OCR-free methods face a trade-off between capacity and precision: end-to-end models scale poorly with document length, while visual retrieval-based pipelines are brittle and passive. We propose Doc- $V^*$ , an OCR-free agentic framework that casts multi-page DocVQA as sequential evidence aggregation. Doc- $V^*$ begins with a thumbnail overview, then actively navigates via semantic retrieval and targeted page fetching, and aggregates evidence in a structured working memory for grounded reasoning. Trained by imitation learning from expert trajectories and further optimized with Group Relative Policy Optimization, Doc- $V^*$ balances answer accuracy with evidence-seeking efficiency. Across five benchmarks, Doc- $V^*$ outperforms open-source baselines and approaches proprietary models, improving out-of-domain performance by up to 47.9% over RAG baseline. Other results reveal effective evidence aggregation with selective attention, not increased input pages.

Figures & tables

Figure 1: The Doc- $V^{*}$ agent workflow for multi-page document VQA. It adopts an active perception paradigm by planning from a global thumbnail view and iteratively deciding when to fetch high-resolution pages or perform semantic searches, aggregating evidence in a structured working memory for grounded answering.

Figure 2: Overview of the training pipeline for Doc- $V^{*}$ . (a) Training data construction. Documents and queries are paired to generate thumbnail-guided reasoning trajectories, followed by quality filtering. (b) Supervised fine-tuning (SFT). (c) Reinforcement learning with GRPO.

Method	Backbone	Param	Paradigm	DUDE (ANLS)	MPDocVQA (ANLS)	SlideVQA (F1)	MMLong. (Acc)	LongDoc. (Acc)
Closed Source
Gemini-1.5-Pro	-	-	E2E	46.0	-	-	28.2	50.9
GPT-4o mini	-	-	E2E	46.5	-	60.7	28.6	-
GPT-4o	-	-	E2E	54.1	67.4	65.8	42.8	64.5
GPT-4.1	-	-	E2E	50.2	-	74.7	45.6	-
Claude-3.7-Sonnet	-	-	E2E	58.1	-	76.3	33.9	-

Table 1: Comparison of different methods on five long-context and multi-page document understanding benchmarks. The results are reported on DUDE (ANLS), MPDocVQA (ANLS), SlideVQA (F1), MMLongBench-Doc (Acc), and LongDocURL (Acc). “Param.” denotes the parameter scale (referring specifically to the Generator for RAG methods). “Backbone” indicates the underlying LLM or LVLM used. “Paradigm” categorizes methods into End-to-End ( E2E ), Retrieval-Augmented Generation ( RAG ), or Agent . The best and second-best results among Open Source methods are highlighted in bold and underlined , respectively. Scores marked with an asterisk ( ∗ ) indicate that the method’s backbone was supervised fine-tuned on that specific benchmark’s training set. Red subscripts in parentheses indicate the absolute performance gain over the baseline (Qwen2.5-VL).

Figure 3: Efficiency–effectiveness trade-off across SlideVQA , LongDocURL , and MMLongBench-Doc .The top row reports Page-F1, measuring the quality of page selection under different input budgets, while the bottom row shows downstream task performance. For Doc-

V^{*}

, Page-F1 is computed based on the pages that the model explicitly predicts as relevant, i.e., the model outputs a set of relevant_pages , which are then compared against the ground-truth evidence pages to compute F1.

Retriever	Model	Avg. Pages	Page-F1	Overall	SIN	MUL	UNA
ColQwen	Qwen2.5-VL	6.0	30.9	35.5	37.0	13.4	70.4
ColQwen	Doc- $V^{*}$	5.6	49.7	42.1	54.6	23.5	45.7
BGE-Large	Qwen2.5-VL	9.0	17.6	33.0	31.2	9.8	77.1
BGE-Large	Doc- $V^{*}$	8.4	34.0	36.3	45.7	18.5	45.7
BM25	Qwen2.5-VL	10.0	20.5	32.9	32.7	11.3	71.3
BM25	Doc- $V^{*}$	9.2	36.8	37.5	48.4	20.4	43.0

Table 2: Comparison of different retrievers on MMLongBench-Doc.

Figure 4: Accuracy vs. document length under different methods (RAG uses top-k = 5 retrieval).

Figure 5: (a) : Comparison of average inference latency per sample across different methods. (b) : Comparison of average peak GPU memory consumption per sample under different methods.

Cognitive Modules			MMLong.	LongDoc.	SlideVQA
T	A	M	(Acc)	(Acc)	(F1)
✓	✓	✓	39.8	53.0	73.8
✗	✓	✓	$34.9_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}4.9)}}}$	$46.3_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}6.7)}}}$	$68.3_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}5.5)}}}$
✓	✗	✓	$35.9_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}4.7)}}}$	$49.5_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}3.5)}}}$	$71.8_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}2.0)}}}$
✓	✓	✗	$36.4_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}3.4)}}}$	$47.1_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}5.9)}}}$	$69.8_{{\color[rgb]{0.1328,0.5469,0.1328}\textbf{(\text{-}4.0)}}}$

Table 3: Ablation study on the cognitive modules of the Doc-

V^{*}

agent. T : Global Thumbnail Overview; A : Page-by-page content analyis; M : Memory.

Setting	Acc $\uparrow$	Avg. Pages $\downarrow$
Doc- $V^{*}$	39.8	6.4
w/o Retrieval	34.9	14.2
w/o Fetch	35.2	5.9

Table 4: Action ablation study on MMLongBench-Doc. Removing either retrieval or fetch leads to clear performance degradation.

Metric	SlideVQA		LongDoc.		MMLong.
Metric	RP	FP	RP	FP	RP	FP
Ratio	97.6	4.1	99.8	3.6	94.0	14.7
Recall	95.7	70.9	83.4	37.3	75.4	55.9
Precision	39.0	81.2	32.7	36.6	33.1	49.9
F1	54.1	72.9	44.4	31.9	42.1	49.6

Table 5: Page-level analysis of agent tool usage and retrieval quality across three benchmarks. RP denotes pages retrieved by the retrieval_page , while FP denotes pages obtained via the fetch_page . Ratio indicates the proportion of samples in which the corresponding tool is invoked. Recall , Precision , and F1 are computed at the page level.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Average document length across datasets. The figure reports the average number of pages per document for MP-DocVQA, DUDE, SlideVQA, LongDocURL, and MMLongBench-Doc, illustrating the increasing document length and context complexity from standard document QA benchmarks to long-context multi-modal settings.

Method	Retriever	Backbone	Param	OCR-Free	Paradigm	Training Dataset
Method	Retriever	Backbone	Param	OCR-Free	Paradigm	DUDE	MPDocVQA	SlideVQA
HiVT5 (PR)	-	DiT / T5	0.3B	$\times$	E2E	$\times$	$\checkmark$	$\times$
CREAM (ACM MM’24)	bge-large	Pix2Struct / LLaMa2	7B	$\times$	RAG	$\checkmark$	$\checkmark$	$\times$
mPLUG-DocOwl2 (ACL’25)	-	ViT / LLaMa	8B	$\checkmark$	E2E	$\checkmark$	$\checkmark$	$\times$
M3DocRAG (arXiv’24)	Colpali	Qwen2-VL	7B	$\checkmark$	RAG	$\times$	$\times$	$\times$
VisRAG (ICLR’25)	VisRAG-Ret	MiniCPM-V 2.6	8B	$\checkmark$	RAG	$\times$	$\times$	$\times$

Appendix

Table 6: Detailed configurations of Open Source baselines. “Retriever” denotes the model used for page retrieval. “Param” refers to the parameter size of the LLM backbone. “Paradigm” categorizes methods into End-to-End ( E2E ), Retrieval-Augmented Generation ( RAG ), or Agent . The columns under “Training Dataset” indicate whether the backbone was fine-tuned (

\checkmark

) on the corresponding benchmark’s training set or evaluated in a zero-shot setting (

\times

K	Avg. Pages	Overall	Breakdown
K	Avg. Pages	Overall	SIN	MUL	UNA
Adaptive	5.6	42.1	54.6	23.5	45.7
1	3.0	40.5	51.6	17.0	56.1
2	4.3	39.7	54.4	20.7	39.9
3	5.4	40.1	53.3	23.0	40.8
4	6.5	41.1	53.3	24.0	43.5

Appendix

Table 7: Impact of K on MMLongBench-Doc. “Adaptive” denotes the document-adaptive setting

K=\min(\lceil N/10\rceil,4)

, where

N

is the total number of pages.

Iteration	Avg. Pages	Overall	Breakdown
Iteration	Avg. Pages	Overall	SIN	MUL	UNA
3	4.2	41.1	54.0	22.5	44.4
4	4.6	41.2	54.3	23.0	43.5
5	4.9	41.4	54.3	23.4	43.5
6	5.2	41.5	54.4	24.0	44.0
7	5.6	42.1	54.6	23.5	45.7

Appendix

Table 8: Impact of maximum interaction steps on MMLongBench-Doc.

Figure 7: Case 1 in SlideVQA between different methods

Figure 8: Case 2 in SlideVQA between different methods

Figure 9: Case 3 in SlideVQA between different methods

Explore similar work

Aug 11, 2026

InSight-doc: Agentic Visual Perception for Long-Document Understanding

Long-document understanding often requires reasoning over many visually rich pages, making inference costly and prone to context rot. In this work, we propose InSight-doc, an agentic visual perception framework that treats visual resolution as an adaptive reasoning-time resource. InSight-doc starts from low resolution and selectively zooms into high-resolution regions for finer evidence, without relying on any external retriever. To train such an agent, we construct an active-perception corpus of 17.9K high-quality SFT examples with region-level zoom-in trajectories, accompanied by 19.2K hard RL examples. Through SFT+RL, InSight-doc-8B improves the baseline by 4.3--16.4 accuracy points over document VQA benchmarks. On long documents, it reduces hallucination by more than 40% and inference latency by 41%--68% while maintaining an accuracy lead. Our code, datasets, and model are released at https://github.com/m-Just/InSight-doc .

Kaican Li, Weiyan Xie, Lewei Yao +4

1The Hong Kong University of Science and Technology · 2Huawei

Document Understanding Visual Question Answering Tool-Using Vision-Language Agents Active Visual Perception Document Image Understanding

Ask PDF

Sep 7, 2026

Capability-Routed Visual Retrieval and Evidence Threading for Long-Context Document Question Answering

Annual reports, diligence packs, and infographic dashboards bury numbers in page images: axes, cell grids, and footnotes that OCR pipelines flatten and that page-level visual retrievers still treat as interchangeable in-context examples. We keep a frozen Qwen2.5-VL-7B-Instruct generator and a ColPali / VisRAG-Ret page index, and insert three modules. A capability-aware visual router (CAVR) tags each retrieved page as text, table, chart, layout, or mixed and mixes specialist experts before generation. Weak-to-strong page selection (WSPS) distils a frozen 7B answerability teacher into a 3B selection head so ranking is no longer a single InfoNCE score. Visual evidence threading (VET) builds layout-anchored paths of length at most three and lets the generator read the thread rather than a flat top-

k

list. On gold-page DocVQA / ChartQA / InfographicVQA the 7B system reaches 96.3 / 90.1 / 85.4. Under the VisRAG top-3 protocol the mean generation accuracy is 62.74 versus 59.39 for the same backbone with concatenation. On MMLongBench-Doc retrieve-then-read, F1 moves from 19.2 to 22.6 and multi-page accuracy from 16.4 to 21.2. ViDoRe nDCG@5 after WSPS reranking is 83.6, with TAT-DQA financial reports at 70.4.

Amirul Rahman, Aisha Karim, Kenji Nakamura +1

University of Malaya

Document QA Multimodal RAG Visual Document Retrieval Document Image Understanding Long-Context VLMs LLM Reranking

Ask PDF

Jul 31, 2026

HierDoc: Hierarchical Page-to-Region Evidence Routing for Long-Document Visual Question Answering

Multi-page document visual question answering requires locating sparse evidence at both the page and region levels. Existing approaches typically emphasize one level over the other: page-centric methods focus on page acquisition, with region operations serving mainly as navigation aids, whereas region-centric methods assume that the relevant pages have already been supplied. Consequently, page and region selection remain disconnected rather than forming successive evidence decisions. We propose HierDoc, a hierarchical evidence-routing framework that formulates long-document evidence acquisition as two-stage set prediction from pages to regions. A page policy selects evidence pages from the full document; these pages are then parsed for semantic elements, after which a region policy selects the elements passed to a downstream answer model. Both answer-agnostic policies are optimized with stage-wise GRPO using granularity-specific structured-set rewards. The answer model receives selected full pages together with selected region crops and OCR or table text, preserving global context while emphasizing fine-grained evidence. Across the evaluated benchmarks, HierDoc achieves state-of-the-art or competitive performance among open-weight systems, improving LongDocURL by 16.87% relative to the strongest reported open-weight baseline. Controlled ablations further show that selected regional evidence improves the page-only system in accuracy and F1 by 5.51% and 4.82%, respectively. These results demonstrate the benefit of organizing coarse page routing and fine-grained region routing as successive, separately optimized stages of a unified evidence-acquisition process.

Rongjian Gu, Wengang Zhou, Junyu Xiong +4

University of Science and Technology of China · 2iFLYTEK Research · 3Anhui Transport Consulting & Design Institute Co. Ltd., Hefei, China

Visual Question Answering Document QA Evidence Retrieval

Ask PDF