Capability-Routed Visual Retrieval and Evidence Threading for Long-Context Document Question Answering

Amirul Rahman; Aisha Karim; Kenji Nakamura; Yi-Fan Ng

cs.CVSep 7, 2026

Capability-Routed Visual Retrieval and Evidence Threading for Long-Context Document Question Answering

Authors: Amirul Rahman, Aisha Karim, Kenji Nakamura, Yi-Fan Ng

Organizations: University of Malaya

Abstract

Annual reports, diligence packs, and infographic dashboards bury numbers in page images: axes, cell grids, and footnotes that OCR pipelines flatten and that page-level visual retrievers still treat as interchangeable in-context examples. We keep a frozen Qwen2.5-VL-7B-Instruct generator and a ColPali / VisRAG-Ret page index, and insert three modules. A capability-aware visual router (CAVR) tags each retrieved page as text, table, chart, layout, or mixed and mixes specialist experts before generation. Weak-to-strong page selection (WSPS) distils a frozen 7B answerability teacher into a 3B selection head so ranking is no longer a single InfoNCE score. Visual evidence threading (VET) builds layout-anchored paths of length at most three and lets the generator read the thread rather than a flat top- $k$ list. On gold-page DocVQA / ChartQA / InfographicVQA the 7B system reaches 96.3 / 90.1 / 85.4. Under the VisRAG top-3 protocol the mean generation accuracy is 62.74 versus 59.39 for the same backbone with concatenation. On MMLongBench-Doc retrieve-then-read, F1 moves from 19.2 to 22.6 and multi-page accuracy from 16.4 to 21.2. ViDoRe nDCG@5 after WSPS reranking is 83.6, with TAT-DQA financial reports at 70.4.

Figures & tables

Figure 1: Ours (CAVR+WSPS+VET). ColPali / VisRAG-Ret recall page images. CAVR routes each page through text, table, chart, and layout experts. WSPS reranks with a frozen 7B teacher. VET builds layout-anchored threads that Qwen2.5-VL-7B-Instruct reads under LoRA.

Method	Size	DocVQA	ChartQA	InfoVQA
Qwen-VL Bai et al. (2023)	9.6B	65.1	65.7	35.4
UReader Ye et al. (2023)	7.1B	65.4	59.3	42.2
Monkey Liu et al. (2024)	9.8B	66.5	—	36.1
DocOwl-1.5 Hu et al. (2024)	8.1B	81.6	70.5	50.4
DocOwl-1.5-Chat	8.1B	82.2	70.2	50.7
Qwen2.5-VL-3B Bai et al. (2025)	3B	93.9	84.0	77.1

Table 1: Gold-page OCR-free generation. Main comparison: open models

{\leq}10

B. Larger and closed models are references. Monkey ChartQA was not reported by Hu et al.

Method	Size	DocVQA
Donut ∗ Kim et al. (2022)	${<}1$ B	67.5
Pix2Struct ${}^{\ast}_{\mathrm{large}}$ Lee et al. (2023)	1.3B	76.6
DocOwl	7.1B	62.2
Qwen-VL Bai et al. (2023)	9.6B	65.1
UReader Ye et al. (2023)	7.1B	65.4
Monkey Liu et al. (2024)	9.8B	66.5

Table 2: OCR-free DocVQA from Hu et al. Table 3. Asterisks: task-specific fine-tunes, not generalists.

Method	val ANLS	val Acc.	test ANLS	test Acc.
LoRRA	0.110	7.22	0.112	7.63
M4C (orig.)	0.292	18.34	0.306	18.75
M4C, dyn 500	0.385	24.73	0.391	24.81
BERT-base ft.	0.556	45.6	0.574	47.6
BERT-large ft.	0.594	49.28	0.610	51.08
BERT-large-SQuAD	0.462	36.72	0.475	38.26

Table 3: Original DocVQA extractors (Mathew et al. Tables 2–3). Historical floor only.

Method	H	M	Overall
TaPas	28.72	53.84	41.28
T5	25.12	56.96	41.04
VL-T5	26.24	56.88	41.56
VisionTaPas	29.60	61.44	45.52
VisionTaPas †	24.84	61.60	43.72
VisionTaPas pretr.	32.56	61.60	47.08

Table 4: ChartQA dataset paper (Masry et al. Table 6), relaxed accuracy (%).

Method	val ANLS	test ANLS	val Acc.	test Acc.
Random	0.006	0.005	0.00	0.00
Majority	0.041	0.035	2.21	1.73
M4C orig.	0.107	0.119	4.81	4.87
M4C, 300 OCR	0.136	0.143	5.86	6.58
LayoutLM, no vis.	0.212	0.225	13.40	15.32
LayoutLM, in-domain	0.250	0.272	18.14	19.74

Table 5: InfographicVQA dataset paper (Mathew et al. Tables 4–6).

Method	ArxivQA	ChartQA	DocVQA	InfoVQA	PlotQA	SlideVQA	Avg
MiniCPM-V 2.0 concat Yao et al. (2024b)	59.19	22.22	24.87	20.33	16.92	30.22	28.96
MiniCPM-V 2.0 w. sel.	60.78	31.75	38.41	28.69	17.03	36.33	35.50
LLaVA-OneVision-7B Li et al. (2024)	64.20	42.10	58.40	46.80	32.50	47.30	48.55
InternVL2-8B Chen et al. (2024b)	65.80	47.20	64.10	50.20	35.80	49.50	52.10
Qwen2-VL-7B Wang et al. (2024)	66.90	51.40	68.20	53.10	37.60	51.20	54.73
MiniCPM-V 2.6	67.77	53.97	70.90	54.46	38.93	50.72	56.12

Table 6: VisRAG protocol, same VisRAG-Ret, top-3 visual generation accuracy. Oracle, OCR, and GPT-4o are references.

Method	Retriever	Reader	ACC	F1
SigLIP + 7B Zhai et al. (2023)	SigLIP	Qwen2.5-VL-7B	16.8	15.4
BiPali + 7B Faysse et al. (2025)	BiPali	Qwen2.5-VL-7B	17.4	16.1
ColPali + MiniCPM-V 2.6	ColPali	MiniCPM-V 2.6	18.2	16.8
VisRAG-Ret + MiniCPM-V 2.6	VisRAG-Ret	MiniCPM-V 2.6	18.6	17.3
ColPali + InternVL2-8B	ColPali	InternVL2-8B	19.5	18.1
ColPali + Qwen2.5-VL concat	ColPali	Qwen2.5-VL-7B	20.8	19.2

Table 7: MMLongBench-Doc retrieve-then-read (

k{=}3

). Concatenation of all pages, OCR+LLM, and GPT-4o are references.

Method	SIN	MUL	UNA	CHA	TAB
ColPali + concat	23.6	16.4	14.8	14.2	18.5
Ours (CAVR+WSPS+VET)	26.8	21.2	17.5	19.6	22.4

Table 8: MMLongBench-Doc evidence-page types under retrieve-then-read. MUL is cross-page.

Method	ArxivQ	DocQ	InfoQ	TabF	TATQ	Shift	AI	Energy	Gov.	Health.	Avg
OCR BM25 Robertson and Zaragoza (2009)	31.6	36.8	62.9	46.5	62.7	64.3	92.8	85.9	83.9	87.2	65.5
OCR BGE-M3 Xiao et al. (2023)	31.4	25.7	60.1	70.8	50.5	73.2	90.2	83.6	84.9	91.1	66.1
SigLIP Zhai et al. (2023)	43.2	30.3	64.1	58.1	26.2	18.7	62.5	65.7	66.1	79.1	51.4
BiSigLIP	58.5	32.9	70.5	62.7	30.5	26.5	74.3	73.7	74.2	82.3	58.6
BiPali Faysse et al. (2025)	56.5	30.0	67.4	76.9	33.4	43.7	71.2	61.9	73.8	73.6	58.8
ColPali	79.1	54.4	81.8	83.9	65.8	73.2	96.2	91.0	92.7	94.4	81.3

Table 9: ViDoRe nDCG@5. Ours reranks ColPali top-20 with WSPS then takes top-5.

Variant	VisRAG Avg	MMLong F1	ChartQA
Concat top-3	59.39	19.2	87.3
+CAVR	60.62	20.4	89.1
+WSPS	61.08	20.8	88.2
+VET	60.31	20.6	87.8
+CAVR+WSPS	61.88	21.5	89.6
Full (CAVR+WSPS+VET)	62.74	22.6	90.1

Table 10: Ablations on Qwen2.5-VL-7B-Instruct + ColPali. Full matches Tables 1 , 6 , 7 .

$k$	VisRAG Avg	MMLong F1	Note
1	57.12	17.8	under-recall
2	60.85	20.9	near default
3	62.74	22.6	default
5	62.41	22.3	noise starts
8	61.20	21.1	context too long

Table 11: Page budget

k

(other knobs at Full defaults).

$\lambda_{\mathrm{sel}}$	VisRAG Avg	MMLong F1
0.0	61.88	21.5
0.2	62.35	22.1
0.4	62.74	22.6
0.6	62.51	22.4
0.8	61.96	21.8

Table 12: WSPS fusion

\lambda_{\mathrm{sel}}

on Full.

$L$	VisRAG Avg	MMLong MUL	Note
1	61.42	18.6	single page
2	62.38	20.4	short chain
3	62.74	21.2	default
4	62.60	21.0	diminishing
5	62.18	20.1	conflicting hops

Table 13: VET maximum hops

L

. MUL is MMLongBench-Doc cross-page accuracy.

Type	Concat F1	Ours F1	$\Delta$
Industry file	24.6	27.2	$+2.6$
Academic paper	18.4	22.1	$+3.7$
Research report / infographic	16.2	21.0	$+4.8$
Financial report	15.8	21.4	$+5.6$

Table 14: MMLongBench-Doc F1 by document type (retrieve-then-read).

Panel of Figure 2 — Figure 2: Saturation at $k{=}3$ (Figure 2(a) ) and larger gains on financial and infographic reports (Figure 2(b) ).

Config	Page (s)	QA (s)	Trainable	VisRAG Avg
OCR+caption	7.22	—	—	—
SigLIP Zhai et al. (2023)	0.12	2.18	0	—
ColPali concat	0.39	2.51	LoRA 32	59.39
+CAVR	0.42	2.63	$+18$ M	60.62
+WSPS	0.41	2.70	$+12$ M	61.08
Ours (full)	0.46	2.84	$+41$ M	62.74

Table 15: Indexing and QA cost. Page encoding follows Faysse et al. (L4, batch 4). QA latency: A100, batch 1,

k{=}3

Panel of Figure 3 — Figure 3: Where the three modules help (Figure 3(a) ) and the A100 cost of that help (Figure 3(b) ).

Explore similar work

What the Reranker Sees: Multi-Aspect Page Annotation for Long-Document Multimodal Question Answering
Aug 14, 2026Guanchen Wu, Jiayuan Ding, Subhabrata Mukherjee +1Multimodal QA Visual Question Answering
Department of Computer Science, Emory University, Atlanta, GA, USA · Hippocratic AI, Palo Alto, CA, USA
Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA
Apr 15, 2026Yuanlei Zheng, Pei Fu, Hang Li +9Document QA VLM Reasoning
School of Software Engineering, Huazhong University of Science and Technology · MiLM Plus, Xiaomi Inc. · School of Computer Science and Technology, Huazhong University of Science and Technology +1
VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA
Jun 15, 2026Young Rok Jang, Hyesoo Kong, Kyunghwan An +3VLM Evaluation Multimodal Large Language Models
LG AI Research

Aug 14, 2026J/K move · Enter open · S save

What the Reranker Sees: Multi-Aspect Page Annotation for Long-Document Multimodal Question Answering

Guanchen Wu, Jiayuan Ding, Subhabrata Mukherjee, Carl Yang

Department of Computer Science, Emory University, Atlanta, GA, USA · Hippocratic AI, Palo Alto, CA, USA

Multimodal QA Visual Question Answering Multimodal Reranking Visual Document Retrieval

Open paper AskPDF arXiv

Long-document visual question answering (VQA) over documents of tens to hundreds of pages mixing text, tables, charts, and figures typically follows retrieve-then-read pipelines. In our setting, the bottleneck shifts from retrieval recall to reranker-side evidence selection: on MMLongBench-Doc, BGE-M3 reaches Recall@20 = 0.86 but only F1@5 = 0.254, and even the visual retriever ColPali reaches only F1@5 = 0.332; a text-only rerank LLM seeing only raw snippets misses table, chart, and layout evidence even when the upstream retriever encoded images. We propose Trident, with two complementary components: Trident-R, a retriever-agnostic LLM reranker that converts each candidate into an LLM-readable semantic record, including a visual caption, section path, entity tags, multi-axis concept hits, and a text snippet, then performs a single adaptive-K rerank call; and Trident-S, a generation-side module that prompts the VLM under topical, entity, and structural lenses before synthesis. On two long-document datasets, the annotation+rerank protocol substantially improves retrieval F1 across five heterogeneous pools, with every reranked pool exceeding the strongest adaptive-K baseline PageIndex. An LLM rerank without the annotation barely changes first-hit ranking, indicating the lift comes from the structured annotation. Trident-S targets open-ended synthesis questions by design, adding up to 6.6 points in generation accuracy on these questions. The best Trident configuration is the strongest downstream QA pipeline in our evaluation, with rankings consistent across two LLM judges (kappa = 0.913).

Abstract

Explore similar work

What the Reranker Sees: Multi-Aspect Page Annotation for Long-Document Multimodal Question Answering

Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA

VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA