Locate, Judge, Read: Role-Specialized Tool Agents over a Knowledge-Point Index for Open-Domain Multimodal Document QA

Wenlin Lin; Fanzhe Wei

cs.AIOct 9, 2026

Locate, Judge, Read: Role-Specialized Tool Agents over a Knowledge-Point Index for Open-Domain Multimodal Document QA

Authors: Wenlin Lin, Fanzhe Wei

Organizations: Metask Lab

Abstract

Question answering over large heterogeneous document collections requires first finding the right passages and then reading them, yet current retrieval-augmented agents give one model both kinds of tools and let it schedule itself. We present KnowForge, a pipeline separated by job: a hybrid-retrieval knowledge-point index; a Locator agent that can only search; a code-scheduled bridge judge that forces the missing second hop; a typed, size-bounded evidence pool that merges every search's ranking; a Deep Reader agent that can only read text, images and tables; and a synthesizer with a format checker. With a self-hosted Qwen3.6-35B-A3B (3B active), KnowForge reaches F1 47.5 on the full open-domain M3DocVQA (2,441 questions, 3,368 PDFs), +11.0 over the published M3DocRAG system and +7.1 over one-shot retrieve-and-answer on the same index and model, and 53.5 with DeepSeek-V4-Flash. On MultiHop-RAG it reaches 73.5 accuracy, above the strongest published RAG and GraphRAG results (71.2); on the single-document MMLongBench-Doc it lifts the same model from 34.8 to 47.0, level with GPT-4o reading page images (46.3). Per-job ablations on the full benchmarks attribute the gain: the Deep Reader is worth 3.8, 9.8 and 0 points on the three corpora, tracking how visual each is; merging both agents into one with all tools and twice the rounds loses 1.9 F1 because it keeps searching instead of reading; a round-budget sweep shows a step at four Locator rounds and a plateau beyond; query latency scales with model calls rather than corpus size. We serve the three indexed benchmarks as read-only knowledge bases (free registration) and release all prompts and the runner code: https://github.com/metask-ai/knowforge-bench

Figures & tables

Panel of Figure 1 — Figure 1: The KnowForge pipeline. Stages 2 and 5 are agents with disjoint tool sets (blue); 1, 3 and 6 are code-scheduled (grey; 3 and 6 make one fixed model call each); 4 is the typed, size-bounded merge of all searches from stages 2–3 (orange).

System	Multi-round	Roles split by	Disjoint tool sets	Forced 2nd hop	Typed evidence pool	Open-domain full-set eval	Per-role ablation
M3DocRAG Cho et al. (2024)	–	–	–	–	–	✓	–
HiKEY Shin et al. (2026)	–	–	–	–	–	✓	$\circ$
SimpleDoc Jain et al. (2025)	✓	–	–	–	–	–	$\circ$
Search-o1 Li et al. (2025a)	✓	stage (model-triggered)	$\circ$	–	–	–	$\circ$
WebThinker Li et al. (2025b)	✓	stage (model-triggered)	✓	–	$\circ$	–	$\circ$
DeepRead Li et al. (2026)	✓	–	–	–	–	–	$\circ$

Table 1: Design comparison with the closest systems, from their published descriptions.

\circ

= partial; n/r = not reported; “open-domain full-set eval” = results on the complete M3DocVQA corpus in one index (single-document systems marked –).

Panel of Figure 2 — Figure 2: Published systems (grey) and ours (red; self-hosted 35B-A3B unless marked Flash). M3DocVQA and MultiHop-RAG are multi-document (full sets); MMLongBench-Doc is one document per question. Sources: Text RAG and M3DocRAG Cho et al. (2024) ; HiKEY Shin et al. (2026) reports EM only (27.5 with a 7B reader vs. our 7B one-shot 31.7); GPT-4 RAG Tang and Yang (2024) ; RAG / HippoRAG2 (Llama-3.1-8B) and GraphRAG local (Llama-3.1-70B) Han et al. (2025a) ; MDocAgent, M3DocRAG-style (“M3DocRAG-st.”) and MM-Doc-R1 (RL-trained Qwen3-8B) Lin et al. (2026) ; GPT-4o reading pages Ma et al. (2024b) ; MAGE-RAG Zuo et al. (2026) ; “frontier” = best 2026 public-leaderboard entry (Qwen3.6 Plus). On MMLongBench-Doc the pipeline lifts the same 35B-A3B model from 34.8 to 47.0 (+12.2), level with GPT-4o; the systems above it read charts with frontier or RL-trained vision models. Settings are not equalised; per-configuration numbers in Appendix B .

	F1	Doc@all	Search/q	Read/q	No-read
S4 one-shot	40.4	73.3	1	0	100%
P1: no Deep Reader	43.7	83.3	3.52	0	100%
P0: one agent, all tools, 8 rounds	45.6	86.0	4.81	0.34	74%
M5: two agents, 4+4	47.5	84.6	3.58	0.71	58%

Table 2: Role ablation on full M3DocVQA. Search/q and Read/q are mean tool calls per question; No-read is the share of questions on which no reading tool was ever called. P0 finds documents slightly better than M5 and answers worse.

Panel of Figure 3 — Figure 3: Round-cap sweep on mid300 (Table 3 ). Retrieval (right axis) rises and saturates; answer F1 (left axis) is flat for caps 1–3, steps up at 4+4 and does not rise at 5+5 or 6+6. Single-role cuts show the Reader cap matters below 3 and the Locator cap below 4.

$L{+}R$	F1	Doc found	R@all	Loc calls	Read calls
0+0 (S4)	43.1	71.0	83.0	–	–
1+0	44.7	78.0	87.7	2.11	0
1+1	46.4	77.3	87.6	2.09	0.24
2+2	46.7	80.7	90.1	2.45	0.38
3+3	46.2	82.0	90.5	3.08	0.48
4+4 (3 runs)	50.9	83.8	91.5	3.45–3.58	0.67–0.73

Table 3: Round-cap sweep on mid300 (Qwen3.6). Caps are upper bounds; the last two columns are mean calls actually made. Single-role cuts (4+1, 1+4, 3+4, 4+3) are in Figure 3 and Appendix C .

MMLongBench-Doc (1,082 q; judge acc., strict in parentheses)
	S4	1+1	M5 4+4	P1 no reader
All	34.8 (32.8)	41.5 (38.8)	47.0 (44.1)	37.2 (35.0)
Answerable (859)	16.6	24.9	31.7	19.9
Unanswerable (223)	95.1	92.4	91.9	93.3
Table (143)	22.4	37.8	46.9	26.6
Chart (107)	15.0	21.5	32.7	16.8

Table 4: Transfer. Same tools, same prompts up to one benchmark-specific rule paragraph and the answer-format checker. On the chart-heavy MMLongBench-Doc the Deep Reader is the largest single factor; on the text-only MultiHop-RAG it contributes nothing and the cheaper P1 ties M5.

System (serial, 100 q)	p50 / p90	first event	answer start	calls	F1
Qwen3.6 one-shot	4.6 / 19.6 s	–	1.7 s	2	31.1
Qwen3.6 pipeline M5	27.3 / 65.3 s	1.2 s	23.5 s	$\approx$ 9	43.2
Flash API one-shot	6.0 / 13.4 s	–	2.4 s	2	39.3
Flash API pipeline M5	31.9 / 47.2 s	1.7 s	27.6 s	$\approx$ 9	50.5

Table 5: Serial latency on the first 100 mid300 questions, one question at a time, no other load. “First event” is the first tool call (what a user would see first); “answer start” is the first synthesizer token. The 27 s of the pipeline is nine model calls of 3–4.6 s; the API model is slower per call but has a tighter p90 because the self-hosted server is shared with production traffic.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Config	Description	R@1/3/8 (1st search)	R@all	Doc found	Ans. visible	EM	F1	vs. ref.	Time p50
S0	closed book, Qwen2-VL-7B	–	–	–	–	17.2	22.0
S0	closed book, Qwen3.6	–	–	–	–	26.8	32.2
S0	closed book, Flash	–	–	–	–	35.6	43.0
S1	dense only, one shot	53.5/72.5/80.3	82.6	71.4	66.6	32.8	39.7		4.3s
S3	dense+BM25, one shot	56.3/77.4/84.5	86.3	75.6	68.3	33.3	40.6	176:156 vs S1
S4v1	dense+BM25+graph, one shot, our prompt	56.3/75.0/82.0	84.5	73.3	66.4	33.3	40.2	174:165 vs S1

Appendix

Table 6: All full-set M3DocVQA runs (2,441 questions). “vs. ref.” gives question flips won:lost against the named reference. Time is the median per question under the 4-way concurrent harness; serial latencies are in § 5.8 .

F1 by bucket	image	table	text	multi	single
S0 closed, Qwen3.6	22.2	25.1	40.9	21.9	39.1
S0 closed, Flash	29.1	35.6	54.8	32.6	50.0
S1 dense one shot	24.6	28.4	54.2	28.5	–
S3 hybrid one shot	25.6	30.7	54.8	31.1	–
S4 one shot (official)	23.5	30.4	53.7	27.7	49.0
S4 one shot, 7B (official)	25.8	28.3	49.3	27.7	44.9

Appendix

Table 7: M3DocVQA F1 by evidence modality and hop count. Bucket sizes: image 940, table 1,265, text 1,216 (a question can carry several evidence modalities), multi-hop 980, single-hop 1,461; – = not computed for that run.

Cap $L{+}R$	Loc. calls	Loc. at cap	Read calls	Zero reads	Read at cap	p50 / p90
0+0	–	–	–	–	–	4s / 24s
1+0	2.11	100%	0	100%	–	10s / 38s
1+1	2.09	100%	0.24	76%	70%	18s / 60s
2+2	2.45	70%	0.38	67%	40%	27s / 80s
3+3	3.08	56%	0.48	63%	22%	33s / 102s
1+4	2.15	100%	0.77	58%	19%	23s / 88s

Appendix

Table 8: Actual tool usage under each round cap (mid300, Qwen3.6, 4-way concurrent harness). Locator calls include the bridge judge’s follow-up searches (at most 2). The Deep Reader makes no call on 55–76% of questions under every cap; its call count depends only on its own cap, not on the Locator’s.

Corpus	Source	Docs	Markdown chars	Images	OCR pages	Embeddings	Pre-proc. tokens	Extraction tokens (text / mm)	Total tokens
M3DocVQA	1.02 GB PDF	3,368	57.7M	12,556	17,223	201,957	44.3M	72.1M / 21.7M	138M
MMLongBench-Doc	666 MB PDF	135	11.8M	7,189	3,657	34,234	8.4M	6.7M / 5.2M	20M
MultiHop-RAG	5 MB Markdown	609	5.2M	0	0	7,608	0	3.1M / 0	3.1M

Appendix

Table 9: One-time index construction cost by corpus, from the system’s usage ledger. Pre-processing covers file classification and heading-tree extraction; extraction covers knowledge-point extraction (text) and image/table description (multimodal, “mm”). Relation typing runs on recalled pairs only and is included in the text figure. Wall clock: M3DocVQA about 12 hours at the highest concurrency preset; MMLongBench-Doc a median of 120 s per document (6.6 hours in total at the normal preset); MultiHop-RAG 62 minutes.

MMLongBench-Doc (judge acc.)	n	S4	1+1	M5	P1
Single-page	499	25	35	43	26
Cross-page	360	11	19	25	18
Unanswerable	223	95	92	92	93
Format Int	299	16	22	29	21
Format Str	250	24	35	43	26
Format Float	159	25	36	45	30

Appendix

Table 10: MMLongBench-Doc by position, answer format and evidence source (the benchmark’s own bucket protocol, in which a question with several sources counts in each). Earlier runs of the previous pipeline version on the same questions scored 46.2 / 44.3 / 45.4 (Qwen3.6) and 49.0 / 48.7 (Flash).

MultiHop-RAG, M5	n	R@1	R@3	R@8	R@all
Inference	816	20.5	38.8	58.3	73.0
Comparison	856	32.4	47.6	60.4	91.6
Temporal	583	30.4	50.3	65.6	88.9
All non-null	2,255	27.6	45.1	61.0	84.2

Appendix

Table 11: MultiHop-RAG document-level retrieval for the full pipeline: recall of gold evidence articles in the first search (R@

k

) and in the union of all searches (R@all). The official benchmark reports chunk-level Hits@

k

(best published: 0.747 at

k{=}10

with a reranker), so the two are not directly comparable. Inference questions have low first-search recall because they typically need three or four articles.

Document QA

Document question answering uses AI to answer questions from text, tables, images, and structured content in documents.

See all 89 papers

IndexRAG: Index-Time Reasoning for Multi-Hop Retrieval-Augmented Generation
Mar 17, 2026Zhenghua Bao, Yi ShiMulti-Hop QA Retrieval-Augmented Generation
Continuum AI Shanghai, China · Continuum AI San Francisco, CA, USA
SPD-RAG: Sub-Agent Per Document Retrieval-Augmented Generation
Mar 9, 2026Yagiz Can Akay, Muhammed Yusuf Kartal, Esra Alparslan +2Multi-Document QA Retrieval-Augmented Generation
TOBB University of Economics and Technology · OSTIM Technical University
DocNavRAG: Document-Structured Graph RAG with Stateful Evidence Construction for Complex Document Question Answering
Aug 3, 2026Dongyang Xie, Yao Tian, Hao Zhang +5Graph Retrieval-Augmented Generation Agentic RAG
School of Computer Science, Wuhan University · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong +1

Mar 17, 2026J/K move · Enter open · S save

IndexRAG: Index-Time Reasoning for Multi-Hop Retrieval-Augmented Generation

Zhenghua Bao, Yi Shi

Continuum AI Shanghai, China · Continuum AI San Francisco, CA, USA

Multi-Hop QA Retrieval-Augmented Generation Multi-Document QA

Open paper AskPDF arXiv

Multi-hop question answering (QA) requires reasoning across multiple documents, yet existing retrieval-augmented generation (RAG) approaches address this either through graph-based methods requiring additional online processing or iterative multi-step reasoning. We present IndexRAG, a novel approach that shifts cross-document reasoning from online inference to offline indexing. IndexRAG identifies bridge entities shared across documents and generates bridging facts as independently retrievable units, requiring no additional training or fine-tuning. Experiments on three widely-used multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA, MuSiQue) show that IndexRAG improves F1 over Naive RAG by 4.6 points on average, while requiring only single-pass retrieval and a single LLM call at inference time. When combined with IRCoT, IndexRAG achieves the best average performance among all evaluated methods, including graph-based baselines such as HippoRAG2 and FastGraphRAG, while relying on a flat vector index. Our code is available at https://github.com/Continuum-AI-Corp/IndexRAG .

Abstract

Document QA

Explore similar work

IndexRAG: Index-Time Reasoning for Multi-Hop Retrieval-Augmented Generation

SPD-RAG: Sub-Agent Per Document Retrieval-Augmented Generation

DocNavRAG: Document-Structured Graph RAG with Stateful Evidence Construction for Complex Document Question Answering