Question answering over large heterogeneous document collections requires first finding the right passages and then reading them, yet current retrieval-augmented agents give one model both kinds of tools and let it schedule itself. We present KnowForge, a pipeline separated by job: a hybrid-retrieval knowledge-point index; a Locator agent that can only search; a code-scheduled bridge judge that forces the missing second hop; a typed, size-bounded evidence pool that merges every search's ranking; a Deep Reader agent that can only read text, images and tables; and a synthesizer with a format checker. With a self-hosted Qwen3.6-35B-A3B (3B active), KnowForge reaches F1 47.5 on the full open-domain M3DocVQA (2,441 questions, 3,368 PDFs), +11.0 over the published M3DocRAG system and +7.1 over one-shot retrieve-and-answer on the same index and model, and 53.5 with DeepSeek-V4-Flash. On MultiHop-RAG it reaches 73.5 accuracy, above the strongest published RAG and GraphRAG results (71.2); on the single-document MMLongBench-Doc it lifts the same model from 34.8 to 47.0, level with GPT-4o reading page images (46.3). Per-job ablations on the full benchmarks attribute the gain: the Deep Reader is worth 3.8, 9.8 and 0 points on the three corpora, tracking how visual each is; merging both agents into one with all tools and twice the rounds loses 1.9 F1 because it keeps searching instead of reading; a round-budget sweep shows a step at four Locator rounds and a plateau beyond; query latency scales with model calls rather than corpus size. We serve the three indexed benchmarks as read-only knowledge bases (free registration) and release all prompts and the runner code: https://github.com/metask-ai/knowforge-bench
Figures & tables
Figure 1: The KnowForge pipeline. Stages 2 and 5 are agents with disjoint tool sets (blue); 1, 3 and 6 are code-scheduled (grey; 3 and 6 make one fixed model call each); 4 is the typed, size-bounded merge of all searches from stages 2–3 (orange).
System
Multi-round
Roles split by
Disjoint tool sets
Forced 2nd hop
Typed evidence pool
Open-domain full-set eval
Per-role ablation
M3DocRAG Cho et al. (2024)
–
–
–
–
–
✓
–
HiKEY Shin et al. (2026)
–
–
–
–
–
✓
∘
SimpleDoc Jain et al. (2025)
✓
–
–
–
–
–
∘
Search-o1 Li et al. (2025a)
✓
stage (model-triggered)
∘
–
–
–
∘
WebThinker Li et al. (2025b)
✓
stage (model-triggered)
✓
–
∘
–
∘
DeepRead Li et al. (2026)
✓
–
–
–
–
–
∘
Table 1: Design comparison with the closest systems, from their published descriptions. ∘ = partial; n/r = not reported; “open-domain full-set eval” = results on the complete M3DocVQA corpus in one index (single-document systems marked –).
Figure 2: Published systems (grey) and ours (red; self-hosted 35B-A3B unless marked Flash). M3DocVQA and MultiHop-RAG are multi-document (full sets); MMLongBench-Doc is one document per question. Sources: Text RAG and M3DocRAG Cho et al. (2024) ; HiKEY Shin et al. (2026) reports EM only (27.5 with a 7B reader vs. our 7B one-shot 31.7); GPT-4 RAG Tang and Yang (2024) ; RAG / HippoRAG2 (Llama-3.1-8B) and GraphRAG local (Llama-3.1-70B) Han et al. (2025a) ; MDocAgent, M3DocRAG-style (“M3DocRAG-st.”) and MM-Doc-R1 (RL-trained Qwen3-8B) Lin et al. (2026) ; GPT-4o reading pages Ma et al. (2024b) ; MAGE-RAG Zuo et al. (2026) ; “frontier” = best 2026 public-leaderboard entry (Qwen3.6 Plus). On MMLongBench-Doc the pipeline lifts the same 35B-A3B model from 34.8 to 47.0 (+12.2), level with GPT-4o; the systems above it read charts with frontier or RL-trained vision models. Settings are not equalised; per-configuration numbers in Appendix B .
F1
Doc@all
Search/q
Read/q
No-read
S4 one-shot
40.4
73.3
1
0
100%
P1: no Deep Reader
43.7
83.3
3.52
0
100%
P0: one agent, all tools, 8 rounds
45.6
86.0
4.81
0.34
74%
M5: two agents, 4+4
47.5
84.6
3.58
0.71
58%
Table 2: Role ablation on full M3DocVQA. Search/q and Read/q are mean tool calls per question; No-read is the share of questions on which no reading tool was ever called. P0 finds documents slightly better than M5 and answers worse.
Figure 3: Round-cap sweep on mid300 (Table 3 ). Retrieval (right axis) rises and saturates; answer F1 (left axis) is flat for caps 1–3, steps up at 4+4 and does not rise at 5+5 or 6+6. Single-role cuts show the Reader cap matters below 3 and the Locator cap below 4.
L+R
F1
Doc found
R@all
Loc calls
Read calls
0+0 (S4)
43.1
71.0
83.0
–
–
1+0
44.7
78.0
87.7
2.11
0
1+1
46.4
77.3
87.6
2.09
0.24
2+2
46.7
80.7
90.1
2.45
0.38
3+3
46.2
82.0
90.5
3.08
0.48
4+4 (3 runs)
50.9
83.8
91.5
3.45–3.58
0.67–0.73
Table 3: Round-cap sweep on mid300 (Qwen3.6). Caps are upper bounds; the last two columns are mean calls actually made. Single-role cuts (4+1, 1+4, 3+4, 4+3) are in Figure 3 and Appendix C .
S4
1+1
M5 4+4
P1 no reader
MMLongBench-Doc (1,082 q; judge acc., strict in parentheses)
All
34.8 (32.8)
41.5 (38.8)
47.0 (44.1)
37.2 (35.0)
Answerable (859)
16.6
24.9
31.7
19.9
Unanswerable (223)
95.1
92.4
91.9
93.3
Table (143)
22.4
37.8
46.9
26.6
Chart (107)
15.0
21.5
32.7
16.8
Table 4: Transfer. Same tools, same prompts up to one benchmark-specific rule paragraph and the answer-format checker. On the chart-heavy MMLongBench-Doc the Deep Reader is the largest single factor; on the text-only MultiHop-RAG it contributes nothing and the cheaper P1 ties M5.
System (serial, 100 q)
p50 / p90
first event
answer start
calls
F1
Qwen3.6 one-shot
4.6 / 19.6 s
–
1.7 s
2
31.1
Qwen3.6 pipeline M5
27.3 / 65.3 s
1.2 s
23.5 s
≈ 9
43.2
Flash API one-shot
6.0 / 13.4 s
–
2.4 s
2
39.3
Flash API pipeline M5
31.9 / 47.2 s
1.7 s
27.6 s
≈ 9
50.5
Table 5: Serial latency on the first 100 mid300 questions, one question at a time, no other load. “First event” is the first tool call (what a user would see first); “answer start” is the first synthesizer token. The 27 s of the pipeline is nine model calls of 3–4.6 s; the API model is slower per call but has a tighter p90 because the self-hosted server is shared with production traffic.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Config
Description
R@1/3/8 (1st search)
R@all
Doc found
Ans. visible
EM
F1
vs. ref.
Time p50
S0
closed book, Qwen2-VL-7B
–
–
–
–
17.2
22.0
S0
closed book, Qwen3.6
–
–
–
–
26.8
32.2
S0
closed book, Flash
–
–
–
–
35.6
43.0
S1
dense only, one shot
53.5/72.5/80.3
82.6
71.4
66.6
32.8
39.7
4.3s
S3
dense+BM25, one shot
56.3/77.4/84.5
86.3
75.6
68.3
33.3
40.6
176:156 vs S1
S4v1
dense+BM25+graph, one shot, our prompt
56.3/75.0/82.0
84.5
73.3
66.4
33.3
40.2
174:165 vs S1
Appendix
Table 6: All full-set M3DocVQA runs (2,441 questions). “vs. ref.” gives question flips won:lost against the named reference. Time is the median per question under the 4-way concurrent harness; serial latencies are in § 5.8 .
F1 by bucket
image
table
text
multi
single
S0 closed, Qwen3.6
22.2
25.1
40.9
21.9
39.1
S0 closed, Flash
29.1
35.6
54.8
32.6
50.0
S1 dense one shot
24.6
28.4
54.2
28.5
–
S3 hybrid one shot
25.6
30.7
54.8
31.1
–
S4 one shot (official)
23.5
30.4
53.7
27.7
49.0
S4 one shot, 7B (official)
25.8
28.3
49.3
27.7
44.9
Appendix
Table 7: M3DocVQA F1 by evidence modality and hop count. Bucket sizes: image 940, table 1,265, text 1,216 (a question can carry several evidence modalities), multi-hop 980, single-hop 1,461; – = not computed for that run.
Cap L+R
Loc. calls
Loc. at cap
Read calls
Zero reads
Read at cap
p50 / p90
0+0
–
–
–
–
–
4s / 24s
1+0
2.11
100%
0
100%
–
10s / 38s
1+1
2.09
100%
0.24
76%
70%
18s / 60s
2+2
2.45
70%
0.38
67%
40%
27s / 80s
3+3
3.08
56%
0.48
63%
22%
33s / 102s
1+4
2.15
100%
0.77
58%
19%
23s / 88s
Appendix
Table 8: Actual tool usage under each round cap (mid300, Qwen3.6, 4-way concurrent harness). Locator calls include the bridge judge’s follow-up searches (at most 2). The Deep Reader makes no call on 55–76% of questions under every cap; its call count depends only on its own cap, not on the Locator’s.
Corpus
Source
Docs
Markdown chars
Images
OCR pages
Embeddings
Pre-proc. tokens
Extraction tokens (text / mm)
Total tokens
M3DocVQA
1.02 GB PDF
3,368
57.7M
12,556
17,223
201,957
44.3M
72.1M / 21.7M
138M
MMLongBench-Doc
666 MB PDF
135
11.8M
7,189
3,657
34,234
8.4M
6.7M / 5.2M
20M
MultiHop-RAG
5 MB Markdown
609
5.2M
0
0
7,608
0
3.1M / 0
3.1M
Appendix
Table 9: One-time index construction cost by corpus, from the system’s usage ledger. Pre-processing covers file classification and heading-tree extraction; extraction covers knowledge-point extraction (text) and image/table description (multimodal, “mm”). Relation typing runs on recalled pairs only and is included in the text figure. Wall clock: M3DocVQA about 12 hours at the highest concurrency preset; MMLongBench-Doc a median of 120 s per document (6.6 hours in total at the normal preset); MultiHop-RAG 62 minutes.
MMLongBench-Doc (judge acc.)
n
S4
1+1
M5
P1
Single-page
499
25
35
43
26
Cross-page
360
11
19
25
18
Unanswerable
223
95
92
92
93
Format Int
299
16
22
29
21
Format Str
250
24
35
43
26
Format Float
159
25
36
45
30
Appendix
Table 10: MMLongBench-Doc by position, answer format and evidence source (the benchmark’s own bucket protocol, in which a question with several sources counts in each). Earlier runs of the previous pipeline version on the same questions scored 46.2 / 44.3 / 45.4 (Qwen3.6) and 49.0 / 48.7 (Flash).
MultiHop-RAG, M5
n
R@1
R@3
R@8
R@all
Inference
816
20.5
38.8
58.3
73.0
Comparison
856
32.4
47.6
60.4
91.6
Temporal
583
30.4
50.3
65.6
88.9
All non-null
2,255
27.6
45.1
61.0
84.2
Appendix
Table 11: MultiHop-RAG document-level retrieval for the full pipeline: recall of gold evidence articles in the first search (R@ k ) and in the union of all searches (R@all). The official benchmark reports chunk-level Hits@ k (best published: 0.747 at k=10 with a reranker), so the two are not directly comparable. Inference questions have low first-search recall because they typically need three or four articles.
Multi-hop question answering (QA) requires reasoning across multiple documents, yet existing retrieval-augmented generation (RAG) approaches address this either through graph-based methods requiring additional online processing or iterative multi-step reasoning. We present IndexRAG, a novel approach that shifts cross-document reasoning from online inference to offline indexing. IndexRAG identifies bridge entities shared across documents and generates bridging facts as independently retrievable units, requiring no additional training or fine-tuning. Experiments on three widely-used multi-hop QA benchmarks (HotpotQA, 2WikiMultiHopQA, MuSiQue) show that IndexRAG improves F1 over Naive RAG by 4.6 points on average, while requiring only single-pass retrieval and a single LLM call at inference time. When combined with IRCoT, IndexRAG achieves the best average performance among all evaluated methods, including graph-based baselines such as HippoRAG2 and FastGraphRAG, while relying on a flat vector index. Our code is available at https://github.com/Continuum-AI-Corp/IndexRAG .
Zhenghua Bao, Yi Shi
Continuum AI Shanghai, China · Continuum AI San Francisco, CA, USA
Answering complex, real-world queries often requires synthesizing facts scattered across vast document corpora. In these settings, standard retrieval-augmented generation (RAG) pipelines suffer from incomplete evidence coverage, while long-context large language models (LLMs) struggle to reason reliably over massive inputs. We introduce SPD-RAG, a hierarchical multi-agent framework for exhaustive cross-document question answering that decomposes the problem along the document axis. Each document is processed by a dedicated document-level agent operating only on its own content, enabling focused retrieval, while a coordinator dispatches tasks to relevant agents and aggregates their partial answers. Agent outputs are synthesized by merging partial answers through a token-bounded synthesis layer (which supports recursive map-reduce for massive corpora). This document-level specialization with centralized fusion improves scalability and answer quality in heterogeneous multidocument settings while yielding a modular, extensible retrieval pipeline. On the LOONG benchmark (EMNLP 2024) for long-context multi-document QA, SPD-RAG achieves an Avg Score of 58.1 (GPT-5 evaluation), outperforming Normal RAG (33.0) and Agentic RAG (32.8) while using only 38% of the API cost of a full-context baseline (68.0).
Yagiz Can Akay, Muhammed Yusuf Kartal, Esra Alparslan +2
TOBB University of Economics and Technology · OSTIM Technical University
Answering complex questions over large document collections requires assembling complementary evidence across sections and documents. GraphRAG offers structured retrieval but typically uses fixed traversal, while agentic RAG operates over weakly structured interfaces. Our key insight is that agents should navigate document structure within and across documents rather than repeatedly search from scratch. We introduce DocNavRAG, which organizes document hierarchies and cross-region relations into a navigable graph, exposes graph operations for locating, navigating, expanding, and fetching, and maintains an evolving evidence state to guide retrieval until sufficient evidence is collected. Across four long- and multi-document QA benchmarks, DocNavRAG improves answer quality and context sufficiency over the strongest baseline by 7.8% and 17.7% on average.
Dongyang Xie, Yao Tian, Hao Zhang +5
School of Computer Science, Wuhan University · The Hong Kong University of Science and Technology · The Chinese University of Hong Kong +1