Organizations: Beijing Institute of Technology, Beijing, China · Chinese Academy of Sciences, China · Zhejiang University, Hangzhou, China · Westlake University, Hangzhou, China · Fudan University, Shanghai, China · National University of Defense Technology, Changsha, China
Web-based applications increasingly rely on understanding long multimodal documents, where answering questions often requires combining information across pages, tables, and figures. Agentic retrieval-augmented generation addresses these tasks through iterative retrieval and reasoning. However, Agentic execution is not always necessary, introducing additional computation and potentially incorrect answers. Our empirical analysis reveals that Agentic execution provides query-dependent benefits, while additional operations may improve, preserve, or degrade answer accuracy. These findings motivate controlling both Agent invocation and subsequent execution. To address these challenges, we propose RouteDRAG, a learning-based framework comprising four components. Unified Document Index organizes multimodal content into a graph-backed index and derives query representations from initial retrieval. Adaptive Path Routing preserves exact answering and predicts the relative benefit of Agent over shorter execution paths. Dynamic Agent Execution maintains an evolving workspace and refines Planner proposals to control tool use and termination. Sequential Policy Learning first trains operation control from observed outcomes and then trains path routing using the resulting Agent's performance. Across two benchmarks, RouteDRAG achieves an average accuracy improvement of 3.47 percentage points over shorter-path-only execution while assigning only 79.3% of queries to Agent. Notably, it also surpasses uniform Agent execution on both benchmarks while requiring fewer answering calls. Our code is now available at https://github.com/Ereshkigal1/RouteDRAG.
Figures & tables
Figure 1. Why selective Agentic execution? (a) Initial retrieval can suffice, making Agent execution redundant. (b) Others require linking information through Agent execution. (c) Further retrieval can distract an Agent with sufficient information.
Figure 2. Empirical analysis of Agentic execution. (a) Accuracy and answering-token usage across document categories for Standard and Agent. (b) Paired query outcomes, illustrating the query-dependent benefits of Agentic execution. (c) Answer correctness changes after an additional operation compared with immediate termination.
Figure 3. Overview of RouteDRAG. A shared document index supports learned path routing and Agent execution control. Actor-aligned policy learning uses paired execution outcomes to train routing and action decisions in stages.
Support source
Support requirement
Type
Method
TXT
LAY
CHA
TAB
FIG
SIN
MUL
UNA
Overall
Direct
Direct MLLM
30.76 1.88
36.30 2.09
28.58 1.72
20.52 2.16
28.46 1.94
36.30 1.66
21.36 2.31
52.46 1.83
35.00
Text RAG
BM25
34.86 1.19
19.20 1.37
17.33 1.12
31.88 1.48
14.22 1.06
34.00 0.94
16.98 1.56
76.64 1.21
37.98
BGE-M3
33.72 1.32
23.17 1.18
13.96 1.09
35.82 1.41
16.15 1.02
33.15 0.96
19.99 1.53
73.36 1.24
37.86
Multimodal
M3DocRAG ( Cho et al., 2024 )
43.47 1.79
31.81 1.63
39.23 1.51
33.45 1.91
37.33 1.43
53.08 1.32
25.38 1.72
25.00 1.98
37.58
EVisRAG ( Sun et al., 2025 )
38.29 1.22
33.21 1.34
38.17 1.11
39.84 1.46
38.90 0.98
51.55 0.92
25.03 1.28
58.20 1.57
44.27
Table 1. Answer accuracy (%) on MMLongBench-Doc. Best and second-best results are bold and underlined, respectively.
Document domain
Question type
Type
Method
Acad.
Finance
Gov.
Law
News
TXT
MM
META
UNA
Overall
Direct
Direct MLLM
30.36 2.14
39.58 2.38
68.92 1.92
65.97 2.26
33.14 2.47
52.43 1.85
22.40 2.55
49.22 2.08
63.71 2.31
44.55
Text RAG
BM25
61.06 1.45
57.64 1.62
74.32 1.27
73.82 1.73
87.21 1.18
89.81 1.06
58.44 1.84
39.53 1.91
80.65 1.39
68.24
BGE-M3
55.78 1.58
54.51 1.42
67.57 1.31
71.20 1.64
86.63 1.22
82.77 1.15
52.92 1.78
42.25 1.96
79.03 1.37
64.52
Multimodal
M3DocRAG ( Cho et al., 2024 )
61.06 2.03
65.28 1.87
66.89 1.74
71.20 2.18
81.40 1.65
85.68 1.52
68.51 1.96
53.10 2.26
57.90 2.07
67.88
EVisRAG ( Sun et al., 2025 )
67.30 1.43
65.75 1.56
70.65 1.28
72.92 1.72
82.47 1.16
82.44 1.08
72.65 1.47
45.84 1.85
78.45 1.31
70.69
Table 2. Answer accuracy (%) on DocBench. Bold and underlined values indicate the best and second-best results, respectively.
DocBench
MMLongBench-Doc
Method
Acc. ↑
Agent% ↓
Act./Q ↓
Calls/Q ↓
Tok./Q ↓
Acc. ↑
Agent% ↓
Act./Q ↓
Calls/Q ↓
Tok./Q ↓
w/o Adaptive Path Routing
73.77
15.88
0.300
2.97
29.14
50.32
16.50
0.379
3.31
33.15
w/o Dynamic Agent Execution
74.19
100.00
1.418
7.11
49.30
50.25
100.00
1.850
8.23
56.53
w/o Agent Routing
73.41
–
–
2.01
24.07
49.31
–
–
2.23
27.14
All Agent Routing
74.68
100.00
1.685
7.46
52.08
52.52
100.00
2.148
8.25
57.02
RouteDRAG-Efficient
74.05
11.52
0.171
2.58
27.36
50.87
16.41
0.348
3.20
32.18
Table 3. Component ablation. Accuracy and Agent use are percentages; tokens are in thousands. Dashes mark the no-Agent setting. Best and second-best measured values are bold and underlined.
Figure 4. Sensitivity to regularization strength and routing threshold on (a) DocBench and (b) MMLongBench-Doc. Cells report accuracy (%); colors show percentage-point changes from α=100 and zero threshold offset. Two three-by-three heatmaps show accuracy on DocBench and MMLongBench-Doc as regularization and the development-calibrated threshold offset vary.
Figure 6. Practical efficiency comparison with representative RAG baselines. (a–b) Answer accuracy versus inference latency on DocBench and MMLongBench-Doc. (c–d) Peak GPU memory consumption across different methods.
VLM Backbone
Agent
Random
Ours
Calls Red. (%)
Qwen3-VL-4B
69.60 / 45.92
68.20/43.21
70.15 / 46.65
13.8/14.2
Qwen3-VL-8B
74.68 / 52.52
73.66/49.88
75.86 / 53.80
16.4/14.9
Qwen2.5-VL-7B
72.78 / 48.49
70.40/45.92
73.32 / 48.30
14.1/14.4
InternVL3-8B
73.50 / 50.23
71.10/47.59
74.23 / 51.05
15.0/14.3
Gemma-3-12B
71.96 / 47.57
69.54/45.97
71.87 / 48.49
13.7/15.2
Table 4. Backbone generalization on DocBench and MMLongBench-Doc. Accuracy (%) and relative reduction in answering model calls compared with Uniform Agent (%).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11
DocBench
MMLongBench-Doc
Configuration
Acc. ↑
Agent% ↓
Calls/Q ↓
Acc. ↑
Agent% ↓
Calls/Q ↓
RouteDRAG-Efficient
74.05
11.52
2.58
50.87
16.41
3.20
RouteDRAG-Medium
74.73
42.65
4.22
52.29
57.56
5.72
RouteDRAG-High
75.18
59.53
5.09
52.93
71.13
6.57
RouteDRAG
75.86
78.49
6.24
53.80
80.02
7.02
Appendix
Table A1. Accuracy–cost trade-offs of per-query routing configurations. Accuracy and actual Agent usage are percentages.
Long-document multimodal question answering requires more than retrieving relevant chunks from a large document. Different queries require different evidence behavior. Existing multimodal RAG systems improve evidence access through text chunks, page images, graph links, or heterogeneous document elements, but they often apply a largely query-agnostic evidence-use strategy. We present TAP-RAG, a task-aware policy-controlled RAG framework for long-document multimodal QA. TAP-RAG contains a main controller, the Task-Aware Policy Controller (TAPC), and two policy-guided evidence executors: Task-Aware Query-Guided Flow Diffusion (TA-QFD) and Task-Aware Visual Enhancement (TAVE). For each query, TAPC predicts the task prior, estimates visual/local/global evidence signals, and produces an executable policy. TA-QFD then expands textual and structural evidence over the multimodal document graph, while TAVE selectively inspects page images when visual or layout evidence is needed. A guarded synthesis stage fuses text, visual, and structural evidence and abstains when support is insufficient. On DocBench and MMLongBench-Doc, TAP-RAG achieves the best overall accuracy among the compared systems, improving over a matched multimodal-RAG baseline by +9.1 points (61.1 to 70.2) and +4.5 points (42.2 to 46.7), respectively.
Zhong Ji, Keqi Jin, Yan Zhang +1
School of Electrical and Information Engineering, Tianjin University · The International Joint Institute of Tianjin University, Fuzhou, Tianjin University · Department of Automation, Tsinghua University
Multi-modal retrieval-augmented generation (RAG) is a key technique for visually rich long document understanding. Existing multi-modal RAG methods are progressively advancing toward multi-agent systems: they first retrieve relevant pages based on a query, and then iteratively understand information within those pages. However, these methods typically rely on fixed workflows and lack the ability to dynamically scale computation at test time, often leading to insufficient evidence. To address this, we propose D2-ScaleAgent, an agentic framework that introduces a dual-dimensional scaling paradigm for retrieval and reasoning. The core of D2-ScaleAgent is a Verifier agent-driven dynamic routing loop based on the intrinsic difficulty of the query, centered around a continuously updated evidence bank that serves as the agent's dynamic working memory: when retrieval needs to be expanded, the agent routes outward (retrieval scaling), decomposing the query into attributes and performing parallel page retrieval, followed by adaptive pruning to ensure comprehensive evidence coverage. When fine-grained reasoning is required, the agent routes inward (reasoning scaling), dynamically selecting sub-agents with varying granularity and count to extract evidence from pages. Finally, D2-ScaleAgent achieves logical closure over the evidence chain. Extensive experiments demonstrate that D2-ScaleAgent is effective on long and visually rich document benchmarks like MMLongBench-Doc, LongDocURL, etc.
Hao Zhang, Longrong Yang, Lunhao Duan +3
Zhejiang University · Alibaba Group · University of Science and Technology of China
Answering complex, real-world queries often requires synthesizing facts scattered across vast document corpora. In these settings, standard retrieval-augmented generation (RAG) pipelines suffer from incomplete evidence coverage, while long-context large language models (LLMs) struggle to reason reliably over massive inputs. We introduce SPD-RAG, a hierarchical multi-agent framework for exhaustive cross-document question answering that decomposes the problem along the document axis. Each document is processed by a dedicated document-level agent operating only on its own content, enabling focused retrieval, while a coordinator dispatches tasks to relevant agents and aggregates their partial answers. Agent outputs are synthesized by merging partial answers through a token-bounded synthesis layer (which supports recursive map-reduce for massive corpora). This document-level specialization with centralized fusion improves scalability and answer quality in heterogeneous multidocument settings while yielding a modular, extensible retrieval pipeline. On the LOONG benchmark (EMNLP 2024) for long-context multi-document QA, SPD-RAG achieves an Avg Score of 58.1 (GPT-5 evaluation), outperforming Normal RAG (33.0) and Agentic RAG (32.8) while using only 38% of the API cost of a full-context baseline (68.0).
Yagiz Can Akay, Muhammed Yusuf Kartal, Esra Alparslan +2
TOBB University of Economics and Technology · OSTIM Technical University