Conventional Retrieval-Augmented Generation (RAG) systems often struggle with complex multi-hop queries over long documents due to their single-pass retrieval. We introduce MM-Doc-R1, a novel framework that employs an agentic, vision-aware workflow to address long document visual question answering through iterative information discovery and synthesis. To incentivize the information seeking capabilities of our agents, we propose Similarity-based Policy Optimization (SPO), addressing baseline estimation bias in existing multi-turn reinforcement learning (RL) algorithms like GRPO. Our core insight is that in multi-turn RL, the more semantically similar two trajectories are, the more accurate their shared baseline estimation becomes. Leveraging this, SPO calculates a more precise baseline by similarity-weighted averaging of rewards across multiple trajectories, unlike GRPO which inappropriately applies the initial state's baseline to all intermediate states. This provides a more stable and accurate learning signal for our agents, leading to superior training performance that surpasses GRPO. Our experiments on the MMLongbench-Doc benchmark show that MM-Doc-R1 outperforms previous baselines by 10.4%. Furthermore, SPO demonstrates superior performance over GRPO, boosting results by 5.0% with Qwen3-8B and 6.1% with Qwen3-4B. These results highlight the effectiveness of our integrated framework and novel training algorithm in advancing the state-of-the-art for complex, long-document visual question answering.
Figures & tables
Figure 1: Introduction to MM-Doc-R1. MM-Doc-R1 employs a seeker for iterative key information retrieval within documents, leveraging a VLM (Visual Language Model) as a reading tool to ensure accurate processing of visual elements.
Figure 2: Detailed framework of MM-Doc-R1. The framework operates in three sequential stages. First, the planner module formulates a reasoning plan based on the preprocessed document TOC and the user query. Subsequently, the information seeker executes a multi-turn retrieval and reading process, leveraging the “search" and “read" tools to gather relevant information. Finally, the collected knowledge is integrated by the answer agent , which generates a coherent and contextually accurate response.
Figure 3: SPO and GRPO’s advantage estimation. The bottom panel shows the baseline computation of SPO and GRPO.
Evidence Modality
Evidence Count
Overall
Method
Text
Layout
Chart
Table
Figure
Single
Multi
Unans.
ACC
F1
Human Performance
Human
—
—
—
—
—
—
—
—
65.8
66.0
Upper Bounds (Ground-Truth Evidence)
Qwen2.5-VL-7B
33.8
38.7
31.8
32.3
34.1
46.6
20.5
92.8
46.8
41.7
Qwen3-8B
44.3
37.7
25.7
59.3
22.8
42.7
35.7
89.2
49.6
46.9
Table 1: Performance comparison on the MMLongBench-Doc dataset (1,082 questions). The evaluation includes text-based RAG methods (BM25, BGE-M3), multi-modal RAG (ColQwen2.5-7B-multilingual), and agent-based systems (Mdoc Agent, M3doc RAG). Text-based methods use the top-4 retrieved pages and Qwen3-8B for answer generation. Multi-modal RAG uses the top-4 retrieved image pages and Qwen2.5-VL-7B. Agent-based methods operate over the top-5 retrieved pages, we use Qwen2.5-VL-7B as the VLM and Qwen3-8B as the LLM. Metrics include overall Accuracy (ACC) and F1, as well as performance on sub-categories by evidence modality (Text, Layout, Chart, Table, Figure), number of required evidences (Single, Multi), and unanswerable questions. Best scores among baselines and our methods are marked in bold, second-best in underline, considering only “RAG Baselines", and “Ours MM-Doc-R1" sections.
Figure 4: Comparison of SPO and GRPO in Training; the base model in this figure is Qwen3-8B.
Model
ACC
F1
Unanswerable
MM-Doc-R1 (Qwen3-8B)
45.7
41.5
73.9
w/o TOC
44.3
39.9
73.5
w/o Read pages’ OCR
43.4
38.7
74.7
w/o VLM Read
42.1
37.3
81.6
Table 2: Ablation study of MM-Doc-R1 components on MMLongBench-Doc, using Qwen3-8B. All metrics are in %.
Model
average pages
Recall
BM25
5
42.0
BGE-M3
5
51.3
Colqwen-2.5-VL-7B
5
65.4
MM-Doc-R1(SPO)
3.35
66.3
Table 3: Recall Performance
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Avg
Min
Max
ColQwen2.5 RAG (one-pass)
∼ 40
–
–
MM-Doc-R1 (Qwen3-4B, no train)
84.47
10
724
MM-Doc-R1 (Qwen3-8B, no train)
109.32
11
397
MM-Doc-R1 (Qwen3-8B, SPO)
154.44
23
316
Appendix
Table 4: Per-question latency (seconds) on MMLongBench-Doc.