PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers

Lei Xiong; Huaying Yuan; Zheng Liu; Zhao Cao; Zhicheng Dou

cs.AIApr 13, 2026

PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers

Authors: Lei Xiong, Huaying Yuan, Zheng Liu, Zhao Cao, Zhicheng Dou

Organizations: Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China

Abstract

Leveraging Multi-modal Large Language Models (MLLMs) to accelerate frontier scientific research is promising, yet how to rigorously evaluate such systems remains unclear. Existing benchmarks mainly focus on single-document understanding, whereas real scientific workflows require integrating evidence from multiple papers, including their text, tables, and figures. As a result, multi-modal, multi-document scientific reasoning remains underexplored and lacks systematic evaluation. To address this gap, we introduce PaperScope, a multi-modal multi-document benchmark designed for agentic deep research. PaperScope presents three advantages: (1) Structured scientific grounding. It is built on a knowledge graph of over 2,000 AI papers spanning three years, providing a structured foundation for research-oriented queries. (2) Semantically dense evidence construction. It integrates semantically related key information nodes and employs optimized random-walk article selector to sample thematically coherent paper sets, thereby ensuring adequate semantic density and task complexity. (3) Multi-task evaluation of scientific reasoning. It contains over 2,000 QA pairs across reasoning, retrieval, summarization, and problem solving, enabling evaluation of multi-step scientific reasoning. Experimental results show that even advanced systems such as OpenAI Deep Research and Tongyi Deep Research achieve limited scores on PaperScope, highlighting the difficulty of long-context retrieval and deep multi-source reasoning. PaperScope thus provides a rigorous benchmark alongside a scalable pipeline for constructing large-scale multi-modal, multi-source deep research datasets.

Figures & tables

Benchmarks	Avg. Papers	Avg. Pages	Cross-Page	Multi-Modal	Multi-Doc	Capability	Open Form	Agent
LSDU Benchmark
DocVQA	-	1	✗	✗	✗	U	✗	✗
ChartQA	-	1	✗	✗	✗	U	✗	✗
InfoVQA	-	1	✗	✗	✗	U	✗	✗
Charxiv	-	1	✗	✗	✗	U	✗	✗
ArxivQA	-	1	✗	✗	✗	U	✗	✗

Table 1: Comparison of existing benchmarks and our proposed Agentic Deep Research Benchmark . Symbols: ✓ indicates support, ✗ indicates lack of support. Abbreviations: U = Understanding, SRea = Single-resource reasoning, MRea = Multi-resource reasoning, Retr = Retrieval, Solu = Solution, Think = Thinking, G = Generation.

Figure 1: Visualized Examples of PaperScope Bench: Sub-task illustrations from four meta-tasks. The icons in the center represent the various capabilities required by the agent. In each case, the icons placed next to specific stages indicate the particular capabilities needed at that stage. Thinking refers to the reasoning and decomposition of the underlying intent of a given query. Understanding denotes multi-modal comprehension of the document content.

Category	Sub-Task	Num
Multi-Doc Reasoning Task	Figure-table-chart comparison	200
	Figure-table-chart reasoning	200
	Formula reasoning	200
	Algorithm reasoning	200
	Full paper reasoning	200
Topic Induction Task	Implicit topic induction	200

Table 2: Statistics of task categories and subtasks.

Figure 2: The overview of construction methodology of the PaperScope Bench.

Agent Type	Model	Reasoning	Induction	Summary	Solution	Score
LLM-based ReAct	WebWatcher 32B	4	0	46.74	26.78	18.70
	OpenAI 4o-mini	6	25.49	53.26	22.1	23.74
	Gemini-2.5-flash-thinking	7	13.33	38.40	29.71	19.32
	OpenAI GPT-5.1	0	0	42	51.84	17.78
	Gemini 2.5 pro	3	7.02	47.54	40.39	20.50
	GLM 4.5V	0	0	37.32	32.45	14.44

Table 3: Performance of different agents across Reasoning, Induction, Summary, and Solution tasks. The best score on each dataset is shown in bold.

Figure 3: The ablation results of PaperScope Bench in reasoning task.

Figure 4: A case study comparing the capabilities of different model types in tool use and reasoning. The figure illustrates the behavioral distinctions and performance gaps between models with no tool support (left), shallow tool calling (middle), and strong reasoning combined with tool calling (right) when addressing complex retrieval tasks. For brevity, intermediate tool return results and the FileVisit tool interactions are omitted.

Setting	Combinations (Uniqueness)	Max Quality Score	Interpretation
(a) Walk Length ( $L$ ) — finding the semantic sweet spot (with $W=10000,\beta=0.3$ )
$L=10$	2	60.33	Too Short: Fails to escape local nodes.
$L=100$ (Default)	10,000 (Max)	1,564.19	Optimal: Best balance of diversity and efficiency.
$L=500$	10,000 (Max)	26,430.99	Diminishing Returns: Deep links but introduces noise.
$L=1000$	-	-	Divergence: Paths drift too far, failing to converge.
(b) Bias Probability ( $\beta$ ) — exploration vs. exploitation (with $L=100,W=10000$ )

Table 4: Ablation analysis of ORWAS hyperparameters. Parameter tuning significantly impacts the uniqueness (discovery rate) of document combinations. Our default settings yield the maximum combination diversity without incurring excessive computational overhead.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Corpus Split	#Documents	Source Venues	#QA Pairs
Corpus-A	500	5 conferences	1100
Corpus-B	1500	5 conferences	1100
Test Corpus	202	5 conferences	200

Appendix

Table 5: High-level statistics of PaperScope corpus.

1 Doc	2 Docs	3 Docs	4 Docs	5+ Docs
200	200	400	200	1400

Appendix

Table 6: Distribution of the number of supporting documents per question.

Statistic	Graph-Test	Graph-Docs 500	Graph-Docs 1500
#Nodes	30962	31348	85152
#Edges	1990449	1907081	12735443
Density	0.0042	0.0039	0.0025
Diameter	3	4	8
Avg. Degree	128.57	121.67	125.81
Avg. Path Length	2.041	2.17	4.14

Appendix

Table 7: Structural statistics of the constructed semantic graphs.

Figure 5: Visualization of selected semantic graphs.

Stage	Time Complexity	Space Complexity
Random Walk	$O(W\cdot L)$	$O(L)$
Aggregation	$O(\|A\|\cdot\|E\|)$	$O(\|A\|)$
Combination Filter	$\min(C,\binom{N}{k})$	Linear in output

Appendix

Table 8: Complexity analysis of ORWAS.

Figure 6: The prompts used for induction task QAs construction.

Figure 7: The prompts used for solution task QAs construction.

Figure 8: The prompts used for summary trend task QAs construction.

Task Type	Pearson ( $r$ )	Spearman ( $\rho$ )	P-value
Summary	0.6175	0.6072	$<0.0001$
Solution	0.6385	0.6263	$<0.0001$

Appendix

Table 9: Correlation between automated evaluation and human expert judgments based on 120 trajectories.

Model	Full Score	Text-only	Drop (%)
WebWatcher 32B	4	1	75.0%
Gemini 2.5 pro	3	1	66.7%
Kimi k2	12	3	75.0%
MiroThinker-v1.0-30B	3	0	100.0%
OpenAI o3 (Deep Res)	13	3	76.9%
Grok 4	36	5	86.1%

Appendix

Table 10: Text-only ablation results. The massive performance drop indicates that the benchmark heavily relies on visual information rather than simple text matching.

Component	Configuration
GPUs	8 $\times$ H100
Inference Engine	vLLM
Max retries	10
Max excute time	2h30mins
Max tokens	128000
Top p	0.95

Appendix

Table 11: Experimental configuration for baseline evaluation.

Figure 9: The prompts used for summary task evaluation.

Figure 10: The prompts used for solution task evaluation.

Figure 11: The prompts used for solution task evaluation.

Model	Max Score	Min Score	Macro Avg.
OpenAI o3 deep research	79	35	59.15
kimi-k2	73	11	49.85
Grok-4	65	32	48.27
Gemini-2.5-pro	51.5	19	40.39
DR Tulu-8B	51	0	38.71
Qwen3-VL	59	25	37.38

Appendix

Table 12: Performance comparison of models on solution tasks showing Max Score, Min Score, and Macro Average.

Model	Explicit Topic	Implicit Topic	Macro Avg.
deepseek-V3.1	26.67	25.93	26.32
OpenAI 4o-mini	18.52	33.33	25.49
kimi-k2	20	29.17	24.07
Grok-4	20	20	20
Gemini-2.5-flash-thinking	13.33	13.33	13.33
Qwen3-VL	16.67	10	13.33

Appendix

Table 13: Performance comparison of models on Explicit Topic and Implicit Topic tasks.

Task	Gemini-2.5	GPT-5.1	MM-r1	Tulu-8b	Miro-30B
Reasoning
Algorithm	5	0	10	6	5
Fig-Tab-Cha Com	0	0	0	10	0
Fig-Tab-Cha Rea	10	0	5	5	5
Formula	10	0	20	0	5
Paper	10	0	5	0	0

Appendix

Table 14: Performance comparison (Transposed View). Note: Model names are abbreviated for brevity.

Figure 12: a case study of Grok-4.

Figure 13: a case study of Grok-4.

Figure 14: a case study of Grok-4.

Failure Mode	Definition	Count	Ratio
Extraction Error	Correct document retrieved, but unable to read precise numerical values from charts/tables.	15	37.5%
Reasoning Error	Values extracted correctly, but multi-step calculation or logical inference fails.	9	22.5%
Retrieval Failure	Fails to search (7.5%) or retrieves the wrong document (12.5%).	8	20.0%
Hallucination	Outputs speculative guesses despite successful retrieval.	6	15.0%
Context Limit	Key information truncated due to long multi-document concatenation.	2	5.0%

Appendix

Table 15: Error attribution for Grok-4 based on 40 failure cases.

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,494 papers

DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections
Aug 20, 2025Jiwon Park, Seohyun Pyeon, Jinwoo Kim +4Multi-Hop QA Scientific QA
Pohang University of Science and Technology, Pohang-si, South Korea · The University of Melbourne, Melbourne, Australia · The University of Western Australia, Perth, Australia
SciMDR: Advancing Scientific Multimodal Document Reasoning
Mar 12, 2026Ziyu Chen, Yilun Zhao, Chengye Wang +3Multimodal Reasoning Multimodal QA
University of Chicago · Yale University · TCS Research
AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery
Apr 28, 2026Lei Xiong, Kun Luo, Ziyi Xia +15Deep Research Agents Scientific Discovery
AutoResearchBench Team*

Aug 20, 2025J/K move · Enter open · S save

DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections

Jiwon Park, Seohyun Pyeon, Jinwoo Kim, Rina Carines Cabral+3

Pohang University of Science and Technology, Pohang-si, South Korea · The University of Melbourne, Melbourne, Australia · The University of Western Australia, Perth, Australia

Multi-Hop QA Scientific QA Multimodal QA Document QA

Open paper AskPDF arXiv

Despite rapid progress in large language models (LLMs), current QA benchmarks still overlook the core challenge of real-world scientific information seeking: synthesizing multimodal evidence scattered across multiple documents and structural formats. Existing QAs remain narrow in scope, relying on unimodal text and short-span reasoning that fail to capture the complexity of real information-seeking. We introduce DocHop-QA, a benchmark of 11,379 instances for evaluating multimodal, multi-document, multi-hop scientific QA. Built from publicly available PubMed articles, DocHop-QA incorporates textual passages, tables, and layout cues, enabling cross-document inference without explicit hyperlinks. To scale realistic QA construction, we develop an LLM-driven generation pipeline grounded in 11 scientific reasoning concepts, producing diverse and coherent question-answer pairs. To highlight the utility and versatility of the dataset, we propose a task-driven evaluation framework spanning four settings, including generative answering, multimodal evidence integration and structured index prediction. Experiments show that current models struggle with DocHop-QA's long-context, multi-evidence demands, establishing it as a rigorous testbed for advancing next-generation scientific QA systems.

Abstract

AI Agent Benchmarks

Explore similar work

DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections

SciMDR: Advancing Scientific Multimodal Document Reasoning

AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery