Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning

Rongjin Li; Zichen Tang; Xianghe Wang; Xinyi Hu; Zhengyu Wang; Zhengyu Lu; Yiling Huang; Jiayuan Chen

cs.AIMar 27, 2026

Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning

Authors: Rongjin Li, Zichen Tang, Xianghe Wang, Xinyi Hu, Zhengyu Wang, Zhengyu Lu, Yiling Huang, Jiayuan Chen,

Organizations: Beijing University of Posts and Telecommunications

Abstract

With the rapid progress of multimodal large language models (MLLMs), AI already performs well at literature retrieval and certain reasoning tasks, serving as a capable assistant to human researchers, yet it remains far from autonomous research. The fundamental reason is that current work on academic paper reasoning is largely confined to a search-oriented paradigm centered on pre-specified targets, with reasoning grounded in relevance retrieval, which struggles to support researcher-style full-document understanding, reasoning, and verification. To bridge this gap, we propose ScholScan, a new benchmark for academic paper reasoning. ScholScan introduces a scan-oriented task setting that asks models to read and cross-check entire papers like human researchers, scanning the document to identify consistency issues. The benchmark comprises 1,800 carefully annotated questions drawn from nine error categories across 13 natural-science domains and 715 papers, and provides detailed annotations for evidence localization and reasoning traces, together with a unified evaluation protocol. We assessed 15 models across 24 input configurations and conducted a fine-grained analysis of MLLM capabilities for all error categories. Across the board, retrieval-augmented generation (RAG) methods yield no significant improvements, revealing systematic deficiencies of current MLLMs on scan-oriented tasks and underscoring the challenge posed by ScholScan. We expect ScholScan to be the leading and representative work of the scan-oriented task paradigm.

Figures & tables

Figure 1: A comparison between search-oriented and scan-oriented task paradigms. Unlike the former, the scan-oriented paradigm provides no pre-specified targets, requiring the model to actively scan the entire paper, construct a document-level evidence view.

Figure 2: Left : Overview of ScholScan. Right : Comparison to related benchmarks. Mod. : Modalities; Para. : Task Paradigm; Eval. : Evaluation Focus; T : Text; I : Image; TD : Text Document; MD : Multimodal Document; P : Process; O : Outcome; Dom. : Academic Domain Coverage.

Figure 3: Sampled ScholScan examples with 9 error categories, covering the whole process of scientific research, each requiring the model to perform thorough cross-source evidence-based reasoning.

MLLM (Image Input)
Models	Avg.	RQD	DI	SG	MO	DHP	CF	IC	RCA	LE
Proprietary MLLMs
GPT-5	19.2	10.1	9.7	28.2	14.6	26.6	13.8	25.3	25.3	6.9
Gemini 2.5 Pro	15.6	11.9	12.6	35.7	12.3	27.0	4.6	14.7	15.2	7.4
Doubao-Seed-1.6-thinking	10.2	3.4	3.5	22.3	7.5	15.1	10.2	12.2	10.9	3.3
Doubao-Seed-1.6	9.9	3.0	4.4	29.2	4.9	15.0	6.3	17.9	8.0	3.9

Table 1: Model performance across 9 error categories (scaled by 100). RQD : Research Question & Definitions; DI : Design & Identifiability; SG : Sampling & Generalizability; MO : Measurement & Operationalization; DHP : Data Handling & Preprocessing; CF : Computation & Formulae; IC : Inference & Conclusions; RCA : Referential & Citation Alignment; LE : Language & Expression.

Figure 4: Spearman correlation matrix among the 9 error categories.

Figure 5: Left : Distribution of omission and hallucination errors. Right : Average reasoning steps and evidence locations involved in the answer generation, compared against the gold reference.

Figure 6: Model performance trends across reasoning steps and evidence locations (scaled by 100).

Text Input (Base Model: Qwen3-Thinking)
Models	Avg.	RQD	DI	SG	MO	DHP	CF	IC	RCA	LE
Baseline	17.4	8.9	16.2	31.9	15.1	23.7	5.6	22.3	21.1	2.3
Oracle	24.5	20.6	27.9	43.6	21.3	40.8	7.4	26.9	26.0	1.9
BM25	16.7	9.7	13.7	33.0	17.3	23.8	6.8	25.4	16.5	3.0
Contriever	16.6	9.7	18.2	33.7	10.7	20.8	6.4	18.5	19.8	1.8
BGE-M3	11.3	8.6	7.5	24.8	9.1	15.4	5.3	15.6	11.4	1.0

Table 2: Overall scores of RAG methods across the 9 error categories (scaled by 100).

Text Input (Base Model: Qwen3-Thinking)
Models	MRR@5	Recall@5
BM25	0.41	0.48
Contriever	0.31	0.39
NV-Embed-v2	0.30	0.38
BGE-M3	0.16	0.21
Image Input (Base Model: Llama 4 Maverick)

Table 3: Retrieval performance of RAG methods.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	$S$	$S_{\mathrm{location}}$	$S_{\mathrm{reasoning}}$	$P_{\mathrm{unrelated\_err}}$
Correlation Coefficients	$0.841$	$0.806$	$0.842$	$0.954$

Appendix

Table 4: Spearman’s correlation coefficients among

S

S_{\mathrm{location}}

S_{\mathrm{reasoning}}

, and

P_{\mathrm{unrelated\_err}}

MLLM (Image Input)
Model	Avg.	RQD	DI	SG	MO	DHP	CF	IC	RCA	LE
GPT-5	21.2	12.6	11.8	33.5	15.4	26.6	16.0	27.1	27.0	6.4
Gemini 2.5 Pro	19.7	15.0	23.2	44.7	13.2	31.4	7.8	17.3	17.8	12.8
Doubao-Seed-1.6	12.0	4.8	5.9	36.2	7.5	13.3	5.7	19.9	9.9	5.5
Doubao-Seed-1.6-thinking	11.6	5.1	6.9	28.0	7.7	14.3	10.5	15.6	10.8	4.5
Grok 4	4.2	0.0	1.3	20.5	2.6	5.5	1.2	3.9	2.6	0.8

Appendix

Table 5: Model performance under Qwen3-32B evaluation (scaled by 100).

MLLM (Image Input)
Model	Avg.	RQD	DI	SG	MO	DHP	CF	IC	RCA	LE
GPT-5	20.5	11.2	12.0	32.4	17.9	27.4	10.5	25.9	27.4	3.1
Gemini 2.5 Pro	14.6	9.1	11.3	34.2	11.4	28.2	4.9	12.6	14.8	5.3
Doubao-Seed-1.6	10.9	5.3	3.9	34.0	6.1	15.6	6.1	17.9	8.0	4.7
Doubao-Seed-1.6-thinking	10.3	3.2	4.7	26.4	7.3	14.2	9.2	14.8	9.3	3.1
Grok 4	3.9	0.5	1.8	15.9	1.7	4.1	1.5	1.1	4.4	0.0

Appendix

Table 6: Model performance under Gemini 2.5 Flash evaluation (scaled by 100).

Model	$\lambda{=}0.6$	$\lambda{=}0.8$	$\lambda{=}1.0$	$\mu{=}0.85$	$\mu{=}0.9$	$\mu{=}0.95$
GPT-5	19.3	19.2	19.0	18.5	19.2	19.9
Gemini 2.5 Pro	15.8	15.6	15.3	15.0	15.6	16.1
Doubao-Seed-1.6-thinking	10.4	10.2	10.0	9.9	10.2	10.5
Doubao-Seed-1.6	10.1	9.9	9.8	9.7	9.9	10.2
Grok 4	4.0	4.0	3.9	3.9	4.0	4.1

Appendix

Table 7: Sensitivity under image input: variations of

\lambda

and

\mu

(scaled by 100).

Model	$\gamma{=}0.4$	$\gamma{=}0.6$	$\gamma{=}0.8$	$q{=}1.0$	$q{=}1.5$	$q{=}2.0$
GPT-5	19.4	19.2	19.0	19.3	19.2	19.1
Gemini 2.5 Pro	15.7	15.6	15.5	15.6	15.6	15.5
Doubao-Seed-1.6-thinking	10.4	10.2	10.0	10.3	10.2	10.1
Doubao-Seed-1.6	10.1	9.9	9.8	10.0	9.9	9.9
Grok 4	4.0	4.0	4.0	4.0	4.0	4.0

Appendix

Table 8: Sensitivity under image input: variations of

\gamma

and

q

(scaled by 100).

Model	$\lambda{=}0.6$	$\lambda{=}0.8$	$\lambda{=}1.0$	$\mu{=}0.85$	$\mu{=}0.9$	$\mu{=}0.95$
Gemini 2.5 Pro	30.6	30.2	29.9	29.1	30.2	31.5
GPT-5	22.7	22.5	22.3	21.4	22.5	23.7
Grok 4	21.1	20.8	20.6	20.2	20.8	21.4
Doubao-Seed-1.6-thinking	15.6	15.3	15.0	14.8	15.3	15.8
Doubao-Seed-1.6	14.1	13.9	13.7	13.6	13.9	14.3
Claude Sonnet 4	6.0	5.9	5.8	5.6	5.9	6.1

Appendix

Table 9: Sensitivity under text input: variations of

\lambda

and

\mu

(scaled by 100).

Model	$\gamma{=}0.4$	$\gamma{=}0.6$	$\gamma{=}0.8$	$q{=}1.0$	$q{=}1.5$	$q{=}2.0$
Gemini 2.5 Pro	30.7	30.2	29.9	30.5	30.2	30.1
GPT-5	23.4	22.5	21.9	23.0	22.5	22.2
Grok 4	21.0	20.8	20.7	20.9	20.8	20.8
Doubao-Seed-1.6-thinking	15.6	15.3	15.1	15.5	15.3	15.2
Doubao-Seed-1.6	14.2	13.9	13.7	14.1	13.9	13.8
Claude Sonnet 4	6.3	5.9	5.6	6.1	5.9	5.7

Appendix

Table 10: Sensitivity under text input: variations of

\gamma

and

q

(scaled by 100).

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Aug 27, 2026

Not Just Reason, Not Just Scan: Reinforcement Learning for Proactive Scientific Error Verification over Academic Paper

Multimodal large language models (MLLMs) are increasingly capable scientific assistants, yet they remain far from fully autonomous research. This transition requires models to actively inspect academic papers, build global evidence views, and make traceable judgments without prespecified issues or evidence. However, existing work provides limited task paradigms or training studies for such issue- and evidence-absent verification. We study this challenge through scientific error detection, where models must determine whether errors exist and justify them with evidence-based reasoning. To fill this gap, we present VERA-RL, a reinforcement-learning formulation for scientific error detection over academic papers. Following a Reason--Verify--Scan progression, we construct VERA-13K, a 12,900-sample dataset organized into 4,300 matched chains, covering 6 scientific-error categories across the research workflow and broad natural-science domains. We further introduce fine-grained rewards for reasoning completeness, evidence alignment, and error precision. Training Qwen3-VL-8B with VERA-RL substantially improves verifiable reasoning, approaching flagship MLLMs such as Gemini 3 Pro and Qwen3-VL-235B-A22B on Scan.

Rongjin Li, Yuanxin Liu, Hao Zhou +3

Beijing University of Posts and Telecommunications · State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University · WeChat AI, Tencent Inc., China

Language Model Error Detection Scientific Reasoning in Language Models Reinforcement Learning with Verifiable Rewards

Ask PDF

Apr 13, 2026

PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers

Leveraging Multi-modal Large Language Models (MLLMs) to accelerate frontier scientific research is promising, yet how to rigorously evaluate such systems remains unclear. Existing benchmarks mainly focus on single-document understanding, whereas real scientific workflows require integrating evidence from multiple papers, including their text, tables, and figures. As a result, multi-modal, multi-document scientific reasoning remains underexplored and lacks systematic evaluation. To address this gap, we introduce PaperScope, a multi-modal multi-document benchmark designed for agentic deep research. PaperScope presents three advantages: (1) Structured scientific grounding. It is built on a knowledge graph of over 2,000 AI papers spanning three years, providing a structured foundation for research-oriented queries. (2) Semantically dense evidence construction. It integrates semantically related key information nodes and employs optimized random-walk article selector to sample thematically coherent paper sets, thereby ensuring adequate semantic density and task complexity. (3) Multi-task evaluation of scientific reasoning. It contains over 2,000 QA pairs across reasoning, retrieval, summarization, and problem solving, enabling evaluation of multi-step scientific reasoning. Experimental results show that even advanced systems such as OpenAI Deep Research and Tongyi Deep Research achieve limited scores on PaperScope, highlighting the difficulty of long-context retrieval and deep multi-source reasoning. PaperScope thus provides a rigorous benchmark alongside a scalable pipeline for constructing large-scale multi-modal, multi-source deep research datasets.

Lei Xiong, Huaying Yuan, Zheng Liu +2

Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China

AI Agent Benchmarks Multimodal Model Evaluation Scientific Reasoning in Language Models Multimodal QA

Ask PDF

Jun 18, 2026

ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments

Academic paper search is a core step in scientific research, and LLM-based search agents are emerging as a promising paradigm for iterative, intent-driven literature exploration. However, existing benchmarks offer limited coverage of research intents and insufficient support for scalable answer construction and reproducible multi-turn evaluation. We propose ScholarQuest, a taxonomy-guided benchmark constructed from over 1,000 computer science topics and organized around four research intents. To construct answer sets at scale, we combine multi-source retrieval, citation expansion, and multi-stage relevance adjudication. We further build ScholarBase, a million-scale literature environment with unified retrieval and citation-expansion interfaces, together with a shared multi-turn search harness. We benchmark complete literature search systems and compare language models within this harness, evaluating retrieval coverage, search efficiency, and robustness across research intents. Results show that agentic methods outperform single-shot retrieval baselines, yet the best-performing search system achieves only 0.312 Recall@100 and 0.352 Recall@All. Comparisons under the shared harness reveal intent-dependent model strengths, with higher Planner token consumption not consistently yielding higher recall. We release the benchmark, retrieval backend, and evaluation harness for reproducible research

Tingyue Pan, Mingyue Cheng, Daoyu Wang +4

State Key Lab of Cognitive Intelligence, University of Science and Technology of China

Agentic Search Information Retrieval IR Evaluation Agent Benchmarks

Ask PDF

Abstract

LLM Evaluation

Explore similar work

Not Just Reason, Not Just Scan: Reinforcement Learning for Proactive Scientific Error Verification over Academic Paper

PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers

ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments