Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning

Rongjin Li; Zichen Tang; Xianghe Wang; Xinyi Hu; Zhengyu Wang; Zhengyu Lu; Yiling Huang; Jiayuan Chen

cs.AIMar 27, 2026

Not Search, But Scan: Benchmarking MLLMs on Scan-Oriented Academic Paper Reasoning

Authors: Rongjin Li, Zichen Tang, Xianghe Wang, Xinyi Hu, Zhengyu Wang, Zhengyu Lu, Yiling Huang, Jiayuan Chen,

Organizations: Beijing University of Posts and Telecommunications

Abstract

With the rapid progress of multimodal large language models (MLLMs), AI already performs well at literature retrieval and certain reasoning tasks, serving as a capable assistant to human researchers, yet it remains far from autonomous research. The fundamental reason is that current work on academic paper reasoning is largely confined to a search-oriented paradigm centered on pre-specified targets, with reasoning grounded in relevance retrieval, which struggles to support researcher-style full-document understanding, reasoning, and verification. To bridge this gap, we propose ScholScan, a new benchmark for academic paper reasoning. ScholScan introduces a scan-oriented task setting that asks models to read and cross-check entire papers like human researchers, scanning the document to identify consistency issues. The benchmark comprises 1,800 carefully annotated questions drawn from nine error categories across 13 natural-science domains and 715 papers, and provides detailed annotations for evidence localization and reasoning traces, together with a unified evaluation protocol. We assessed 15 models across 24 input configurations and conducted a fine-grained analysis of MLLM capabilities for all error categories. Across the board, retrieval-augmented generation (RAG) methods yield no significant improvements, revealing systematic deficiencies of current MLLMs on scan-oriented tasks and underscoring the challenge posed by ScholScan. We expect ScholScan to be the leading and representative work of the scan-oriented task paradigm.

Figures & tables

Figure 1: A comparison between search-oriented and scan-oriented task paradigms. Unlike the former, the scan-oriented paradigm provides no pre-specified targets, requiring the model to actively scan the entire paper, construct a document-level evidence view.

Figure 2: Left : Overview of ScholScan. Right : Comparison to related benchmarks. Mod. : Modalities; Para. : Task Paradigm; Eval. : Evaluation Focus; T : Text; I : Image; TD : Text Document; MD : Multimodal Document; P : Process; O : Outcome; Dom. : Academic Domain Coverage.

Figure 3: Sampled ScholScan examples with 9 error categories, covering the whole process of scientific research, each requiring the model to perform thorough cross-source evidence-based reasoning.

MLLM (Image Input)
Models	Avg.	RQD	DI	SG	MO	DHP	CF	IC	RCA	LE
Proprietary MLLMs
GPT-5	19.2	10.1	9.7	28.2	14.6	26.6	13.8	25.3	25.3	6.9
Gemini 2.5 Pro	15.6	11.9	12.6	35.7	12.3	27.0	4.6	14.7	15.2	7.4
Doubao-Seed-1.6-thinking	10.2	3.4	3.5	22.3	7.5	15.1	10.2	12.2	10.9	3.3
Doubao-Seed-1.6	9.9	3.0	4.4	29.2	4.9	15.0	6.3	17.9	8.0	3.9

Table 1: Model performance across 9 error categories (scaled by 100). RQD : Research Question & Definitions; DI : Design & Identifiability; SG : Sampling & Generalizability; MO : Measurement & Operationalization; DHP : Data Handling & Preprocessing; CF : Computation & Formulae; IC : Inference & Conclusions; RCA : Referential & Citation Alignment; LE : Language & Expression.

Figure 4: Spearman correlation matrix among the 9 error categories.

Figure 5: Left : Distribution of omission and hallucination errors. Right : Average reasoning steps and evidence locations involved in the answer generation, compared against the gold reference.

Figure 6: Model performance trends across reasoning steps and evidence locations (scaled by 100).

Text Input (Base Model: Qwen3-Thinking)
Models	Avg.	RQD	DI	SG	MO	DHP	CF	IC	RCA	LE
Baseline	17.4	8.9	16.2	31.9	15.1	23.7	5.6	22.3	21.1	2.3
Oracle	24.5	20.6	27.9	43.6	21.3	40.8	7.4	26.9	26.0	1.9
BM25	16.7	9.7	13.7	33.0	17.3	23.8	6.8	25.4	16.5	3.0
Contriever	16.6	9.7	18.2	33.7	10.7	20.8	6.4	18.5	19.8	1.8
BGE-M3	11.3	8.6	7.5	24.8	9.1	15.4	5.3	15.6	11.4	1.0

Table 2: Overall scores of RAG methods across the 9 error categories (scaled by 100).

Text Input (Base Model: Qwen3-Thinking)
Models	MRR@5	Recall@5
BM25	0.41	0.48
Contriever	0.31	0.39
NV-Embed-v2	0.30	0.38
BGE-M3	0.16	0.21
Image Input (Base Model: Llama 4 Maverick)

Table 3: Retrieval performance of RAG methods.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	$S$	$S_{\mathrm{location}}$	$S_{\mathrm{reasoning}}$	$P_{\mathrm{unrelated\_err}}$
Correlation Coefficients	$0.841$	$0.806$	$0.842$	$0.954$

Appendix

Table 4: Spearman’s correlation coefficients among

S

S_{\mathrm{location}}

S_{\mathrm{reasoning}}

, and

P_{\mathrm{unrelated\_err}}

MLLM (Image Input)
Model	Avg.	RQD	DI	SG	MO	DHP	CF	IC	RCA	LE
GPT-5	21.2	12.6	11.8	33.5	15.4	26.6	16.0	27.1	27.0	6.4
Gemini 2.5 Pro	19.7	15.0	23.2	44.7	13.2	31.4	7.8	17.3	17.8	12.8
Doubao-Seed-1.6	12.0	4.8	5.9	36.2	7.5	13.3	5.7	19.9	9.9	5.5
Doubao-Seed-1.6-thinking	11.6	5.1	6.9	28.0	7.7	14.3	10.5	15.6	10.8	4.5
Grok 4	4.2	0.0	1.3	20.5	2.6	5.5	1.2	3.9	2.6	0.8

Appendix

Table 5: Model performance under Qwen3-32B evaluation (scaled by 100).

MLLM (Image Input)
Model	Avg.	RQD	DI	SG	MO	DHP	CF	IC	RCA	LE
GPT-5	20.5	11.2	12.0	32.4	17.9	27.4	10.5	25.9	27.4	3.1
Gemini 2.5 Pro	14.6	9.1	11.3	34.2	11.4	28.2	4.9	12.6	14.8	5.3
Doubao-Seed-1.6	10.9	5.3	3.9	34.0	6.1	15.6	6.1	17.9	8.0	4.7
Doubao-Seed-1.6-thinking	10.3	3.2	4.7	26.4	7.3	14.2	9.2	14.8	9.3	3.1
Grok 4	3.9	0.5	1.8	15.9	1.7	4.1	1.5	1.1	4.4	0.0

Appendix

Table 6: Model performance under Gemini 2.5 Flash evaluation (scaled by 100).

Model	$\lambda{=}0.6$	$\lambda{=}0.8$	$\lambda{=}1.0$	$\mu{=}0.85$	$\mu{=}0.9$	$\mu{=}0.95$
GPT-5	19.3	19.2	19.0	18.5	19.2	19.9
Gemini 2.5 Pro	15.8	15.6	15.3	15.0	15.6	16.1
Doubao-Seed-1.6-thinking	10.4	10.2	10.0	9.9	10.2	10.5
Doubao-Seed-1.6	10.1	9.9	9.8	9.7	9.9	10.2
Grok 4	4.0	4.0	3.9	3.9	4.0	4.1

Appendix

Table 7: Sensitivity under image input: variations of

\lambda

and

\mu

(scaled by 100).

Model	$\gamma{=}0.4$	$\gamma{=}0.6$	$\gamma{=}0.8$	$q{=}1.0$	$q{=}1.5$	$q{=}2.0$
GPT-5	19.4	19.2	19.0	19.3	19.2	19.1
Gemini 2.5 Pro	15.7	15.6	15.5	15.6	15.6	15.5
Doubao-Seed-1.6-thinking	10.4	10.2	10.0	10.3	10.2	10.1
Doubao-Seed-1.6	10.1	9.9	9.8	10.0	9.9	9.9
Grok 4	4.0	4.0	4.0	4.0	4.0	4.0

Appendix

Table 8: Sensitivity under image input: variations of

\gamma

and

q

(scaled by 100).

Model	$\lambda{=}0.6$	$\lambda{=}0.8$	$\lambda{=}1.0$	$\mu{=}0.85$	$\mu{=}0.9$	$\mu{=}0.95$
Gemini 2.5 Pro	30.6	30.2	29.9	29.1	30.2	31.5
GPT-5	22.7	22.5	22.3	21.4	22.5	23.7
Grok 4	21.1	20.8	20.6	20.2	20.8	21.4
Doubao-Seed-1.6-thinking	15.6	15.3	15.0	14.8	15.3	15.8
Doubao-Seed-1.6	14.1	13.9	13.7	13.6	13.9	14.3
Claude Sonnet 4	6.0	5.9	5.8	5.6	5.9	6.1

Appendix

Table 9: Sensitivity under text input: variations of

\lambda

and

\mu

(scaled by 100).

Model	$\gamma{=}0.4$	$\gamma{=}0.6$	$\gamma{=}0.8$	$q{=}1.0$	$q{=}1.5$	$q{=}2.0$
Gemini 2.5 Pro	30.7	30.2	29.9	30.5	30.2	30.1
GPT-5	23.4	22.5	21.9	23.0	22.5	22.2
Grok 4	21.0	20.8	20.7	20.9	20.8	20.8
Doubao-Seed-1.6-thinking	15.6	15.3	15.1	15.5	15.3	15.2
Doubao-Seed-1.6	14.2	13.9	13.7	14.1	13.9	13.8
Claude Sonnet 4	6.3	5.9	5.6	6.1	5.9	5.7

Appendix

Table 10: Sensitivity under text input: variations of

\gamma

and

q

(scaled by 100).

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Not Just Reason, Not Just Scan: Reinforcement Learning for Proactive Scientific Error Verification over Academic Paper
Aug 27, 2026Rongjin Li, Yuanxin Liu, Hao Zhou +3Language Model Error Detection Scientific Reasoning in Language Models
Beijing University of Posts and Telecommunications · State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University · WeChat AI, Tencent Inc., China
PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers
Apr 13, 2026Lei Xiong, Huaying Yuan, Zheng Liu +2AI Agent Benchmarks Multimodal Model Evaluation
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China
ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments
Jun 18, 2026Tingyue Pan, Mingyue Cheng, Daoyu Wang +4Agentic Search Information Retrieval
State Key Lab of Cognitive Intelligence, University of Science and Technology of China

Aug 27, 2026J/K move · Enter open · S save

Not Just Reason, Not Just Scan: Reinforcement Learning for Proactive Scientific Error Verification over Academic Paper

Rongjin Li, Yuanxin Liu, Hao Zhou, Fandong Meng+2

Beijing University of Posts and Telecommunications · State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University · WeChat AI, Tencent Inc., China

Language Model Error Detection Scientific Reasoning in Language Models Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) are increasingly capable scientific assistants, yet they remain far from fully autonomous research. This transition requires models to actively inspect academic papers, build global evidence views, and make traceable judgments without prespecified issues or evidence. However, existing work provides limited task paradigms or training studies for such issue- and evidence-absent verification. We study this challenge through scientific error detection, where models must determine whether errors exist and justify them with evidence-based reasoning. To fill this gap, we present VERA-RL, a reinforcement-learning formulation for scientific error detection over academic papers. Following a Reason--Verify--Scan progression, we construct VERA-13K, a 12,900-sample dataset organized into 4,300 matched chains, covering 6 scientific-error categories across the research workflow and broad natural-science domains. We further introduce fine-grained rewards for reasoning completeness, evidence alignment, and error precision. Training Qwen3-VL-8B with VERA-RL substantially improves verifiable reasoning, approaching flagship MLLMs such as Gemini 3 Pro and Qwen3-VL-235B-A22B on Scan.

Abstract

LLM Evaluation

Explore similar work

Not Just Reason, Not Just Scan: Reinforcement Learning for Proactive Scientific Error Verification over Academic Paper

PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers

ScholarQuest: A Taxonomy-Guided Benchmark for Agentic Academic Paper Search in Open Literature Environments