EpiBench: Benchmarking Multi-turn Research Workflows for Multimodal Agents

Xuan Dong; Huanyang Zheng; Tianhao Niu; Zhe Han; Pengzhan Li; Bofei Liu; Zhengyang Liu; Guancheng Li

cs.CLApr 7, 2026

EpiBench: Benchmarking Multi-turn Research Workflows for Multimodal Agents

Authors: Xuan Dong, Huanyang Zheng, Tianhao Niu, Zhe Han, Pengzhan Li, Bofei Liu, Zhengyang Liu, Guancheng Li,

Organizations: Harbin Institute of Technology

Abstract

Scientific research follows multi-turn, multi-step workflows that require proactively searching the literature, consulting figures and tables, and integrating evidence across papers to align experimental settings and support reproducible conclusions. This joint capability is not systematically assessed in existing benchmarks, which largely under-evaluate proactive search, multi-evidence integration and sustained evidence use over time. In this work, we introduce EpiBench, an episodic multi-turn multimodal benchmark that instantiates short research workflows. Given a research task, agents must navigate across papers over multiple turns, align evidence from figures and tables, and use the accumulated evidence in the memory to answer objective questions that require cross paper comparisons and multi-figure integration. EpiBench introduces a process-level evaluation framework for fine-grained testing and diagnosis of research agents. Our experiments show that even the leading model achieves an accuracy of only 29.23% on the hard split, indicating substantial room for improvement in multi-turn, multi-evidence research workflows, providing an evaluation platform for verifiable and reproducible research agents.

Figures & tables

Benchmark	Cross-paper Integration	Multimodal Integration	Proactive Search	Evidence Memory	Multi-turn Dialogue
Non-paper benchmarks
MMCR [ 34 ]	$\times$	$\times$	$\times$	$\times$	$\checkmark$
Vision-DeepResearch [ 36 ]	$\times$	$\checkmark$	$\checkmark$	$\times$	$\times$
MMDU [ 12 ]	$\times$	$\checkmark$	$\times$	$\times$	$\checkmark$
VisChainBench [ 13 ]	$\times$	$\checkmark$	$\times$	$\times$	$\times$
MMDeepResearch [ 7 ]	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\times$

Table 1: Comparison of agentic reasoning benchmarks by key capabilities.

\checkmark

indicates supported, and

\times

indicates not supported.

Figure 1: EpiBench motivation and key features. Top: A human research workflow proceeds iteratively from posing a question to searching papers, inspecting figures and tables, aligning evidence across papers, archiving findings, and validating the final claim. Bottom: EpiBench mirrors this workflow with four benchmark features: multi-turn coherence, active paper search, cross-paper grounding over figures and tables, and evidence memory that supports reuse in later turns.

Figure 2: Example episode and tool-use workflow in EpiBench . A research task is instantiated as a multi-turn question chain. At each turn, the agent uses tools to search for relevant papers, open PDFs, and locate and extract target figures or tables. Retrieved evidence is stored in a memory pool and reused in later turns. The final turn requires multi-evidence fusion across papers, combining visual and textual evidence in the memory pool to produce the structured answer.

Panel of Figure 3 — Figure 3: Dataset overview of EpiBench . Left: Statistics of the EpiBench dataset by difficulty level. Middle: Data distribution across categories. Right: Episode counts across subject areas stratified by difficulty.

Base LMM	Easy			Hard			Average
Base LMM	ESR	Acc final	Acc pre	ESR	Acc final	Acc pre	ESR	Acc final	Acc pre
Open-source models
Qwen3-VL-235B-Instr.	16.98	41.51	70.67	8.16	51.02	49.77	12.75	46.08	55.17
Qwen3-VL-235B-Think.	30.00	65.00	76.09	3.23	43.55	43.84	13.73	51.96	51.90
GLM-4.5V	8.11	51.35	38.95	0.00	30.77	29.17	2.94	38.24	31.59
Kimi-K2.5	52.94	73.53	89.77	26.15	49.23	79.51	35.35	57.58	81.91

Table 2: Main results on EpiBench . We report episode success rate (ESR, %), accuracy on the final turn (Acc final , %), and accuracy on all non-final turns (Acc pre , %), stratified by difficulty. Instr. denotes instruction-tuned models and Think. denotes thinking-mode models.

Base LMM	Easy		Hard		Average
	EC $\uparrow$	MG $\downarrow$	EC $\uparrow$	MG $\downarrow$	EC $\uparrow$	MG $\downarrow$
Open-source models
Qwen3-VL-235B-Instr.	64.63	1.41	63.48	1.03	63.85	1.23
Qwen3-VL-235B-Think.	54.55	1.02	45.66	1.00	48.63	1.01
GLM-4.5V	24.49	2.60	31.20	2.60	29.31	2.60
Kimi-K2.5	81.30	1.49	84.91	1.20	84.13	1.35

Table 3: Process-level diagnostics on EpiBench . We report evidence correctness (EC, %) and minimality gap (MG) by difficulty. Higher EC is better. Lower MG is better. Instr. denotes instruction-tuned models and Think. denotes thinking-mode models.

Figure 4: Failure-type distribution (%) across models.

Panel of Figure 5 — Figure 5: Ablations on memory-centric constraints. (a) Allowing tool use in the final turn yields a large recovery in ESR and Acc final , indicating that multimodal evidence cached across turns is often insufficient for reliable final fusion under the memory-only protocol. (b) Removing PDF text RAG has mixed effects and generally smaller impact, suggesting that the primary bottleneck lies in multimodal evidence selection and alignment rather than text retrieval alone.

Panel of Figure 6 — Figure 6: Effect of step budget $S_{\max}$ on episodic performance. (a) ESR increases from 5 to 10 steps but saturates from 10 to 15. (b) Max-steps error rate drops sharply from 5 to 10, indicating fewer budget-induced terminations, while additional budget yields diminishing returns.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Human revision intensity from GPT-5.2 drafts to the final benchmark release. We group episodes into five categories, ranging from minimal intervention ( Kept , Light Edit ) to substantial restructuring ( Final-Q Rewrite , Full-Q Rewrite , Episode Rebuild ). Most episodes, especially in the hard split, undergo major expert revision before inclusion in EpiBench .

Panel of Figure 8 — Figure 8: Statistics of benchmark composition. Left: distribution of episode turn lengths. Middle: distribution of task categories, including constraint-style identification, cross-paper comparison, and evidence aggregation. Right: distribution of proactive search initiation types, including direct-title search, citation-based search, and indirect-hint-driven search.

Judge Pair	Turn Agr. $\uparrow$	Turn $\kappa$ $\uparrow$	Episode Agr. $\uparrow$	Episode $\kappa$ $\uparrow$
GPT vs. Gemini	98.7	0.97	100.0	1.00
GPT vs. Human	100.0	1.00	100.0	1.00
Human vs. Human	100.0	1.00	100.0	1.00

Appendix

Table 4: Agreement analysis between the primary GPT-5.2-based judge, an independent Gemini-2.5-Pro judge, and Ph.D.-level human judges on a sampled subset covering 30% of the predictions from all eight models in the main experiments. Higher is better for all metrics.

Figure 9: Representative failed questions grouped by error type. Each row shows a compressed execution trace from a real failed episode. The highlighted red module marks the first error under our trace-based attribution rule, while later incorrect outputs are shown only as downstream consequences. From top to bottom, the examples illustrate retrieval failure, reasoning failure, perception failure, and reading-memory failure. The omitted Others category mainly includes max-step termination, tool execution failures, and runtime or parsing issues.

Tool	Function	Input	Output
Web Search	Discovers candidate papers, resolves paper identities from indirect cues, and follows citation leads through ranked web retrieval.	Natural-language query, citation cue, keyword phrase	Ranked result list with titles, snippets, and URLs
PDF Extractor	Downloads a paper PDF and parses it into a normalized cached representation for later access.	Paper identifier such as title, URL, or arXiv ID	Parsed markdown document, metadata, and cache handle
PDF Extractor RAG	Retrieves the most relevant text spans from a cached paper without rereading the entire document.	Paper identifier plus text query	Ranked text spans with section context and lightweight provenance
Figure Extractor	Extracts a target figure from a cached paper when visual grounding is required.	Paper identifier plus figure index or figure ID	Figure image, caption context, nearby text, and provenance
Table Extractor	Extracts a target table from a cached paper and returns a structured representation for downstream reasoning.	Paper identifier plus table index or table ID	Table content, structured markdown, nearby text, and provenance
Code Executor	Executes code for deterministic calculations, lightweight post-processing, or verification in extended settings.	Code snippet with optional serialized inputs	Execution result, derived values, and runtime logs such as stdout or stderr

Appendix

Table 5: Detailed specification of the tool environment. The first four entries correspond to the core benchmark toolchain used in the main text, with Figure Extractor and Table Extractor serving as explicit decompositions of Content Extractor . Code Executor is listed separately because it is not part of the agent’s core reasoning process, but only a simple auxiliary execution utility, and is therefore not included in the default benchmark protocol.

Model	Input ( $\times 10^{7}$ )	Output ( $\times 10^{6}$ )	Total ( $\times 10^{7}$ )	Duration ( $\times 10^{4}$ s)
Kimi-K2.5	3.58	1.87	3.77	5.68
GLM-4.5V	1.52	1.04	1.62	3.61
Grok-4.1	3.99	2.08	4.20	3.38
Gemini-2.5-Pro	3.49	1.08	3.60	2.64
GPT-5.2	2.10	2.06	2.31	3.64
GPT-5-Mini	3.25	1.43	3.39	3.52

Appendix

Table 6: Aggregate token usage and runtime statistics over representative full-benchmark runs. Input and total tokens are reported in units of

\times 10^{7}

, output tokens are reported in units of

\times 10^{6}

, and duration is reported in units of

\times 10^{4}

AI Agent Benchmarks

AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.

See all 1,494 papers

PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers
Apr 13, 2026Lei Xiong, Huaying Yuan, Zheng Liu +2AI Agent Benchmarks Multimodal Model Evaluation
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China
Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agents
Sep 10, 2026Minghao Guo, Meng Cao, Sui Zhao +10Deep Research Agents Long-Horizon Agent Evaluation
Mohamed bin Zayed University of Artificial Intelligence · University of Science and Technology of China · Zhejiang University +1
Sci-MMR: Benchmarking Multi-Step Evidence-Grounded Scientific Reasoning in Multimodal Agents
Sep 10, 2026Jiaqiang Li, Yajie Yang, Zhiheng Xi +15Evidence-Grounded Reasoning Scientific Reasoning
Fudan NLP Lab · Fudan NLP Group

Apr 13, 2026J/K move · Enter open · S save

PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers

Lei Xiong, Huaying Yuan, Zheng Liu, Zhao Cao+1

Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China

AI Agent Benchmarks Multimodal Model Evaluation Scientific Reasoning in Language Models Multimodal QA

Open paper AskPDF arXiv

Leveraging Multi-modal Large Language Models (MLLMs) to accelerate frontier scientific research is promising, yet how to rigorously evaluate such systems remains unclear. Existing benchmarks mainly focus on single-document understanding, whereas real scientific workflows require integrating evidence from multiple papers, including their text, tables, and figures. As a result, multi-modal, multi-document scientific reasoning remains underexplored and lacks systematic evaluation. To address this gap, we introduce PaperScope, a multi-modal multi-document benchmark designed for agentic deep research. PaperScope presents three advantages: (1) Structured scientific grounding. It is built on a knowledge graph of over 2,000 AI papers spanning three years, providing a structured foundation for research-oriented queries. (2) Semantically dense evidence construction. It integrates semantically related key information nodes and employs optimized random-walk article selector to sample thematically coherent paper sets, thereby ensuring adequate semantic density and task complexity. (3) Multi-task evaluation of scientific reasoning. It contains over 2,000 QA pairs across reasoning, retrieval, summarization, and problem solving, enabling evaluation of multi-step scientific reasoning. Experimental results show that even advanced systems such as OpenAI Deep Research and Tongyi Deep Research achieve limited scores on PaperScope, highlighting the difficulty of long-context retrieval and deep multi-source reasoning. PaperScope thus provides a rigorous benchmark alongside a scalable pipeline for constructing large-scale multi-modal, multi-source deep research datasets.

Abstract

AI Agent Benchmarks

Explore similar work

PaperScope: A Multi-Modal Multi-Document Benchmark for Agentic Deep Research Across Massive Scientific Papers

Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agents

Sci-MMR: Benchmarking Multi-Step Evidence-Grounded Scientific Reasoning in Multimodal Agents