Automatic Inter-document Multi-hop Scientific QA Generation

Seungmin Lee; Dongha Kim; Yuni Jeon; Junyoung Koh; Min Song

cs.CLMar 15, 2026

Automatic Inter-document Multi-hop Scientific QA Generation

Authors: Seungmin Lee, Dongha Kim, Yuni Jeon, Junyoung Koh, Min Song

Organizations: Yonsei University, Seoul, Republic of Korea · OnomaAI, Seoul, Republic of Korea

Abstract

Existing automatic scientific question generation studies mainly focus on single-document factoid QA, overlooking the inter-document reasoning crucial for scientific understanding. We present AIM-SciQA, an automated framework for generating multi-document, multi-hop scientific QA datasets. AIM-SciQA extracts single-hop QAs using large language models (LLMs) with machine reading comprehension and constructs cross-document relations based on embedding-based semantic alignment while selectively leveraging citation information. Applied to 8,211 PubMed Central papers, it produced 411,409 single-hop and 13,672 multi-hop QAs, forming the IM-SciQA dataset. Human and automatic validation confirmed high factual consistency, and experimental results demonstrate that IM-SciQA effectively differentiates reasoning capabilities across retrieval and QA stages, providing a realistic and interpretable benchmark for retrieval-augmented scientific reasoning. We further extend this framework to construct CIM-SciQA, a citation-guided variant achieving comparable performance to the Oracle setting, reinforcing the dataset's validity and generality.

Figures & tables

Figure 1: Example of Inter-document Multi-hop QA: ① the model retrieves relevant documents and ② integrates complementary evidence to answer.

Dataset	M Hop	M Doc	#QA	QA Gen	Free-form	Domain	Source
PubMedQA	X	X	211k	Human	X	Biomedical	PubMed
SciQAG-24D	X	X	180k	LLM	O	Scientific Literature	Scientific Literature
WikiHop	O	O	51k	Rule-based	X	General/Wiki	Wiki
HotpotQA	O	O	113k	Human	O	General/Wiki	Wiki
DocHop-QA	O	O	11k	LLM	X	Biomedical/Scientific	PubMed
M3SciQA	O	O	1.4k	Human	X	NLP Papers	NLP Papers (+cited)

Table 1: Comparison with other datasets. M hop : Multi Hop, M Doc : Multi Documents, Q Gen : Question Generation method, Free-form : answers are open-ended.

Figure 2: Illustration of the relation construction process in AIM-SciQA. For IM-SciQA, candidate relations are created from papers sharing overlapping keywords with the source paper ( implicit semantic similarity relationships ), while CIM-SciQA constructs relations directly from explicit citation links ( explicit citation relationships ). From these candidate relations, QA-based relation construction (Section 4.3 ) process selects promising single-hop QA (made by Section 4.2 ) pairs across papers to form inter-document QA candidates, and a paper cluster is built to define the target paper and its corresponding retrieval QA .

Figure 3: Average Single-hop QA Count by Section

Figure 4: Total Single-hop QA Count by Section

Item	Value
Count	411,409
Avg. Question Length	88.75
Avg. Answer Length	126.92
Avg. QA per Paper	55.67
Avg. QA per Section	10.49

Table 2: Statistics of the Single-hop QA

Item	Dev	Test
Count	13,372	300
Avg. Retrieval Q Length	15.05	15.16
Avg. Inter-doc Q Length	28.68	28.65
Avg. Combined Q Length	48.52	49.21
Avg. Inter-doc A Length	28.13	29.98
Avg. Combined A Length	43.00	45.33

Table 3: Statistics of the Multi-hop QA

Figure 5: Results of multi-dimensional MHQA quality validation

Criterion	Description
Fluency	Whether the Complete QA is grammatically correct and naturally written.
Completeness	Whether the Complete QA can be answered solely based on the provided QA pair.
Cross-reference Necessity	Whether both papers are essential to answer the Complete QA.
Relational Appropriateness	Whether the two papers are conceptually suitable to be combined into a single QA.
Overall Decision	Synthesizes the above four criteria to make the final decision.

Table 4: Criteria of MHQA quality validation.

Evidence (E)	For instance, the U.S. CDC N1 assay had a recall of 99.0% and the U.S. CDC N2 assay had a 99.2% recall.
Meta	PMC10021650
Q1	What was the recall rate for the U.S. CDC N1 assay?
A1	The recall rate for the U.S. CDC N1 assay was 99.0%.
Q2	What was the recall rate for the U.S. CDC N2 assay?
A2	The recall rate for the U.S. CDC N2 assay was 99.2%.

Table 5: Example of Multi-QA Evidence

Model	Paper Cluster (N=30)			Random Cluster (N=30)			Full Papers (N=8211)
Model	Hit@1	Hit@3	MRR@5	Hit@1	Hit@3	MRR@5	Hit@1	Hit@50	MRR@50
GritLM-7B Muennighoff et al. (2024)	0.220	0.353	0.306	0.177	0.303	0.248	0.010	0.130	0.023
Jina-v2-base-en Günther et al. (2023)	0.197	0.350	0.280	0.183	0.340	0.266	0.020	0.117	0.030
e5-base-v2 Wang et al. (2022)	0.273	0.493	0.395	0.307	0.537	0.428	0.017	0.157	0.028
e5-mistral-7b-instruct Wang et al. (2023)	0.300	0.473	0.390	0.273	0.430	0.356	0.030	0.203	0.058
Qwen3-Embedding-8B Zhang et al. (2025)	0.407	0.597	0.554	0.483	0.607	0.555	0.080	0.397	0.136

Table 6: Retrieval performance across different settings. Bold indicates the best performance, and underline denotes the second best.

Model	Oracle Setting				Realistic Setting
Model	Acc	F1	ROUGE	BERTScore	Acc	F1	ROUGE	BERTScore
Human Expert	0.975	0.324	0.319	0.447	–	–	–	–
gemma-3-1b-it Team et al. (2025)	0.265	0.337	0.301	0.340	0.260	0.332	0.292	0.328
Llama-3.2-3B-it Grattafiori et al. (2024)	0.57	0.421	0.403	0.435	0.515	0.399	0.380	0.403
gemma-3-4b-it Team et al. (2025)	0.590	0.430	0.403	0.465	0.515	0.418	0.390	0.453
gemma-3-27b-it Team et al. (2025)	0.750	0.517	0.495	0.556	0.670	0.483	0.461	0.519

Table 7: Comparison of model performance under Oracle and Realistic settings. Bold indicates the best performance, and underline denotes the second best (based on Accuracy). Accuracy in Oracle setting is measured over 100 QA instances.

Model	Acc	F1	ROUGE	BERTScore
gemma-3-1b-it Team et al. (2025)	0.446	0.330	0.295	0.366
gemma-3-4b-it Team et al. (2025)	0.635	0.366	0.329	0.405
Llama-3.2-3B-it Grattafiori et al. (2024)	0.608	0.367	0.339	0.379
gemma-3-27b-it Team et al. (2025)	0.798	0.427	0.402	0.449
DeepSeek-R1 Guo et al. (2025)	0.784	0.131	0.205	0.110
Qwen3-30B-A3B-it Yang et al. (2025)	0.824	0.425	0.392	0.439

Table 8: CIM-SciQA Results (Realistic Setting)

Scientific QA

Scientific question answering develops systems that answer research and domain-science questions using reliable evidence from papers, data, and figures.

See all 63 papers

DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections
Aug 20, 2025Jiwon Park, Seohyun Pyeon, Jinwoo Kim +4Multi-Hop QA Scientific QA
Pohang University of Science and Technology, Pohang-si, South Korea · The University of Melbourne, Melbourne, Australia · The University of Western Australia, Perth, Australia
LitTraceQA: A Benchmark for Multi-Stage Grounding and Verification in Scientific Question Answering
Aug 7, 2026Xuye Liu, Yimu Wang, Peng Shi +7Evidence-Grounded Generation Scientific QA
University of Waterloo · 2Amazon · 3City University of Hong Kong +1
ResearchQA: Benchmarking Citation-Grounded Question-Answering on Scientific Papers
Jul 13, 2026Saba Imran, Debanjum Singh SolankyLLM Evaluation Citation Verification
Khoj Inc.

Aug 20, 2025J/K move · Enter open · S save

DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections

Jiwon Park, Seohyun Pyeon, Jinwoo Kim, Rina Carines Cabral+3

Pohang University of Science and Technology, Pohang-si, South Korea · The University of Melbourne, Melbourne, Australia · The University of Western Australia, Perth, Australia

Multi-Hop QA Scientific QA Multimodal QA Document QA

Open paper AskPDF arXiv

Despite rapid progress in large language models (LLMs), current QA benchmarks still overlook the core challenge of real-world scientific information seeking: synthesizing multimodal evidence scattered across multiple documents and structural formats. Existing QAs remain narrow in scope, relying on unimodal text and short-span reasoning that fail to capture the complexity of real information-seeking. We introduce DocHop-QA, a benchmark of 11,379 instances for evaluating multimodal, multi-document, multi-hop scientific QA. Built from publicly available PubMed articles, DocHop-QA incorporates textual passages, tables, and layout cues, enabling cross-document inference without explicit hyperlinks. To scale realistic QA construction, we develop an LLM-driven generation pipeline grounded in 11 scientific reasoning concepts, producing diverse and coherent question-answer pairs. To highlight the utility and versatility of the dataset, we propose a task-driven evaluation framework spanning four settings, including generative answering, multimodal evidence integration and structured index prediction. Experiments show that current models struggle with DocHop-QA's long-context, multi-evidence demands, establishing it as a rigorous testbed for advancing next-generation scientific QA systems.

Abstract

Scientific QA

Explore similar work

DocHop-QA: Towards Multi-Hop Reasoning over Multimodal Document Collections

LitTraceQA: A Benchmark for Multi-Stage Grounding and Verification in Scientific Question Answering

ResearchQA: Benchmarking Citation-Grounded Question-Answering on Scientific Papers