Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific Articles

Sai Koneru; Jian Wu; Sarah Rajtmajer

cs.CLMar 22, 2026

Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific Articles

Authors: Sai Koneru, Jian Wu, Sarah Rajtmajer

Organizations: Pennsylvania State University, University Park, Pennsylvania, USA · Old Dominion University, Norfolk, Virginia, USA

Abstract

Extracting hypotheses and their supporting statistical evidence from full-text scientific articles is central to the synthesis of empirical findings, but remains difficult due to document length and the distribution of scientific arguments across sections of the paper. The work studies a sequential full-text extraction setting, where the statement of a primary finding in an article's abstract is linked to (i) a corresponding hypothesis statement in the paper body and (ii) the statistical evidence that supports or refutes that hypothesis. This formulation induces a challenging within-document retrieval setting in which many candidate paragraphs are topically related to the finding but differ in rhetorical role, creating hard negatives for retrieval and extraction. Using a two-stage retrieve-and-extract framework, we conduct a controlled study of retrieval design choices, varying context quantity, context quality (standard Retrieval Augmented Generation, reranking, and a fine-tuned retriever paired with reranking), as well as an oracle paragraph setting to separate retrieval failures from extraction limits across four Large Language Model extractors. We find that targeted context selection consistently improves hypothesis extraction relative to full-text prompting, with gains concentrated in configurations that optimize retrieval quality and context cleanliness. In contrast, statistical evidence extraction remains substantially harder. Even with oracle paragraphs, performance remains moderate, indicating persistent extractor limitations in handling hybrid numeric-textual statements rather than retrieval failures alone.

Figures & tables

Figure 1. An example claim trace spanning the Abstract, Introduction, and Results sections of a paper. Our two-stage pipeline uses an abstract-level finding to extract a hypothesis statement (from the Introduction, in this case) and then locate statistical evidence (Results).

Figure 2. Schematic of the sequential retrieve and extract pipeline. Stage 1 (top) uses the abstract finding to extract the Hypothesis, which then serves as part of a composite query for extracting the Statistical Evidence in Stage 2 (bottom).

LLM	Method	Hypothesis			Evidence
		P	R	F1	P	R	F1
llama 3.2	Zero-Shot (Full text)	0.34	0.40	0.36	0.18	0.20	0.19
	CoT (Full text)	0.21	0.34	0.26	0.19	0.22	0.20
	RAG (k=5)	0.45	0.36	0.40	0.17	0.22	0.19
	RAG (k=10)	0.50	0.40	0.44	0.18	0.22	0.20
	RAG (k=20)	0.50	0.41	0.45	0.20	0.21	0.21

Table 1. Results for sequential two-stage extraction of hypothesis and corresponding evidence from full text of scientific articles. We compare hypothesis and statistical evidence extraction performance (Precision, Recall, F1) across four LLMs, comparing full-text baselines against retrieval-augmented strategies with varying context quantity (

k

) and retrieval quality. Oracle settings indicate performance upper bounds given perfect paragraph selection. Best non-oracle F1 per model and subtask is bolded, and oracle results are reported as upper bounds.

Panel of Figure 4 — Figure 4. Hypothesis extraction F1 versus relevant-sentence proportion (RSP), binned. Points show mean F1 within each bin and point size is proportional to the number of papers in the bin.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5. Calibration of the cosine similarity threshold using a precision recall curve. The plot shows the values at different threshold values for the Gemini embedding model. The selected threshold of 0.89 corresponds to the point of maximum F1 score (0.72) on manually labeled set.

Panel of Figure 8 — Figure 8. Per-paper transition analysis for evidence extraction under quantity vs. quality changes. Left column varies retrieved context quantity ( $k{=}5\rightarrow 10\rightarrow 20\rightarrow$ full text). Right column varies retrieval quality at fixed context size ( $k{=}5$ ) (RAG $\rightarrow$ RAG+Reranker $\rightarrow$ fine-tuned retriever+reranker $\rightarrow$ oracle). Bars report mean per-paper $\Delta$ F1 within each retrieval-outcome bucket; labels above bars show the fraction of papers in that bucket. For quality transitions, we additionally report GT_Lost when the gold evidence paragraph is present in the source context but absent in the target.

LLM	Method	Evidence
		P	R	F1
llama 3.2	RAG (k=5)	0.19	0.25	0.22
	RAG (k=10)	0.24	0.32	0.27
	RAG (k=20)	0.25	0.29	0.26
	Full text	0.21	0.28	0.24
	RAG + Reranker (k=5)	0.29	0.43	0.35

Appendix

Table 2. Evidence extraction with ground-truth hypotheses. We isolate the evidence stage by providing the extractor the human-annotated (ground-truth) hypothesis and varying only the evidence-stage context configuration (RAG

k\in\{5,10,20\}

, reranking, and fine-tuned retrieval). Reported Precision/Recall/F1 are computed with the same component-based evaluator as in the main results.

Document Information Extraction

Document Information Extraction turns text, tables, forms, and images into structured, searchable data.

See all 290 papers

Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models
Aug 19, 2026Valentin Romanov, Monique Bax, Steven NiedererLLM Information Extraction LLM Reliability
National Heart & Lung Institute, Faculty of Medicine, Imperial College London, United Kingdom · Cambridge Stem Cell Institute, JeDrey Cheah Biomedical Centre, Cambridge Biomedical Campus, University of Cambridge, Cambridge, United Kingdom · Department of Medicine, University of Cambridge, Cambridge, United Kingdom +1
From Articles to Premises: Building PrimeFacts, an Extraction Methodology and Resource for Fact-Checking Evidence
May 7, 2026Premtim Sahitaj, Jawan Kolanowski, Ariana Sahitaj +7Automated Fact-Checking LLM Information Extraction
Technische Universität Berlin, Quality and Usability Lab, Berlin, Germany · Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI), Berlin, Germany · Harz University of Applied Sciences, Wernigerode, Germany +1
Factorized Hypothesis Search for Evidence-to-Taxonomy Retrieval
Aug 6, 2026Linhai Ma, Ethan F. Wei, Xueqing Peng +3Query Reformulation Information Retrieval
The Fin AI, USA · Yale University, USA · Cardiff University, UK

Aug 19, 2026J/K move · Enter open · S save

Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models

Valentin Romanov, Monique Bax, Steven Niederer

National Heart & Lung Institute, Faculty of Medicine, Imperial College London, United Kingdom · Cambridge Stem Cell Institute, JeDrey Cheah Biomedical Centre, Cambridge Biomedical Campus, University of Cambridge, Cambridge, United Kingdom · Department of Medicine, University of Cambridge, Cambridge, United Kingdom+1

LLM Information Extraction LLM Reliability LLM Prompting AI Agents for Scientific Discovery

Open paper AskPDF arXiv

Accurately extracting nuanced, contextualized data from research articles is laborious and time intensive. Here, we investigate the performance of frontier, browser-based large language models (LLMs) to extract highly contextualized information. We demonstrate four escalating workflows, 1) given an expert curated prompt and research articles, most frontier LLMs perform well at data extraction, however can struggle with interpreting scientific context and nuance, 2) given simple instructions, LLMs can author their own prompts which were almost as eNective as expert-written prompts, 3) autonomous discovery of research literature was diNicult, agents either missed or hallucinated references, and 4) LLMs can create new datasets from published guidelines that closely match human-expert judges, but still require a human-in-the-loop. Together, these findings define an auditable division of labour in which experts specify the evidence standard, models cross-check repeated extractions and researchers resolve disputed cases, providing a practical route to scaling scientific data curation without relinquishing expert oversight.

Abstract

Document Information Extraction

Explore similar work

Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models

From Articles to Premises: Building PrimeFacts, an Extraction Methodology and Resource for Fact-Checking Evidence

Factorized Hypothesis Search for Evidence-to-Taxonomy Retrieval