cs.CLMar 22, 2026

Context Selection for Hypothesis and Statistical Evidence Extraction from Full-Text Scientific Articles

Authors: Sai Koneru, Jian Wu, Sarah Rajtmajer

Organizations: Pennsylvania State University, University Park, Pennsylvania, USA · Old Dominion University, Norfolk, Virginia, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Extracting hypotheses and their supporting statistical evidence from full-text scientific articles is central to the synthesis of empirical findings, but remains difficult due to document length and the distribution of scientific arguments across sections of the paper. The work studies a sequential full-text extraction setting, where the statement of a primary finding in an article's abstract is linked to (i) a corresponding hypothesis statement in the paper body and (ii) the statistical evidence that supports or refutes that hypothesis. This formulation induces a challenging within-document retrieval setting in which many candidate paragraphs are topically related to the finding but differ in rhetorical role, creating hard negatives for retrieval and extraction. Using a two-stage retrieve-and-extract framework, we conduct a controlled study of retrieval design choices, varying context quantity, context quality (standard Retrieval Augmented Generation, reranking, and a fine-tuned retriever paired with reranking), as well as an oracle paragraph setting to separate retrieval failures from extraction limits across four Large Language Model extractors. We find that targeted context selection consistently improves hypothesis extraction relative to full-text prompting, with gains concentrated in configurations that optimize retrieval quality and context cleanliness. In contrast, statistical evidence extraction remains substantially harder. Even with oracle paragraphs, performance remains moderate, indicating persistent extractor limitations in handling hybrid numeric-textual statements rather than retrieval failures alone.

Figures & tables

Figure 1. An example claim trace spanning the Abstract, Introduction, and Results sections of a paper. Our two-stage pipeline uses an abstract-level finding to extract a hypothesis statement (from the Introduction, in this case) and then locate statistical evidence (Results).
Figure 2. Schematic of the sequential retrieve and extract pipeline. Stage 1 (top) uses the abstract finding to extract the Hypothesis, which then serves as part of a composite query for extracting the Statistical Evidence in Stage 2 (bottom).
LLMMethodHypothesisEvidence
  PRF1PRF1
llama 3.2Zero-Shot (Full text)0.340.400.360.180.200.19
CoT (Full text)0.210.340.260.190.220.20
RAG (k=5)0.450.360.400.170.220.19
RAG (k=10)0.500.400.440.180.220.20
RAG (k=20)0.500.410.450.200.210.21
Table 1. Results for sequential two-stage extraction of hypothesis and corresponding evidence from full text of scientific articles. We compare hypothesis and statistical evidence extraction performance (Precision, Recall, F1) across four LLMs, comparing full-text baselines against retrieval-augmented strategies with varying context quantity ( kk ) and retrieval quality. Oracle settings indicate performance upper bounds given perfect paragraph selection. Best non-oracle F1 per model and subtask is bolded, and oracle results are reported as upper bounds.
Figure 4. Hypothesis extraction F1 versus relevant-sentence proportion (RSP), binned. Points show mean F1 within each bin and point size is proportional to the number of papers in the bin.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5. Calibration of the cosine similarity threshold using a precision recall curve. The plot shows the values at different threshold values for the Gemini embedding model. The selected threshold of 0.89 corresponds to the point of maximum F1 score (0.72) on manually labeled set.
Figure 8. Per-paper transition analysis for evidence extraction under quantity vs. quality changes. Left column varies retrieved context quantity ( k=5→10→20→k{=}5\rightarrow 10\rightarrow 20\rightarrow full text). Right column varies retrieval quality at fixed context size ( k=5k{=}5 ) (RAG →\rightarrow RAG+Reranker →\rightarrow fine-tuned retriever+reranker →\rightarrow oracle). Bars report mean per-paper Δ\Delta F1 within each retrieval-outcome bucket; labels above bars show the fraction of papers in that bucket. For quality transitions, we additionally report GT_Lost when the gold evidence paragraph is present in the source context but absent in the target.
LLMMethodEvidence
  PRF1
llama 3.2RAG (k=5)0.190.250.22
RAG (k=10)0.240.320.27
RAG (k=20)0.250.290.26
Full text0.210.280.24
RAG + Reranker (k=5)0.290.430.35
Appendix
Table 2. Evidence extraction with ground-truth hypotheses. We isolate the evidence stage by providing the extractor the human-annotated (ground-truth) hypothesis and varying only the evidence-stage context configuration (RAG k∈{5,10,20}k\in\{5,10,20\} , reranking, and fine-tuned retrieval). Reported Precision/Recall/F1 are computed with the same component-based evaluator as in the main results.
More in this topic

Document Information Extraction

Document Information Extraction turns text, tables, forms, and images into structured, searchable data.

See all 290 papers

Explore similar work

CardsList
Aug 19, 2026

Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models

Accurately extracting nuanced, contextualized data from research articles is laborious and time intensive. Here, we investigate the performance of frontier, browser-based large language models (LLMs) to extract highly contextualized information. We demonstrate four escalating workflows, 1) given an expert curated prompt and research articles, most frontier LLMs perform well at data extraction, however can struggle with interpreting scientific context and nuance, 2) given simple instructions, LLMs can author their own prompts which were almost as eNective as expert-written prompts, 3) autonomous discovery of research literature was diNicult, agents either missed or hallucinated references, and 4) LLMs can create new datasets from published guidelines that closely match human-expert judges, but still require a human-in-the-loop. Together, these findings define an auditable division of labour in which experts specify the evidence standard, models cross-check repeated extractions and researchers resolve disputed cases, providing a practical route to scaling scientific data curation without relinquishing expert oversight.

Valentin Romanov, Monique Bax, Steven Niederer

National Heart & Lung Institute, Faculty of Medicine, Imperial College London, United Kingdom · Cambridge Stem Cell Institute, JeDrey Cheah Biomedical Centre, Cambridge Biomedical Campus, University of Cambridge, Cambridge, United Kingdom · Department of Medicine, University of Cambridge, Cambridge, United Kingdom +1

LLM Information ExtractionLLM ReliabilityLLM PromptingAI Agents for Scientific Discovery
AskPDF
May 7, 2026

From Articles to Premises: Building PrimeFacts, an Extraction Methodology and Resource for Fact-Checking Evidence

Fact-checking articles encode rich supporting evidence and reasoning, yet this evidence remains largely inaccessible to automated verification systems due to unstructured presentation. We introduce PrimeFacts, a methodology and resource for extracting fine-grained evidence from full fact-checking articles. We compile 13,106 PolitiFact articles with claims, verdicts, and all referenced sources, and we identify 49,718 in-article hyperlinks as natural anchors to pinpoint key evidence. Our framework leverages large language models (LLMs) to rewrite these anchor sentences into stand-alone, context-independent premises and investigates the extraction of additional implicit evidence. In evaluations on cross-article evidence retrieval and claim verification, the extracted premises substantially improve performance. Decontextualized evidence yields higher retrievability, achieving up to a 30 percent relative gain in Mean Reciprocal Rank over verbatim sentences, and using the evidence for verdict prediction raises Macro-F1 by 10-20 points over the baseline. These gains are consistent across different verdict granularities (2-class vs. 5-class) and model architectures. A qualitative analysis indicates that the decontextualized premises remain faithful to the original sources. Our work highlights the promise of reusing fact-checkers' evidence for automation and provides a large-scale resource of structured evidence from real-world fact-checks.

Premtim Sahitaj, Jawan Kolanowski, Ariana Sahitaj +7

Technische Universität Berlin, Quality and Usability Lab, Berlin, Germany · Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI), Berlin, Germany · Harz University of Applied Sciences, Wernigerode, Germany +1

Automated Fact-CheckingLLM Information ExtractionEvidence Retrieval
AskPDF
Aug 6, 2026

Factorized Hypothesis Search for Evidence-to-Taxonomy Retrieval

Large-taxonomy retrieval often assumes that the input already expresses the target concept. In many settings, however, the input is indirect evidence, such as a table cell whose meaning depends on its row, column, datatype, and context. We call this mismatch the retrieval readiness gap. Our analysis shows that the current index retrieves the target reliably when its semantics are explicit, while raw evidence often leaves it deep in the ranking. We propose Factorized Hypothesis Search (FHS), which maintains multiple partial interpretations over named semantic dimensions. These hypotheses support structured query rendering, multi-hypothesis retrieval, and dimension-level candidate verification. On both financial taxonomy tagging and CodiEsp clinical coding tasks, FHS achieves the best Recall@1, MRR, and final accuracy among the non-oracle methods. Replacing the factorized hypothesis path with a free-text ensemble causes the largest drop in head-ranking performance, while sequential refinement provides no additional gain over FHS's strong parallel first round.

Linhai Ma, Ethan F. Wei, Xueqing Peng +3

The Fin AI, USA · Yale University, USA · Cardiff University, UK

Query ReformulationInformation RetrievalEvidence Retrieval
AskPDF