Large language model-assisted discovery of cohorts from scientific literature

Moritz Sturm; Lisa M. Berg; Inken Berg; Harishny Sarma; Jasmin Hartmann; Denissa Girschik; Gemma Roig; Christine M. Freitag

cs.IRAug 16, 2026

Large language model-assisted discovery of cohorts from scientific literature

Authors: Moritz Sturm, Lisa M. Berg, Inken Berg, Harishny Sarma, Jasmin Hartmann, Denissa Girschik, Gemma Roig, Christine M. Freitag,

Organizations: Department of Child and Adolescent Psychiatry, Psychosomatic Medicine and Psychotherapy, University Hospital Frankfurt, Goethe University Frankfurt, Frankfurt am Main, Germany · Department of Computer Science, Goethe University Frankfurt, Frankfurt am Main, Germany · The Hessian Center for Artificial Intelligence (hessian.AI), Darmstadt, Germany

Abstract

Background: Planning multi-study analyses requires identifying cohorts with the relevant participants, phenotypes, and data modalities. This process commonly relies on prior knowledge, cohort catalogues, and manual literature searches. We developed a complementary question-driven framework that searches relevant scientific literature and extracts explicit cohort names. Methods: The framework first generates multiple PubMed queries from configurable vocabularies and templates and retrieves the resulting scientific literature automatically through the PubMed API. A large language model then screens the retrieved titles and abstracts and extracts explicit cohort names using a prompt tailored to the research question. The extracted names are deduplicated with human review. Configurable code, prompts, and example outputs are available at https://gitlab.rz.uni-frankfurt.de/cap_molgenlab/literature-cohort-discovery. Evaluation: As a use case, we applied the framework to youth aggression genetics. From 5,400 generated PubMed queries, the framework retrieved 5,254 unique records and identified 188 candidate cohorts. Manual screening using predefined criteria, including participant age and genetic-data availability, retained 44 eligible cohorts. Automated LLM-based name extraction was within the agreement range of human annotators. We also searched four established cohort catalogues using the same research question. Their combined results contained 27 of the 44 eligible cohorts, while 17 were not returned by any cohort catalogue search. Conclusion: The framework converts research-question-specific vocabulary into screenable cohort inventories via a large, automated literature search. It can be adapted across populations, phenotypes, data modalities, and study designs, and provides a literature-based complement to curated cohort catalogues.

Figures & tables

Fig 1: Overview of the cohort-discovery workflow. First, a research question is translated into question-specific vocabularies, PubMed query templates, and extraction instructions. Publications are then retrieved through the PubMed API, followed by large language model (LLM)-assisted extraction of cohort names. Finally, extracted name variants are clustered as potential aliases and manually reviewed to produce the candidate cohort inventory. Red outlines denote automated processing steps.

Comparison	n	Positives	Precision	Recall	F1-score	Balanced accuracy
GPT vs. GOLD (R1 and R2 agree)	176	26	1.000	0.769	0.870	0.885
GPT vs. R1	199	30	0.885	0.767	0.821	0.874
GPT vs. R2	199	45	0.885	0.511	0.648	0.746
R1 vs. R2 (human-human)	199	30	0.578	0.867	0.693	0.877

Table 1: Performance metrics for LLM-based database name extraction. The generative pre-trained transformer (GPT) was compared against the gold standard (agreement between human rater 1 [R1] and human rater 2 [R2]) and against each individual rater. The last row shows the human-human baseline (R1 vs. R2).

Fig 2: Catalogue search returns and overlap with the literature-derived cohort inventory. Grey bars show the cohort identities retrieved from each catalogue; blue and orange bars show matches to the 188 literature-derived candidates and 44 eligible cohorts, respectively. Counts are catalogue-specific, and the same cohort may occur in more than one catalogue. CatMH denotes Catalogue of Mental Health Measures; dbGaP, Database of Genotypes and Phenotypes.

Fig S1: Example workflow for AI-assisted cohort-name extraction in the youth-aggression application. Titles and abstracts retrieved through the PubMed API were inserted into a structured prompt and submitted to the GPT-4o API. The model returned a structured JSON response containing an INCLUDE / DISCARD decision and any extracted cohort names. Extracted names were linked to the corresponding PMID and added to the candidate list. The prompt, publication record, and model output shown are based on real instances; the abstract for PMID 39661216 was shortened for display. The complete prompt and implementation are available in the accompanying code repository (see Data provenance statement ).

Biomedical IR

Biomedical information retrieval finds and ranks relevant evidence in scientific literature, clinical records, databases, and biomedical knowledge resources.

See all 43 papers

Jul 29, 2026

Scientific Knowledge Discovery in the Age of Large Language Models

The rapid growth of scholarly literature has made identifying relevant publications increasingly difficult, and conventional search systems still depend heavily on manually formulated queries and effortful manual inspection. Generative large language models (LLMs) offer a more flexible alternative, supporting literature retrieval and the screening of candidate studies against eligibility criteria. This chapter surveys 34 peer-reviewed papers applying generative LLMs to these two tasks, identified via a Boolean search over the OpenAIRE Graph (1,589 records screened to 34 inclusions). Reviewed studies are characterised by LLMs employed, model access and adaptation, prompting and architectural techniques, ground-truth sources, and evaluation metrics.

Eleni Adamidi, Serafeim Chatzopoulos, Thanasis Vergoulis

IMSI, ATHENA RC

AI-Assisted Scientific Research Information Retrieval Large Language Model-Guided Scientific Discovery

Ask PDF

Aug 19, 2026

Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models

Accurately extracting nuanced, contextualized data from research articles is laborious and time intensive. Here, we investigate the performance of frontier, browser-based large language models (LLMs) to extract highly contextualized information. We demonstrate four escalating workflows, 1) given an expert curated prompt and research articles, most frontier LLMs perform well at data extraction, however can struggle with interpreting scientific context and nuance, 2) given simple instructions, LLMs can author their own prompts which were almost as eNective as expert-written prompts, 3) autonomous discovery of research literature was diNicult, agents either missed or hallucinated references, and 4) LLMs can create new datasets from published guidelines that closely match human-expert judges, but still require a human-in-the-loop. Together, these findings define an auditable division of labour in which experts specify the evidence standard, models cross-check repeated extractions and researchers resolve disputed cases, providing a practical route to scaling scientific data curation without relinquishing expert oversight.

Valentin Romanov, Monique Bax, Steven Niederer

National Heart & Lung Institute, Faculty of Medicine, Imperial College London, United Kingdom · Cambridge Stem Cell Institute, JeDrey Cheah Biomedical Centre, Cambridge Biomedical Campus, University of Cambridge, Cambridge, United Kingdom · Department of Medicine, University of Cambridge, Cambridge, United Kingdom +1

LLM Information Extraction LLM Reliability LLM Prompting AI Agents for Scientific Discovery

Ask PDF

Oct 15, 2026

ReviewPilot: A Multi-Agent System for Screening and Data Extraction in Systematic Reviews

Systematic literature reviews follow the PRISMA process of screening scholarly works and extracting evidence from included studies. Deep research tools based on large language models (LLMs) can search and summarize the literature but do not record which studies are excluded or where extracted values come from. We develop a conversational multi-agent system called ReviewPilot that automates the PRISMA workflow from paper screening to information extraction. Researchers can tailor criteria and schemas through dialogue, and a project memory keeps their configurations and revisions. Each screening decision comes with its reason and each extracted value with its source. On epidemiological expert reviews, ReviewPilot outperforms a single-prompt LLM and AgentSLR across three LLMs, with screening macro F1 up to 0.776 and extraction F1 up to 0.788. Case studies in biomedicine and human-computer interaction show close agreement with human annotators, suggesting that ReviewPilot can transfer beyond epidemiology. Code is available at https://github.com/ZhaoqianXue/ReviewPilot.

Zhaoqian Xue, Shaoze Zhou, Xuexin Li +10

University of Arizona · Florida International University · Vanderbilt University +7

Systematic Review Automation AI-Assisted Scientific Research Multi-Agent LLM Systems LLM Information Extraction

Ask PDF

Abstract

Biomedical IR

Explore similar work

Scientific Knowledge Discovery in the Age of Large Language Models

Self-prompting and cross-model consensus enable reproducible data extraction from scientific literature with large language models

ReviewPilot: A Multi-Agent System for Screening and Data Extraction in Systematic Reviews