Organizations: University of Arizona · Florida International University · Vanderbilt University · University of California, Irvine · New York University · University of Illinois Urbana-Champaign · Broad Institute of MIT and Harvard · University of Cincinnati · University of South Florida · Vanderbilt University Medical Center
Systematic literature reviews follow the PRISMA process of screening scholarly works and extracting evidence from included studies. Deep research tools based on large language models (LLMs) can search and summarize the literature but do not record which studies are excluded or where extracted values come from. We develop a conversational multi-agent system called ReviewPilot that automates the PRISMA workflow from paper screening to information extraction. Researchers can tailor criteria and schemas through dialogue, and a project memory keeps their configurations and revisions. Each screening decision comes with its reason and each extracted value with its source. On epidemiological expert reviews, ReviewPilot outperforms a single-prompt LLM and AgentSLR across three LLMs, with screening macro F1 up to 0.776 and extraction F1 up to 0.788. Case studies in biomedicine and human-computer interaction show close agreement with human annotators, suggesting that ReviewPilot can transfer beyond epidemiology. Code is available at https://github.com/ZhaoqianXue/ReviewPilot.
Figures & tables
Figure 1: Manual and ReviewPilot-assisted systematic review workflows, aligned with PRISMA 2020 stages ( Page et al., 2021 ) . ReviewPilot coordinates search, screening, and extraction while supporting researcher inspection and revision. Project memory retains configurations, outputs, and corrections.
Figure 2: ReviewPilot architecture and workflow. (1) The Lead Agent coordinates five review stages using project memory. Researchers configure stages and correct decisions or values. In the upper right, the Prompt Agent prepares screening guidance, an extraction schema, and coding rules without reviewer labels. (2) Screening returns a rationale and one of five likelihood levels. (3) Extraction returns values with verbatim excerpts. The label verified means the excerpt is found in the PDF text. Orange marks screening rationales and extraction evidence.
Macro
Macro
Macro
Review
Cost ↓
Method
precision ↑
recall ↑
F1 ↑
share ↓
(USD)
gpt-6-luna
Single-prompt LLM
0.685
0.828
0.696
0.354
0.13
AgentSLR
0.694
0.833
0.713
0.331
0.30
ReviewPilot
0.734
0.861
0.766
0.274
0.23
DeepSeek V4.1 Flash
Table 1: Screening results pooled over four PERG pathogens (5,614 records with 854 expert inclusions). Precision, recall, and F1 are macro-averaged over the include and exclude classes. Review share is the proportion of records retained for full-text review. Cost is USD per 1,000 records. Bold marks the best value within each model (highest for ↑ , lowest for ↓ ).
Verified-
correct
Cost ↓
Method
Precision ↑
Recall ↑
F1 ↑
rate ↑
(USD)
gpt-6-luna
Single-prompt LLM
0.714
0.837
0.771
0.528
0.0025
AgentSLR
0.723
0.782
0.751
0.449
0.0078
ReviewPilot
0.741
0.841
0.788
0.713
0.0030
Table 2: Extraction results on PERG (178 papers, or 177 for GLM-5.3-Flash after excluding the same paper for all methods). Precision, recall, and F1 are computed from counts pooled across papers and the five fields. Verified-correct rate requires the complete normalized prediction to match the reference set and at least one excerpt to be located in the paper. Cost is USD per paper, excluding coding-rule generation. Bold marks the best value (highest for ↑ , lowest for ↓ ).
Figure 3: HCI workflow in PRISMA 2020 format ( Page et al., 2021 ) . Squares track records. Green denotes retention, pink exclusion, grey removal or non-retrieval. Outlines identify numbered examples. Included squares show the first assigned category in right-panel order. Counts include all assignments. The right panel shows agents, tools, outputs, researcher actions, system processing times, and LLM costs of completed-workflow calls.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Screening
Extraction
Pathogen
Records
Incl.
Pathogen
Papers
Lassa
667
108
Ebola
79
Nipah
736
132
Lassa
6
SARS
2,047
244
SARS
36
Zika
2,164
370
Zika
57
Total
5,614
854
Total
178
Appendix
Table B.1: PERG data used in § 4 . Incl.: expert title/abstract inclusions.
Field
What it records
Values
model_type
The type of transmission model
Compartmental; Branching process; Agent / Individual based; Other
stoch_deter
Whether the model is stochastic or deterministic
Deterministic; Stochastic
code_available
Whether the authors share the model’s code
True; False
transmission_route
The transmission routes the model includes
Vector/Animal to human; Human to human (direct contact); Human to human (direct non-sexual contact); Sexual; Airborne or close contact; Animal reservoir to human
Table B.2: The five PERG extraction fields evaluated in § 4.3 . A paper’s gold value is the set of values over all its models. The transmission_route and interventions_type fields often take several values.
Model
Provider
Input
Output
gpt-6-luna
OpenAI
0.10
0.50
DeepSeek V4.1 Flash
Together AI
0.30
1.20
GLM-5.3-Flash
Together AI
0.15
0.50
Appendix
Table D.1: Models and prices (USD per million tokens). The Together AI identifiers are deepseek-ai/DeepSeek-V4.1-Flash and zai-org/GLM-5.3-Flash . Reasoning tokens are billed as output.
Component
Kept from the public release
Changed for this evaluation
Code
Commit 3111fcf; the prompt, parser and pipeline source files are checked against it at run time.
None.
Screening prompt and parser
Abstract-screening template, study objectives and decision parser.
None: the published template is used verbatim.
Screening requests
One request per record.
gpt-6-luna: reasoning effort medium instead of the published high; output cap 8,192 tokens. Together AI models: provider-default reasoning and temperature, streamed, output cap 32,768 (DeepSeek) and 65,536 (GLM) tokens.
Reasoning effort medium instead of the published high; output cap 8,192 (gpt-6-luna) or 32,768 (Together AI) tokens; a per-paper call budget.
Extraction input
—
The PDF text read by the application, with the same 400,000-character truncation as the other methods, instead of AgentSLR’s own document conversion.
Scoring
—
Per-model predictions are aggregated to the five paper-level fields as for the reference values; other AgentSLR fields are not scored; papers rejected by its internal screening stay in the evaluation with empty predictions.
Appendix
Table D.2: Changes made to AgentSLR’s public implementation for § 4 .
gpt-6-luna
DeepSeek V4.1 Flash
GLM-5.3-Flash
Macro
Macro
Macro
Macro
Macro
Macro
Macro
Macro
Macro
Method
precision ↑
recall ↑
F1 ↑
precision ↑
recall ↑
F1 ↑
precision ↑
recall ↑
F1 ↑
Lassa (667 records, 108 inclusions)
Single-prompt LLM
0.742
0.868
0.773
0.704
0.847
0.717
0.678
0.823
0.663
AgentSLR
0.745
0.857
0.777
0.765
0.882
0.800
0.715
0.867
0.730
ReviewPilot
0.756
0.870
0.789
0.777
0.882
0.812
0.757
0.887
0.791
Appendix
Table E.1: Screening results per pathogen. Precision, recall, and F1 are macro-averaged over the include and exclude classes, as in Table 1 . Bold marks the best value within each model.
Figure F.1: Biomedical review workflow in the PRISMA 2020 format ( Page et al., 2021 ) . The layout and record colors follow Figure 3 . The 3,000 records yield 1,341 included studies after screening and full-text assessment; 336 reports are not retrieved and seven are retrieved but not extracted, shown in dark grey within the not-retrieved box. Numbered examples show a survey exclusion (1), a reversed exclusion of a medical-exam evaluation (2), and extraction of model names and expert evaluation with source excerpts (3). Category counts include every assignment to a study. System processing times and LLM costs of completed-workflow calls are shown beside each stage.
Biomedicine
HCI
Comparison
Cohen’s κ↑
Cohen’s κ↑
Annotators A vs. B
0.860
0.733
ReviewPilot vs. annotator A
0.905
0.667
ReviewPilot vs. annotator B
0.907
0.793
ReviewPilot vs. shared label
0.975
0.844
Appendix
Table F.1: Screening agreement (Cohen’s κ ; Cohen, 1960 ) in separate audit-pool evaluations (87 biomedical and 90 HCI records). A and B are the independent annotators. Shared-label agreement uses only the 81 biomedical and 78 HCI records on which the annotators agree, with one and six ReviewPilot disagreements, respectively.
Topic
Screening inspections
Extraction source checks
HCI
40 exclusions checked: 19 flagged for an unverified quote or rule, and 21 sampled in proportion to exclusion reasons. Three exclusions reversed.
Eight eligible papers sampled from four strata, plus three selected by a consistency check. Three field values corrected; one missing excerpt added.
Biomedicine
56 exclusions checked: 33 flagged for an unverified quote or rule, and 23 sampled in proportion to exclusion reasons. Four exclusions reversed.
Eight eligible papers sampled from four strata. Three field values corrected; two missing excerpts added.
Appendix
Table F.2: Researcher inspections in the complete workflows. Flagged records and stratified paper samples are selected for checking. These counts do not estimate corpus-level error rates.
Topic
Inclusion (all must hold)
Exclusion (any applies)
Biomedicine
(1) The study develops, adapts, applies or evaluates a large language model. (2) The application is in biomedical research or clinical care. (3) The study reports original empirical work: an experiment, evaluation, deployment or user study.
(1) Review, survey, editorial, commentary or position paper without original empirical results. (2) No large language model is developed, adapted, applied or evaluated. (3) The application is outside biomedical research and clinical care.
HCI
(1) The study develops, adapts, applies or evaluates a large language model. (2) The study addresses human–computer interaction: interactive systems, interfaces, user experience, or how people use or perceive LLM-based tools. (3) The study reports original empirical work: a system evaluation, user study, experiment or deployment.
(1) Review, survey, editorial, commentary or position paper without original empirical results. (2) No large language model is developed, adapted, applied or evaluated. (3) No human interaction or user aspect is studied.
Appendix
Table F.3: Eligibility criteria for the case-study screening audits and both complete workflows (Section 5 ).
Topic
Field
Type
Values
Leave empty when
Both
llm_models
Text (list)
Model names as written in the paper, e.g., GPT-4 or Llama 2 70B
The paper names no specific model
Both
adaptation
Text (list)
Prompting only; Fine-tuning; Retrieval-augmented generation (RAG); Agent or tool use; Other
The paper does not say how the model is used
Biomedicine
biomedical_task
Text (list)
Clinical QA or exam; Documentation and summarization; Information extraction; Diagnosis or decision support; Patient communication; Biomedical research; Other
No biomedical task is stated
Biomedicine
data_source
Text (list)
EHR or clinical notes; Exam or benchmark questions; Biomedical literature; Patient messages; Synthetic cases; Other
The data source is not stated
Biomedicine
expert_evaluation
Boolean
True: clinicians or domain experts evaluate the outputs. False: evaluation is automatic only.
Output evaluation is not described
HCI
application_area
Text (list)
Writing; Programming; Education; Accessibility; Creativity and design; Information seeking; Health; Collaboration; Other
No application area is stated
Appendix
Table F.4: Case-study extraction schema, with five substantive fields per topic. List fields may contain multiple values. Unreported information is left empty, without inference or default values. Categorical list fields use the listed labels and Other for values outside them. Model names are free text.
Figure H.1: Search Setup. In the conversation, the researcher describes the topic and asks to merge two concepts and to set the date range and per-source limits. The canvas shows the resulting research question, the two required concepts, the Boolean query, the three sources with 1,000 results each, and the date range.
Figure H.3: Reviewing screening decisions. (a) The review dialog for one of the four exam-evaluation studies that the researcher reinstated (example 2 in Figure F.1 ), with the saved decision, the researcher’s rationale and approved criterion, and the controls for changing a decision. (b) Two rows of the screening list: a model decision with its likelihood label, rationale, decisive rule, and a quote verified in the title or abstract, and a reinstated record marked as human reviewed.
Figure H.4: Full-Text Retrieval. PDFs are retrieved for 1,369 of 1,705 records. Each of the 336 failures is listed with its reason and can be retried or supplied manually.
Figure H.6: Inspecting an extracted value of the radiology study in Figure F.1 (example 3). The excerpt supporting llm_models = GPT-4o is located verbatim on page 3 of the PDF and shown beside the page text. The researcher can correct the value with a supporting quote and a reason.
Figure H.8: The PRISMA 2020 flow that ReviewPilot generates for the run, from 3,000 identified records to 1,341 included and categorized studies, with exclusion counts by reason.
This study presents OpenExtract, an open-source pipeline for automated data extraction in large-scale systematic literature reviews. The pipeline queries large language models (LLMs) to predict data entries based on relevant sections of scientific articles. To test the efficacy of OpenExtract, we apply it to a systematic literature review in digital health and compare its outputs with those of human researchers. OpenExtract achieves precision and recall scores of > 0.8 in this task, indicating that it can be effective at extracting data automatically and efficiently. OpenExtract: https://github.com/JimAchterbergLUMC/OpenExtract.
Jim Achterberg, Bram Van Dijk, Jing Meng +8
Leiden University Medical Center, The Netherlands · Leiden Institute of Advanced Computer Science, Leiden University, The Netherlands · University of Warwick, UK +2
Systematic reviews require sustained human judgment across thousands of records, yet existing evaluations of large language models (LLMs) typically examine review stages in isolation. We introduce SciLitBench, a multi-stage benchmark spanning title and abstract screening, full-text screening, and schema-guided data extraction, with 42,981 retrieved records, 1,012 full texts, and annotations for 888 included papers. Across 22 open-weight LLMs from six model families, explicit inclusion and exclusion criteria improve title and abstract screening F2 by 28.8%, while researcher-authored rationales improve full-text screening by 15%. Data extraction reveals a different reliability regime: performance declines from 0.97 accuracy for publication year to 0.37 Jaccard overlap for computational approach, while the strongest models recover only 30% of annotated evaluation evidence and 25% of limitations. SciLitBench identifies a practical boundary between high-recall screening and evidence-complete extraction and provides a reproducible resource for evaluating LLM-assisted evidence synthesis.
Miguel Zabaleta, Baihan Lin
Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA
Systematic reviews traditionally have taken considerable amounts of human time and energy to complete, in part due to the extensive number of titles and abstracts that must be reviewed for potential inclusion. Recently, researchers have begun to explore how to use large language models (LLMs) to make this process more efficient. However, research to date has shown inconsistent results. We posit this is because prompting alone may not provide sufficient context for the model(s) to perform well. In this study, we fine-tune a small 1.2 billion parameter open-weight LLM specifically for study screening in the context of a systematic review in which humans rated more than 8500 titles and abstracts for potential inclusion. Our results showed strong performance improvements from the fine-tuned model, with the weighted F1 score improving 80.79% compared to the base model. When run on the full dataset of 8,277 studies, the fine-tuned model had 86.40% agreement with the human coder, a 91.18% true positive rate, a 86.38% true negative rate, and perfect agreement across multiple inference runs. Taken together, our results show that there is promise for fine-tuning LLMs for title and abstract screening in large-scale systematic reviews.