cs.CLOct 15, 2026
SaveReviewPilot: A Multi-Agent System for Screening and Data Extraction in Systematic Reviews
Organizations: University of Arizona · Florida International University · Vanderbilt University · University of California, Irvine ·
Abstract
Systematic literature reviews follow the PRISMA process of screening scholarly works and extracting evidence from included studies. Deep research tools based on large language models (LLMs) can search and summarize the literature but do not record which studies are excluded or where extracted values come from. We develop a conversational multi-agent system called ReviewPilot that automates the PRISMA workflow from paper screening to information extraction. Researchers can tailor criteria and schemas through dialogue, and a project memory keeps their configurations and revisions. Each screening decision comes with its reason and each extracted value with its source. On epidemiological expert reviews, ReviewPilot outperforms a single-prompt LLM and AgentSLR across three LLMs, with screening macro F1 up to 0.776 and extraction F1 up to 0.788. Case studies in biomedicine and human-computer interaction show close agreement with human annotators, suggesting that ReviewPilot can transfer beyond epidemiology. Code is available at https://github.com/ZhaoqianXue/ReviewPilot.
Figures & tables
Figure 1: Manual and ReviewPilot-assisted systematic review workflows, aligned with PRISMA 2020 stages ( Page et al., 2021 ) . ReviewPilot coordinates search, screening, and extraction while supporting researcher inspection and revision. Project memory retains configurations, outputs, and corrections.
Figure 2: ReviewPilot architecture and workflow. (1) The Lead Agent coordinates five review stages using project memory. Researchers configure stages and correct decisions or values. In the upper right, the Prompt Agent prepares screening guidance, an extraction schema, and coding rules without reviewer labels. (2) Screening returns a rationale and one of five likelihood levels. (3) Extraction returns values with verbatim excerpts. The label verified means the excerpt is found in the PDF text. Orange marks screening rationales and extraction evidence.
| Macro | Macro | Macro | Review | Cost | |
| Method | precision | recall | F1 | share | (USD) |
| gpt-6-luna | |||||
| Single-prompt LLM | 0.685 | 0.828 | 0.696 | 0.354 | 0.13 |
| AgentSLR | 0.694 | 0.833 | 0.713 | 0.331 | 0.30 |
| ReviewPilot | 0.734 | 0.861 | 0.766 | 0.274 | 0.23 |
| DeepSeek V4.1 Flash | |||||
Table 1: Screening results pooled over four PERG pathogens (5,614 records with 854 expert inclusions). Precision, recall, and F1 are macro-averaged over the include and exclude classes. Review share is the proportion of records retained for full-text review. Cost is USD per 1,000 records. Bold marks the best value within each model (highest for , lowest for ).
| Verified- | |||||
| correct | Cost | ||||
| Method | Precision | Recall | F1 | rate | (USD) |
| gpt-6-luna | |||||
| Single-prompt LLM | 0.714 | 0.837 | 0.771 | 0.528 | 0.0025 |
| AgentSLR | 0.723 | 0.782 | 0.751 | 0.449 | 0.0078 |
| ReviewPilot | 0.741 | 0.841 | 0.788 | 0.713 | 0.0030 |
Table 2: Extraction results on PERG (178 papers, or 177 for GLM-5.3-Flash after excluding the same paper for all methods). Precision, recall, and F1 are computed from counts pooled across papers and the five fields. Verified-correct rate requires the complete normalized prediction to match the reference set and at least one excerpt to be located in the paper. Cost is USD per paper, excluding coding-rule generation. Bold marks the best value (highest for , lowest for ).
Figure 3: HCI workflow in PRISMA 2020 format ( Page et al., 2021 ) . Squares track records. Green denotes retention, pink exclusion, grey removal or non-retrieval. Outlines identify numbered examples. Included squares show the first assigned category in right-panel order. Counts include all assignments. The right panel shows agents, tools, outputs, researcher actions, system processing times, and LLM costs of completed-workflow calls.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Screening | Extraction | |||
|---|---|---|---|---|
| Pathogen | Records | Incl. | Pathogen | Papers |
| Lassa | 667 | 108 | Ebola | 79 |
| Nipah | 736 | 132 | Lassa | 6 |
| SARS | 2,047 | 244 | SARS | 36 |
| Zika | 2,164 | 370 | Zika | 57 |
| Total | 5,614 | 854 | Total | 178 |
Table B.1: PERG data used in § 4 . Incl.: expert title/abstract inclusions.
| Field | What it records | Values |
|---|---|---|
| model_type | The type of transmission model | Compartmental; Branching process; Agent / Individual based; Other |
| stoch_deter | Whether the model is stochastic or deterministic | Deterministic; Stochastic |
| code_available | Whether the authors share the model’s code | True; False |
| transmission_route | The transmission routes the model includes | Vector/Animal to human; Human to human (direct contact); Human to human (direct non-sexual contact); Sexual; Airborne or close contact; Animal reservoir to human |
| interventions_type | The interventions the model includes | 17 values, e.g., Quarantine, Behaviour changes, Vaccination, Treatment, Hospitals, Vector/Animal control, Safe burials, Contact tracing, Other |
Table B.2: The five PERG extraction fields evaluated in § 4.3 . A paper’s gold value is the set of values over all its models. The transmission_route and interventions_type fields often take several values.
| Model | Provider | Input | Output |
|---|---|---|---|
| gpt-6-luna | OpenAI | 0.10 | 0.50 |
| DeepSeek V4.1 Flash | Together AI | 0.30 | 1.20 |
| GLM-5.3-Flash | Together AI | 0.15 | 0.50 |
Table D.1: Models and prices (USD per million tokens). The Together AI identifiers are deepseek-ai/DeepSeek-V4.1-Flash and zai-org/GLM-5.3-Flash . Reasoning tokens are billed as output.
| Component | Kept from the public release | Changed for this evaluation |
|---|---|---|
| Code | Commit 3111fcf; the prompt, parser and pipeline source files are checked against it at run time. | None. |
| Screening prompt and parser | Abstract-screening template, study objectives and decision parser. | None: the published template is used verbatim. |
| Screening requests | One request per record. | gpt-6-luna: reasoning effort medium instead of the published high; output cap 8,192 tokens. Together AI models: provider-default reasoning and temperature, streamed, output cap 32,768 (DeepSeek) and 65,536 (GLM) tokens. |
| Extraction pipeline | Responses-API extraction, tool schemas, output validation, internal full-text screening and provenance loop. | Reasoning effort medium instead of the published high; output cap 8,192 (gpt-6-luna) or 32,768 (Together AI) tokens; a per-paper call budget. |
| Extraction input | — | The PDF text read by the application, with the same 400,000-character truncation as the other methods, instead of AgentSLR’s own document conversion. |
| Scoring | — | Per-model predictions are aggregated to the five paper-level fields as for the reference values; other AgentSLR fields are not scored; papers rejected by its internal screening stay in the evaluation with empty predictions. |
Table D.2: Changes made to AgentSLR’s public implementation for § 4 .
| gpt-6-luna | DeepSeek V4.1 Flash | GLM-5.3-Flash | |||||||
| Macro | Macro | Macro | Macro | Macro | Macro | Macro | Macro | Macro | |
| Method | precision | recall | F1 | precision | recall | F1 | precision | recall | F1 |
| Lassa (667 records, 108 inclusions) | |||||||||
| Single-prompt LLM | 0.742 | 0.868 | 0.773 | 0.704 | 0.847 | 0.717 | 0.678 | 0.823 | 0.663 |
| AgentSLR | 0.745 | 0.857 | 0.777 | 0.765 | 0.882 | 0.800 | 0.715 | 0.867 | 0.730 |
| ReviewPilot | 0.756 | 0.870 | 0.789 | 0.777 | 0.882 | 0.812 | 0.757 | 0.887 | 0.791 |
Table E.1: Screening results per pathogen. Precision, recall, and F1 are macro-averaged over the include and exclude classes, as in Table 1 . Bold marks the best value within each model.
Figure F.1: Biomedical review workflow in the PRISMA 2020 format ( Page et al., 2021 ) . The layout and record colors follow Figure 3 . The 3,000 records yield 1,341 included studies after screening and full-text assessment; 336 reports are not retrieved and seven are retrieved but not extracted, shown in dark grey within the not-retrieved box. Numbered examples show a survey exclusion (1), a reversed exclusion of a medical-exam evaluation (2), and extraction of model names and expert evaluation with source excerpts (3). Category counts include every assignment to a study. System processing times and LLM costs of completed-workflow calls are shown beside each stage.
| Biomedicine | HCI | |
|---|---|---|
| Comparison | Cohen’s | Cohen’s |
| Annotators A vs. B | 0.860 | 0.733 |
| ReviewPilot vs. annotator A | 0.905 | 0.667 |
| ReviewPilot vs. annotator B | 0.907 | 0.793 |
| ReviewPilot vs. shared label | 0.975 | 0.844 |
Table F.1: Screening agreement (Cohen’s ; Cohen, 1960 ) in separate audit-pool evaluations (87 biomedical and 90 HCI records). A and B are the independent annotators. Shared-label agreement uses only the 81 biomedical and 78 HCI records on which the annotators agree, with one and six ReviewPilot disagreements, respectively.
| Topic | Screening inspections | Extraction source checks |
|---|---|---|
| HCI | 40 exclusions checked: 19 flagged for an unverified quote or rule, and 21 sampled in proportion to exclusion reasons. Three exclusions reversed. | Eight eligible papers sampled from four strata, plus three selected by a consistency check. Three field values corrected; one missing excerpt added. |
| Biomedicine | 56 exclusions checked: 33 flagged for an unverified quote or rule, and 23 sampled in proportion to exclusion reasons. Four exclusions reversed. | Eight eligible papers sampled from four strata. Three field values corrected; two missing excerpts added. |
Table F.2: Researcher inspections in the complete workflows. Flagged records and stratified paper samples are selected for checking. These counts do not estimate corpus-level error rates.
| Topic | Inclusion (all must hold) | Exclusion (any applies) |
|---|---|---|
| Biomedicine | (1) The study develops, adapts, applies or evaluates a large language model. (2) The application is in biomedical research or clinical care. (3) The study reports original empirical work: an experiment, evaluation, deployment or user study. | (1) Review, survey, editorial, commentary or position paper without original empirical results. (2) No large language model is developed, adapted, applied or evaluated. (3) The application is outside biomedical research and clinical care. |
| HCI | (1) The study develops, adapts, applies or evaluates a large language model. (2) The study addresses human–computer interaction: interactive systems, interfaces, user experience, or how people use or perceive LLM-based tools. (3) The study reports original empirical work: a system evaluation, user study, experiment or deployment. | (1) Review, survey, editorial, commentary or position paper without original empirical results. (2) No large language model is developed, adapted, applied or evaluated. (3) No human interaction or user aspect is studied. |
Table F.3: Eligibility criteria for the case-study screening audits and both complete workflows (Section 5 ).
| Topic | Field | Type | Values | Leave empty when |
|---|---|---|---|---|
| Both | llm_models | Text (list) | Model names as written in the paper, e.g., GPT-4 or Llama 2 70B | The paper names no specific model |
| Both | adaptation | Text (list) | Prompting only; Fine-tuning; Retrieval-augmented generation (RAG); Agent or tool use; Other | The paper does not say how the model is used |
| Biomedicine | biomedical_task | Text (list) | Clinical QA or exam; Documentation and summarization; Information extraction; Diagnosis or decision support; Patient communication; Biomedical research; Other | No biomedical task is stated |
| Biomedicine | data_source | Text (list) | EHR or clinical notes; Exam or benchmark questions; Biomedical literature; Patient messages; Synthetic cases; Other | The data source is not stated |
| Biomedicine | expert_evaluation | Boolean | True: clinicians or domain experts evaluate the outputs. False: evaluation is automatic only. | Output evaluation is not described |
| HCI | application_area | Text (list) | Writing; Programming; Education; Accessibility; Creativity and design; Information seeking; Health; Collaboration; Other | No application area is stated |
Table F.4: Case-study extraction schema, with five substantive fields per topic. List fields may contain multiple values. Unreported information is left empty, without inference or default values. Categorical list fields use the listed labels and Other for values outside them. Model names are free text.
Figure H.1: Search Setup. In the conversation, the researcher describes the topic and asks to merge two concepts and to set the date range and per-source limits. The canvas shows the resulting research question, the two required concepts, the Boolean query, the three sources with 1,000 results each, and the date range.
Figure H.3: Reviewing screening decisions. (a) The review dialog for one of the four exam-evaluation studies that the researcher reinstated (example 2 in Figure F.1 ), with the saved decision, the researcher’s rationale and approved criterion, and the controls for changing a decision. (b) Two rows of the screening list: a model decision with its likelihood label, rationale, decisive rule, and a quote verified in the title or abstract, and a reinstated record marked as human reviewed.
Figure H.4: Full-Text Retrieval. PDFs are retrieved for 1,369 of 1,705 records. Each of the 336 failures is listed with its reason and can be retried or supplied manually.
Figure H.6: Inspecting an extracted value of the radiology study in Figure F.1 (example 3). The excerpt supporting llm_models = GPT-4o is located verbatim on page 3 of the PDF and shown beside the page text. The researcher can correct the value with a supporting quote and a reason.
Figure H.8: The PRISMA 2020 flow that ReviewPilot generates for the run, from 3,000 identified records to 1,341 included and categorized studies, with exclusion counts by reason.