Evaluating AI-based Scientific Knowledge Synthesis with Epidemiological Systematic Reviews
Authors: Shreyansh Padarha, Ryan Othniel Kearns, Tristan Naidoo, Lingyi Yang, Łukasz Borchmann, Piotr BŁaszczyk, Christian Morgenstern, Ruth McCabe, Sangeeta Bhatia, Philip H. Torr, Jakob Foerster, Scott A. Hale, Thomas Rawson, Anne Cori, Elizaveta Semenova, Adam Mahdi
Organizations: University of Oxford · Imperial College London · University of Nottingham · Snowflake AI Research · Independent
Systematic literature reviews (SLRs) are a demanding and high-stakes form of scientific knowledge synthesis that remains underspecified as an evaluation setting for large language models (LLMs). We introduce AgentSLR, a large-scale evaluation harness comprising an SLR automation workflow and an expert annotated dataset covering 16,248 articles, designed to test LLM capabilities across the stages of SLRs in epidemiology. Reference annotations were derived from peer-reviewed studies on WHO priority pathogens and produced by domain experts. The harness evaluates each review stage as a separate unit with dedicated metrics enabling targeted failure analysis. We evaluated five frontier reasoning models and found that no single model dominated across all tasks, showing sub-task specialisation often hidden by aggregate benchmarks. Structured data extraction is a major bottleneck, with no model exceeding an average field-level F1 of 0.67. Estimated costs vary substantially, by up to 96 times across evaluated models. Documented failure modes suggest that the evaluated models are not yet reliable enough for unsupervised deployment in epidemiology, where findings can inform public policy.
Figures & tables
Figure 1: AgentSLR evaluation harness (LLM-assisted workflow). (a) Article Search and Retrieval queries bibliographic databases with domain-specific Boolean searches and obtains PDFs from open-access sources. (b) Title and Abstract Screening uses LLMs to filter articles using expert-designed inclusion and exclusion criteria. (c) PDF-to-Markdown Conversion uses OCR to convert PDFs to markdown files. (d) Full-text Screening applies the same approach as in Title and Abstract Screening but with stricter criteria. (e) Data Extraction employs multi-stage tool-calling with schema validation to extract structured epidemiological data (parameters, models, outbreaks). (f) Report Generation synthesises extracted data through artefact generation followed by LLM refinement.
Pathogen
PERG *
AgentSLR Matched
∙ Marburg virus
2,593
801 (30.9%)
∙ Ebola virus
11,605
4,119 (35.5%)
∙ Lassa fever
2,131
667 (31.3%)
∙ SARS-CoV-1
12,280
2,047 (16.7%)
∙ Zika virus
10,510
2,164 (20.6%)
∙ MERS-CoV
19,656
5,714 (29.1%)
Table 1: Released dataset (benchmark) record counts. AgentSLR Matched denotes the downloaded article subset that matched PERG labelled records and was released on HuggingFace. The coloured symbols represent the progress of PERG’s SLRs per priority pathogen: published ∙ , conducting data extraction ∙ , and yet to begin screening ∙ , as of March 2026.
Figure 2: Stage-wise model performance on AgentSLR harness. Each model is run on individual pathogens with AgentSLR harness. Averages are computed over the pathogens evaluated at each stage, following reference data availability described in Section 4.1 . Error bars indicate one standard deviation across pathogens. No single model dominates across all stages: Kimi-K2.5 and gpt-oss-120b lead screening, while extraction leaders vary by data type. Full metrics are provided in Appendix E .
Parameters
Models
Outbreaks
Average
Model
∙
∙
∙
∙
∙
∙
∙
∙
∙
∙
∙
∙
gpt-oss-120b
0.66 ± 0.07
0.59 ± 0.08
0.54 ± 0.03
0.91 ± 0.05
0.68 ± 0.05
0.67 ± 0.03
0.61 ± 0.13
0.69 ± 0.31
0.79 ± 0.02
0.75 ± 0.15
0.65 ± 0.13
0.64 ± 0.10
GPT-5.2
0.66 ± 0.07
0.50 ± 0.07
0.59 ± 0.03
0.90 ± 0.05
0.72 ± 0.04
0.67 ± 0.03
0.66 ± 0.12
0.80 ± 0.05
0.84 ± 0.04
0.76 ± 0.14
0.65 ± 0.14
0.67 ± 0.10
DeepSeek-V3.2
0.60 ± 0.07
0.56 ± 0.04
0.50 ± 0.03
0.87 ± 0.04
0.92 ± 0.08
0.65 ± 0.09
0.65 ± 0.04
0.78 ± 0.19
0.75 ± 0.02
0.72 ± 0.14
0.75 ± 0.19
0.61 ± 0.11
Kimi-K2.5
0.72 ± 0.09
0.62 ± 0.05
0.56 ± 0.02
0.92 ± 0.04
0.81 ± 0.04
0.68 ± 0.04
0.64 ± 0.07
0.87 ± 0.11
0.78 ± 0.07
0.78 ± 0.14
0.75 ± 0.12
0.65 ± 0.10
GLM-4.7
0.72 ± 0.09
0.61 ± 0.05
0.54 ± 0.02
0.93 ± 0.07
0.93 ± 0.06
0.68 ± 0.05
0.68 ± 0.03
0.72 ± 0.29
0.77 ± 0.02
0.80 ± 0.13
0.76 ± 0.18
0.64 ± 0.10
Table 2: Structured evidence extraction performance. F 1 scores are averaged across pathogens for five frontier models and three extraction tasks: ∙ Flagging, ∙ Counts, and ∙ field-level Extraction. Standard deviations ( ± ) are reported across pathogens.
Figure 3: Recall of article screening strategies across pathogens. Two ablation strategies (direct full-text and human-conditioned screening) with AgentSLR ( GPT-OSS-120B ) offer better recall (or ‘fetch rate’) than standard AI-based two-stage screening, with confidence intervals overlapping across most pathogens. Full article screening metrics results are reported in Appendix E.3 .
Data Type
Flagging Precision
Extraction Accuracy
Competence (1–7)
Parameters
0.66± 0.06
0.77± 0.05
4.23± 0.20
Models
0.40± 0.07
0.83± 0.05
2.80± 0.32
Outbreaks
0.61± 0.09
0.80± 0.07
3.90± 0.45
Table 3: Expert-rated holistic evaluation of data extraction quality. We report expert-rated flagging precision, field-level extraction accuracy, and perceived competence (AgentSLR with gpt-oss-120b ) for parameter, model, and outbreak extractions, aggregated across six epidemiologists. Values show mean ± standard error. Competence is rated on a 1–7 Likert scale (1: Incompetent, 7: Autonomous).
Figure 4: Cost vs. average performance per AgentSLR pathogen run. Each point shows a model’s mean macro F1 across all pipeline stages against estimated total cost per run (USD, log10 scale), with error bars denoting one standard deviation across stages. Full cost and token breakdowns are provided in Appendix D.2 .
Figure 5: SLR completion time (Human vs. AgentSLR). Using gpt-oss-120b across the full SLR creation pipeline takes 20 hours versus 385 hours for manual review, a 19.3× wall-clock speed-up and 58× reduction in calendar days. Full runtime statistics are reported in Appendix D .
Appendix figures & tables40 assets
Supplementary material from the paper’s appendix.
Appendix
otto-SR [ 20 ]
A4SLR [ 22 ]
MetaBeeAI [ 21 ]
AgentSLR (ours)
Domain
Clinical evidence
Health economics
Biology
Epidemiology
Methodological focus
Cochrane-style intervention reviews
HEOR and HTA use cases
Bee ecotoxicology
WHO-priority pathogen epidemiology
Extraction method
LLM extraction with post-hoc human correction
Prompt-based with domain templates
Multi-pass chunk retrieval
Tool-calling with JSON schema constraints
Open code
×
×
✓
✓
Open weight
×
×
×
✓
Human reference
✓
✓
✓
✓
Appendix
Table 4: Comparison against related work. LLM-based SLR pipelines differ substantially in domain, extraction methodology, and evaluation rigour, making direct quantitative comparison methodologically ill-posed. ✓ and × denote presence and absence of each property. Open code : source code is publicly released. Open weight : compatible with open-weight models. Human reference : evaluation uses expert-curated annotations as reference labels (proxy). Independent evaluation : performance is not assessed using LLM-as-a-judge. Stage evaluation : evaluation is decomposed into stage-specific metrics enabling failure attribution. AgentSLR workflow is the only framework satisfying all five properties.
Study
Evaluated capability
Reference signal
Relation to AgentSLR
TrialReview Bench [ 23 ]
Clinical SR search, screening, extraction and synthesis
Included studies and study characteristics from 100 cancer treatment reviews
Broad clinical workflow evaluation, focused on intervention evidence rather than epidemiological parameters, transmission models and outbreaks.
LLM4 SCREENLIT [ 24 ]
Screening evaluation methodology
Reanalysis of confusion matrix based screening studies
Directly motivates recall focused and cost sensitive screening metrics, but does not provide a biomedical extraction dataset.
Evidence Bench [ 25 ]
Evidence sentence retrieval within biomedical papers
Expert written evidence summaries and sentence annotations
Strong evidence grounding benchmark, but it tests locating supporting sentences rather than study selection or schema grounded records.
Med Evidence [ 26 ]
Matching Cochrane review conclusions from source studies
284 human curated conclusion questions from 100 Cochrane reviews
Tests cross study conclusion reasoning, while abstracting away article screening and structured extraction.
DeepScholar Bench [ 42 ]
Generative related work synthesis from live scholarly search
Human written related work sections and automated synthesis, retrieval and citation metrics
Evaluates long form academic synthesis, but not clinical or epidemiological review labels.
Mushtaq et al. [ 43 ]
Judging completed SLR manuscripts
PRISMA aligned scores on five SLRs
Evaluates manuscript quality after a review exists, not evidence retrieval, screening or extraction.
Appendix
Table 5: Evaluation studies adjacent to AgentSLR. We compare each study by the capability evaluated, the reference signal used, and its relation to epidemiological SLR evidence handling.
Stage
AgentSLR (Open-Access)
PERG Population
Δ F1
Abstract Screening
0.877 [0.853, 0.899]
0.906 [0.878, 0.930]
− 0.028 [ − 0.062, 0.010]
Full-text Screening
0.860 [0.830, 0.886]
0.918 [0.890, 0.942]
− 0.058 [ − 0.093, − 0.017]
Parameter Extraction
0.728 [0.654, 0.799]
0.697 [0.607, 0.781]
+ 0.031 [ − 0.079, 0.142]
Appendix
Table 6: Representativeness of the open-access evaluation subset across pipeline stages. Macro F1 with 95% bootstrap confidence intervals for the AgentSLR open-access evaluation sample and a matched sample of 1,004 closed-access articles from the broader PERG population corpus, retrieved via institutional access across Ebola, Lassa, SARS-CoV-1, and Zika. Both groups are processed using gpt-oss-120b under identical conditions. Δ F1 denotes the signed difference (open-access minus population). Differences are small across stages (range: −5.8 to +3.1 pp). The full-text screening interval excludes zero, indicating lower performance for the open-access subset at this stage. The abstract screening and parameter extraction intervals include zero. Overall, these results suggest broad comparability between the open-access subset and the broader PERG population corpus, with a modest full-text screening gap.
Table 7: Optimal bipartite matching example: 2 PERG reference models, 3 LLM-extracted models. (a) Input field values (two fields shown for illustration). (b) Pairwise similarity matrix S . (c) Optimal matching; LLM M 2 is unmatched (FP). (d) Per-cell similarity calculations for entries of S .
Pathogen
PERG Total
PERG Invalid
Invalid (%)
AI-extracted Total
Lassa
52
2
3.85
19
Ebola
294
46
15.7
239
SARS
112
8
7.14
85
Zika
229
53
23.1
132
Appendix
Table 8: Validation statistics for PERG reference data and AI-extracted transmission model annotations across four pathogens. PERG entries contained invalid field values due to manual data entry inconsistencies, while AI-extracted values showed no invalid entries due to structured schema enforcement during extraction. The AI-extracted numbers below are with gpt-oss-120b , but we find across all LLMs the number of invalid entries are 0.
Pathogen
PERG Total
PERG Invalid
Invalid (%)
AI-Extracted Total
Lassa
30
0
0.00
62
Zika
159
15
9.43
240
Appendix
Table 9: Validation statistics for PERG reference data and AI-extracted outbreak annotations across two pathogens. PERG entries contained invalid field values due to manual data entry inconsistencies, while AI-extracted values showed 0% invalid entries due to structured schema enforcement during extraction.
Pathogen
Title & Abstract Screening
Full-text Screening
Data Extraction
Ebola
11,605
1,674
522
Lassa
2,131
512
193
SARS
12,280
878
289
Zika
10,510
1,343
574
Average
9,132
1,102
395
Appendix
Table 10: Estimated number of articles reviewed by human annotators at PERG across successive stages of each systematic literature review (SLR). Counts for Title and Abstract Screening correspond to records remaining after deduplication and exclusion of entries with missing or empty abstract metadata. Full-text Screening includes articles flagged as potentially relevant during abstract screening and advanced for full-text review. Data Extraction represents articles deemed suitable for extracting structured, task-relevant evidence. All values are estimates intended to reflect annotator workload at each phase rather than finalised inclusion totals.
Stage
Articles (Avg.)
AgentSLR (s/article)
PERG (s/article)
AgentSLR (Hours)
PERG (Hours)
Article Retrieval
9,132
0.63
0
1.6
0.00
Title & Abstract Screening
9,132
0.63
45
1.6
114.2
PDF-to-Markdown Conversion
9,132
1.1
0
2.8
0.00
Full-text Screening
1,102
2.0
240
0.62
73.5
Data Extraction
395
122.1
1,800
13.4
197.5
Total
–
–
–
20.0
385.1
Appendix
Table 11: Comparison of average human time investment (PERG) versus automated processing time (AgentSLR workflow) across systematic literature review stages. The table reports average articles processed across the four pathogens (Ebola, Lassa, SARS, Zika), average per-article time (in seconds), and total processing time (in hours), highlighting efficiency gains from automation. AgentSLR workflow timings are computed using gpt-oss-120b as the underlying model. PDF-to-Markdown conversion is applied to all 9,132 retrieved articles to preserve the option of direct full-text screening across the complete corpus.
Figure 6: Articles processed, scaled token and costs by pipeline stage across models. We report the average number of articles reaching each stage, and the corresponding total input tokens, output tokens, and USD cost per stage averaged across models. Token totals are computed by multiplying per article token usage (Table 12 ) by the average article counts per stage (Table 10 ). Parameter extraction dominates overall compute, with substantially higher input and output token totals than other stages. Title and abstract screening processes the largest volume of articles but contributes comparatively less to total cost.
Model
Title & Abstract Screening
Article Screening (AI Cond.)
Parameter Extraction
Model Extraction
Outbreak Extraction
Overall
GPT-OSS-120B (High)
Input Tok.
2.3K
16.9K
510.2K
35.9K
49.9K
615.3K
Output Tok.
1.2K
1.1K
19.8K
1.9K
2.6K
26.7K
Cost (USD)
<0.01
<0.01
0.02
<0.01
<0.01
0.02
GPT-5.2 (High)
Input Tok.
2.2K
13.1K
961.1K
31.5K
32.4K
1040.4K
Output Tok.
0.6K
1.4K
91.1K
2.1K
3.5K
98.7K
Cost (USD)
0.01
0.04
2.95
0.08
0.10
3.20
Appendix
Table 12: Per article token usage and estimated cost by stage and model. We report mean input tokens, output tokens, and USD cost for processing a single article at each stage. Green marks the minimum and Red marks the maximum within each row and subcolumn.
Pathogen
gpt-oss-120b
GPT-5.2
DeepSeek-V3.2
Kimi-K2.5
GLM-4.7
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Marburg
0.80
0.64
0.69
0.97
0.58
0.62
0.97
0.55
0.58
0.79
0.65
0.69
0.88
0.61
0.66
Ebola
0.74
0.75
0.75
0.76
0.64
0.68
0.80
0.61
0.64
0.79
0.79
0.79
0.88
0.72
0.77
Lassa
0.82
0.72
0.75
0.78
0.63
0.66
0.84
0.60
0.63
0.84
0.77
0.80
0.88
0.68
0.73
SARS
0.78
0.76
0.77
0.77
0.62
0.65
0.82
0.62
0.66
0.80
0.78
0.79
0.89
0.73
0.78
Zika
0.73
0.77
0.75
0.70
0.62
0.64
0.73
0.63
0.66
0.78
0.79
0.79
0.76
0.69
0.72
Appendix
Table 13: Title and abstract screening metrics across LLMs. Green and Red denote the best- and worst-performing pathogens for each model (in terms of F1 score). Bold indicates the best-performing model for each pathogen, and Underline indicates the second-best. P=precision ; R=recall ; F1=F1-Score .
Pathogen
gpt-oss-120b
GPT-5.2
DeepSeek-V3.2
Kimi-K2.5
GLM-4.7
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Marburg
0.75
0.76
0.75
0.76
0.59
0.59
0.37
0.49
0.42
0.66
0.66
0.66
0.86
0.72
0.76
Ebola
0.73
0.84
0.77
0.61
0.60
0.60
0.68
0.59
0.55
0.72
0.74
0.71
0.75
0.75
0.75
Lassa
0.79
0.78
0.78
0.66
0.63
0.63
0.63
0.54
0.47
0.74
0.75
0.74
0.77
0.73
0.73
SARS
0.71
0.85
0.76
0.60
0.58
0.58
0.73
0.61
0.59
0.67
0.69
0.66
0.73
0.72
0.72
Zika
0.66
0.79
0.69
0.50
0.50
0.50
0.61
0.55
0.52
0.63
0.64
0.61
0.60
0.59
0.59
Appendix
Table 14: Full-text screening metrics across LLMs. Green and Red denote the best- and worst-performing pathogens for each model (in terms of F1 score). Bold indicates the best-performing model for each pathogen, and Underline indicates the second-best. P=precision ; R=recall ; F1=F1-Score .
Pathogen
Type
gpt-oss-120b
GPT-5.2
DeepSeek-V3.2
Kimi-K2.5
GLM-4.7
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Ebola
Flagging
0.60
0.92
0.72
0.58
0.93
0.71
0.49
0.91
0.64
0.67
0.90
0.77
0.72
0.82
0.77
Counts
0.79
0.47
0.59
0.46
0.80
0.58
0.57
0.59
0.58
0.52
0.73
0.61
0.59
0.65
0.62
Extraction
0.48
0.54
0.50
0.58
0.57
0.57
0.54
0.47
0.49
0.55
0.57
0.55
0.51
0.56
0.52
Average
0.62
0.64
0.60
0.54
0.77
0.62
0.54
0.65
0.57
0.58
0.73
0.64
0.61
0.68
0.64
Lassa
Flagging
0.56
0.98
0.71
0.58
1.00
0.73
0.54
0.91
0.68
0.70
0.94
0.81
0.77
0.87
0.82
Appendix
Table 15: Parameter extraction metrics across LLMs. Average denotes means across sub-tasks; Overall denotes means across pathogens. Green and Red denote the best- and worst-performing pathogens for each model (in terms of F1 score). Bold indicates the best-performing model for each pathogen, and Underline indicates the second-best. P=precision ; R=recall ; F1=F1-Score .
Pathogen
Type
gpt-oss-120b
GPT-5.2
DeepSeek-V3.2
Kimi-K2.5
GLM-4.7
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Ebola
Flagging
0.92
0.92
0.92
0.89
0.90
0.89
0.87
0.86
0.86
0.93
0.93
0.93
0.95
0.94
0.95
Counts
0.50
1.00
0.67
0.56
1.00
0.71
0.81
0.99
0.89
0.63
0.99
0.77
0.88
0.99
0.93
Extraction
0.59
0.72
0.64
0.62
0.74
0.66
0.57
0.65
0.61
0.60
0.71
0.64
0.62
0.71
0.66
Average
0.67
0.88
0.74
0.69
0.88
0.76
0.75
0.83
0.79
0.72
0.88
0.78
0.81
0.88
0.84
Lassa
Flagging
0.95
0.99
0.97
0.95
0.99
0.97
0.96
0.85
0.89
0.95
0.99
0.97
1.00
1.00
1.00
Appendix
Table 16: Model extraction metrics across LLMs. Green and Red denote the best- and worst-performing pathogens for each model (in terms of F1 score). Average denotes means across sub-tasks; Overall denotes means across pathogens. Bold indicates the best-performing model for each pathogen, and Underline indicates the second-best. P=precision ; R=recall ; F1=F1-Score .
Pathogen
Type
gpt-oss-120b
GPT-5.2
DeepSeek-V3.2
Kimi-K2.5
GLM-4.7
P
R
F1
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Lassa
Flagging
0.69
0.82
0.70
0.72
0.84
0.74
0.61
0.65
0.62
0.67
0.77
0.69
0.65
0.68
0.66
Counts
0.83
1.00
0.91
0.62
1.00
0.77
0.83
1.00
0.91
0.90
1.00
0.95
1.00
0.86
0.92
Extraction
0.85
0.73
0.77
0.84
0.79
0.81
0.75
0.78
0.76
0.84
0.83
0.83
0.83
0.76
0.78
Average
0.79
0.85
0.80
0.73
0.88
0.78
0.73
0.81
0.77
0.80
0.87
0.82
0.83
0.76
0.79
Zika
Flagging
0.58
0.71
0.53
0.59
0.75
0.57
0.65
0.82
0.68
0.61
0.80
0.59
0.67
0.87
0.70
Appendix
Table 17: Outbreak extraction metrics across LLMs. Average denotes means across sub-tasks; Overall denotes means across pathogens. Green and Red denote the best- and worst-performing pathogens for each model (in terms of F1 score). Bold indicates the best-performing model for each pathogen, and Underline indicates the second-best. P=precision ; R=recall ; F1=F1-Score .
Metric
Lassa
Ebola
SARS
Zika
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Flagging
0.56
0.98
0.71
0.60
0.92
0.72
0.50
0.81
0.62
0.40
0.96
0.57
Count
1.00
0.35
0.51
0.79
0.47
0.59
0.80
0.61
0.69
0.72
0.47
0.57
Appendix
Table 18: Flagging and Count classification metrics for parameter extraction with AgentSLR harness ( gpt-oss-120b ). P=precision ; R=recall ; F1=F1-Score .
Group
Field
Lassa
Ebola
SARS
Zika
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Value
value
0.22
0.22
0.22
0.20
0.20
0.20
0.23
0.23
0.23
0.14
0.14
0.14
unit
0.50
0.43
0.46
0.62
0.35
0.44
0.69
0.61
0.65
0.65
0.43
0.52
method
1.00
0.89
0.94
0.48
0.78
0.59
0.76
0.83
0.79
0.86
0.80
0.83
Average
0.57
0.51
0.54
0.44
0.44
0.41
0.56
0.56
0.56
0.55
0.46
0.50
Uncertainty
value type
0.38
0.43
0.40
0.30
0.33
0.32
0.35
0.57
0.43
0.12
0.22
0.16
Appendix
Table 19: Field-level precision, recall, and F1 for Extraction on parameters with AgentSLR harness ( gpt-oss-120b ). Group corresponds to the sub-stage of parameter extraction where the field is collected. The final row shows averages across all fields. P=precision ; R=recall ; F1=F1-Score .
Lassa
Ebola
SARS
Zika
P
R
F1
P
R
F1
P
R
F1
P
R
F1
Flagging
Article Flagging
0.95
0.99
0.97
0.92
0.92
0.92
0.86
0.86
0.86
0.87
0.89
0.88
Counts
Model Count
0.60
1.00
0.75
0.50
1.00
0.67
0.49
0.97
0.65
0.48
0.98
0.65
Extraction
Appendix
Table 20: Precision, recall, and F1 metrics for transmission model screening and extraction across four pathogens with AgentSLR harness ( gpt-oss-120b ). Screening includes article flagging and model count accuracy. Extraction evaluates field-level accuracy for matched model pairs, covering core structural characteristics (model type, stochastic vs deterministic, theoretical vs data-fitted, code availability) and more complex multi-value fields (transmission routes, assumptions, interventions). Strong performance is observed for core model characteristics, while extraction of assumptions, interventions, and transmission routes remains more challenging. P=precision ; R=recall ; F1=F1-Score .
Metric
Field
Lassa
Zika
P
R
F1
P
R
F1
Flagging
Article Flagging
0.69
0.82
0.75
0.58
0.71
0.64
Counts
Outbreak Counts
0.83
1.00
0.91
0.49
0.45
0.47
Extraction
Temporal Features
0.83
0.74
0.78
0.85
0.82
0.83
Geographic and Spatial Features
0.75
0.78
0.76
0.75
0.75
0.75
Case Burden
0.82
0.75
0.79
0.93
0.93
0.93
Appendix
Table 21: Outbreak screening and extraction with AgentSLR harness ( gpt-oss-120b ) across feature categories. Screening measured article flagging (identifying papers containing outbreaks) and outbreak count accuracy (extracting the correct number of outbreaks per paper). Extraction evaluated field-level accuracy for matched outbreak pairs across four epidemiological categories: temporal features (start/end dates), geographic features (country, specific location), case burden (confirmed cases, deaths), and epidemiological context (detection mode, pre-outbreak status, ongoing status, asymptomatic transmission). Overall metrics represent the average across all extraction fields. P=precision ; R=recall ; F1=F1-Score .
Lassa
Zika
P
R
F1
P
R
F1
Temporal Features
Start Year
0.89
0.80
0.84
0.90
0.82
0.86
Start Month
0.78
0.78
0.78
0.80
0.80
0.80
Start Day
0.86
0.67
0.75
0.95
0.95
0.95
End Month
0.78
0.78
0.78
0.65
0.65
0.65
Appendix
Table 22: Outbreak feature extraction by category for AgentSLR harness ( gpt-oss-120b ). Each row shows field-level performance within the four major epidemiological categories. Temporal and case burden features showed consistently high performance, while location-specific fields and epidemiological context features showed greater variability. P=precision ; R=recall ; F1=F1-Score .
Pathogen
P
R
F1
Marburg
0.80
0.64
0.69
Ebola
0.74
0.75
0.75
Lassa
0.82
0.72
0.75
SARS
0.78
0.76
0.77
Zika
0.73
0.77
0.75
MERS
0.83
0.74
0.78
Appendix
Table 23: Precision, recall, and F1 for title-and-abstract screening across seven pathogens with AgentSLR harness ( gpt-oss-120b ). Metrics summarise how well the abstract-stage classifier retained studies judged relevant under PERG screening criteria, reported for each pathogen and overall. P=precision ; R=recall ; F1=F1-Score .
Pathogen
AI Screen (Abstract) → AI Screen (Full-text)
Human Screen (Abstract) → AI Screen (Full-text)
AI Screen (Direct Full-text)
P
R
F1
P
R
F1
P
R
F1
Marburg
0.75
0.76
0.75
0.77
0.83
0.80
0.64
0.82
0.69
Ebola
0.73
0.84
0.77
0.86
0.97
0.91
0.67
0.93
0.72
Lassa
0.79
0.78
0.78
0.83
0.94
0.88
0.71
0.91
0.77
SARS
0.71
0.85
0.76
0.80
0.95
0.86
0.64
0.91
0.68
Zika
0.66
0.79
0.69
0.81
0.91
0.85
0.64
0.85
0.67
Appendix
Table 24: Full-text screening performance on AgentSLR harness ( gpt-oss-120b ) under three operational strategies for identifying relevant articles. Metrics compare a two-stage AI pipeline (AI abstract → AI full-text), a mixed workflow (human abstract → AI full-text), and direct AI full-text screening, reported for each pathogen and overall. R=recall ; F1=F1-Score .
Item
Score
Parameters
Overall — Flagging precision
0.66
Overall — Extraction accuracy
0.77
Precision by class
Attack rate
0.25
Growth rate
1.00
Appendix
Table 25: Extended expert validation results. Results are reported as expert-rated flagging precision and expert-rated extraction accuracy. Within each section, rows are ordered from overall scores to subgroup scores and then field-level scores.
Pathogen
PATHOGEN_IDENTIFIER
ADDITIONAL_TERMS
EXCLUSION_CRITERIA
Marburg virus
Marburg virus
—
—
Ebola virus
Ebola
—
—
Lassa virus
Lassa
—
—
SARS-CoV-1
SARS OR SARS-CoV-1 OR “Severe acute respiratory syndrome"
—
NOT (COVID-19 OR SARS-CoV-2)
Zika virus
zika
OR (“extrinsic incubation period" OR “EIP" OR “vector competence" OR “vectorial capacity") †
—
Nipah virus
Nipah
—
—
Appendix
Table 26: Pathogen-specific modifications to the standardised search query . All databases share consistent pathogen identifiers and exclusion criteria; PubMed/Europe PMC use wildcard forms while OpenAlex uses expanded variants.
Table 33
Priority
Source & Endpoint
Rate Limit
Cached
1
OpenAlex Direct PDF URL
30 req/s
No
Metadata field openalex_pdf_url
2
Europe PMC Fulltext API
20 req/s
Yes
ebi.ac.uk/europepmc/webservices/rest/search
3
Unpaywall API
50 req/s
Yes
api.unpaywall.org/v2/{DOI}?email={EMAIL}
Appendix
Table 29: PDF retrieval sources in cascading priority order. Sources are queried sequentially until success or exhaustion. Identifier cross-referencing via NCBI PMC ID Converter API precedes all download attempts (10 req/s, cached).
Parameter type
Parameter class
Description
Attack rate
Attack rate
Proportion of a population that becomes infected during an outbreak.
Secondary attack rate
Attack rate
Proportion of contacts of a primary case who become infected.
Doubling time
Doubling time
Time required for the number of infections to double.
Growth rate
Growth rate
Exponential rate at which new infections increase over time.
Evolutionary rate
Mutations
Rate of genetic change in a population over time, typically substitutions per site per year.
Mutation rate
Mutations
Frequency at which new genetic mutations arise per site per replication cycle.
Appendix
Table 30: Valid parameters for extraction, according to PERG’s process.
Parameter Class
Screening Details
Attack rate
The attack rate is the proportion of an at-risk population contracting the disease during a specified time interval. It is often reported as a percentage or rate, e.g. 52 people per 10,000 people.
Growth rate
The epidemic growth rate is a key metric that reflects how quickly the number of infections is changing day by day in a population. It is a time-dependent measure, usually expressed as a percentage or a rate per unit of time (e.g. per day), and is crucial for monitoring the speed and trajectory of an outbreak.
Human delay
These parameters all refer to time intervals in the natural history of infection of the host.
Mutation rate
Mutation rates, like substitution rate or evolutionary rate, describe the speed at which genetic changes accumulate in a population.
Relative contribution
This parameter is intended for pathogens (e.g. MERS) where there is both human to human (h2h) and animal to human (a2h) transmission, and aims to capture the relative magnitude of these two routes of infection in humans. We expect these to be proportions or percentages. E.g. a study might estimate 60% of infections in humans to be from h2h infection.
Reproduction number
We are extracting either the basic reproduction number R0 or the effective reproduction number Re.
Appendix
Table 31: Screening details for each parameter class. This is inputted into the “Parameter Class Screening Details” section of the Parameter Screening Prompt below.
Parameter class
Variable
Type
Allowed values
Description
Attack rate
value
Float
–
The value of the attack rate.
unit
Enum
percentage; rate
The unit of the provided attack rate.
type
Enum
primary; secondary
Whether primary or secondary attack rate.
rate denominator
Integer; Null
–
The denominator of the value, if the parameter is provided as a rate.
Doubling time
value
Float
–
The value of the doubling time, in days.
Growth rate
value
Float
–
The value of the growth rate.
Appendix
Table 32: Schemas used for value extraction tool calls for each parameter class. Here “–” means that any values of the correct type are allowed.
Variable
Type
Allowed values
Description
population sex
Enum
female; male; both; unspecified
The sex composition of your study population. If you have 99 men and 1 woman you would still put both in this option.
population sample type
Enum
community based; hospital based; household based; housing estate based; population based; school based; travel based; trade or business based; contact based; mixed settings; other; unspecified
How was the study conducted?
population group
Enum
healthcare workers; farmers; outdoor workers; animal workers; butchers; abattoir workers; pregnant women; children; sex workers; people who inject drugs; household contacts of survivors; persons under investigation; general population; persons with symptoms; mixed settings; unspecified; other
Demographic i.e. who was sampled?
population sample size
Integer; Null
–
Number of participants/samples tested etc.
population age min
Integer; Null
–
These must be number fields. If your sample is people over 18 you would put age min = 18 and leave age max blank.
population age max
Integer; Null
–
These must be number fields. If your sample is people over 18 you would put age min = 18 and leave age max blank.
Appendix
Table 33: The schema for the population context extraction tool call.
Category
Description
Structural Properties
Model type (compartmental, agent-based, branching process) and compartmental architecture (SIS, SIR, SEIR, etc.). Whether the model is stochastic or deterministic.
Epidemiological Features
Primary transmission routes (airborne, direct contact, vector-borne, sexual). Spatial heterogeneity and spillover dynamics from animal reservoirs.
Behavioural Assumptions
Mixing patterns (homogeneous or heterogeneous), age-dependent susceptibility, cross-immunity between pathogens, and temporal variation in transmission rates.
Theoretical vs. Fitted
Whether the model was fitted to actual data or uses parameters from literature or arbitrary values.
Intervention Categories
Control measures evaluated including vaccination, quarantine, vector control, treatment, contact tracing, behaviour changes, and various vector management strategies.
Reproducibility Indicators
Code availability, programming language used, data sharing status, and presence of documentation (README).
Appendix
Table 34: Model characteristic categories targeted by the extraction pipeline.
Primary modeling framework used for transmission dynamics.
compartmental_type
Enum
SIS; SIR; SEIR; SEIR-SEI; SAIR-SEI; Not compartmental; Other compartmental
Specific compartmental model architecture if applicable. Use “Not compartmental” for non-compartmental models.
stoch_deter
Enum; Null
Stochastic; Deterministic
Whether the model is stochastic or deterministic. Only extract if explicitly stated. Null if not specified.
transmission_route
List[Enum]
Airborne or close contact; Human to human (direct contact); Human to human (direct non-sexual contact); Vector/Animal to human; Sexual; Unspecified
Primary pathway(s) through which pathogen transmission occurs. Multiple routes can be selected.
uncertainty_was_considered
Boolean; Null
True; False
Whether uncertainty was considered through stochasticity, multiple models, or parameter variation (e.g. sensitivity analyses, Bayesian analysis). Null if not specified.
spatial_model
Boolean; Null
True; False
Whether the model explicitly incorporated spatial or geographic heterogeneity. Null if not specified.
Appendix
Table 35: Model extraction schema with field specifications, data types, allowed values, and descriptions.
Outbreak Characteristic
Description
Outbreak start day
Day of outbreak onset (1–31). Extracted as stated in paper.
Outbreak start month
Month of outbreak onset. Extracted as stated in paper.
Outbreak start year
Year of outbreak onset. Extracted as stated in paper.
Outbreak end day
Day of outbreak conclusion (1–31). Extracted as stated in paper.
Outbreak end month
Month of outbreak conclusion. Extracted as stated in paper.
Outbreak end year
Year of outbreak conclusion. Extracted as stated in paper.
Appendix
Table 36: Outbreak field descriptions and meanings.
Field Name
Type
Allowed Values
Description
outbreak_start_day
Integer; Null
1-31
Day of outbreak onset. Null if not provided.
outbreak_start_month
String (Enum); Null
Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec
Month of outbreak onset. Null if not provided.
outbreak_start_year
Integer; Null
Integer year
Year of outbreak onset. Null if not provided.
outbreak_end_day
Integer; Null
1-31
Day of outbreak conclusion. Null if not provided.
outbreak_end_month
String (Enum); Null
Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec
Month of outbreak conclusion. Null if not provided.
outbreak_end_year
Integer; Null
Integer year
Year of outbreak conclusion. Null if not provided.
Appendix
Table 37: Outbreak extraction schema with field specifications, data types, allowed values, and descriptions.
Artefact
Path (relative to repo root)
Purpose
Outbreak report (Markdown)
writeup/ p /outbreaks_writeup.md
Human-readable draft with embedded figures and tables.
Outbreak report (PDF)
writeup/ p /outbreaks_writeup.pdf
Portable rendering for sharing and archiving.
Figures directory
writeup/ p /figures/
Generated plots referenced by Markdown (e.g., temporal distribution, geographic spread, case counts).
Summary tables (embedded)
(in outbreaks_writeup.md )
Count and proportion tables computed from Dp\textscO .
Content manifest
writeup/ p /content_manifest.json
Machine-readable inventory of figures, tables, and dataset statistics.
Appendix
Table 38: Artefact inventory for outbreak report generation (per pathogen p ). All artefacts are derived from the extracted outbreak dataset Dp\textscO .
Figure 7: Ebola living reviews generated by AgentSLR. Both reviews follow a structured format: evidence-based descriptions citing supporting figures and tables, followed by interpretation blocks explicitly labelled as AI-generated synthesis.
Figure 8: Preliminary living reviews for pathogens without completed PERG validation. RVF and CCHF represent pathogens for which PERG has not yet commenced systematic screening.
Type
Transmission-Modelling Review
Outbreak Surveillance Review
Figures
Model architecture distribution (compartmental, branching process, agent-based); Stochasticity classification; Transmission route breakdown; Code availability
Geographic burden (choropleth maps for cases and deaths); Outbreak timeline by country; Detection mode distribution
Tables
Model type counts and proportions; Deterministic vs stochastic breakdown; Transmission routes with sample sizes; Modelling assumptions; Intervention categories; Spatial scale indicators; Code availability and language
Outbreak source categories; Detection methodology breakdown; Ongoing outbreaks at extraction; Case burden stratified by confirmation status; Sex disaggregation where reported
Appendix
Table 39: Key artefacts in AgentSLR workflow generated living reviews. Each pathogen generates two review types with consistent visualisation and evidence table structures. The text-based LLM uses manifests, with summary statistics of the figures to write its interpretation.
Figure 9: The AgentSLR extraction and validation tool interface. The dual-pane view presents the source document ( left ) alongside structured extraction fields ( right ). AI-predicted entries are pre-filled and accompanied by highlighted, AI-tagged evidence excerpts from the manuscript. Reviewers can accept, revise, or reject individual fields, with validation status indicators (e.g. “AI Match” or “Revised”) reflecting whether human intervention was required.
Figure 10: Review management and submission tracking interfaces. (a) The study review list displays papers awaiting expert validation, along with associated model and outbreak counts and direct links to extracted excerpts. (b) The verification view presents finalised AI-assisted extractions, enabling reviewers to explicitly reject predictions or verify and save corrected entries, which are then recorded for downstream quality control and system evaluation.
Systematic literature reviews follow the PRISMA process of screening scholarly works and extracting evidence from included studies. Deep research tools based on large language models (LLMs) can search and summarize the literature but do not record which studies are excluded or where extracted values come from. We develop a conversational multi-agent system called ReviewPilot that automates the PRISMA workflow from paper screening to information extraction. Researchers can tailor criteria and schemas through dialogue, and a project memory keeps their configurations and revisions. Each screening decision comes with its reason and each extracted value with its source. On epidemiological expert reviews, ReviewPilot outperforms a single-prompt LLM and AgentSLR across three LLMs, with screening macro F1 up to 0.776 and extraction F1 up to 0.788. Case studies in biomedicine and human-computer interaction show close agreement with human annotators, suggesting that ReviewPilot can transfer beyond epidemiology. Code is available at https://github.com/ZhaoqianXue/ReviewPilot.
Zhaoqian Xue, Shaoze Zhou, Xuexin Li +10
University of Arizona · Florida International University · Vanderbilt University +7
Systematic reviews require sustained human judgment across thousands of records, yet existing evaluations of large language models (LLMs) typically examine review stages in isolation. We introduce SciLitBench, a multi-stage benchmark spanning title and abstract screening, full-text screening, and schema-guided data extraction, with 42,981 retrieved records, 1,012 full texts, and annotations for 888 included papers. Across 22 open-weight LLMs from six model families, explicit inclusion and exclusion criteria improve title and abstract screening F2 by 28.8%, while researcher-authored rationales improve full-text screening by 15%. Data extraction reveals a different reliability regime: performance declines from 0.97 accuracy for publication year to 0.37 Jaccard overlap for computational approach, while the strongest models recover only 30% of annotated evaluation evidence and 25% of limitations. SciLitBench identifies a practical boundary between high-recall screening and evidence-complete extraction and provides a reproducible resource for evaluating LLM-assisted evidence synthesis.
Miguel Zabaleta, Baihan Lin
Department of AI and Human Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA · Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, USA
Context: Study screening in systematic literature reviews is costly, inconsistency-prone, and risk-asymmetric, since false negatives can compromise validity. Despite rapid uptake of Large Language Models (LLMs), there is limited evidence on how such models behave during the study screening phase, particularly regarding the choice of specific LLMs and their comparison with classical models. Objective: To assess LLM performance and variability in screening, quantify the impact of input metadata (abstract, title, keywords), and compare LLMs with classical classifiers under a shared protocol. Methods: We analyzed 12 LLMs from 4 providers (OpenAI, Google Gemini, Anthropic, Llama) and 4 classical models (Logistic Regression, Support Vector Classification, Random Forest, and Naive Bayes) on 2 real Systematic Literature Reviews (SLRs), totaling 518 papers. The experimental design investigated 3 critical dimensions: (i) LLMs performance variability, (ii) the impact of input feature composition (abstract, title, and keywords) on LLM performance, and (iii) the real gain of using LLMs instead of more traditional classification models. Results: LLMs exhibited substantial heterogeneity and residual non-determinism even at temperature zero. Abstract availability was decisive: removing it consistently degraded performance, while adding title and/or keywords to the abstract yielded no robust gains. Compared to classical models, performance differences were not consistent enough to support generalizable LLM superiority. Discussion: LLM adoption should be justified by operational and governance constraints (reproducibility, cost, metadata availability), supported by pilot validation and explicit reporting of variability and input configuration.