cs.IRMar 20, 2026
SaveEvaluating AI-based Scientific Knowledge Synthesis with Epidemiological Systematic Reviews
Organizations: University of Oxford · Imperial College London · University of Nottingham · Snowflake AI Research ·
Abstract
Systematic literature reviews (SLRs) are a demanding and high-stakes form of scientific knowledge synthesis that remains underspecified as an evaluation setting for large language models (LLMs). We introduce AgentSLR, a large-scale evaluation harness comprising an SLR automation workflow and an expert annotated dataset covering 16,248 articles, designed to test LLM capabilities across the stages of SLRs in epidemiology. Reference annotations were derived from peer-reviewed studies on WHO priority pathogens and produced by domain experts. The harness evaluates each review stage as a separate unit with dedicated metrics enabling targeted failure analysis. We evaluated five frontier reasoning models and found that no single model dominated across all tasks, showing sub-task specialisation often hidden by aggregate benchmarks. Structured data extraction is a major bottleneck, with no model exceeding an average field-level F1 of 0.67. Estimated costs vary substantially, by up to 96 times across evaluated models. Documented failure modes suggest that the evaluated models are not yet reliable enough for unsupervised deployment in epidemiology, where findings can inform public policy.
Figures & tables
Figure 1: AgentSLR evaluation harness (LLM-assisted workflow). (a) Article Search and Retrieval queries bibliographic databases with domain-specific Boolean searches and obtains PDFs from open-access sources. (b) Title and Abstract Screening uses LLMs to filter articles using expert-designed inclusion and exclusion criteria. (c) PDF-to-Markdown Conversion uses OCR to convert PDFs to markdown files. (d) Full-text Screening applies the same approach as in Title and Abstract Screening but with stricter criteria. (e) Data Extraction employs multi-stage tool-calling with schema validation to extract structured epidemiological data (parameters, models, outbreaks). (f) Report Generation synthesises extracted data through artefact generation followed by LLM refinement.
| Pathogen | PERG * | AgentSLR Matched |
| Marburg virus | 2,593 | 801 (30.9%) |
| Ebola virus | 11,605 | 4,119 (35.5%) |
| Lassa fever | 2,131 | 667 (31.3%) |
| SARS-CoV-1 | 12,280 | 2,047 (16.7%) |
| Zika virus | 10,510 | 2,164 (20.6%) |
| MERS-CoV | 19,656 | 5,714 (29.1%) |
Table 1: Released dataset (benchmark) record counts. AgentSLR Matched denotes the downloaded article subset that matched PERG labelled records and was released on HuggingFace. The coloured symbols represent the progress of PERG’s SLRs per priority pathogen: published , conducting data extraction , and yet to begin screening , as of March 2026.
Figure 2: Stage-wise model performance on AgentSLR harness. Each model is run on individual pathogens with AgentSLR harness. Averages are computed over the pathogens evaluated at each stage, following reference data availability described in Section 4.1 . Error bars indicate one standard deviation across pathogens. No single model dominates across all stages: Kimi-K2.5 and gpt-oss-120b lead screening, while extraction leaders vary by data type. Full metrics are provided in Appendix E .
| Parameters | Models | Outbreaks | Average | |||||||||
| Model | ||||||||||||
| gpt-oss-120b | 0.66 0.07 | 0.59 0.08 | 0.54 0.03 | 0.91 0.05 | 0.68 0.05 | 0.67 0.03 | 0.61 0.13 | 0.69 0.31 | 0.79 0.02 | 0.75 0.15 | 0.65 0.13 | 0.64 0.10 |
| GPT-5.2 | 0.66 0.07 | 0.50 0.07 | 0.59 0.03 | 0.90 0.05 | 0.72 0.04 | 0.67 0.03 | 0.66 0.12 | 0.80 0.05 | 0.84 0.04 | 0.76 0.14 | 0.65 0.14 | 0.67 0.10 |
| DeepSeek-V3.2 | 0.60 0.07 | 0.56 0.04 | 0.50 0.03 | 0.87 0.04 | 0.92 0.08 | 0.65 0.09 | 0.65 0.04 | 0.78 0.19 | 0.75 0.02 | 0.72 0.14 | 0.75 0.19 | 0.61 0.11 |
| Kimi-K2.5 | 0.72 0.09 | 0.62 0.05 | 0.56 0.02 | 0.92 0.04 | 0.81 0.04 | 0.68 0.04 | 0.64 0.07 | 0.87 0.11 | 0.78 0.07 | 0.78 0.14 | 0.75 0.12 | 0.65 0.10 |
| GLM-4.7 | 0.72 0.09 | 0.61 0.05 | 0.54 0.02 | 0.93 0.07 | 0.93 0.06 | 0.68 0.05 | 0.68 0.03 | 0.72 0.29 | 0.77 0.02 | 0.80 0.13 | 0.76 0.18 | 0.64 0.10 |
Table 2: Structured evidence extraction performance. F 1 scores are averaged across pathogens for five frontier models and three extraction tasks: Flagging, Counts, and field-level Extraction. Standard deviations ( ) are reported across pathogens.
Figure 3: Recall of article screening strategies across pathogens. Two ablation strategies (direct full-text and human-conditioned screening) with AgentSLR ( GPT-OSS-120B ) offer better recall (or ‘fetch rate’) than standard AI-based two-stage screening, with confidence intervals overlapping across most pathogens. Full article screening metrics results are reported in Appendix E.3 .
| Data Type | Flagging Precision | Extraction Accuracy | Competence (1–7) |
| Parameters | 0.06 | 0.05 | 0.20 |
| Models | 0.07 | 0.05 | 0.32 |
| Outbreaks | 0.09 | 0.07 | 0.45 |
Table 3: Expert-rated holistic evaluation of data extraction quality. We report expert-rated flagging precision, field-level extraction accuracy, and perceived competence (AgentSLR with gpt-oss-120b ) for parameter, model, and outbreak extractions, aggregated across six epidemiologists. Values show mean standard error. Competence is rated on a 1–7 Likert scale (1: Incompetent, 7: Autonomous).
Figure 4: Cost vs. average performance per AgentSLR pathogen run. Each point shows a model’s mean macro across all pipeline stages against estimated total cost per run (USD, scale), with error bars denoting one standard deviation across stages. Full cost and token breakdowns are provided in Appendix D.2 .
Figure 5: SLR completion time (Human vs. AgentSLR). Using gpt-oss-120b across the full SLR creation pipeline takes hours versus hours for manual review, a wall-clock speed-up and reduction in calendar days. Full runtime statistics are reported in Appendix D .
Appendix figures & tables40 assets
Supplementary material from the paper’s appendix.
Appendix
| otto-SR [ 20 ] | A4SLR [ 22 ] | MetaBeeAI [ 21 ] | AgentSLR (ours) | |
| Domain | Clinical evidence | Health economics | Biology | Epidemiology |
| Methodological focus | Cochrane-style intervention reviews | HEOR and HTA use cases | Bee ecotoxicology | WHO-priority pathogen epidemiology |
| Extraction method | LLM extraction with post-hoc human correction | Prompt-based with domain templates | Multi-pass chunk retrieval | Tool-calling with JSON schema constraints |
| Open code | ||||
| Open weight | ||||
| Human reference |
Table 4: Comparison against related work. LLM-based SLR pipelines differ substantially in domain, extraction methodology, and evaluation rigour, making direct quantitative comparison methodologically ill-posed. and denote presence and absence of each property. Open code : source code is publicly released. Open weight : compatible with open-weight models. Human reference : evaluation uses expert-curated annotations as reference labels (proxy). Independent evaluation : performance is not assessed using LLM-as-a-judge. Stage evaluation : evaluation is decomposed into stage-specific metrics enabling failure attribution. AgentSLR workflow is the only framework satisfying all five properties.
| Study | Evaluated capability | Reference signal | Relation to AgentSLR |
| TrialReview Bench [ 23 ] | Clinical SR search, screening, extraction and synthesis | Included studies and study characteristics from 100 cancer treatment reviews | Broad clinical workflow evaluation, focused on intervention evidence rather than epidemiological parameters, transmission models and outbreaks. |
| LLM4 SCREENLIT [ 24 ] | Screening evaluation methodology | Reanalysis of confusion matrix based screening studies | Directly motivates recall focused and cost sensitive screening metrics, but does not provide a biomedical extraction dataset. |
| Evidence Bench [ 25 ] | Evidence sentence retrieval within biomedical papers | Expert written evidence summaries and sentence annotations | Strong evidence grounding benchmark, but it tests locating supporting sentences rather than study selection or schema grounded records. |
| Med Evidence [ 26 ] | Matching Cochrane review conclusions from source studies | 284 human curated conclusion questions from 100 Cochrane reviews | Tests cross study conclusion reasoning, while abstracting away article screening and structured extraction. |
| DeepScholar Bench [ 42 ] | Generative related work synthesis from live scholarly search | Human written related work sections and automated synthesis, retrieval and citation metrics | Evaluates long form academic synthesis, but not clinical or epidemiological review labels. |
| Mushtaq et al. [ 43 ] | Judging completed SLR manuscripts | PRISMA aligned scores on five SLRs | Evaluates manuscript quality after a review exists, not evidence retrieval, screening or extraction. |
Table 5: Evaluation studies adjacent to AgentSLR. We compare each study by the capability evaluated, the reference signal used, and its relation to epidemiological SLR evidence handling.
| Stage | AgentSLR (Open-Access) | PERG Population | F1 |
| Abstract Screening | 0.877 [0.853, 0.899] | 0.906 [0.878, 0.930] | 0.028 [ 0.062, 0.010] |
| Full-text Screening | 0.860 [0.830, 0.886] | 0.918 [0.890, 0.942] | 0.058 [ 0.093, 0.017] |
| Parameter Extraction | 0.728 [0.654, 0.799] | 0.697 [0.607, 0.781] | 0.031 [ 0.079, 0.142] |
Table 6: Representativeness of the open-access evaluation subset across pipeline stages. Macro F1 with 95% bootstrap confidence intervals for the AgentSLR open-access evaluation sample and a matched sample of 1,004 closed-access articles from the broader PERG population corpus, retrieved via institutional access across Ebola, Lassa, SARS-CoV-1, and Zika. Both groups are processed using gpt-oss-120b under identical conditions. F1 denotes the signed difference (open-access minus population). Differences are small across stages (range: to pp). The full-text screening interval excludes zero, indicating lower performance for the open-access subset at this stage. The abstract screening and parameter extraction intervals include zero. Overall, these results suggest broad comparability between the open-access subset and the broader PERG population corpus, with a modest full-text screening gap.
Table 7: Optimal bipartite matching example: 2 PERG reference models, 3 LLM-extracted models. (a) Input field values (two fields shown for illustration). (b) Pairwise similarity matrix . (c) Optimal matching; LLM M 2 is unmatched (FP). (d) Per-cell similarity calculations for entries of .
| Pathogen | PERG Total | PERG Invalid | Invalid (%) | AI-extracted Total |
| Lassa | 52 | 2 | 3.85 | 19 |
| Ebola | 294 | 46 | 15.7 | 239 |
| SARS | 112 | 8 | 7.14 | 85 |
| Zika | 229 | 53 | 23.1 | 132 |
Table 8: Validation statistics for PERG reference data and AI-extracted transmission model annotations across four pathogens. PERG entries contained invalid field values due to manual data entry inconsistencies, while AI-extracted values showed no invalid entries due to structured schema enforcement during extraction. The AI-extracted numbers below are with gpt-oss-120b , but we find across all LLMs the number of invalid entries are 0.
| Pathogen | PERG Total | PERG Invalid | Invalid (%) | AI-Extracted Total |
| Lassa | 30 | 0 | 0.00 | 62 |
| Zika | 159 | 15 | 9.43 | 240 |
Table 9: Validation statistics for PERG reference data and AI-extracted outbreak annotations across two pathogens. PERG entries contained invalid field values due to manual data entry inconsistencies, while AI-extracted values showed 0% invalid entries due to structured schema enforcement during extraction.
| Pathogen | Title & Abstract Screening | Full-text Screening | Data Extraction |
| Ebola | 11,605 | 1,674 | 522 |
| Lassa | 2,131 | 512 | 193 |
| SARS | 12,280 | 878 | 289 |
| Zika | 10,510 | 1,343 | 574 |
| Average | 9,132 | 1,102 | 395 |
Table 10: Estimated number of articles reviewed by human annotators at PERG across successive stages of each systematic literature review (SLR). Counts for Title and Abstract Screening correspond to records remaining after deduplication and exclusion of entries with missing or empty abstract metadata. Full-text Screening includes articles flagged as potentially relevant during abstract screening and advanced for full-text review. Data Extraction represents articles deemed suitable for extracting structured, task-relevant evidence. All values are estimates intended to reflect annotator workload at each phase rather than finalised inclusion totals.
| Stage | Articles (Avg.) | AgentSLR (s/article) | PERG (s/article) | (Hours) | (Hours) |
| Article Retrieval | 9,132 | 0.63 | 0 | 1.6 | 0.00 |
| Title & Abstract Screening | 9,132 | 0.63 | 45 | 1.6 | 114.2 |
| PDF-to-Markdown Conversion | 9,132 | 1.1 | 0 | 2.8 | 0.00 |
| Full-text Screening | 1,102 | 2.0 | 240 | 0.62 | 73.5 |
| Data Extraction | 395 | 122.1 | 1,800 | 13.4 | 197.5 |
| Total | – | – | – | 20.0 | 385.1 |
Table 11: Comparison of average human time investment (PERG) versus automated processing time (AgentSLR workflow) across systematic literature review stages. The table reports average articles processed across the four pathogens (Ebola, Lassa, SARS, Zika), average per-article time (in seconds), and total processing time (in hours), highlighting efficiency gains from automation. AgentSLR workflow timings are computed using gpt-oss-120b as the underlying model. PDF-to-Markdown conversion is applied to all 9,132 retrieved articles to preserve the option of direct full-text screening across the complete corpus.
Figure 6: Articles processed, scaled token and costs by pipeline stage across models. We report the average number of articles reaching each stage, and the corresponding total input tokens, output tokens, and USD cost per stage averaged across models. Token totals are computed by multiplying per article token usage (Table 12 ) by the average article counts per stage (Table 10 ). Parameter extraction dominates overall compute, with substantially higher input and output token totals than other stages. Title and abstract screening processes the largest volume of articles but contributes comparatively less to total cost.
| Model | Title & Abstract Screening | Article Screening (AI Cond.) | Parameter Extraction | Model Extraction | Outbreak Extraction | Overall | |
| GPT-OSS-120B (High) | Input Tok. | 2.3K | 16.9K | 510.2K | 35.9K | 49.9K | 615.3K |
| Output Tok. | 1.2K | 1.1K | 19.8K | 1.9K | 2.6K | 26.7K | |
| Cost (USD) | 0.02 | 0.02 | |||||
| GPT-5.2 (High) | Input Tok. | 2.2K | 13.1K | 961.1K | 31.5K | 32.4K | 1040.4K |
| Output Tok. | 0.6K | 1.4K | 91.1K | 2.1K | 3.5K | 98.7K | |
| Cost (USD) | 0.01 | 0.04 | 2.95 | 0.08 | 0.10 | 3.20 |
Table 12: Per article token usage and estimated cost by stage and model. We report mean input tokens, output tokens, and USD cost for processing a single article at each stage. Green marks the minimum and Red marks the maximum within each row and subcolumn.
| Pathogen | gpt-oss-120b | GPT-5.2 | DeepSeek-V3.2 | Kimi-K2.5 | GLM-4.7 | ||||||||||
| Marburg | 0.80 | 0.64 | 0.69 | 0.97 | 0.58 | 0.62 | 0.97 | 0.55 | 0.58 | 0.79 | 0.65 | 0.69 | 0.88 | 0.61 | 0.66 |
| Ebola | 0.74 | 0.75 | 0.75 | 0.76 | 0.64 | 0.68 | 0.80 | 0.61 | 0.64 | 0.79 | 0.79 | 0.79 | 0.88 | 0.72 | 0.77 |
| Lassa | 0.82 | 0.72 | 0.75 | 0.78 | 0.63 | 0.66 | 0.84 | 0.60 | 0.63 | 0.84 | 0.77 | 0.80 | 0.88 | 0.68 | 0.73 |
| SARS | 0.78 | 0.76 | 0.77 | 0.77 | 0.62 | 0.65 | 0.82 | 0.62 | 0.66 | 0.80 | 0.78 | 0.79 | 0.89 | 0.73 | 0.78 |
| Zika | 0.73 | 0.77 | 0.75 | 0.70 | 0.62 | 0.64 | 0.73 | 0.63 | 0.66 | 0.78 | 0.79 | 0.79 | 0.76 | 0.69 | 0.72 |
Table 13: Title and abstract screening metrics across LLMs. Green and Red denote the best- and worst-performing pathogens for each model (in terms of score). Bold indicates the best-performing model for each pathogen, and Underline indicates the second-best. ; ; .
| Pathogen | gpt-oss-120b | GPT-5.2 | DeepSeek-V3.2 | Kimi-K2.5 | GLM-4.7 | ||||||||||
| Marburg | 0.75 | 0.76 | 0.75 | 0.76 | 0.59 | 0.59 | 0.37 | 0.49 | 0.42 | 0.66 | 0.66 | 0.66 | 0.86 | 0.72 | 0.76 |
| Ebola | 0.73 | 0.84 | 0.77 | 0.61 | 0.60 | 0.60 | 0.68 | 0.59 | 0.55 | 0.72 | 0.74 | 0.71 | 0.75 | 0.75 | 0.75 |
| Lassa | 0.79 | 0.78 | 0.78 | 0.66 | 0.63 | 0.63 | 0.63 | 0.54 | 0.47 | 0.74 | 0.75 | 0.74 | 0.77 | 0.73 | 0.73 |
| SARS | 0.71 | 0.85 | 0.76 | 0.60 | 0.58 | 0.58 | 0.73 | 0.61 | 0.59 | 0.67 | 0.69 | 0.66 | 0.73 | 0.72 | 0.72 |
| Zika | 0.66 | 0.79 | 0.69 | 0.50 | 0.50 | 0.50 | 0.61 | 0.55 | 0.52 | 0.63 | 0.64 | 0.61 | 0.60 | 0.59 | 0.59 |
Table 14: Full-text screening metrics across LLMs. Green and Red denote the best- and worst-performing pathogens for each model (in terms of score). Bold indicates the best-performing model for each pathogen, and Underline indicates the second-best. ; ; .
| Pathogen | Type | gpt-oss-120b | GPT-5.2 | DeepSeek-V3.2 | Kimi-K2.5 | GLM-4.7 | ||||||||||
| Ebola | Flagging | 0.60 | 0.92 | 0.72 | 0.58 | 0.93 | 0.71 | 0.49 | 0.91 | 0.64 | 0.67 | 0.90 | 0.77 | 0.72 | 0.82 | 0.77 |
| Counts | 0.79 | 0.47 | 0.59 | 0.46 | 0.80 | 0.58 | 0.57 | 0.59 | 0.58 | 0.52 | 0.73 | 0.61 | 0.59 | 0.65 | 0.62 | |
| Extraction | 0.48 | 0.54 | 0.50 | 0.58 | 0.57 | 0.57 | 0.54 | 0.47 | 0.49 | 0.55 | 0.57 | 0.55 | 0.51 | 0.56 | 0.52 | |
| Average | 0.62 | 0.64 | 0.60 | 0.54 | 0.77 | 0.62 | 0.54 | 0.65 | 0.57 | 0.58 | 0.73 | 0.64 | 0.61 | 0.68 | 0.64 | |
| Lassa | Flagging | 0.56 | 0.98 | 0.71 | 0.58 | 1.00 | 0.73 | 0.54 | 0.91 | 0.68 | 0.70 | 0.94 | 0.81 | 0.77 | 0.87 | 0.82 |
Table 15: Parameter extraction metrics across LLMs. Average denotes means across sub-tasks; Overall denotes means across pathogens. Green and Red denote the best- and worst-performing pathogens for each model (in terms of score). Bold indicates the best-performing model for each pathogen, and Underline indicates the second-best. ; ; .
| Pathogen | Type | gpt-oss-120b | GPT-5.2 | DeepSeek-V3.2 | Kimi-K2.5 | GLM-4.7 | ||||||||||
| Ebola | Flagging | 0.92 | 0.92 | 0.92 | 0.89 | 0.90 | 0.89 | 0.87 | 0.86 | 0.86 | 0.93 | 0.93 | 0.93 | 0.95 | 0.94 | 0.95 |
| Counts | 0.50 | 1.00 | 0.67 | 0.56 | 1.00 | 0.71 | 0.81 | 0.99 | 0.89 | 0.63 | 0.99 | 0.77 | 0.88 | 0.99 | 0.93 | |
| Extraction | 0.59 | 0.72 | 0.64 | 0.62 | 0.74 | 0.66 | 0.57 | 0.65 | 0.61 | 0.60 | 0.71 | 0.64 | 0.62 | 0.71 | 0.66 | |
| Average | 0.67 | 0.88 | 0.74 | 0.69 | 0.88 | 0.76 | 0.75 | 0.83 | 0.79 | 0.72 | 0.88 | 0.78 | 0.81 | 0.88 | 0.84 | |
| Lassa | Flagging | 0.95 | 0.99 | 0.97 | 0.95 | 0.99 | 0.97 | 0.96 | 0.85 | 0.89 | 0.95 | 0.99 | 0.97 | 1.00 | 1.00 | 1.00 |
Table 16: Model extraction metrics across LLMs. Green and Red denote the best- and worst-performing pathogens for each model (in terms of score). Average denotes means across sub-tasks; Overall denotes means across pathogens. Bold indicates the best-performing model for each pathogen, and Underline indicates the second-best. ; ; .
| Pathogen | Type | gpt-oss-120b | GPT-5.2 | DeepSeek-V3.2 | Kimi-K2.5 | GLM-4.7 | ||||||||||
| Lassa | Flagging | 0.69 | 0.82 | 0.70 | 0.72 | 0.84 | 0.74 | 0.61 | 0.65 | 0.62 | 0.67 | 0.77 | 0.69 | 0.65 | 0.68 | 0.66 |
| Counts | 0.83 | 1.00 | 0.91 | 0.62 | 1.00 | 0.77 | 0.83 | 1.00 | 0.91 | 0.90 | 1.00 | 0.95 | 1.00 | 0.86 | 0.92 | |
| Extraction | 0.85 | 0.73 | 0.77 | 0.84 | 0.79 | 0.81 | 0.75 | 0.78 | 0.76 | 0.84 | 0.83 | 0.83 | 0.83 | 0.76 | 0.78 | |
| Average | 0.79 | 0.85 | 0.80 | 0.73 | 0.88 | 0.78 | 0.73 | 0.81 | 0.77 | 0.80 | 0.87 | 0.82 | 0.83 | 0.76 | 0.79 | |
| Zika | Flagging | 0.58 | 0.71 | 0.53 | 0.59 | 0.75 | 0.57 | 0.65 | 0.82 | 0.68 | 0.61 | 0.80 | 0.59 | 0.67 | 0.87 | 0.70 |
Table 17: Outbreak extraction metrics across LLMs. Average denotes means across sub-tasks; Overall denotes means across pathogens. Green and Red denote the best- and worst-performing pathogens for each model (in terms of score). Bold indicates the best-performing model for each pathogen, and Underline indicates the second-best. ; ; .
| Metric | Lassa | Ebola | SARS | Zika | ||||||||
| Flagging | 0.56 | 0.98 | 0.71 | 0.60 | 0.92 | 0.72 | 0.50 | 0.81 | 0.62 | 0.40 | 0.96 | 0.57 |
| Count | 1.00 | 0.35 | 0.51 | 0.79 | 0.47 | 0.59 | 0.80 | 0.61 | 0.69 | 0.72 | 0.47 | 0.57 |
Table 18: Flagging and Count classification metrics for parameter extraction with AgentSLR harness ( gpt-oss-120b ). ; ; .
| Group | Field | Lassa | Ebola | SARS | Zika | ||||||||
| Value | value | 0.22 | 0.22 | 0.22 | 0.20 | 0.20 | 0.20 | 0.23 | 0.23 | 0.23 | 0.14 | 0.14 | 0.14 |
| unit | 0.50 | 0.43 | 0.46 | 0.62 | 0.35 | 0.44 | 0.69 | 0.61 | 0.65 | 0.65 | 0.43 | 0.52 | |
| method | 1.00 | 0.89 | 0.94 | 0.48 | 0.78 | 0.59 | 0.76 | 0.83 | 0.79 | 0.86 | 0.80 | 0.83 | |
| Average | 0.57 | 0.51 | 0.54 | 0.44 | 0.44 | 0.41 | 0.56 | 0.56 | 0.56 | 0.55 | 0.46 | 0.50 | |
| Uncertainty | value type | 0.38 | 0.43 | 0.40 | 0.30 | 0.33 | 0.32 | 0.35 | 0.57 | 0.43 | 0.12 | 0.22 | 0.16 |
Table 19: Field-level precision, recall, and for Extraction on parameters with AgentSLR harness ( gpt-oss-120b ). Group corresponds to the sub-stage of parameter extraction where the field is collected. The final row shows averages across all fields. ; ; .
| Lassa | Ebola | SARS | Zika | |||||||||
| Flagging | ||||||||||||
| Article Flagging | 0.95 | 0.99 | 0.97 | 0.92 | 0.92 | 0.92 | 0.86 | 0.86 | 0.86 | 0.87 | 0.89 | 0.88 |
| Counts | ||||||||||||
| Model Count | 0.60 | 1.00 | 0.75 | 0.50 | 1.00 | 0.67 | 0.49 | 0.97 | 0.65 | 0.48 | 0.98 | 0.65 |
| Extraction | ||||||||||||
Table 20: Precision, recall, and metrics for transmission model screening and extraction across four pathogens with AgentSLR harness ( gpt-oss-120b ). Screening includes article flagging and model count accuracy. Extraction evaluates field-level accuracy for matched model pairs, covering core structural characteristics (model type, stochastic vs deterministic, theoretical vs data-fitted, code availability) and more complex multi-value fields (transmission routes, assumptions, interventions). Strong performance is observed for core model characteristics, while extraction of assumptions, interventions, and transmission routes remains more challenging. ; ; .
| Metric | Field | Lassa | Zika | ||||
| Flagging | Article Flagging | 0.69 | 0.82 | 0.75 | 0.58 | 0.71 | 0.64 |
| Counts | Outbreak Counts | 0.83 | 1.00 | 0.91 | 0.49 | 0.45 | 0.47 |
| Extraction | Temporal Features | 0.83 | 0.74 | 0.78 | 0.85 | 0.82 | 0.83 |
| Geographic and Spatial Features | 0.75 | 0.78 | 0.76 | 0.75 | 0.75 | 0.75 | |
| Case Burden | 0.82 | 0.75 | 0.79 | 0.93 | 0.93 | 0.93 | |
Table 21: Outbreak screening and extraction with AgentSLR harness ( gpt-oss-120b ) across feature categories. Screening measured article flagging (identifying papers containing outbreaks) and outbreak count accuracy (extracting the correct number of outbreaks per paper). Extraction evaluated field-level accuracy for matched outbreak pairs across four epidemiological categories: temporal features (start/end dates), geographic features (country, specific location), case burden (confirmed cases, deaths), and epidemiological context (detection mode, pre-outbreak status, ongoing status, asymptomatic transmission). Overall metrics represent the average across all extraction fields. ; ; .
| Lassa | Zika | |||||
| Temporal Features | ||||||
| Start Year | 0.89 | 0.80 | 0.84 | 0.90 | 0.82 | 0.86 |
| Start Month | 0.78 | 0.78 | 0.78 | 0.80 | 0.80 | 0.80 |
| Start Day | 0.86 | 0.67 | 0.75 | 0.95 | 0.95 | 0.95 |
| End Month | 0.78 | 0.78 | 0.78 | 0.65 | 0.65 | 0.65 |
Table 22: Outbreak feature extraction by category for AgentSLR harness ( gpt-oss-120b ). Each row shows field-level performance within the four major epidemiological categories. Temporal and case burden features showed consistently high performance, while location-specific fields and epidemiological context features showed greater variability. ; ; .
| Pathogen | |||
| Marburg | 0.80 | 0.64 | 0.69 |
| Ebola | 0.74 | 0.75 | 0.75 |
| Lassa | 0.82 | 0.72 | 0.75 |
| SARS | 0.78 | 0.76 | 0.77 |
| Zika | 0.73 | 0.77 | 0.75 |
| MERS | 0.83 | 0.74 | 0.78 |
Table 23: Precision, recall, and for title-and-abstract screening across seven pathogens with AgentSLR harness ( gpt-oss-120b ). Metrics summarise how well the abstract-stage classifier retained studies judged relevant under PERG screening criteria, reported for each pathogen and overall. ; ; .
| Pathogen | AI Screen (Abstract) AI Screen (Full-text) | Human Screen (Abstract) AI Screen (Full-text) | AI Screen (Direct Full-text) | ||||||
| Marburg | 0.75 | 0.76 | 0.75 | 0.77 | 0.83 | 0.80 | 0.64 | 0.82 | 0.69 |
| Ebola | 0.73 | 0.84 | 0.77 | 0.86 | 0.97 | 0.91 | 0.67 | 0.93 | 0.72 |
| Lassa | 0.79 | 0.78 | 0.78 | 0.83 | 0.94 | 0.88 | 0.71 | 0.91 | 0.77 |
| SARS | 0.71 | 0.85 | 0.76 | 0.80 | 0.95 | 0.86 | 0.64 | 0.91 | 0.68 |
| Zika | 0.66 | 0.79 | 0.69 | 0.81 | 0.91 | 0.85 | 0.64 | 0.85 | 0.67 |
Table 24: Full-text screening performance on AgentSLR harness ( gpt-oss-120b ) under three operational strategies for identifying relevant articles. Metrics compare a two-stage AI pipeline (AI abstract AI full-text), a mixed workflow (human abstract AI full-text), and direct AI full-text screening, reported for each pathogen and overall. ; .
| Item | Score |
|---|---|
| Parameters | |
| Overall — Flagging precision | |
| Overall — Extraction accuracy | |
| Precision by class | |
| Attack rate | |
| Growth rate | |
Table 25: Extended expert validation results. Results are reported as expert-rated flagging precision and expert-rated extraction accuracy. Within each section, rows are ordered from overall scores to subgroup scores and then field-level scores.
| Pathogen | PATHOGEN_IDENTIFIER | ADDITIONAL_TERMS | EXCLUSION_CRITERIA |
| Marburg virus | Marburg virus | — | — |
| Ebola virus | Ebola | — | — |
| Lassa virus | Lassa | — | — |
| SARS-CoV-1 | SARS OR SARS-CoV-1 OR “Severe acute respiratory syndrome" | — | NOT (COVID-19 OR SARS-CoV-2) |
| Zika virus | zika | OR (“extrinsic incubation period" OR “EIP" OR “vector competence" OR “vectorial capacity") † | — |
| Nipah virus | Nipah | — | — |
Table 26: Pathogen-specific modifications to the standardised search query . All databases share consistent pathogen identifiers and exclusion criteria; PubMed/Europe PMC use wildcard forms while OpenAlex uses expanded variants.
| Priority | Source & Endpoint | Rate Limit | Cached |
| 1 | OpenAlex Direct PDF URL | 30 req/s | No |
| Metadata field openalex_pdf_url | |||
| 2 | Europe PMC Fulltext API | 20 req/s | Yes |
| ebi.ac.uk/europepmc/webservices/rest/search | |||
| 3 | Unpaywall API | 50 req/s | Yes |
| api.unpaywall.org/v2/{DOI}?email={EMAIL} |
Table 29: PDF retrieval sources in cascading priority order. Sources are queried sequentially until success or exhaustion. Identifier cross-referencing via NCBI PMC ID Converter API precedes all download attempts (10 req/s, cached).
| Parameter type | Parameter class | Description |
| Attack rate | Attack rate | Proportion of a population that becomes infected during an outbreak. |
| Secondary attack rate | Attack rate | Proportion of contacts of a primary case who become infected. |
| Doubling time | Doubling time | Time required for the number of infections to double. |
| Growth rate | Growth rate | Exponential rate at which new infections increase over time. |
| Evolutionary rate | Mutations | Rate of genetic change in a population over time, typically substitutions per site per year. |
| Mutation rate | Mutations | Frequency at which new genetic mutations arise per site per replication cycle. |
Table 30: Valid parameters for extraction, according to PERG’s process.
| Parameter Class | Screening Details |
| Attack rate | The attack rate is the proportion of an at-risk population contracting the disease during a specified time interval. It is often reported as a percentage or rate, e.g. 52 people per 10,000 people. |
| Growth rate | The epidemic growth rate is a key metric that reflects how quickly the number of infections is changing day by day in a population. It is a time-dependent measure, usually expressed as a percentage or a rate per unit of time (e.g. per day), and is crucial for monitoring the speed and trajectory of an outbreak. |
| Human delay | These parameters all refer to time intervals in the natural history of infection of the host. |
| Mutation rate | Mutation rates, like substitution rate or evolutionary rate, describe the speed at which genetic changes accumulate in a population. |
| Relative contribution | This parameter is intended for pathogens (e.g. MERS) where there is both human to human (h2h) and animal to human (a2h) transmission, and aims to capture the relative magnitude of these two routes of infection in humans. We expect these to be proportions or percentages. E.g. a study might estimate 60% of infections in humans to be from h2h infection. |
| Reproduction number | We are extracting either the basic reproduction number R0 or the effective reproduction number Re. |
Table 31: Screening details for each parameter class. This is inputted into the “Parameter Class Screening Details” section of the Parameter Screening Prompt below.
| Parameter class | Variable | Type | Allowed values | Description |
| Attack rate | value | Float | – | The value of the attack rate. |
| unit | Enum | percentage; rate | The unit of the provided attack rate. | |
| type | Enum | primary; secondary | Whether primary or secondary attack rate. | |
| rate denominator | Integer; Null | – | The denominator of the value, if the parameter is provided as a rate. | |
| Doubling time | value | Float | – | The value of the doubling time, in days. |
| Growth rate | value | Float | – | The value of the growth rate. |
Table 32: Schemas used for value extraction tool calls for each parameter class. Here “–” means that any values of the correct type are allowed.
| Variable | Type | Allowed values | Description |
| population sex | Enum | female; male; both; unspecified | The sex composition of your study population. If you have 99 men and 1 woman you would still put both in this option. |
| population sample type | Enum | community based; hospital based; household based; housing estate based; population based; school based; travel based; trade or business based; contact based; mixed settings; other; unspecified | How was the study conducted? |
| population group | Enum | healthcare workers; farmers; outdoor workers; animal workers; butchers; abattoir workers; pregnant women; children; sex workers; people who inject drugs; household contacts of survivors; persons under investigation; general population; persons with symptoms; mixed settings; unspecified; other | Demographic i.e. who was sampled? |
| population sample size | Integer; Null | – | Number of participants/samples tested etc. |
| population age min | Integer; Null | – | These must be number fields. If your sample is people over 18 you would put age min = 18 and leave age max blank. |
| population age max | Integer; Null | – | These must be number fields. If your sample is people over 18 you would put age min = 18 and leave age max blank. |
Table 33: The schema for the population context extraction tool call.
| Category | Description |
| Structural Properties | Model type (compartmental, agent-based, branching process) and compartmental architecture (SIS, SIR, SEIR, etc.). Whether the model is stochastic or deterministic. |
| Epidemiological Features | Primary transmission routes (airborne, direct contact, vector-borne, sexual). Spatial heterogeneity and spillover dynamics from animal reservoirs. |
| Behavioural Assumptions | Mixing patterns (homogeneous or heterogeneous), age-dependent susceptibility, cross-immunity between pathogens, and temporal variation in transmission rates. |
| Theoretical vs. Fitted | Whether the model was fitted to actual data or uses parameters from literature or arbitrary values. |
| Intervention Categories | Control measures evaluated including vaccination, quarantine, vector control, treatment, contact tracing, behaviour changes, and various vector management strategies. |
| Reproducibility Indicators | Code availability, programming language used, data sharing status, and presence of documentation (README). |
Table 34: Model characteristic categories targeted by the extraction pipeline.
| Field Name | Type | Allowed Values | Description |
| model_type | Enum | Compartmental; Branching process; Agent/Individual based; Other; Unspecified | Primary modeling framework used for transmission dynamics. |
| compartmental_type | Enum | SIS; SIR; SEIR; SEIR-SEI; SAIR-SEI; Not compartmental; Other compartmental | Specific compartmental model architecture if applicable. Use “Not compartmental” for non-compartmental models. |
| stoch_deter | Enum; Null | Stochastic; Deterministic | Whether the model is stochastic or deterministic. Only extract if explicitly stated. Null if not specified. |
| transmission_route | List[Enum] | Airborne or close contact; Human to human (direct contact); Human to human (direct non-sexual contact); Vector/Animal to human; Sexual; Unspecified | Primary pathway(s) through which pathogen transmission occurs. Multiple routes can be selected. |
| uncertainty_was_considered | Boolean; Null | True; False | Whether uncertainty was considered through stochasticity, multiple models, or parameter variation (e.g. sensitivity analyses, Bayesian analysis). Null if not specified. |
| spatial_model | Boolean; Null | True; False | Whether the model explicitly incorporated spatial or geographic heterogeneity. Null if not specified. |
Table 35: Model extraction schema with field specifications, data types, allowed values, and descriptions.
| Outbreak Characteristic | Description |
|---|---|
| Outbreak start day | Day of outbreak onset (1–31). Extracted as stated in paper. |
| Outbreak start month | Month of outbreak onset. Extracted as stated in paper. |
| Outbreak start year | Year of outbreak onset. Extracted as stated in paper. |
| Outbreak end day | Day of outbreak conclusion (1–31). Extracted as stated in paper. |
| Outbreak end month | Month of outbreak conclusion. Extracted as stated in paper. |
| Outbreak end year | Year of outbreak conclusion. Extracted as stated in paper. |
Table 36: Outbreak field descriptions and meanings.
| Field Name | Type | Allowed Values | Description |
|---|---|---|---|
| outbreak_start_day | Integer; Null | 1-31 | Day of outbreak onset. Null if not provided. |
| outbreak_start_month | String (Enum); Null | Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec | Month of outbreak onset. Null if not provided. |
| outbreak_start_year | Integer; Null | Integer year | Year of outbreak onset. Null if not provided. |
| outbreak_end_day | Integer; Null | 1-31 | Day of outbreak conclusion. Null if not provided. |
| outbreak_end_month | String (Enum); Null | Jan, Feb, Mar, Apr, May, Jun, Jul, Aug, Sep, Oct, Nov, Dec | Month of outbreak conclusion. Null if not provided. |
| outbreak_end_year | Integer; Null | Integer year | Year of outbreak conclusion. Null if not provided. |
Table 37: Outbreak extraction schema with field specifications, data types, allowed values, and descriptions.
| Artefact | Path (relative to repo root) | Purpose |
| Outbreak report (Markdown) | writeup/ /outbreaks_writeup.md | Human-readable draft with embedded figures and tables. |
| Outbreak report (PDF) | writeup/ /outbreaks_writeup.pdf | Portable rendering for sharing and archiving. |
| Figures directory | writeup/ /figures/ | Generated plots referenced by Markdown (e.g., temporal distribution, geographic spread, case counts). |
| Summary tables (embedded) | (in outbreaks_writeup.md ) | Count and proportion tables computed from . |
| Content manifest | writeup/ /content_manifest.json | Machine-readable inventory of figures, tables, and dataset statistics. |
Table 38: Artefact inventory for outbreak report generation (per pathogen ). All artefacts are derived from the extracted outbreak dataset .
Figure 7: Ebola living reviews generated by AgentSLR. Both reviews follow a structured format: evidence-based descriptions citing supporting figures and tables, followed by interpretation blocks explicitly labelled as AI-generated synthesis.
Figure 8: Preliminary living reviews for pathogens without completed PERG validation. RVF and CCHF represent pathogens for which PERG has not yet commenced systematic screening.
| Type | Transmission-Modelling Review | Outbreak Surveillance Review |
| Figures | Model architecture distribution (compartmental, branching process, agent-based); Stochasticity classification; Transmission route breakdown; Code availability | Geographic burden (choropleth maps for cases and deaths); Outbreak timeline by country; Detection mode distribution |
| Tables | Model type counts and proportions; Deterministic vs stochastic breakdown; Transmission routes with sample sizes; Modelling assumptions; Intervention categories; Spatial scale indicators; Code availability and language | Outbreak source categories; Detection methodology breakdown; Ongoing outbreaks at extraction; Case burden stratified by confirmation status; Sex disaggregation where reported |
Table 39: Key artefacts in AgentSLR workflow generated living reviews. Each pathogen generates two review types with consistent visualisation and evidence table structures. The text-based LLM uses manifests, with summary statistics of the figures to write its interpretation.
Figure 9: The AgentSLR extraction and validation tool interface. The dual-pane view presents the source document ( left ) alongside structured extraction fields ( right ). AI-predicted entries are pre-filled and accompanied by highlighted, AI-tagged evidence excerpts from the manuscript. Reviewers can accept, revise, or reject individual fields, with validation status indicators (e.g. “AI Match” or “Revised”) reflecting whether human intervention was required.
Figure 10: Review management and submission tracking interfaces. (a) The study review list displays papers awaiting expert validation, along with associated model and outbreak counts and direct links to extracted excerpts. (b) The verification view presents finalised AI-assisted extractions, enabling reviewers to explicitly reject predictions or verify and save corrected entries, which are then recorded for downstream quality control and system evaluation.