EpiLens: Lightweight RAG for Epidemiological Knowledge Acquisition

Vincenzo Perri; Ciro Cattuto; Daniela Paolotti

cs.CLOct 19, 2026

EpiLens: Lightweight RAG for Epidemiological Knowledge Acquisition

Authors: Vincenzo Perri, Ciro Cattuto, Daniela Paolotti

Organizations: ISI Foundation, Turin, Italy.

Abstract

To prepare for future outbreaks, researchers need to know what studies have been done and what data they use. Mapping a large literature by hand is slow, while large language models alone can misstate or conflate sources. We present EpiLens, a lightweight, open-source retrieval-augmented generation package that applies a fixed retrieval template and output schema to every paper in a corpus and links outputs to source passages. We build a corpus of 2,297 papers from the sources of a multi-pathogen parameter study and evaluate EpiLens's paper-classification workflow against 214 manually labelled papers. Without task-specific training data, the classifier matches or exceeds the tested supervised and zero-shot baselines across four tasks. The study's early extraction sheets cite published sources for 327 estimates. We link each estimate to a cited document or to corpus studies included by a review. Then, we use the classifier's labels to compare the papers associated with different parameters and pathogens. EpiLens labels more papers linked to reproduction numbers than fatality risks as model-based (54% versus 22%, compared with 62% versus 24% in the manual labels). The gap is consistent with the frequent use of transmission models to estimate reproduction numbers and persists within each of five pathogens compared. Beyond the contrast between parameters, the model-based share of papers linked to incubation-period estimates differs by pathogen: 55% for SARS-CoV-2 versus 6% for mpox.

Figures & tables

Panel of Figure 1 — Figure 1: Division of labour between EpiLens and the researcher. Large-scale, unstructured outbreak literature (left) is systematically processed by EpiLens to resolve a corpus into distinct, structured dimensions. The dimensions shown (data type, accessibility, geography, and paper type) are the examples we carry through this paper. They are not fixed by the system, which the user configures for their own questions. By handling the mechanical extraction and provenance of these variables, the system provides filtered evidence to the researcher (right). This separation lets the researcher move from labour-intensive sorting of documents to higher-level work: defining label vocabularies, auditing edge cases, and interpreting the resulting patterns.

Panel of Figure 2 — Figure 2: RAG workflow used in EpiLens. Dashed arrows indicate one-time ingestion and indexing of the corpus. Solid arrows indicate per-task interrogation (query embedding, search, and generation) with structured outputs linked to source chunks.

	Floor	gemini-2.5-pro		gemini-2.5-flash		Nemotron
Task	Majority	$T{=}0$	$T{=}1$	$T{=}0$	$T{=}1$	$T{=}0$
Geography	0.353	$0.928_{\,0.89\text{--}0.95}$	$0.925_{\,0.89\text{--}0.95}$	$0.875_{\,0.83\text{--}0.91}$	$0.871_{\,0.83\text{--}0.90}$	$0.886_{\,0.84\text{--}0.92}$
Data access.	0.472	$0.802_{\,0.76\text{--}0.84}$	$0.775_{\,0.73\text{--}0.81}$	$0.670_{\,0.62\text{--}0.72}$	$0.684_{\,0.65\text{--}0.72}$	$0.660_{\,0.62\text{--}0.70}$
Data type	0.843	$0.950_{\,0.92\text{--}0.97}$	$0.941_{\,0.91\text{--}0.96}$	$0.916_{\,0.88\text{--}0.94}$	$0.913_{\,0.88\text{--}0.94}$	$0.877_{\,0.84\text{--}0.91}$
Paper type	0.593	$0.903_{\,0.86\text{--}0.93}$	$0.901_{\,0.86\text{--}0.93}$	$0.877_{\,0.83\text{--}0.91}$	$0.880_{\,0.84\text{--}0.91}$	$0.848_{\,0.80\text{--}0.89}$

Table 1: Accuracy of the served backends. 0 0 footnotetext: Mean Jaccard similarity on the

214

-paper evaluation set over

R{=}5

runs per configuration. The subscript is a

95\%

bootstrap interval over papers (BCa,

B{=}10\,000

), computed after averaging each paper’s score over the runs: it shows how precisely these papers determine the score. Each interval describes one configuration; methods are compared with the paired tests of Section 11.4 . Run-to-run standard deviations are in Table 1 . The Floor column (set off on the left) is the majority-class baseline, a single deterministic value included as a reference point for reading the scores. Nemotron is an open-weight backend (NVIDIA Nemotron-120B) run through the identical pipeline at

T{=}0

. Paper type is a single-label task. The remaining tasks are multi-label.

	Qwen2.5-32B (local)				Gemini pro		Gemini flash		Nemotron
	fully local		partially local
Task	$T{=}0$	$T{=}1$	$T{=}0$	$T{=}1$	$T{=}0$	$T{=}1$	$T{=}0$	$T{=}1$	$T{=}0$
Geography	100.0	80.4	100.0	78.5	94.9	93.0	92.1	81.3	86.9
Data access.	100.0	17.8	100.0	20.6	63.6	51.4	66.8	32.2	44.9
Data type	100.0	79.4	100.0	83.6	89.3	90.2	94.9	89.3	86.9
Paper type	100.0	87.4	100.0	86.9	87.9	86.0	89.7	76.2	82.7

Table 2: Label stability across repeated runs. 0 0 footnotetext: Percentage of papers whose predicted label set was identical across five repeated runs under each configuration. A prediction is considered stable if the full label set is unchanged across runs. The Qwen2.5-32B columns are the self-hosted backend of Section 10.3 : fully local embeds the index locally with BAAI/bge-m3 , while partially local reuses the Gemini-embedded index. Both are perfectly stable at

T{=}0

, returning an identical label set for every paper across all five repeats. Gemini pro and Gemini flash are gemini-2.5-pro and gemini-2.5-flash . Nemotron is the open-weight backend (NVIDIA Nemotron-120B) at

T{=}0

. The majority floor is omitted here because it is deterministic.

Figure 3: Ablation results for two generators. The upper panel uses the served gemini-2.5-flash model and the lower panel the self-hosted qwen2.5:32b-instruct , both at

T{=}0

. Within each panel, the model, prompt, and post-processing are fixed. The four conditions are title and abstract only, title and abstract plus

k{=}15

randomly selected body-text chunks, title and abstract plus

k{=}15

semantically retrieved chunks, and title, abstract, and the whole parsed body. Comparing conditions 1 and 2 shows the effect of adding body text. Comparing conditions 2 and 3 shows the effect of targeted selection with the same number of chunks.

Reference
Method	Requirements		Jaccard by task
	Body?	Task sup.	Paper type	Geo- graphy	Data type	Data access.
Majority prior (floor)	No	Weak	0.593	0.353	0.843	0.472
Zero-shot (no training labels)
Prototype (TF-IDF)	No	None	0.407	0.283	0.315	0.217
Prototype (SPECTER)	No	None	0.126	0.165	0.515	0.360

Table 3: Performance and requirements of the baseline approaches. 0 0 footnotetext: Sample-averaged Jaccard similarity

J

on the fixed

214

-paper evaluation set. Body? indicates whether parsed body text is required. No generally means title and abstract, while the majority floor uses no text. Task sup. records the task-specific input used to construct or fit the method: None uses no labelled papers, Weak uses a class prior, seeds, or partial paper labels, and Manual uses a fully manually labelled training set. All scores use the same manual labels. Prototype rows contrast uncalibrated similarity with threshold calibration. BERTopic uses the best topic count and oracle cluster-to-label alignment. Best supervised is the per-task maximum over the supervised grid. Both are favourable descriptive upper bounds rather than single deployable configurations. For EpiLens, the mean is over

R{=}5

runs at

T{=}0

, with its

95\%

bootstrap interval over papers beneath it, as in Table 1 . Other rows are deterministic evaluations or cross-validated estimates. Scores are rounded to three decimals. Paired tests are reported in Table 1 . Full method specifications are given in Section 11 .

Figure 4: Sources of parameter estimates. Estimates of each parameter, by source: a review whose included studies are in our corpus, or a single study or other document in our corpus. Parameters are ordered by the share of their estimates taken from reviews.

Figure 5: Profiles of associated papers by parameter. The first row describes all 2,297 corpus papers, and each other row the papers behind the estimates of one parameter, as labelled by EpiLens (model outputs, not estimates). Rows are ordered by the share of model-based papers, and

n

is the number of papers. (a) Percentage of the papers with each paper type. Each paper has one, and model-based papers are those labelled forecast or inference. (b, c) Percentage of the papers with each data type and each continent. A paper can have several. “No data” means no empirical data, and “no location” means irrelevant or unclear. (d) Fractional allocation of the corpus: each paper has total weight one, divided equally among the pathogen–parameter pairs whose estimates it lies behind, so that the row weights sum to

100\%

Figure 6: Papers linked to cited sources by pathogen. The first row describes all 2,297 corpus papers. Each other row describes papers linked to sources cited for that pathogen across all parameters in the seed study. The panels show the same measures as in Figure 5 . Pathogens with at least

10

associated papers are shown. Influenza A subtypes are merged here and listed separately in Table 1 . CCHF: Crimean–Congo haemorrhagic fever. RVF: Rift Valley fever.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Paper	Data sources extracted (name and link or accession where given)
Lauer et al. ( Lauer et al., 2020 ) COVID-19 incubation	• News and public health reports • Johns Hopkins Center for Systems Science and Engineering • HopkinsIDD/ncov_incubation GitHub repository ( https://github.com/HopkinsIDD/ncov_incubation ) • Zenodo record 3692048 ( https://zenodo.org/record/3692048 )
Althaus ( Althaus, 2014 ) Ebola $R_{0}$	• World Health Organisation (WHO) • Chowell et al. (study)
Nguku et al. ( Nguku et al., 2010 ) Rift Valley fever, Kenya	• International Livestock Research Institute (ILRI) GIS site ( http://www.ilri.org/gis ) • Kenya Soil Survey, Report E1, 1982 • LANDSAT images obtained in 1987 by JICA for the Kenya National Water Master Plan • National Oceanic and Atmospheric Administration’s (NOAA) Surface Data Global Summary of the Day file ( http://www.cdo.ncdc.noaa.gov/cdo/dataproduct ) • 1999 Kenya Census • Genbank
Nakoune et al. ( Nakoune et al., 2017 ) Monkeypox, CAR	• DDBJ/EMBL/GenBank database • KP849469, A27L gene (reference sequence) • Standardized tools for data collection (Ministry of Health and World Health Organization) • Consultation registers, health records, and hospitalization forms • Institut Pasteur de Bangui (IPB) reports of suspected cases

Appendix

Table 1: A second structured workflow composed with the classifier: the data-source Precision Miner. 0 0 footnotetext: Rights-safe output of the find_data_sources miner on four corpus papers, one per pathogen family. The released summary contains identifiers, data-source names, and links or accessions. It deliberately omits source sentences, retrieved text, prompts, and model traces.

Axis	Input	Reasoning	Visible	Total
		Output (billable)
Paper type	$4172$	$1552$	$151$	$5875$
Geography	$3356$	$913$	$169$	$4437$
Data type	$3213$	$926$	$123$	$4261$
Data accessibility	$3568$	$1514$	$155$	$5237$
All axes (mean)	$3577$	$1226$	$149$	$4952$

Appendix

Table 2: Mean token usage per classification case in the cost proxy. 0 0 footnotetext: Measured on the random-chunk ablation with gemini-2.5-flash ,

T{=}0

, and

k{=}15

. The targeted workflow uses the same prompts, schemas, generator, retrieval budget, and repair path but different evidence selection. Totals include repair invocations.

Scope	Cases	Gen. in	Gen. out	Embed (1 $\times$ )
Evaluation sample (214 papers)	$856$	$3.1M$	$1.2M$	$1.3M$
Full corpus (2297 papers)	$9188$	$32.9M$	$12.6M$	$14.5M$

Appendix

Table 3: Projected token usage of one EpiLens classification pass over the four axes. 0 0 footnotetext: Projected from the ablation-based per-case proxy in Table 2 . Generation is incurred per pass. Corpus embedding is a one-time estimate.

	gemini-2.5-pro		gemini-2.5-flash		Nemotron
Metric	$T{=}0.0$	$T{=}1.0$	$T{=}0.0$	$T{=}1.0$	$T{=}0.0$
Geography
Jaccard	$0.9283_{\pm 0.0044}$	$0.9248_{\pm 0.0049}$	$0.8748_{\pm 0.0069}$	$0.8709_{\pm 0.0044}$	$0.886_{\pm 0.011}$
Micro precision	$0.8816_{\pm 0.0091}$	$0.8845_{\pm 0.0082}$	$0.794_{\pm 0.011}\phantom{0}\phantom{\scriptstyle 0}$	$0.780_{\pm 0.016}\phantom{0}\phantom{\scriptstyle 0}$	$0.816_{\pm 0.014}$
Micro recall	$0.9457_{\pm 0.0027}$	$0.9434_{\pm 0.0021}$	$0.9279_{\pm 0.0044}$	$0.9240_{\pm 0.0044}$	$0.930_{\pm 0.012}$
Micro F 1	$0.9125_{\pm 0.0058}$	$0.9130_{\pm 0.0051}$	$0.8559_{\pm 0.0073}$	$0.8461_{\pm 0.0099}$	$0.870_{\pm 0.012}$

Appendix

Table 1: Aggregate agreement metrics by task and configuration.

Geography
Label	$n$	Pro	Flash	Nemotron	Qwen (local)
Asia	85	$0.9714_{\pm 0.0039}$	$0.9382_{\pm 0.0043}$	$0.9434_{\pm 0.0057}$	$0.927_{\pm 0.000}$
Africa	67	$0.9592_{\pm 0.0082}$	$0.9362_{\pm 0.0047}$	$0.9460_{\pm 0.0080}$	$0.913_{\pm 0.000}$
Europe	42	$0.8825_{\pm 0.0060}$	$0.8049_{\pm 0.0092}$	$0.876_{\pm 0.012}\phantom{0}\phantom{\scriptstyle 0}$	$0.812_{\pm 0.000}$
North America	24	$0.7783_{\pm 0.0083}$	$0.683_{\pm 0.024}\phantom{0}\phantom{\scriptstyle 0}$	$0.709_{\pm 0.033}\phantom{0}\phantom{\scriptstyle 0}$	$0.731_{\pm 0.000}$
Irrelevant	15	$0.864_{\pm 0.026}\phantom{0}\phantom{\scriptstyle 0}$	$0.636_{\pm 0.065}\phantom{0}\phantom{\scriptstyle 0}$	$0.30_{\pm 0.11}\phantom{0}\phantom{\scriptstyle 0}\phantom{0}\phantom{\scriptstyle 0}$	$0.640_{\pm 0.000}$

Appendix

Table 2: Per-label F 1 of EpiLens against the manual labels. 0 0 footnotetext: F 1 of each label against the manual labels on the

214

evaluation papers at

T{=}0

: the mean over five runs, with the standard deviation as a subscript.

n

is the number of papers that carry the label in the manual labels. Labels that never occur there are not shown. Pro and Flash are gemini-2.5-pro and gemini-2.5-flash , Nemotron is nemotron-3-super-120b-a12b , and Qwen (local) is the fully local qwen2.5:32b-instruct configuration, whose repeats are identical. For labels carried by few papers, F 1 rests on a handful of decisions.

	Floor	Qwen2.5-32B (self-hosted)		Served backends
Task	Majority	fully local	partially local	Nemotron	flash	pro
Geography	0.353	$0.869_{\pm 0.000}$	$0.854_{\pm 0.000}$	$0.886_{\pm 0.011}\phantom{0}\phantom{\scriptstyle 0}$	$0.8748_{\pm 0.0069}$	$0.9283_{\pm 0.0044}$
Data access.	0.472	$0.539_{\pm 0.000}$	$0.563_{\pm 0.000}$	$0.660_{\pm 0.014}\phantom{0}\phantom{\scriptstyle 0}$	$0.670_{\pm 0.022}\phantom{0}\phantom{\scriptstyle 0}$	$0.802_{\pm 0.023}\phantom{0}\phantom{\scriptstyle 0}$
Data type	0.843	$0.861_{\pm 0.000}$	$0.875_{\pm 0.000}$	$0.8767_{\pm 0.0086}$	$0.9158_{\pm 0.0063}$	$0.950_{\pm 0.018}\phantom{0}\phantom{\scriptstyle 0}$
Paper type	0.593	$0.794_{\pm 0.000}$	$0.785_{\pm 0.000}$	$0.848_{\pm 0.011}\phantom{0}\phantom{\scriptstyle 0}$	$0.8766_{\pm 0.0097}$	$0.9028_{\pm 0.0084}$

Appendix

Table 3: A fully local, fully open-weight deployment of the same workflow. 0 0 footnotetext: Sample-averaged Jaccard on the

214

-paper evaluation set, as mean with the standard deviation across

R{=}5

repeated runs shown as a subscript. Every column is at

T{=}0

. Table 4 and Table 5 report the same configurations at

T{=}1

as well. Both Qwen2.5-32B columns generate with qwen2.5:32b-instruct run locally through Ollama on our own hardware, and differ only in how the retrieval index is embedded: fully local embeds with BAAI/bge-m3 on the same machine, so no stage of the pipeline contacts a remote service, whereas partially local reuses the Gemini-embedded index shared with the served-backend runs. Everything else is held fixed across all columns (dense retrieval, workflow_top_k

{=}15

, grobid parses, identical prompts, schema-constrained decoding). The Floor column is the deterministic majority-class baseline. Nemotron (NVIDIA Nemotron-120B) and the two Gemini backends, flash and pro , are reproduced from Table 1 for reference. Both self-hosted configurations return identical predictions on all four axes across all five repeats, so their SD is exactly zero ( Section 10.3 ).

	Floor	Qwen2.5-32B (open weight, self-hosted generation)
		fully local (bge-m3)		partially local (Gemini index)
Task	Majority	$T{=}0$	$T{=}1$	$T{=}0$	$T{=}1$
Geography	0.353	$0.869_{\pm 0.000}$	$0.869_{\pm 0.014}\phantom{0}\phantom{\scriptstyle 0}$	$0.854_{\pm 0.000}$	$0.8677_{\pm 0.0062}$
Data access.	0.472	$0.539_{\pm 0.000}$	$0.516_{\pm 0.027}\phantom{0}\phantom{\scriptstyle 0}$	$0.563_{\pm 0.000}$	$0.542_{\pm 0.021}\phantom{0}\phantom{\scriptstyle 0}$
Data type	0.843	$0.861_{\pm 0.000}$	$0.8661_{\pm 0.0061}$	$0.875_{\pm 0.000}$	$0.8785_{\pm 0.0059}$
Paper type	0.593	$0.794_{\pm 0.000}$	$0.775_{\pm 0.013}\phantom{0}\phantom{\scriptstyle 0}$	$0.785_{\pm 0.000}$	$0.779_{\pm 0.012}\phantom{0}\phantom{\scriptstyle 0}$

Appendix

Table 4: Accuracy of the self-hosted configurations. 0 0 footnotetext: Sample-averaged Jaccard as mean with the standard deviation across

R{=}5

repeated runs shown as a subscript, on the

214

-paper evaluation set. Both blocks generate with qwen2.5:32b-instruct run locally through Ollama on our own hardware. Fully local additionally embeds the retrieval index locally with BAAI/bge-m3 , so no stage of the pipeline contacts a remote service. Partially local reuses the Gemini-embedded index built for the served-backend runs, and so remains partially remote. The Floor column is the deterministic majority-class baseline. All

T{=}0

standard deviations are exactly zero: the five repeats return identical label sets for every paper ( Section 10.3 ). Table 1 reports the served backends on the same tasks.

Geography
	Qwen2.5-32B (open weight, self-hosted generation)
	fully local (bge-m3)		partially local (Gemini index)
Metric	$T{=}0.0$	$T{=}1.0$	$T{=}0.0$	$T{=}1.0$
Jaccard	$0.869_{\pm 0.000}$	$0.869_{\pm 0.014}\phantom{0}\phantom{\scriptstyle 0}$	$0.854_{\pm 0.000}$	$0.8677_{\pm 0.0062}$
Micro precision	$0.804_{\pm 0.000}$	$0.8222_{\pm 0.0075}$	$0.785_{\pm 0.000}$	$0.8214_{\pm 0.0066}$
Micro recall	$0.891_{\pm 0.000}$	$0.889_{\pm 0.014}\phantom{0}\phantom{\scriptstyle 0}$	$0.876_{\pm 0.000}$	$0.881_{\pm 0.013}\phantom{0}\phantom{\scriptstyle 0}$

Appendix

Table 5: Full agreement-metric profile of the self-hosted open-weight backend. 0 0 footnotetext: Mean with the standard deviation across

R{=}5

repeated runs shown as a subscript, on the fixed

214

-paper evaluation set. All four columns generate with qwen2.5:32b-instruct run locally through Ollama under the identical workflow, and use schema-constrained decoding. The fully local pair embeds the retrieval index with BAAI/bge-m3 on the same machine, so the pipeline makes no remote call at any stage. The partially local pair reuses the Gemini-embedded index shared with the served-backend runs. Within each pair the columns differ only in sampling temperature. For the single-label paper-type task, micro-averaged scores coincide with subset accuracy. All

T{=}0

standard deviations are exactly zero: the five repeats return identical label sets for every paper.

Figure 1: Label distributions across all 2,297 corpus papers. Panels show paper type, geography, and data type as labelled by gemini-2.5-flash at

T{=}0

. Bar lengths give percentages and the adjacent numbers give paper counts. Paper type is single-label. Geography and data type are multi-label, so their percentages can sum to more than

100\%

. Mobility, health, sentiment, and economic data are the non-traditional categories.

Figure 2: Corpus-scale label distributions from the fully local deployment. The figure reports the same three axes and the same 2,297 papers as Figure 1 , produced on local hardware with qwen2.5:32b-instruct and a BAAI/bge-m3 index at

T{=}0

. Paper type is single-label, whereas geography and data type are multi-label, so their percentages can sum to more than

100\%

Task	$\|E_{\text{pro}}\|$	$\|E_{\text{flash}}\|$	$\|E_{\cap}\|$	$\|E_{\cap}\|/\|E_{\text{pro}}\|$	$\|E_{\cap}\|/\|E_{\text{flash}}\|$
Geography	25	45	20	80.0%	44.4%
Data type	25	32	20	80.0%	62.5%
Data accessibility	95	123	72	75.8%	58.5%
Paper type	34	37	15	44.1%	40.5%

Appendix

Table 6: Containment of ever-wrong sets at

T{=}0.0

. 0 0 footnotetext: For each task,

E

is the set of papers that are incorrectly labeled in at least one of the

R{=}5

runs. For multi-label tasks, incorrect means that the predicted label set is not an exact match to the manual label set.

|E_{\text{pro}}|

and

|E_{\text{flash}}|

report ever-wrong set sizes for the two backends.

|E_{\cap}|

reports overlap. The last two columns divide it by the corresponding backend’s ever-wrong set size.

Task	$\|E_{\text{pro}}\|$	$\|E_{\text{flash}}\|$	$\|E_{\cap}\|$	$\|E_{\cap}\|/\|E_{\text{pro}}\|$	$\|E_{\cap}\|/\|E_{\text{flash}}\|$
Geography	28	59	25	89.3%	42.4%
Data type	30	40	26	86.7%	65.0%
Data accessibility	116	163	106	91.4%	65.0%
Paper type	37	54	23	62.2%	42.6%

Appendix

Table 7: Containment of ever-wrong sets at

T{=}1.0

. 0 0 footnotetext: Definitions as in Table 6 .

Baseline	Paper type	Geography	Data type	Data access.
Majority prior (floor)	$+0.283$ ∗	$+0.521$ ∗	$+0.073$ ∗	$+0.198$ ∗
Zero-shot NLI	$+0.311$ ∗	$+0.166$ ∗	$+0.898$ ∗	$+0.315$ ∗
TF-IDF + SVM	$+0.164$ ∗	$+0.283$ ∗	$+0.062$ ∗	$+0.137$ ∗
SetFit	$+0.143$ ∗	$+0.275$ ∗	$+0.052$ ∗	$+0.096$ ∗
ModernBERT (full body)	$+0.092$ ∗	$+0.222$ ∗	$+0.051$ ∗	$+0.093$ ∗
metadata_llm	$+0.164$ ∗	$+0.049$	$+0.100$ ∗	$+0.429$ ∗

Appendix

Table 1: Paired comparisons of EpiLens ( gemini-2.5-flash ,

T{=}0

) with selected baselines. 0 0 footnotetext: Each cell is the mean per-paper Jaccard difference (EpiLens minus the baseline) on the

N{=}214

evaluation papers. A marker ( ∗ ) denotes a Holm-adjusted

p

-value below

0.05

across eight comparisons per task. Six are shown here. The full-body and leave-one-out TF-IDF variants also enter the correction. Tests use

B{=}10\,000

paired BCa resamples ( Section 11.4 ). They hold the manual labels, predictions, and configurations fixed. The topic-model and Best supervised rows of Table 3 are descriptive because their alignment or configuration was selected using the evaluation papers. Prototype rows are reported as diagnostics and are not tested here.

TF-IDF (title+abstract)
Configuration	Paper type	Geog.	Data type	Data access.
Linear SVM	0.713 / 0.713	0.592 / 0.607	0.854 / 0.871	0.533 / 0.538
Logistic Reg.	0.706 / 0.696	0.593 / 0.575	0.881 / 0.871	0.528 / 0.514
GB trees	0.595 / –	0.453 / –	0.814 / –	0.467 / –
MLP	0.696 / –	0.599 / –	0.856 / –	0.558 / –
TF-IDF (full body)

Appendix

Table 2: Comprehensive supervised configuration grid. 0 0 footnotetext: Sample-averaged Jaccard for every representation

\times

text scope

\times

classifier head combination, all at stratified 5-fold CV over the same

N{=}214

papers. Each cell is shown as “5-fold / LOO”. The gradient-boosted-tree and MLP heads show “–” for the leave-one-out score, because a LOO sweep would refit the head once per held-out paper, which is prohibitively expensive for these non-linear heads, so only the 5-fold score is computed for them. The grid positions EpiLens ( Table 3 ) against the full space of reasonable supervised approaches rather than a single representative. The small 5-fold-to-LOO gaps and the saturation curves ( Figure 2 ) show the baselines are near data-saturated at this sample size.

Figure 2: Saturation of the supervised configuration grid. Learning curves (held-out Jaccard vs. number of labelled training papers) grouped by representation (rows) and task (columns). Within each panel, colour encodes the classifier head (linear SVM, logistic regression, gradient-boosted trees, MLP) and line style the text scope (solid = title+abstract, dashed = full body). The black dotted line is EpiLens (flash,

T{=}0

). Every curve remains below EpiLens within the observed training range, and most curves flatten before the full sample, suggesting that the gap is unlikely to be closed merely by small increases in labelled data.

Reference
Method	Paper type	Geography	Data type	Data access.
Majority prior (floor)	0.593	0.353	0.843	0.472
Unsupervised topic models (best $K$ across sweep, oracle label alignment)
LSA	0.593	0.501	0.843	0.489
LDA	0.757	0.489	0.880	0.484
NMF	0.682	0.576	0.874	0.516

Appendix

Table 3: Corpus-structure audit: how separable each axis is by vocabulary alone. 0 0 footnotetext: Sample-averaged Jaccard

J

for unsupervised topic models on the same

214

-paper evaluation set, reported as the best

J

across the

K

sweep (

K\in\{0.5,\,1,\,2,\,4\}\times|\mathcal{L}|

) under an oracle cluster-to-label alignment, in their original full-text, leave-one-out configuration. These are not classifiers (they discover themes rather than apply the analyst’s scheme), and the best-

K

, oracle-aligned figures are an upper bound on what vocabulary structure alone recovers. The majority floor and EpiLens are shown for reference. Topic models partly recover the lexically regular axes ( paper type , data type ) but sit near the floor on data accessibility , the axis whose signal is a single buried statement.

By parameter
	Papers	Model-based (%)	Simulated data (%)
Reproduction number	$629$	$54$	$48$
Fatality risk	$1549$	$22$	$13$
Serial interval	$539$	$58$	$38$
Incubation period	$845$	$33$	$20$
Dispersion $k$	$137$	$57$	$55$

Appendix

Table 1: Labels of the papers behind the estimates, by parameter and by pathogen. 0 0 footnotetext: Each row counts distinct corpus records associated with estimates of one parameter or pathogen. A record associated with several parameters or pathogens counts for each, so the rows add up to more than All. Model-based and Simulated data: percentage of these papers to which EpiLens assigns the label with gemini-2.5-flash (model outputs, not estimates). Model-based means paper type INFERENCE or FORECAST , and simulated data means data type SYNTHETIC . Shares are not shown for rows with fewer than 10 papers. Section 4.4 compares these labels with the manual labels for the comparisons it draws. Parameters are ordered as in Figure 4 . Bold headings name the pathogen groups in the early extraction sheets shared by the authors of Ward et al. (2025) that cover several pathogens. A/H1N1: estimates not specific to the 2009 pandemic virus. A/H1N1pdm09: the 2009 pandemic virus. A/H1N1 and A/H3N2: estimates reported for both subtypes together. CCHF: Crimean–Congo haemorrhagic fever. RVF: Rift Valley fever. Fatality risk: case or infection fatality risk.

Reproduction number and fatality risk
Pathogen	Parameter	Papers	Model-based (%)
SARS-CoV-2	Reproduction number	$204$	$62$
	Fatality risk	$335$	$52$
MERS-CoV	Reproduction number	$55$	$40$
	Fatality risk	$67$	$36$
Ebola virus	Reproduction number	$117$	$47$

Appendix

Table 2: Model-based shares in the pathogen–parameter comparisons. 0 0 footnotetext: Each row counts distinct corpus papers associated with estimates of the named parameter for that pathogen. A paper can appear in several rows. Model-based is the share labelled INFERENCE or FORECAST by EpiLens with gemini-2.5-flash at

T{=}0

(model outputs, not parameter estimates). Rows show the comparisons discussed in Section 4.4 . The full breakdown is in the reproduction package ( Perri et al., 2026 ) . CCHF: Crimean–Congo haemorrhagic fever. RVF: Rift Valley fever.

LLM Information Extraction

Large language model information extraction converts unstructured documents into structured entities, relations, events, and records.

See all 232 papers

Jan 6, 2026

EpiQAL: Benchmarking Large Language Models in Epidemiological Question Answering and Reasoning

Reliable epidemiological reasoning requires synthesizing study evidence to infer disease burden, transmission dynamics, and intervention effects at the population level. Existing medical question answering benchmarks primarily emphasize clinical knowledge or patient-level reasoning, yet few systematically evaluate evidence-grounded epidemiological inference. We present EpiQAL, to our knowledge the first diagnostic benchmark for epidemiological question answering over research literature, comprising three subsets built from open-access articles across diverse diseases. The three subsets progressively test factual recall, multi-step inference, and conclusion reconstruction under incomplete information, and are constructed through a quality-controlled pipeline combining taxonomy guidance, multi-model verification, and difficulty screening. Experiments on fifteen models spanning open-source and proprietary systems reveal that current LLMs show limited performance on epidemiological reasoning, with multi-step inference posing the greatest challenge. Model rankings shift across subsets, and scale alone does not predict success. Chain-of-Thought prompting benefits multi-step inference but yields mixed results elsewhere. EpiQAL provides fine-grained diagnostic signals for evidence-grounding, inferential reasoning, and conclusion reconstruction.

Mingyang Wei, Dehai Min, Zewen Liu +8

1Emory University · University of Illinois Chicago · 3Microsoft

LLM Evaluation Healthcare Evidence-Grounded Reasoning Biomedical QA Medical Reasoning

Ask PDF

Mar 20, 2026

Evaluating AI-based Scientific Knowledge Synthesis with Epidemiological Systematic Reviews

Systematic literature reviews (SLRs) are a demanding and high-stakes form of scientific knowledge synthesis that remains underspecified as an evaluation setting for large language models (LLMs). We introduce AgentSLR, a large-scale evaluation harness comprising an SLR automation workflow and an expert annotated dataset covering 16,248 articles, designed to test LLM capabilities across the stages of SLRs in epidemiology. Reference annotations were derived from peer-reviewed studies on WHO priority pathogens and produced by domain experts. The harness evaluates each review stage as a separate unit with dedicated metrics enabling targeted failure analysis. We evaluated five frontier reasoning models and found that no single model dominated across all tasks, showing sub-task specialisation often hidden by aggregate benchmarks. Structured data extraction is a major bottleneck, with no model exceeding an average field-level F1 of 0.67. Estimated costs vary substantially, by up to 96 times across evaluated models. Documented failure modes suggest that the evaluated models are not yet reliable enough for unsupervised deployment in epidemiology, where findings can inform public policy.

Shreyansh Padarha, Ryan Othniel Kearns, Tristan Naidoo +13

University of Oxford · Imperial College London · University of Nottingham +2

Systematic Review Automation LLM Evaluation Scientific Workflow Automation LLM Information Extraction

Ask PDF

Oct 2, 2026

EpiWorld: Grounding LLM Policy Agents in Epidemiological World Models

Epidemic intervention policies are textual artefacts that human decision-makers interpret, justify, and revise through natural language, making large language models a natural candidate for epidemic policy reasoning. A naive LLM, however, lacks the epidemic dynamics needed to project intervention consequences, the quantitative surveillance signals required to assess severity, and the institutional constraints that define admissible actions. We present EpiWorld, a closed-loop framework that grounds an LLM policy actor in a learned action-conditioned epidemiological world model and a tiered skill library of public-health protocols, surveillance tools, and adaptive lessons accumulated through after-action analysis. Given a candidate intervention, the world model predicts regional epidemic evolution and enables fast counterfactual rollouts that provide feedback for policy selection and refinement. Outcomes of simulated futures are distilled into reusable lessons while protocol constraints remain fixed, allowing the decision process to improve without sacrificing interpretability or controllability. We evaluate both the world model and the end-to-end framework on retrospective COVID-19 and Influenza datasets: the world model achieves the best out-of-distribution Peak-MAE among all forecasting baselines, and the closed-loop framework reduces cumulative hospitalisation by up to 59% across datasets and by an average of ~16% across six LLM backbones, outperforming reinforcement-learning and optimal-control policy baselines.

Zeeshan Memon, Yiqi Su, Kai Shu +2

Department of Computer Science, Emory University, Atlanta, GA, USA · Department of Computer Science, Virginia Tech, Alexandria, VA, USA

Action-Conditioned World Models Epidemic Forecasting

Ask PDF

Abstract

LLM Information Extraction

Explore similar work

EpiQAL: Benchmarking Large Language Models in Epidemiological Question Answering and Reasoning

Evaluating AI-based Scientific Knowledge Synthesis with Epidemiological Systematic Reviews

EpiWorld: Grounding LLM Policy Agents in Epidemiological World Models