Large language model-generated patterns for epidemiological modeling: a multi-benchmark evaluation framework

Sophia Mao; Keshav Gandhi; Alexander Gutfraind

cs.AIOct 11, 2026

Large language model-generated patterns for epidemiological modeling: a multi-benchmark evaluation framework

Authors: Sophia Mao, Keshav Gandhi, Alexander Gutfraind

Organizations: Division of Epidemiology and Biostatistics, School of Public Health, University of Illinois at Chicago, 1603 W. Taylor St., Chicago, IL, USA · Program for Experimental and Theoretical Modeling, Division of Hepatology, Department of Medicine, Loyola University Chicago, Maywood, IL 60153, USA

Abstract

Representing individual-level behavioral heterogeneity is challenging in epidemiological modeling, particularly when empirical behavioral data are sparse. We assess the potential of large language model (LLM)-generated behavioral outputs in addressing this challenge, using hepatitis C virus-related behaviors among people who inject drugs as a case study. Correspondence with epidemiological patterns was evaluated using external benchmarks: literature-informed association directions, correlations from an agent-based model (ABM), and survey associations. Across three LLMs, unconditional directional recovery across 28 retained literature-informed associations was 46-68%, compared with 82-93% agreement when LLMs were directly asked to classify the same associations. Mean absolute differences from ABM and survey correlation patterns were 0.11-0.13 and 0.15-0.18, respectively. Together, these findings show partial alignment between LLM-generated behavioral relationships and epidemiological evidence, despite higher knowledge-probe agreement. More systematic evaluation is needed before incorporating LLM-generated behavioral outputs into epidemiological models that simulate transmission dynamics.

Figures & tables

Figure 1: Overview of the multi-benchmark evaluation framework

Outcome variable	Definition	Type
Injection frequency	Average weekly injections	Count
Network in-degree	Unique individuals from whom the person received a used syringe	Count
Network out-degree	Unique individuals to whom the person gave a used syringe	Count
Partners	Unique individuals with at least one shared injection event	Count
Non-injection routes	Non-injection use (smoking or snorting)	Count
Overdose	Non-fatal overdoses	Count

Table 1: LLM-generated outcome variables (4-week recall period)

Variable pairs Direction Evidence
Homeless	Public injecting	$\uparrow$	OR = 4.23 (3.17–5.65) (Khezri et al., 2025) (meta-analysis) Positive association reported (Arum et al., 2021) (systematic review)
Homeless	Injection frequency	$\uparrow$	OR = 1.40 (1.08–1.82) (Ballard et al., 2025) Positive association reported (Arum et al., 2021) (systematic review)
Homeless	Receptive sharing	$\uparrow$	Positive association reported (Arum et al., 2021) (systematic review) aOR = 1.93 (1.67–2.23) (Hotton et al., 2021) aPR = 1.26 (1.15–1.39) (Kan et al., 2026) (cohort) RR = 1.18 (0.60–2.39) (Morris et al., 2014) (cohort)
Homeless	Positive HCV status	$\uparrow$	aHR = 1.86 (1.32–2.55) (Morris et al., 2020) (cohort) aRR = 1.64 (1.43–1.89) (Arum et al., 2021) (meta-analysis) aHR = 1.46 (1.21–1.76) (Gahrton et al., 2024) (cohort) aOR = 1.55 (1.50–1.61), currently homeless versus at risk of homelessness among veterans (Byrne et al., 2019) HR = 1.20 (0.60–2.30) (Hahn et al., 2002) (cohort)
Homeless	Overdose	$\uparrow$	OR = 1.64 (1.49–1.81) (Dunne et al., 2025) (meta-analysis) aPR = 1.64 (1.49–1.79) (Marcus et al., 2022) aOR = 1.36 (0.70–2.63) (Gunaratne et al., 2026) aOR = 1.29 (0.82–2.04) (Guarino et al., 2023) Positive association reported (Arum et al., 2021; Austin et al., 2021) (systematic reviews)
Homeless	Injection partners	$\uparrow$	aRR = 1.48 (1.25–1.76) (Hotton et al., 2021)

Table 2: Prespecified literature-informed directions for associations between housing status and outcome variable

Figure 2: Comparison of literature-informed association directions with LLM knowledge-probe and generated-output directions.

Metric	Claude Sonnet 4.6	DeepSeek v3.2	GPT-OSS-120B
LLM outputs vs. literature
Estimable targets	22/28 (78.6%)	17/28 (60.7%)	28/28 (100.0%)
Conditional directional recovery	18/22 (81.8%)	13/17 (76.5%)	19/28 (67.9%)
Unconditional directional recovery	18/28 (64.3%)	13/28 (46.4%)	19/28 (67.9%)
Knowledge-probe agreement	26/28 (92.9%)	25/28 (89.3%)	23/28 (82.1%)
LLM outputs vs. ABM data

Table 3: Evaluation metrics used across benchmarks

Variable pairs Direction Evidence
Public injecting	Positive HCV status	$\uparrow$	OR = 1.55 (1.18–2.02) (Khezri et al., 2025)(meta-analysis) aOR = 1.91 (1.33–2.75) (Eckhardt et al., 2017)
Public injecting	Overdose	$\uparrow$	OR = 2.51 (2.01–3.13) (Khezri et al., 2025) (meta-analysis)
Public injecting	Receptive sharing	$\uparrow$	OR = 2.41 (1.97–2.94) (Khezri et al., 2025)(meta-analysis)
Public injecting	Injection frequency	$\uparrow$	OR range across studies = 1.96–5.34 (Khezri et al., 2025) (systematic review)
Public injecting	Treatment	$\downarrow$	OR = 0.61 (0.48–0.76) for methadone maintenance treatment (longitudinal study reported in Khezri et al., 2025)
Receptive sharing	Positive HCV status	$\uparrow$	HR = 2.1 (1.2–3.7) (Hahn et al., 2002) (cohort) PRR = 1.94 (1.53–2.46) (Pouget et al., 2012) (meta-analysis) aHR = 1.79 (1.44–2.22) (Gahrton et al., 2024) (cohort) Indirect evidence: aRR = 1.53 (1.01–2.31) for syringe sharing, without distinguishing receptive from distributive sharing (Mackesy-Amiti et al., 2023; baseline analysis) aPR = 1.20 (1.09–1.33) (Kan et al., 2026) (cohort)

Table S1: Prespecified literature-informed directions for associations between variable pairs

Figure S1: Comparison of literature-based association directions with LLM knowledge-probe directions and LLM-generated-output correlation directions.

Model	Variable pair	Original	Paraphrased
Claude Sonnet 4.6	Homelessness / Non-injection routes	decrease	uncertain
DeepSeek v3.2	Non-injection routes / HCV status	decrease	increase
GPT-OSS-120B	Treatment / Non-injection routes	mixed	increase
GPT-OSS-120B	Non-injection routes / Overdose	increase	decrease
GPT-OSS-120B	HCV status / Overdose	uncertain	mixed
GPT-OSS-120B	Receptive sharing / Overdose	mixed	increase

Table S2: Variable pairs with different knowledge directions under different prompt wordings.

Figure S2: Comparison of ABM and LLM correlations

Figure S3: Sensitivity of LLM–ABM Correlation MAD to Follow-up Tolerance

Outcome	Claude Sonnet 4.6	DeepSeek v3.2	GPT-OSS-120B
Overdose	49.1	100.0	32.1
Public injecting	100.0	74.6	98.7
Receptive sharing	69.6	94.6	40.2
Injection partners	71.4	90.6	35.3
Network in-degree	97.8	96.9	71.9
Network out-degree	97.3	99.1	67.4

Table S3: Outcome-specific directional agreement (%) between the original and paraphrased prompts across 224 profiles per model.

Human Behavior Modeling

See all 27 papers

Integrating adaptive human behavior into epidemic models with large language models
Aug 30, 2026Yicheng Mao, Haoyang Li, Rob Deardon +1Epidemic Forecasting Large Language Model-Based Social Simulation
Department of Mathematics and Statistics, University of Calgary, University Drive NW, Calgary, T2N 1N4, Canada · Department of Systems & Information Engineering, University of Virginia, Charlottesville, VA, USA · Faculty of Veterinary Medicine, University of Calgary, University Drive NW, Calgary, T2N 1N4, Canada
The Epi-LLM Framework: probing LLM behavioral priors through epidemiological agent-based models
Jun 1, 2026Petra Ferencz, Ava Keeling, Tobias O'Keefe +4Human Behavior Simulation LLM Agent Evaluation
Big Data Institute, Li Ka Shing Center for Health Information and Discovery, University of Oxford, Oxford, United Kingdom · Leverhulme Centre for Demographic Science, Nuffield Department of Population Health, University of Oxford, Oxford, United Kingdom · Pandemic Sciences Institute, Nuffield Department of Medicine, University of Oxford, Oxford, United Kingdom +3
Generating Public Health Responses using Survey-Augmented Large Language Models
Jun 20, 2026Leonardo Marciaga, Thuyen Pham, Julia Rezvani +4Synthetic Data Augmentation Synthetic Data Generation
Illinois Institute of Technology · University of Massachusetts Amherst · Portland State University +4

Aug 30, 2026J/K move · Enter open · S save

Integrating adaptive human behavior into epidemic models with large language models

Yicheng Mao, Haoyang Li, Rob Deardon, Hongru Du

Department of Mathematics and Statistics, University of Calgary, University Drive NW, Calgary, T2N 1N4, Canada · Department of Systems & Information Engineering, University of Virginia, Charlottesville, VA, USA · Faculty of Veterinary Medicine, University of Calgary, University Drive NW, Calgary, T2N 1N4, Canada

Epidemic Forecasting Large Language Model-Based Social Simulation Policy Evaluation

Open paper AskPDF arXiv

Infectious disease transmission is shaped by patterns of human interaction, which adapt as epidemic conditions change. Capturing these context-dependent behaviors remains a fundamental challenge for epidemic models. Here, we recast this challenge by using large language models (LLMs) to represent adaptive human behavior within mechanistic epidemic models. We operationalize this idea through Generative Adaptive Behavioral Layer for Epidemics (GABLE), which adapts LLMs to infer behavioral responses to epidemic and policy conditions and translates them into age-structured contact matrices coupled to a mechanistic epidemic model. Applied to COVID-19 in France, GABLE reproduced responses in population mixing and age-specific contact structures that remained epidemiologically informative. In short-term forecasting, LLM-generated contact matrices outperformed mobility-driven matrices derived from real-world mobility data, with the largest gains at longer horizons. GABLE also extends beyond forecasting to prospective policy evaluation by projecting behavioral and epidemic responses to candidate interventions before implementation. When supplied with subsequently implemented policies, GABLE reproduced epidemic trajectories and generated distinct responses to alternative policy timing and composition. By leveraging LLMs as a flexible behavioral layer, GABLE provides a framework for coupling context-sensitive behavioral generation with epidemic dynamics.

Abstract

Human Behavior Modeling

Explore similar work

Integrating adaptive human behavior into epidemic models with large language models

The Epi-LLM Framework: probing LLM behavioral priors through epidemiological agent-based models

Generating Public Health Responses using Survey-Augmented Large Language Models